A single semantic layer of BI and what it gives on the example of the Microstrategy platform
Every day I communicate with a large number of large companies. Often they express a desire to build a data-driven company within the next few years. Their key strategy is to create DWH / Big data and connect users to it, who will build reports, visualizations and dashboards themselves using self-service tools – Tableau, Power BI or Qlik.
One of the main problems of Tableau, Power BI or Qlik is that they do not have a single semantic layer, a single logical model, a single definition of all attributes, facts and indicators for the entire company’s business intelligence. That is why they grow up in a company without control and governance, giving rise to many versions of the truth in reporting and analytics in the company. After some time, companies are faced with the fact that they can no longer figure out which report to trust – the data does not converge not only in different Tableau, Power BI or Qlik systems, but also between reports in the same system. It turns out that companies invest huge amounts of money in building a single version of the truth at the level of data warehouses, and ultimately end up with an incredible amount of separate and ungoverned reports, visualizations, dashboards that they cannot trust.
Tableau, Power BI, and Qlik are great tools if you want a business user to be able to do self-services local analytics for their department’s needs. However, to build an enterprise-wide data-driven company, you need a tool to organize, structure, and standardize all of your business intelligence.
Since I have been in the field of DWH and BI on the Microstrategy platform for more than 15 years, I tried to systematize and structure the information about the Microstrategy Enterprise Semantic Graph a little.
So, Microstrategy Enterprise Semantic Graph or Semantic Layer – a single layer of attributes, facts, indicators and analytics objects that describes the relationship of business terms with your data sources (QCD, AHD, Big data and others). The user is immersed in the environment of a single terminology, a single business logic for the formation of all indicators in reports and dashboards, without hesitation from which database the data is taken from.
A single layer of business abstraction allows you to:
provide data in a business-friendly language to create business intelligence;
ensure the convergence of data between all company reports in different departments and ensure that any indicator in all company reports is always calculated in the same way (data governance, a single version of the truth);
reduce the complexity of creating new reports and dashboards (the cost of ownership of business intelligence, TCO) due to the repeated reuse of metalayer objects, reduce the complexity of maintaining all existing reports in the event of a change in the business logic of indicators – a change in the algorithm for generating an indicator at the level of a single metadata layer will automatically be reflected in all existing reports;
implement a unified data access security policy managed in one place;
understand at any time who, when, how often and what data is used, for example, in order to place the most requested data in in-memory to achieve better performance or recommend users the most popular content based on the collected user activity statistics;
provide access to a single metalayer through Power BI, Qlik, Tableau, SDK & API, R, Python, or native applications;
Users, analysts, BI developers and data scientists can use a single Microstrategy metalayer for the following tasks:
Combining In-memory cubes, QCD, ACD and Big Data sources in a single semantic layer model
With Microstrategy Enterprise Semantic Graph, you can combine multiple data sources into a single logical model. The user does not need to think about where he gets the data from when building a report – from In-memory cubes, QCD, ACD and other sources.
Building ad-hoc reports, pixel-perfect dashboards, self-service & data discovery in Microstrategy
Based on a single metalayer, the user can independently make a report of any complexity – ad hoc, dashboards, self-service & data discovery, documents, data mining & predictive, freeform, etc. MicroStrategy allows you to implement a design of any complexity in reports and place elements exactly where necessary.
Adding, processing data from external sources and combining them with a single semantic layer
Users can add and process data from external sources, combine them with a single logical model, expanding the horizons of available corporate data for analytics. Microstrategy provides over 200 connectors for different types of sources.
Data visualization based on a single metalayer using Tableau, Power BI or Qlik
To access a single metalayer from Tableau/Qlik/Power BI, you can use the Microstrategy Connector for Tableau/Qlik/Power BI. In addition, we can upload data from the metalayer directly to the database or to a text file.
Building code free native mobile BI applications using Microstrategy Mobile
MicroStrategy allows you to create code free native, functional and intuitive analytical mobile applications in which you can also implement workflow using the write-back functionality and place arbitrary content (videos, pictures, documents, email and web content).
Analytical Data Processing with Built-in Analytic Functions, R or Python
There is a built-in rich library of analytical functions (mathematical, statistical, OLAP and financial functions) with the possibility of expansion. MicroStrategy also offers R and Python open source packages that enable analysts to use machine learning algorithms to quickly build analytics applications based on a single layer of metadata.
Performance monitoring and notifications
Often users send their reports by mail to all interested parties. MicroStrategy can do this on its own. Any report can be sent by mail in different formats: XLS, PDF, or simply insert the report into the body of the letter. The mailing frequency can be adjusted by time, by event, by condition of indicators. Also, the distribution mechanism can be used to upload data to FTP, database, API, send push notifications to mobile devices.
Hyper Intelligence – contextual “zero-click” analytics
Microstrategy provides Hyper Intelligence functionality – contextual analytics on any website, device or application. Allows you to get answers in real time without interrupting your workflow – no clicks required.
Creation of write-back reports and applications
Write-back functionality allows users to write and update data in ERP, CRM source systems and operational databases directly from Microstrategy desktop, web and mobile applications.
Embedded reporting and analytics in your web applications
The Embedding API makes it fairly easy to embed MicroStrategy reports and dashboards into your Web applications.
Geo-analytics
Support for integration with maps, the ability to overlay various layers (branches, regions, divisions, etc.) on an interactive map of Mapbox, Google Maps, ESRI, etc.
Voice assistants and chatbots
It is possible to integrate Microstrategy with voice assistants (Siri, Alexa, Alisa…) or chatbots to quickly get answers to your questions.
Getting data in Microsoft Office applications from a single semantic layer
This add-in allows you to use Microsoft Office to connect to data from the MicroStrategy metalayer. The add-in provides the ability to import and update reports and datasets from MicroStrategy Enterprise Semantic Graph into Microsoft Office.
Discussion of reports and dashboards in the BI system
Users have the ability to communicate with each other through discussions or comments that can be made in the Microstrategy interface. Anyone with access to MicroStrategy reports can add comments related to that report. Users can ask questions and pay attention to specific data in the report itself. All comments are saved in the discussion thread, and a user who has access to the report can see the entire comment history for it.
High performance and scalability
Microstrategy supports linear scaling, clustering, a combination of 2 modes when users work with reports / dashboards – live connect and in-memory, supports multi-level caching, multi-pass SQL engine, the ability to set priorities for different tasks, users, user groups, etc. . The Microstrategy platform is clearly the best choice for large companies with large amounts of data, the number of users and a wide variety of required functionality of the BI solution and data delivery channels to users.
A couple of years ago, @abezugly wrote an article about M.Video-Eldorado’s experience in implementing visualizations in Tableau over SAP HANA https://habr.com/ru/company/mvideo/blog/483656/. After it became finally clear that Tableau, after a data volume of 10 GB with performance, begins to significantly degrade, we were invited to make a pilot on the Microstrategy BI platform. On the pilot in a couple of weeks, we showed that dashboards in Microstrategy with a data volume of 1-2 billion records can be processed in 1-2 seconds, which was the start of a large project to create an operational and analytical reporting system M.video-Eldorado on the Microstrategy platform.
The following video goes into some detail about what a Microstrategy Semantic Graph is and why it is critical to meet changing corporate requirements for agility, security and data governance in projects when implementing business intelligence:
I hope the review and materials were useful to you and allowed you to form an idea of what a single semantic layer is and why all this is needed at all.