What we made JET BI from. Architecture without lyrical digressions

In a previous post, I talked about the evolution of BI systems and how we came to understand that it’s better to create your own platform than to adapt to existing ones.

Today, as promised, I’m talking about the architecture of our new platform, which, I hope, will be the ideal solution for building any BI-systems.

Functional architecture

In the system, two main “Cores” can be functionally distinguished.

Visualization Core and BI (the team and I call it “reading room”) It is engaged in the fact that it filters data, calculates facts and measurements, calculates and caches aggregates, etc. In general – the most ordinary OLAP. Support for in-memory computing is also available. A separate place is occupied by the ETL engine we developed, supporting both standard loading methods (for example, SQL, MongoDB query, unloading from Excel files, etc.), and loading anything from anywhere using JS scripts. How exactly? The fact is that we “bound” the JS loader with a special API, the purpose of which is to provide a set of easy-to-learn methods that are most in demand for queries to different sources, as well as data transformation (for example, transpose, join, add calculated columns, various kinds mathematical and statistical functions).

Tongue

Javascript Are you crazy?

Maybe…

But the choice of JS as the language and the rejection of the invention of another bike, as is often the case in BI platforms, is not accidental. The popularity of the language in itself, the simplicity of its development (at least for ETL tasks) and the large number of specialists in the market turned out to be decisive factors for us. In general, according to the ideology of our platform, the ETL engine is similar to the “download scripts” mechanism used, for example, in QlikView, except for the language in which these scripts are written. I hope that many vendors of BI platforms will sooner or later come to a universal programming language, but for now we are working with what is.

The core of business logic. It is rather a framework that consolidates the software architecture and provides a number of universal APIs with which you can add additional applied information-analytical components to the system.

From what we already have, it can be noted:

• Constructor and handler for data entry forms
• What-if modeling and analysis environment
• Order Management System
• Electronic Document Repository
• Project and project management module

I would like to emphasize that these components are present in the system for a reason and do not live their own lives. They are closely related both to each other and directly to the visualization and reporting system. In fact, they become either suppliers or consumers of data for it. For example, from the order management system, you can make a simple report from scratch from scratch from scratch that displays the status of all orders and a list of the most malicious lazy people. And from the project management module, get the data on the most “stalled” projects and identify the reason for the delays.

Technical architecture

The application backend runs under Node.JS, interspersed with a number of critical (in terms of performance) native components. Like any Node.JS application, the system can be clustered, containerized and deployed in any environment that meets Node requirements.

To store metadata, you can use most of the popular relational DBMSs, we most often deploy on PostgreSQL. The database stores all meta-information about reports, control panels, ETL processes, additional modules, etc. The system can also be used as a tool for filling a third-party data warehouse. For small amounts of data, you can organize visualization in ROLAP mode, that is, directly from source systems. Something like the “associative model” of QlikView is also present. If you select two or more data sets as the data source for the visualizer, then they will be combined according to the column names into a single set.

Frontend is a classic SPA on React, related libraries and additional modules of JET BI itself. Communication with the backend occurs through the most common REST, part of the code is isomorphic, that is, it is used by the front and back. All code is ES7 with type annotations (Flow), run through Babel. We abandoned Typescript in favor of Flow, because when we first started, the latter deduced types a little better.

Quite often (in about 80% of cases) we take ready-made open-source modules and do not invent our own ones (in the backend a little less often). This simplifies and reduces the cost of development and support, but slightly reduces flexibility. There were miscalculations, after which some of the libraries still eventually had to be rewritten on their own.

Conclusion

In the end, I would like to say that I, as an architect, are pleased that the “framework” of the system turned out to be quite solid and stable on the one hand, and on the other hand, universal and with a sufficient margin of flexibility (despite the abundance of the above-mentioned open-source libraries). It’s like a Christmas tree, on which we constantly hang new toys. After all, the tree should withstand toys of various varieties and stripes, and at the same time not fall under their weight. In my opinion, this balance has been achieved in JET BI, which gives confidence that our plans for the development of the system can be implemented.

Albert Nurutdinov, architect of Jet Infosystems