Sizing of multi-level QHD (Part 1 What is sizing)
Greetings, Guest, seeking methods and approaches!
My many years of experience in designing and implementing KHD using products of foreign Vendors has always been associated with the use of their extensive infrastructure and developments that provide and help to perform auxiliary tasks quickly and relatively efficiently. One of such tasks has always been the implementation of the sizing of the KHD being developed. You may ask yourself: “Why “conditionally qualitative”? – the answer here is prosaic and banal: “Sizing tools cannot clearly answer the question of what characteristics to include and how to calculate the sizing of the KHD before the technical specifications are fully formed and all the i's and t's of the future storage are dotted, and of course, no tool can cope with changes in requirements during the project, the clumsiness of developers and the use of non-optimal solutions. As it turned out, after software suppliers left the market and the mass transition to OpenSource solutions, along with the software, the applied solutions for performing KHD sizing “left” too.
The methodology discussed below is based on the following postulates:
The QCD is a multi-level structure with data flows between layers.
Raw source data layer;
A layer of high-quality and harmonized data;
Unified data layer;
Business Transformed Data Layer;
Layer of reporting showcases.
Logic of data movement within the QCD.
The initial data layer is loaded and must store the initial data that enters the KHD unchanged, as they are given by the systems – data suppliers. This is done to solve the problem of reverse analysis of the total indicators to the primary values from which they were formed (the problem extremely resource-intensive and, by its very nature, useless).
The data from the source data layer must undergo a process of cleaning from garbage, bringing it to a single NSI and harmonizing in terms of features and indicators. The output is the same source data as on the source data layer, but in a uniform form, i.e. all data at this level are comparable. I would like to point out separately that sometimes these two concepts of “qualitative” and “harmonized” data are divided into separate layers, but the essence does not change: the data must become comparable to each other for their joint analysis.
The data unification layer solves the problems of transforming harmonized and high-quality data to internal target entities, for example, transforming the Standard Chart of Accounts to the Corporate Chart of Accounts; transforming RAS to IFRS, for example, by applying mapping rules. That is, on the unified data layer, information entities are drawn up that are tailored to solve the final tasks of the data warehouse. It is important to understand that the same KHD object can be reused many times in the future to solve specific problems (all as in OOP).
The Business Data Transformation layer is designed to solve the problem of applying “complex“Business algorithms that cannot be represented as a set of mapping rules and require, for example, the use of complex scripts to implement calculation models or algorithms to generate reports or intermediate results and use them in other calculation models.
The Reporting Display Layer is a set of final objects with data sets materialized for reporting purposes. The layer is a highly aggregated representation of information processed and calculated in the previous steps. An experienced KHD specialist may have a logical remark: why do materialization if you can generate reports on the fly, calculating the necessary data in the server memory. I will answer with a simple example: if the timing of report generation is not critical for you and the server has a large amount of RAM, for example 2Tb, then yes, everything can be calculated in memory, BUT! If you are building business-critical dashboards and reports, the generation time of which should be seconds, then nothing works faster than selecting pre-calculated values from a column table.
Continued: “Part 2 How to Size” here
