import substitution data platform
The issue of import substitution of software from departed vendors has been on the forefront for the last couple of years. We deal with data, so we share our experience of import substitution of data platforms with our enterprise customers. If we try to classify the reasons for import substitution, these could be the reduction of vendor risks, the requirements of the Ministry of Digital Development, the complexity and cost of purchasing certified hardware for imported platforms, and so on. By the way, the demand of development and support teams for the implementation of projects on the Russian and open source technology stack is far from the last place. The Russian job market for Western proprietary vendor software is gradually shrinking, since now the demand of Russian companies for such software is approaching zero and such systems are gradually turning into Legacy. One way or another, Russian large businesses are gradually coming to the task of import substitution and, along with it, to a whole range of complex tasks, ranging from the selection of suitable target systems and technologies and their “finishing”, development, implementation, to meeting the expectations of not always simple business users.
Let's split the problem into two parts – a suitable proven technology stack with team formation and business user satisfaction.
Tech stack
It is important to understand that if we are talking about a data platform on SAP, then this is a whole complex of functions: ELT mechanism, modeling tools, functionality for monitoring data loading processes, even several BI for different tasks from dashboarding to self-service analytics. Below I have provided a picture from the official SAP documentation, which partially visualizes the scale:
In the SAP paradigm, one SAP BW (Business warehouse) consultant set up the entire vertical from data extraction to reporting setup. He could also use low-code modeling tools, load monitoring, and so on. By the way, this is where project teams are built, in which one consultant is responsible for his block, for example, for extraction from SAP MM and simultaneously for the corresponding reporting on purchases and balances, another – for extraction from SAP FI and for reporting on accounts receivable and payable, etc.
At the moment, there is no single solution on the Russian market that would allow a more or less comfortable “move” from a full-featured data platform to SAP.
By the will of project fate, we started working on SAP import substitution tasks back in 2022, and we have been working on target vendor systems, where we propose migrating functionality, for a very long time now and can now share our working experience. With certain assumptions, depending on the infrastructure and landscape, we almost always see a similar block architecture (diagram below) different companies, even regardless of the industry. The solution providers for each block may differ, but the architecture itself has already become a standard. The target platform for migration is highlighted with a gray rectangle. On the left are any sources.
By the way, I will immediately answer the question of whether it is possible to leave data extraction from SAP ERP. Yes, it is possible, and it will work as fast as in standard SAP BW. We conducted webinar on this topic.
From the point of view of the vendors of the blocks themselves, the biggest differences are in BI solutions: there are a huge number of them on the Russian market at the moment. There are much fewer suppliers of data storage solutions, since the technologies are already more or less defined – this is S3 or Hadoop for cold data, Greenplum – for warm data and Clickhouse – for hot, and the suppliers are Arenadata for On-premise and several cloud providers.
The platform itself presents gray blockswhich cover the main components of the platform: loading, storing, visualizing and working with data. In this diagram, the core of the DWH for storing data and showcases are the Arenadata DB and Arenadata QuickMarts products, which provide a comparable or even higher level of platform performance and the possibility of further unlimited growth of data volume. The platform support/development team is also built from here – the team becomes more focused not on functional verticals (finance, logistics, etc.), but rather on horizontals by technology: a data engineer for working with a model, a BI engineer or analyst, and so on.
There is an important nuance – all the gray blocks presented are “bare” databases and functionalities in which you will need to write SQL, Python or Java procedures through special IDEs. This fundamentally changes the requirements for the organization and qualifications of the support/development team, labor costs for development/implementation and time-to-market requests for changes from business users.
The blue block ProPlum is our framework, which we have added to the platform to get closer to the low-code approach of SAP. Framework posted in Open source and allows the following:
Significantly reduce labor costs for support and improvements and increase the speed of development (in the company “Komus” more than 60% of procedures development is implemented through settings without writing SQL code, and this reduces labor costs several times)
Simplify the rotation of employees and the inclusion of new project teams, for example, contractors (everyone works in the same structure and naming convention framework).
Reduce risks and facilitate retraining of the team responsible for supporting and developing the data platform.
By the way, regarding the risks of cutting the SAP BW project team – our clients did not experience them. All members of the support and development teams were able to calmly retrain for new technologies and expand their technological experience. For this purpose, we conduct training for our clients' teams and the ProPlum framework here, of course, is also very helpful for an easier start. As it turned out, leaving SAP is much easier for those who worked with BW and data than for consultants on classic ERP modules.
The other two blue blocks – Apache Superset and Easy Report – are about business user satisfaction.
Satisfaction of business users
We often hear from our clients' business users that after import substitution it should be “no worse”. This is a very complex formulation, essentially a minefield that IT specialists and data engineers need to navigate. We have identified two factors so that the final result that we will receive after the SAP migration project is no worse – this is the absence of blocking factors for work, that is, +/- substitutes for all the functionality that users used, and certain wow effects that will increase the emotional impact of the “dish” that we will ultimately bring.
Apache Superset – we have supplemented the standard Superset with additional functionality, which allows you to get capabilities comparable to advanced Western solutions in terms of front BI, especially when it comes to switching from business with SAP BO. In the near future, we will post our improvements in the Apache Superset part in Open source.
What we added:
Dataset: association of fields with reference books, limited/calculated indicators,
structures and otherPivot Table Widget: what was used in SAP BO Analysis – changing the drill-down for characteristics and key figures, changing the text/key presentation, subtotals, hierarchical presentation, filtering, etc.
Native Filters: new types of filters with master data (text/key),
without master data, hierarchical filterStandard and custom widgets: Expanded functionality of pie chart and column
chart, custom widgets kpi and waterfall chartIntegration with Greenplum and Clickhouse,
The administration functionality and functionality related to information security have been improvedParent-child hierarchy
Easy Report – a proprietary product developed by us that allows you to maximally simplify interaction with data and make requests in any messenger in natural language. Here is a very brief overview of the product’s capabilities:
Receive data summaries in the messenger directly from the data warehouse without developing reports and searching for the required dashboard, and if the service finds the required dashboard in Superset, it will send it directly to the messenger with the required restrictions.
Self-service work with data directly in the storage without knowledge of SQL, accessible to an employee of any qualification.
View descriptions from the data catalog.
The ability to monitor changes in important indicators by receiving notifications and reports in the messenger.
Receive Superset reports directly in the messenger, taking into account the filters from the message.
Easy Report gives, firstly, a wow effect and ease of use for middle and senior management, and secondly, it allows you to easily and cheaply connect a large number of end users, for example, sales managers, such as made by our client Airplane.
Conclusion
Migration from a data platform to software from Russian vendors is not rocket science, but there are many nuances and details that, unfortunately, cannot be covered in one article. It is also impossible to give universal advice. By posting and sharing our developments in Open Source, we have tried to make life as easy as possible for Russian companies that are faced with the task of import substitution of the SAP data platform.
To summarize the experience of our clients, we can say the following:
The task is feasible and we have never encountered such complexity of models and reports that cannot be migrated.
Migration, depending on the volume and bureaucratic procedures, takes from 4 months to about a year.
The labor costs of the specifics and approach of migration, in our experience, can also vary from 200-300 h/d to several thousand when migrating a large volume of reports and models.
Frameworks can facilitate the migration and adaptation process of the Client's current support team.
If your company faces such a task, then we We are holding a webinar on migrationwhere we will share our experience of migrating from SAP BW in a large bank to the architecture described in this article, with key products for loading, storing and processing data from Arenadata and reporting in Apache Superset. The webinar will consist of two parts: a high-level managerial part and a more technical part, where we will talk about the features of data extraction from SAP ERP, modeling in ADB and configuring Apache Superset.
We will also always be happy to answer your questions, make an assessment and share our experience.
