converting data formats easily and with a smile
What is the key difficulty?
At first glance, there is nothing complicated in matching several data fields between different formats. But this is only as long as the number of fields in the XSD or JSON schema does not exceed several dozen, and the integrated services are few. But when the number of services exceeds hundreds and thousands, and some data format schemas contain hundreds of fields, the complexity of the task grows exponentially. Add to this the unavailability of familiar tools, the complexity of data structures, the human factor, frequent updates of format versions and project documentation – and you get an endless stream of document editing. Have you avoided this problem? Then you are a lucky one. And yet, I believe that many will agree that such a problem exists and is worth solving.
Own mapping product
We tried different approaches to mapping. At first, we did it manually in Excel tables or Word documents, without any automation. Comparing the formats of different systems for one integration service could take our analyst a whole day. And this did not guarantee the absence of errors, which then went down the chain to the developers. They, in turn, had to manually transfer the mapping description to the code. And testers had to manually compose example messages for tests.
Then we started using Altova Map Force and Oxygen. They coped with most tasks, although they had some limitations. But the main drawback was the significant cost. There were regularly not enough licenses, we had to wait for purchases. And then the rejection of foreign proprietary software began. And then it turned out that there were simply no suitable Russian solutions for working with XML and JSON schemes.
All this prompted us to create our own tool that would be convenient for our team. This is how XJProfessor appeared. By the way, there is a funny story connected with this name. At first, we wanted to call the product XJProcessor, however, someone made a typo, and an unplanned XJProfessor appeared in many documents and presentations. In the end, they decided to leave it as is.
Functionality
XJProfessor is written using Electron, Vue.js, Spring Boot, D3, Java. Its main task is to simplify format mapping for analysts and developers as much as possible, reduce labor costs and minimize errors. We relied on our experience in solving similar problems and using similar products. We conducted research and interviewed colleagues from various departments. That is, we tried to collect all the features necessary for our work in one product.
In its first versions, XJProfessor allowed you to create, save, and modify mappings between JSON and XSD formats in a graphical interface. Today, its capabilities are broader:
Viewing diagrams in the form of trees, customizing the parameters displayed in the tree (description of fields, type, length, etc.).
Developing mappings between JSON, XSD and OpenAPI formats.
Saving comparisons in a file for further refinement.
Comparison of JSON and XSD schemas.
Generating documentation on mappings (downloading in the form of tables for insertion into documentation).
Generate sample messages based on schemas. Also generate schemas in CSV and HTML formats.
Generate code based on developed mappings in various languages. Currently, these are XSLT, Java, C# and Groovy.
Versions for Windows and Linux.
This is what the matching interface looks like:
You can check for correctness, the tool will show errors:
You can generate documentation in the form of tables and insert them into Confluence, various technical specifications and architectures:
And finally, generate the basis of the code for mapping in the programming languages that are relevant to us:
Additionally, you can compare schemes and generate examples from them:
Generating an example according to the scheme:
Results and benefits
Using XJProfessor has made it possible to speed up the implementation of new integration flows up to 5 times. If previously there were almost always some inaccuracies in large comparisons, errors were found during testing, now everything is the opposite. It has become almost impossible to make a mistake due to carelessness in analysis and development thanks to the convenient graphical interface. In addition, XJProfessor can analyze comparisons for errors, and the developer does not need to manually copy field names from the documentation. This also freed analysts and developers from the same type of large-scale tasks: after all, mechanically sorting through hundreds of details in dozens of services is quite tedious. For us, this is one of the main results of implementing the tool.
Prospects
Our plans include:
Improve the interface. When XJProfessor was created, its visuals were not the most important thing. It was urgent to make projects and new integrations, and for this, functionality was primarily increased. Now we have attracted fellow designers.
Build a web version. For now, the product is implemented as a desktop application. In the future, we see XJProfessor as a SaaS solution. The product can be integrated into corporate tools for analysts and developers.
Add multi-user mode.
Add support for mapping multiple schemas simultaneously on one screen.
Add DBT (data build tool) support.
We are also thinking about automatic generation of documentation based on program code, and are considering creating a full-fledged IDE for working with XSD and JSON schemas.
How do you work with mappings?
If you have experience working with mappings, tell us about it. We are very interested in how fellow analysts solve their problems. We will also be glad to receive feedback and suggestions on the product. You can write to our Telegram channel.
And if you want to test the tool, here is a link to a small one questionnaire. Everyone who fills out the form will get free use of XJProfessor until the end of 2024 🙂
