“Mikhalych, where is the order?” or how we automated the scrap ordering process and tracked it via RFID
In order to improve production quality and continue to grow, businesses have to automate manual operations. But the fight against legacy processes seems simple and easy only at first glance. Finding something that doesn't work well and figuring out how to solve it is not so easy. And weaning employees from doing things the way they are used to is even more difficult.
Hello Habr! Alexander Tamrazov and Dmitry Krivenko, developers from Axenix, are in touch.
In this article we will share a case of how we created and developed a universal production automation system for one of our clients and helped improve business processes. We'll talk about the requirements, business metrics and go through the technical part: architecture, stack, details about backend + frontend + IoT, development approaches. Let's finish the conversation with monitoring and how to scale such a solution.
Client process problems and human factors
The business we started working with had a complex manufacturing process. There were two workshops, between which orders and delivery of goods were carried out. Initially, to place an order, employees of one workshop contacted employees of another workshop by phone. This method caused several problems: the contents of the order were sometimes recorded inaccurately, errors in interpretation occurred, employees could misunderstand each other, and important details were missed. As a result, confusion arose in production. Employees began to keep records in paper journals, but this did not exclude errors. And in order to simply determine where the order is and at what stage of execution it is, the customer’s workshop sent an employee to check.
This process reduced production efficiency as resources were used inefficiently. The customer wanted to ensure transparency of the process so that at any time it was possible to obtain information about the location of transport, their cargo and quickly identify possible problems in logistics or order processing.
We conducted a study and formulated a business problem: track the movement of orders, changes in their statuses and give employees access to a round-the-clock video stream with this information.
Inside the service: logic, architecture, backend and frontend
As a result of the development, the business now has a system with an established process and reasonable automation, where the human factor is reduced to a minimum. This allowed us to reduce the time spent processing orders and reduce the number of manual operations.
Here's what we got at the project level:
Speed and access. The result is a fast, 24/7 online service.
History of orders. Now information about orders from one workshop to another is collected over one year.
User-friendly interface. Employees understand how to work with the service and how to quickly obtain information on any order.
Notifications. Information about orders comes automatically. Changes in order status or location are notified by push notifications.
Additionally, it became possible to collect traffic analytics into dashboards.
From a business point of view, the solution turned out to be convenient also because it is scalable – it can be used for different enterprises, because elements are interchangeable.
Tracking order location using RFID
At the beginning of our work, we examined the entire path along which the client’s transport travels from one workshop to another. To understand the entire movement pattern, we decided to detect and read the location. To achieve this, RFID tags were installed on each carriage.
RFID (Radio Frequency Identification or radio frequency identification) is a method of automatic object identification that allows the use of radio frequency signals to read and write information from tags attached to objects. The tags contain a unique identifier that identifies the cars in the overall diagram.
As soon as the car passes the RFID antenna, its position is read, and workshop employees use cameras to see what is happening with the order on the road.
Backend and architecture
The entire backend is written in Java and Spring Boot. PostgreSQL was used for data storage and Redis for caching.
The system consists of several microservices:
RFID connector. Establishes a connection with RFID readers via a socket and transfers the received data to Kafka for processing.
Data aggregation microservice. Receives information not only from the RFID connector, but also from other sources, such as tracking service, order service, MDM tables and others. This microservice aggregates and combines all data before sending it to Kafka.
Data analytics service. Receives data from Kafka and analyzes it to calculate various effects. Such as calculating the average time for passing sections of the route, downtime in front of workshops, the beginning and end of unloading; analysis of RFID tags for errors and counting the number of arrived and unloaded wagons per day; counting the cars that are currently in processing shops, in material storage shops, and under repair. It also provides data for the tracking microservice.
Tracking service. It also receives data from Kafka and uses it to form trains of cars that the user sees.
BFF (back for front). Serves as a link between analytical and tracking microservices, collecting data for convenient display on the front end.
Order service. Responsible for placing orders and organizing cargo delivery between workshops.
While working on the service, we encountered several operational problems related to technology and the human factor.
Problem 1: RFID device supplier has changed
Previously, the customer already had experience working with RFID technology, but in the process of working on the service, the device supplier changed, and China became the country of production of the readers. The Chinese devices themselves are not bad, but there were difficulties with the documentation.
The customer provided us with a library from the supplier. With its help, we could communicate with readers via socket. But the connection was extremely unstable: the readers constantly lost it and the socket became clogged. Because of this, it was not possible to quickly reconnect, so the readers had to be rebooted.
The supplier did not help in solving the problem, and the customer could not do anything. We decided to rewrite the vendor library manually: we took the pieces of code that were responsible for the unstable connection and changed them so that after losing the connection there was no need to restart the sockets. Everything worked.
Problem 2: the client took over part of the development
RFID is a great technology to track the movement of objects. But what if you need to track the removal of a car from the platform, unloading, weighing and other operations?
The customer wanted to implement a status tracking system independently. New statuses had to be displayed on the general diagram, and shop floor workers had to change them. We agreed, but the tracking system was not ready at the right time – two or three months were not enough. I had to wait.
The first time after the launch, everything worked well, but later some important statuses stopped arriving and the accuracy of the information on the diagram decreased. At the same time, the cars did not lose their cargo, did not leave the platform, and in general everything worked.
From this situation we can conclude that we should have realistically assessed the risks of such a decision. For example, that the customer could delay the release with his development, and we could not be fully responsible for what would happen, which means we increased the chances that the service would not work correctly. And it was worth proposing to implement the tracking system on our own, eliminating the need to manually change the status. For example, place several readers in the areas of unloading, weighing and removing cars from the platform, and thus avoid additional “crutches” in the product.
Problem 3: Force of habit among employees and reversion to old ways of working
We have developed a service for delivering goods between workshops. Workers of one workshop indicated what cargo was needed and in what volume, and workers of another workshop, where the materials were stored, received information and formed a train for transportation. They indicated the car numbers, characteristics and volume of cargo. All information was displayed on a general diagram.
Everything seemed fine. Orders were created, the workers were happy, the train moved, and the customer was delighted. But the human factor came into play. Workers made mistakes, indicated incorrect car numbers or selected cars that should not be used for transportation. Sometimes employees bypassed the system and used the old method of ordering – by phone.
Due to problems with both RFID and the ordering service, the scheme stopped working correctly. The data could not always be trusted. So we decided to develop an algorithm for analyzing and automatically including cars in the train.
The algorithm was implemented as follows.
Step 1: analyze the history of the scanning time of cars on specific readers.
Step 2: compare the time stamps of scanning cars and look for cases when the same car is scanned on the same reader with a time difference that does not exceed a certain value (delta).
If there are such cases, we connect this car with the current train. If we cannot find a suitable train for a car in the scanning history, then we create a new train for this car, which can also be used to search for other cars in the future.
Client part
The front-end part of the service includes an order register with cards for each order. The register provides an overview of all current orders, and cards contain detailed information for each order.
Additionally, we provide a mnemonic diagram – a graphical model on which you can visually track the delivery process. This feature allows the client to easily monitor the movement of orders and track their status.
It is similar to taxi services where customers can track the movement of the vehicle and manage their time efficiently. In the context of our proposal, employees can monitor the movement of their orders through a mimic diagram. The graphical model presents orders along with meta-information and analytical statistics so that employees have more data to manage logistics processes.
Parts of the order mnemonic
The web application is written in React and TypeScript. All components, including mimic diagrams, are simple: they are easy to replace, configure or expand. The mimic can communicate with the equipment either through a backend or through a proxy server to provide a secure connection. There are health-check and error handling mechanisms that ensure round-the-clock availability of the application for the customer and employees.
All components of the mnemonic diagram are represented as SVG components, which ensures ease of rendering in SVG. These components are easily customizable and scalable, and adapt to different types of data.
Data exchange with the backend is carried out using the REST+WebSocket connection. The latter is used as a matching service that notifies about events occurring on the backend, such as when a vehicle passes an RFID tag. This allows you to instantly update data on the front end. And data updates via REST are performed using the BFF service. BFF (Back for Frontend) is a convenient tool that allows you to easily modify data and reduces the load on the frontend, avoiding unnecessary data processing on the client side.
Let's take a closer look at what problems and advantages arose during the development process:
Optimizing rendering in React. Internal optimization methods were used here. They allow you to redraw only those parts of the mnemonic diagram that have changed. For example, if a vehicle moved, we only updated its position without redrawing the entire mnemonic.
Using WebSocket. We wanted to provide real-time updates, but due to the nature of the equipment and transport, this task was difficult to implement. Instead, a unified alert service notifies that the vehicle has changed location. This is displayed on the mimic diagram. But sometimes the number of alerts is excessive due to the RFID tags present on every vehicle. To solve this, we used the browser cache and network request optimization.
Lightweight rendering using SVG. We implemented lightweight rendering, which ensured that the mnemonic loaded quickly the first time you visited a web page.
Using the BFF service. The Back for Frontend service reduced the load on the frontend, facilitating data aggregation and simplifying changes to contracts and data structure.
RFID equipment and CCTV cameras
In the process of solving the problem, we encountered two problems: excessive redrawing and receiving video from cameras on a mnemonic diagram.
We solved the problem of excessive redrawing using cache and memoization. Now, when changes are made to the mnemonic diagram, only the necessary elements are redrawn, which significantly improves performance.
As for receiving video from cameras, the client’s task was to detect vehicles using RFID. We successfully read the location, although not in real time. However, there is a security issue with CCTV cameras. Maximum security of video stream transmission was important to the client. We have created a special proxy server that encrypts video and audio streams from the ws protocol to wss (web socket protocol using encryption). Employees can now receive video feeds securely while meeting customer data security needs.
Service analytics through visualization of metrics
During the development process, we had a need to collect analytics from the front end, including information about the number of visitors and their interaction with the mnemonic diagram. For this purpose we decided to use the Matomo service. This is an opensource library for collecting analytical data, similar to Google Analytics.
However, to meet the client's requirements, we needed to integrate analytics from MATOMO into Grafana. This is where the problem arose, as there was no suitable plugin that could transfer the data. Therefore, we used home-written tools using the MATOMO API and the Json-datasource library for Grafana. This is how it happened to transfer the data and display everything on dashboards.
Results of work at the front
As a result, we managed to achieve flexibility on the front end and maintain a balance between development speed, service availability and reliability. In some cases, in order to speed up development, we abandoned some tests, especially for the mimic diagram. This did not affect the availability of the service, since there was a health-check and sentry for error aggregation that monitored the stability of the service.
The mnemonic diagram itself is unique for each enterprise and its geography, but it is easy to scale, add new objects, change SVG components or rebuild as necessary.
Scalability, flexibility and abstraction are achieved through polymorphism of components. Although some parts of the API contract can be changed, the components always remain functional, independent of each other. Some components may have complex business logic, but most of them simply perform their function without knowing their location on the diagram and other details.
Results
We automated the client's business process, which was previously heavily dependent on the human factor. Let us recall that the business task was to create a system that would enable employees to place orders, see changes in their statuses, and monitor the movement of orders between workshops in real time on a graphical mnemonic diagram. What we got in the end.
First of all, we minimized the human factor in the operation of the service, since this was most often the cause of errors and incorrect collection of order data.
The client’s transport stopped disappearing: RFID monitoring made the route between workshops more transparent and helped identify the place where delays occur. And employees of two workshops received a system that displays the current status of each order. It became convenient for them to see what is happening with orders and optimize their time, depending on this information.
Secondly, inside the service there was analytics on the most problematic areas.
This made it possible to analyze delivery time, taking into account the departure and arrival of the car, parking in front of the workshops, and the beginning and completion of unloading. This solved some delivery issues and helped:
increase delivery speed;
reduce the downtime of trains in front of workshops;
reduce the customer's financial losses.
Third, service capabilities can be easily expanded. For example, if the customer needs to track the movement of vehicles between other workshops, then only new integrations will need to be configured, adding readers on the tracks and integrating them into the system, as well as adding data processing on the data aggregation service. The rest of the processes are already running. This will reduce development time and costs for other workshops and paths.
This solution can be used in any production facility where it is possible to install readers and connect them to the service. And tags can be installed not only on cars, but also on other vehicles – trucks, cars, buses, etc.
Today, our system records more than 200 cars per day. Thanks to automation, the client was able to reduce the downtime of wagons and cargo by 20%. Previously, carriages could stand for a week until one of the employees of another workshop came to check where they were. The number of deliveries per day also increased from 60-70 to 120. And with the help of analytics, the client saw critical problems in the process and was able to increase the speed of unloading raw materials by a third, the time was reduced from 60 to 40 minutes.
