Development for factories. How it was

This article is a response to a previously published article about IT in factories. I read it and realized that I have something to tell about it. The questions are from there and a little more. I will clarify right away that I did not work directly in factories, but worked for a company that provided services for the automation of production lines to various enterprises, but mainly focused on working with scales and dosing.

What brought you to industry?

This was my very first job as a programmer after my very first job as a croupier in a casino. In distant October 2008, I came to an interview with no knowledge of SQL, very basic knowledge of Delphi, and some knowledge that I had managed to acquire before at school and university (the basics of Basic, Prolog, List, C++). A so-so candidate, but no one was eager to work there. They told me straight out: “Just come. We'll teach you everything.” And I came.

It all started with the automation of the first concrete plant, which should be ready in a couple of months, by the New Year. The first thing I did was rewrite what had been done by the previous programmer and figure out what kind of beast this is – RS-232 and the ModbusRTU protocol, which I will have to use to control some MDDVV and Master.

The first pancake was not a lump, but it was a little burnt. But to see how first one piece of iron lying on your desk comes to life, and then an entire line at the plant – is comparable only to the first sex. Result: the operator comes to work, turns on all the iron, the heater in his closet and the heat gun in the room with the dispensers, sits down at the computer, starts the program, selects the desired concrete recipe, sets the mass and presses “Start”. The fairy tale ends here and reality begins. The crushed stone has frozen in the cold and has not yet warmed up. The operator takes a sledgehammer and starts hitting the dispenser. In a few seconds, the crushed stone starts flowing. The operator returns and continues to monitor the process. And this is only the first of dozens of various problems and snags in production. And no difficulties should interfere with the work of the program and the hardware (the electronics cabinet and the control relays).

Interesting tasks

Every time there was something interesting. The most memorable project was, perhaps, the project to create an experimental “analogue” of the American installation for separating material into dirty (exceeding a certain radioactive background) and clean. It all started with the fact that everything could have been done much simpler and cheaper, but they wanted “like they do.” When the installation was ready, it was taken away, but soon (after several months of downtime somewhere in a warehouse, when when ordering it was needed “yesterday”, as usual) it turned out that something was no longer working. So I had a business trip to a closed enterprise in a closed city. i.e. it is not enough to just get into the city, bypassing the checkpoints, but the enterprise is also guarded as if they are breeding supersoldiers there. The checkpoint is more strict than at the airport. And in a warehouse 20 meters from the building, on the door of which there is a sign of radioactive danger, there was our installation, which we successfully revived. Why was it standing idle? We were waiting for the main character – an experimental sensor from outside Moscow, from another scientific center. On the way out of the enterprise I lost my laptop, because it was not properly registered, and you can't take anything out of there. As usual, everything was decided by the morning.

A few months later, the installation finally ended up where it should be (probably). And now it was time to set it up with that very sensor (a kind of special Geiger counter), which could also transmit information. So they called me for the second time and I found myself in St. Petersburg for the first time. In transit. To Sosnovy Bor. To another closed enterprise. At the entrance, they familiarized me with the instructions in case of radioactive contamination, dressed me in a robe and gave me special equipment to take with me, as well as a ticket for lunch. After passing through the checkpoint and for a long time almost across the entire territory, in the very last warehouse with the inscription 0.7 μSv on the wall (if I remember correctly. Although it could have been 7). From the entrance, in a direct line of sight behind the trees, about a couple of kilometers away, the cooling towers of the Leningrad NPP were visible. Half of the room was filled with boxes, which I did not really want to approach, and in the middle of the free space was our installation. It's a bad idea to try to set up delicate equipment in a room where the background noise ten meters away exceeds what is considered “clean” for the material. However, we had both samples and had a productive time.

About other impressions. At the last stage of cement production, there is such a noise from thousands of balls grinding the finished product in huge drums that you can scream at the top of your lungs into the ear of your interlocutor and they won't hear you.

Is it difficult to be a leader in IT?

I wouldn't be able to answer this question if my boss hadn't once said, “I'm tired, I'm leaving.” He started the company when I was still a baby. A year later, he said that he was definitely leaving, but something had changed in my head during that year, and I suggested that he simply re-register the company to me. That's what we did. In short, it was a useful experience.

To be more specific, I was not prepared for this at all. I realized that I hated accounting processes. I didn’t like my job one day a month because of this. Developing software — how easy it is! Being a director without experience and knowledge for this — it’s very difficult! But interesting! You have to find clients, meet with them (often in other cities), negotiate, calculate the cost, discuss the technical specifications, draw up a contract, delegate tasks, negotiate with contractors, assemble equipment, write software (I continued to do this, especially since the processes were already debugged at that time), install equipment, come when someone asks “nothing works for us” and explain how not to do it and how to do it, come again, sign the certificate of completion, get money, come again, but for money, finish the functionality upon request, etc.

My office was at home, the other two employees worked on the fact of having a job (doing their own thing in their free time). It was 2013. We had remote work many years before it became mainstream.

Is it difficult to negotiate with security?

A strange question. Anything can be agreed upon. Even about returning a laptop from a closed enterprise. And the main safety issues in our case are occupational safety. This is not working with signs in an office. There is hardware at work here that can easily take a limb or even a life. Therefore, all possible situations must be foreseen. And an emergency stop button in the program, and a stop button in the relay cabinet, and separate means of protection at the stage of equipment production/installation (emergency stop cable along the conveyor, a sensor for opening the lid of a concrete mixer, etc.). So, on our part, special attention was paid to health protection.

Regarding safety, it is also worth mentioning the legal one. Before we started making an analogue of the American installation, we were sent a contract, which was worth reading very carefully. According to it, for any sneeze we had to bear all sorts of costs, and the customer – nothing. According to the contract, it turned out that we had to start doing it at our own expense, then the customer could change his mind and simply terminate the contract. As a result, we would be left without money with useless hardware. I spent three months talking to their lawyers, explaining why I did not like it. In the end, ours won. Our standard contract was signed with the addition of one paragraph from their contract. The customers were adequate, but even if you trust them completely, in the contract you need to exclude all sorts of points where you can simply be left without money.

Did import substitution influence your work in any way?

The company, from its very foundation, was initially engaged in developing its own controllers, and then switched to using industrial Russian-made ones, and until the very last year, we used only Russian equipment (except for computers, of course). i.e., we were engaged in import substitution when it did not yet exist…

But in one of the last jobs I received an order to count the number of filled bags for cat litter. (Then I learned that some foreign fillers differ from Russian ones only in packaging.) There were three controllers (two of which worked), from which it was necessary to collect information about the amount of dosed material. And I realized that this was a great opportunity to use Raspberry Pi! A Chinese plastic box was bought in a store, a Chinese power supply unit was ordered that fit in it, a Chinese USB/RS-485 adapter from “Alishka” was used (the Russian one was made according to the specification and therefore had problems with starting on Linux, and the Chinese one simply always worked). An LED was also bought and a DIN rail mount for the “Raspberry Pi” was printed on a 3D printer (photo below). As a result, the program on Go began to blink the LED at startup, informing that it was working, and, constantly polling the controllers, saved the received information at the right moment. And then a specially trained employee printed out shift reports using a Delphi program.

Is the development stack in an industrial IT cluster different from the stack in classic IT companies?

Depends on the industry. We had it all very simple. A program in Delphi, a Firebird DB, reports in MS Excel, OOo Calc (the ancestor of Libre Office) or Fast Reports. One development was in Go, as I wrote above.

Not all programs need or can be written as a regular application. Several programs were written for programmable controllers using CoDeSys. There are 5 languages ​​inside that can be combined. Three of them have always been enough for me: flowchart (FBD), Pascal-like (ST) and assembler-like (IL). The peculiarity of developing for controllers is that the written code will be executed in an infinite loop and it must be written taking this peculiarity into account.

In general, anything can be used in an enterprise, depending on the tasks.

How about IT get-togethers for factories?

A strange question, but there is one. IT specialists working for enterprises are no different from others and also attend various events. It's just that we also have a bunch of interesting industrial exhibitions where you can find something new and interesting. For example, a flexible auger for feeding cement. Never used, but sounds intriguing…

And a few words about development

At this job, I understood better than anywhere else what parallelization and synchronization of threads are, and I also applied various popular patterns without even knowing about their existence.

Working with equipment. Only synchronous and infinite. I usually had three queues of commands: for working with weight controllers, for working with control controllers, and errors. All this was displayed on the main form of the application (mnemonic diagram). A separate thread constantly checked the queues, constantly throwing up commands for reading when the command queue ended. Write commands had a higher priority and were added to the beginning. A separate timer on the form constantly read the received data from the devices and displayed it on the mnemonic diagram in the form of animation, loading bars, numbers, etc. We used animation to show the equipment status (open/closed or on/off), in case of a discrepancy between the expected state and the real row, the skull blinked red. Loading bars showed the filling of the dispenser relative to the specified weight. etc.

When batching concrete, you can move step by step, or you can do it smart. If our mixer is 0.5 cubic meters, and we need 1.2, then we batch three times 0.4. i.e. each batcher and mixer must work 3 times. Thus, when the batcher works and the material is dumped into the mixer, it does not wait for the next cycle, but immediately begins its own, if this is not the last one. But they cannot dump until the previous cycle in the mixer has worked. There was quite interesting work with flows, separate for each batcher, dump system and mixer.

But for special cases, a manual mode is also provided, when the operator independently controls the equipment with a cursor and begins dosing after manually entering the required values.

The batching results are saved in the database to be used later for various types of reports. Including manual batching, so that there is no possibility of shipping something to the wrong side. This is usually the task of automation in Russia. Only in one case was an accelerated transition from manual batching to automation made, since the plant received an order to supply concrete for the construction of a bridge and a certain batching accuracy was needed. Our system, even taking into account the errors of the sensors, usually fit within 0.5% on new equipment.

Conclusion

This was my job. Difficult, but incredibly interesting!

I could also write about other business trips. About debugging at the plant from 8 am to midnight, about icy, noisy, dusty rooms. About shouts on the radio: “Stop!” About arguments and swearing. About non-working sensors, equipment that breaks down, a mixer piled high, etc. But it was really exciting and I am glad that I got such an interesting experience.

Since then, I have thought about returning to enterprise development (not only software, but hardware as well), but something told me (especially in recent years) that it was not worth it. Let others do it. Although one day our neighbors will have to rebuild entire cities and they will need a lot of good quality concrete and any help. So we'll see…