Neural network chip with an interface familiar to an Embedded programmer

I usually write series of articles for two reasons. Or I’m participating in some kind of development and, on the one hand, I want to talk about it, but on the other hand, I want to describe something interesting about the technologies used there, since I have mastered them sufficiently. The biggest series was about the Redd complex, which grew into the development of a USB analyzer based on it. Another reason to sit down at the keyboard is that I found some interesting craft.

In today's article (which, I hope, will grow into a series), everything is somewhat different. The craft itself found me. It is being developed in another department of our company. But it was in the next department, not in mine. How it works inside, I'm not supposed to know. But in addition to reading the documentation, I can not only ask the authors about the user interface, but also give them advice on how to change it.

We will talk about a chip that, as planned by the developers, should help Embedded programmers start working with Artificial Intelligence, without plunging into its depths, but operating only with entities familiar to them. Let's look into it in more detail.

Why is this necessary?

If an organization works according to a project model, then someone in it finds a Customer who completes a certain task. If the task is based on technologies that are well known to developers, that’s just great. But in reality, usually from some piece to absolutely everything is new. If you wait for projects where there is no unknown, there will be no work, which means there will be no money. We know what happens to those who wait for princes, rejecting real options… Therefore, usually when a project starts, people sit down and read the documentation, fortunately, usually new things are not at all unprecedented, but simply unusual. You can sit down and figure it out in a reasonable amount of time.

I won’t undertake to speculate on how long this will last, but now more and more often Customers want their systems to have elements related to Artificial Intelligence. And this is where the problem arose. Everything that I said above about “simply unusual” applied to Embedded technologies. A long time ago, when our guys made their first timid attempts to take on projects with AI, it turned out that it would be difficult for a classical programmer there. The whole approach to work needs to be changed there.

Actually, instead of just programming, you need to figure out what type of neural network to choose. Then install the SDK for it. Then start doing something different from the usual programming… In general, learn a lot of new things.

No, those who are immediately accustomed to this will do everything. Let’s say, my son, he’s making some kind of neural networks for his school work. But when I asked him for help with one of my simple ideas, it turned out that he was doing it all at the level of shamanism. He begins to think about which package he will take for work… Maybe this one, maybe that one… Moreover, he is guided by the principle “I feel it in my gut,” and any hardware programmer will say that the gut is not a calibrated measuring instrument. Give him a clear understanding of the processes.

In general, it is difficult for those who are accustomed to working with hardware to change their approach to work. Personally, I just didn't touch these newfangled things.

And so our management came up with the idea of ​​hiding the neural network inside the microcircuit. Make a sort of coprocessor. Here “sort of” and “as if” are not parasitic words. This coprocessor is implemented in hardware. Our implementation is software. Our code with the neural network is sewn into a separately soldered microcontroller, but from the point of view of the end user, everything looks like a real coprocessor. Nobody knows what and how it works inside. The end user is provided with a familiar interface. Now it’s I2C, but there are also SPI in the plans. To access the chip there is a simple API, designed in a style familiar to a system programmer. We send requests to the chip and receive responses.

Actually, this is exactly the thing they gave me for experimentation. And since the developers are interested in as many opinions as possible, they were also asked to describe all this in the form of articles.

What network is used

The best approach from a scientific point of view is to try many different types of networks, make a list of advantages and disadvantages and choose the best or best… But science is inconsistent with reality. But the reality is that it is impossible to allocate an infinite number of employees to an internal project and pay them endless salaries for endless experiments, most of which will go to waste.

But why make big mistakes from scratch if you can study the experience of those who have already made them? This will allow you to discard checking obviously unpromising things and choose those that will allow you to gain a positive experience. And only later, when there is a lot of this experience, the intuition will catch up. And it will be possible to look for other ways. It’s better to start with the one that others praise.

There are many articles and videos online from ST developers who work with ESN (Echo State Network) networks using reservoirs. I’m writing clever words, but if someone asks me in the comments about the details, I won’t be able to say anything. For those who are interested in details, I asked our developers to give me links to theoretical articles. This is what they gave me:

1) Types of Recurrent Neural Networks For Non-linear Dynamic System Modeling
2) A review on TinyML: State-of-the-art and prospects
3) Deep Reservoir Computing: A Critical Analysis
4) Long Short Term Memory-based anomaly detection applied to an industrial dosing pump

True, these ST tanks can only be judged by theoretical calculations. There is no practical code. Moreover, even their theoretical calculations are vague, so they still had to start from scratch. But at least in the direction where you can definitely find something. Why ST? The authority of these developers is quite high. It is unlikely that they will post some kind of fiction and adjust the results to suit their conjectures. If they write that there is success, then there is success. Therefore, our design also uses ESN Network with reservoirs. And in principle, our guys claim that they confirm the figures stated in ST.

What we ultimately recognize

As a result, the resulting chip is able to recognize anomalies in signals. Judging by the conversations of those who work with neural networks all the time, this is a very well-known term. But I repeat: I don’t work with them. And if I’m not the only one, then I’ll explain to my brothers the essence of the problem being solved.

Let's say we have a certain engine. ST people love to talk about the brushless electric motor. Here we are filming either its vibrations or its sound. And we train the network that this captured signal is one normal behavior, and this one is another. And the rest are anomalies. If desired, you can even train which specific signal corresponds to which anomaly.

Well, then, focusing on the signals, you can somehow catch anomalies and use them. The simplest case is to turn off everything in an emergency so that the system does not go haywire. Well, or what other measures to take. The very first experiments were carried out on a fan. The system learned to determine the speed at which it operates by analyzing vibrations, which will be different at different speeds. In one of the following articles, I plan to tell a real case that I once solved to analyze the operation of a machine. But then everything was decided by collecting data and software analysis of the spectrum. It’s a good thing, but collecting new data requires some tinkering. Here it’s captivating that he pressed the button and showed the sound. Pressed it again, showed a different sound. A few presses of the button and the system learned.

Yes, for this specific machine! But she trained remarkably well for him! The network learns quickly, literally in seconds, so you can calibrate anomalies for specific equipment. They collected it, trained it, and then it works. For the production of thousands of products per shift, the need to train each one individually is a disadvantage, but for piece items, precise fitting in place is an advantage. Although, in the case of conveyor production… Well, press the button several times and wait a second. At the television plant in our city, at one time they produced 700 products per shift, for which the adjusters set the geometry for each, and nothing! And here the button can be pressed by the robot.

In general, the range of signals used can be quite wide. One day I met the lead developer from that group in a grocery store. He was clearly on edge. He pressed me against the vegetable counter and excitedly told me that they had tried to recognize gestures using this system. Without any training, the recognition accuracy was 80%. Since then, measures have been taken to improve this very accuracy, but here is the fact. Signals can be the most surprising and unexpected. And perhaps many signals can be classified into classes and anomalies.

An important feature of training

The most important feature of ESN networks is their ease of learning. You don’t need serious computing power, you don’t need large datasets. It is enough to show the network a signal for a few seconds so that it learns to classify it. More precisely, I was told that in general, even milliseconds are enough.

The developers themselves claim that in this case the network is trained not strictly on the same signal, but on a plus or minus signal. Here I even know the smart word “retraining”, this is when the neural network recognizes only those samples that were shown to it and does not perceive the rest. The child told me this, explaining the importance of a large dataset, which in the case of ESN does not exist. I personally am just at the beginning of experimenting with the chip, so I won’t say anything. But in the following articles I will pay special attention to sensitivity to errors. Let's hope everything is okay there.

What kind of chip

This whole thing is implemented in the ESP32-C3 chip. As those responsible for the choice explained, this chip is insanely cheap, but at the same time quite powerful. It is available from a large number of sellers. There is decent documentation on it. In general, why not?

In general, I agree with them. In general, ESP32 is one of the most famous controllers for working with Wi-Fi and Bluetooth, but in our case, none of these functions are used in it. It's just a convenient die with a RISC-V core. Its maximum clock frequency is decent, and there are opportunities for energy saving. But the main thing is the ability to encrypt the code.

Many developers of AI-based solutions like to brag about their results in articles, but at the same time do not share their findings with anyone. Our developers are no exception. They don't want to give away their work. But give out the chip for testing – why not? In older systems, the firmware was closed with a security bit. To update the firmware hidden in this way, the file with it still needs to be given out. We had to come up with some of our own protocol encryption methods. Not always perfectly stable. Hacking a security bit is a favorite topic for many hackers.

Oh, what a commotion there was among my friends when the Chinese opened the AtMega8 protection and made a clone of their board! Older versions of the hardware were compromised and counterfeits were being sold for pennies on TaoBao. It is clear that buyers began to look at them, and not at legal complexes. I helped the guys make a hinged additional protection module containing a Soviet chip that has no imported analogues. The Chinese could not figure out what was there… New versions of the software did not work without this chip, although the project suffered, but at least it did not become scrap.

With ESP32 everything is simpler. We flashed the encryption key into the fuses, after which the controller will decrypt the “firmware” on the fly. A new version has been released – the file with it can be sent to everyone. If you have a crystal with the appropriate key, the firmware will work. The others will look at it, but will not be able to open it. And they won’t be able to use it on crystals purchased elsewhere either. Therefore, the problem of “giving away developments” when using the ESP32-C3 is not worth it. Source code – yes. They don't even give it to me. And binary code – as much as you like! Well, what a wonderful controller!

Development boards

“Large” option

So, we found out that an I²C bus is needed to control the chip. In general, no one bothers you to take and connect everything to a PC via FT232H, but it’s not that simple. Experiments require signals from sensors. In general, you can connect sensors to the same PC, but how convenient will all this be? Therefore, the developers decided that it was better to send signals from a microcontroller on a standalone board. So autonomous that it can even be powered by a battery (but it can, the main source, of course, is the USB cable). Let's slowly improve the flowchart… So. We already have the target chip, control controller and power…

Fine. Now let’s not forget about the very sensors from which influences will be taken for experiments. To do this, the board is equipped with a MEMS gyroscope-accelerometer-magnetometer MPU-9250 to catch various vibrations and other changes in position in space, as well as a pair of MP34DT06 PDM microphones.

To interact with the user, you just need a screen and some buttons.

When anomalies are caught, it is sometimes necessary to indicate this with an audible signal. For this purpose, the system has an I2S DAC with an amplifier and a speaker. Add them to the drawing.

If this system is all so autonomous, then it simply asks for wireless communication, so a module based on ESP32-S3 was chosen as the control controller. To program it, you can use access to the JTAG and virtual COM port via the main USB connector (there is such an interesting thing inside the ESP32-S3), but the USB bus may be occupied by some functionality. So we add a USB-UART adapter to the board to update the firmware of the control controller. And, since the module is used, we output its free lines to the connectors of our board.

As a result, we get the following system:

And the final touch. Let's add a fashionable ability to break off the target chip from the debug board in order to install it somewhere in a working system. To be honest, I have never heard of anyone ever breaking something off a breadboard, but for some reason it is common practice to add such an option. Our developers did not escape the temptation either. Moreover, as an amateur balalaika player (more about this here), I noticed that if the part with the chip is broken off, that half will look like the body of a balalaika. Let me take a photo of the resulting board, otherwise a beautiful rendering was made on the initial picture.

Minimalistic option

You have to pay for everything. Each element soldered onto the “large” version of the module costs money. A penny adds up to a decent amount. When I saw this amount on Ozone, I almost fell out of my chair. But someone may already have an Arduino with a bunch of ready-made sensors! Why overpay? For this case, a minimalistic version of the module has been developed. It looks like this:

To be honest, I’m somehow used to Arduin wiring for connectors with a pitch of 2.54 mm, so the white I2C connector instead of the usual pins confused me. But the developers said that, firstly, the necessary lines are duplicated on the J4 connector, you just need to solder it in… And, secondly, for the Arduino you can buy the newfangled Grover Shield, on which the I2C connector looks exactly like this. Then the connection will look like this:

Well, there are other development boards with just such a connector. Let's say this, also based on ESP32 C3:

Software support

For experiments with the module, two repositories have been created on GitHub. The first one stores everything about the SDK – documents and functionality. This repository is located here

Grovety/grc_sdk

No matter how good a document is, it is still usually very difficult to understand it without real examples. Therefore, in the second repository there is a project that is flashed into the “large” board by default. This repository is located here: Grovety/grc_devboard

The project currently includes three demo applications:

• Gesture recognition (accelerometer based),
• Recognizing rhythms that are usually tapped with a finger,
• Vibration recognition (the developers trained to determine the speed of the fan).

I plan to make a separate analysis about each of them. Today is an overview article, so I simply said that these applications exist, they are published in this repository and can be taken as a basis to understand how to use the SDK.

Where can I buy

Large boards are already on sale on Ozone. You can find them there at this

link

… The price, however, is biting. Not really for amateurs. But the management promised that budget options would appear there in the near future. In the meantime, if you suddenly want to get paid, please contact me in a personal message using the coordinates from your profile or in dialogues, we will try to resolve the issue.

Objectives of the article

Well, the first goal is simple. When I write about my experiences with this chip, I will be able to refer to this general information so that it is at least clear what I am talking about.

But the second goal is more interesting. The development is currently in the active research stage. Any idea can lead the authors to some kind of insight. Under the influence of fresh ideas, they can make a sharp turn to something they didn’t even think about yesterday.
• Where else can this system be used? – Yes, this is the case where it is not the solution that is made to suit the task, but, on the contrary, tasks are sought for the solution. This is not a commercial project, but an internal project to gain experience. By stewing in your own juice, you can make some kind of spherical horse in a vacuum, which in practice no one needs.
• Or maybe someone has not just ideas, but real needs that fall on this concept.
• What if someone gets around to playing this system in practice? And he will give advice on where and what needs to be tweaked within the SDK functionality.
• Or, say, it will catch real problems of retraining… Well, or some other problems that are not visible to developers, but are completely visible to real consumers.

Actually, the main goal of the article is to broaden the horizons of our developers. Of course, they will read the comments. Well, if necessary, you can communicate in personal correspondence.