what tasks for transport video analytics are we solving today
Let's look at several popular areas where transport video analytics services are used:
Let's go through each of them.
Recording traffic violations
This is what makes driving safe. A necessary thing, although getting a fine is always unpleasant (for me too). How does it all work here?
Look at the diagram of the measuring complex with video recording “Kordon-M”:
Previously, photo and video recording systems worked only with radars that recorded speeding. A camera was synchronized with the radar, which, based on a signal from the radar about speeding, photographed the violating car. The system recognized the number for this car, and then the driver received a report with a fine and a photo from the moment of the violation. But there were mistakes, and quite a few – for example, the document might not even show your car. This happened if the number was recognized incorrectly.
Therefore, more modern photo and video recording systems not only detect speeding violations and recognize the number, but also determine the attributes of the car – for example, the model, the relation to emergency and public transport. This information is used to detect violations of the rules (for example, driving in a lane for public transport) and transmit the obtained results to the TsAFAP or TsODD for verification with the PTS databases.
A situation may arise when the attributes do not match the declared ones. For example, the model of your car according to the vehicle registration certificate is Volkswagen Tiguan, but the network recognized that the car with the same number was a Lada Granta. Then a person additionally checks the information. And only after confirmation will the driver be fined – or not, if the data is not confirmed.
Search activities
If your car is stolen (which I wouldn’t wish on anyone), you can try to find it using video streams from cameras installed in the city.
Let's discuss how you can search for a car depending on the situation:
The number on the car remained, it was recorded by one of the cameras. Then the car can be found using the state registration plate recognition. That is, by the state registration plate we can find it in the views from several cameras and, accordingly, track it.
The number was removed or replaced. This is more complicated, you can go two ways:
To search and narrow down the number of candidate vehicles, we will have to use other vehicle attributes: model, color, type, etc. Then we can set filters for attribute values in the system and look at all suitable events of vehicle appearance in the field of view of cameras. Of course, there can be a lot of such events, and networks can also make mistakes and recognize attributes incorrectly in difficult conditions.
You can also search for similar vehicles by vehicle descriptor in video streams from all cameras, without specifying attributes.
Descriptor — is a vector of numbers obtained using a neural network. The neural network encodes in it important external features of the object it sees. It works on the same principle as searching for similar people by faces: we compare two feature vectors from two cars. The more similar these vectors are, the more similar the cars are. This allows us to narrow the search without risking losing any cars due to incorrect attribute recognition.
Intelligent Transport System (ITS)
Such systems are usually installed on toll and new roads to analyze the traffic situation.
This is what the visualization for such a system looks like (example from the Internet, not our software):
We see a division into oncoming and oncoming flows. Zones are highlighted where the system does not search for anything, because the objects there are already too small or these areas are of no interest to us in principle. Here, in the visualization, the vehicle detector already has a classification into the simplest types, which allows us to distinguish between trucks and cars.
In our product, such classification is carried out separately from detection. First, the detector finds all vehicles in the frame. And then, for each of them, classification networks are launched that recognize attributes.
Attributes can be very different: brand, model, type, color, belonging to special equipment, public or emergency transport. For example, our network for vehicle types can identify 14 different types: bicycles, motorcycles, cars, pickups, trucks up to 3.5 tons, trucks up to 12 tons, tractors, road trains, trailers of passenger/truck vehicles, and so on. Together, they cover all types of vehicles that are required by GOST in Russia.
Why do all this? If we know what kind of transport and in what quantity passes on the road, we can predict when it will need to be repaired. For example, heavy trucks with trailers and road trains cause much more damage to the road surface than cars or motorcycles. The more trucks, the faster the road surface will need to be repaired. The flow of vehicles can be aggregated over a certain period of time and the expected damage to the road that will occur can be estimated. And after building a predictive model, we can approach repair work sensibly.
Such systems (ITS) are also installed on high-speed highways, where preventing road accidents is especially important. For example, with the help of ITS, it is possible to quickly determine that there is an accident or fire on the road. The operator will receive a message about this and send an emergency team to the scene. For example, in our product, fire and smoke recognition looks like this:
We divide the images into a grid and use a neural network to analyze each sector separately. The sectors where the network detected fire are highlighted in red, and those where it detected dark or light smoke are highlighted in yellow.
In addition, with the help of ITS, it is possible to determine that there is a parked vehicle, person or animal on the roadway. Then, a message for drivers is displayed on a “smart” billboard several kilometers before a possible collision, informing them that there is an obstacle ahead on the road and it is better to reduce speed.
Access control (checkpoint)
Once we had a big project in Rublevo-Arkhangelskoye, a new smart city, for the construction and further life of which it was necessary to equip “smart” checkpoints. What are they?
This system combines verification of a person and the vehicle they arrive in. That is, the database is filled in advance with information about who will arrive, what they will arrive in, what number they will arrive with, and attributes. This makes it possible, firstly, to verify visitors, and secondly, to separate passenger cars from trucks. This is important because they require different documents, approvals, and different travel distances — all of this must be ensured. As a result, the system not only checked that the person and vehicle correspond to the declared ones and have the right of passage, but also indicated the direction for further movement (along certain lanes) depending on the type of transport.
I would also add to the checkpoint case “smart” barriers with access to residential complexes, shopping centers or any other parking lots. Personally, I really like that as soon as I drive up to the desired location, the car's number is read and the barrier automatically rises.
In Russia, some shopping malls and paid parking lots have implemented this solution: when entering, your number is recognized and linked to a pass that you receive right there in the machine. But after paying for the ride, when leaving, you no longer have to perform acrobatic tricks with the pass: if the number is successfully recognized at the exit, a paper pass is not needed and the barrier will open automatically. Of course, we would like to get rid of papers in this system once and for all.
Such paperless solutions also exist — for example, for parking networks in Hong Kong. There you need to register in the system once, provide information about your vehicle, link a card, and then all parking lots in the network will recognize you. In Russia, our partners have implemented a similar service based on our product. The system recognizes the numbers and attributes of vehicles entering the territory, and you can pay for parking on the website or at the exit with a card. It's beautiful.
At the same time, there are cases that are much worse than a paper pass. Many residential areas in our country do not have a recognition system at all. You need to call the dispatcher (from your mobile or get out of the car and call through the intercom) and wait for him to open the door for you. In 2024, this is an awfully long time for such a routine task.
The task of access to the territory is encountered in a variety of areas. For example, we are currently running a pilot project with Moscow hospitals: LUNA Cars is used to speed up the passage of ambulances to the territory and automate the admission of regular cars for which a pass has been ordered. To distinguish ambulances from all the others, we have a special network for attributes that recognizes all emergency services of the Russian Federation (of course, according to GOST :)).
Seamless tolling
You already know what it is if you have driven, for example, on the Central Ring Road. With barrier-free (or seamless) tolling systems, you do not need to drive through a barrier and stop to pay for travel.
In the case of seamless systems, everything happens automatically. Payment is debited in two ways.
With the help of sensors. They register the transponder installed in your car (of course, you bought it in advance and put it in the car). For example, Avtodor implemented this on the Central Ring Road.
Using video analytics, if you do not have a transponder. The video analytics system on the highway recognizes the number of your vehicle and its type, and a debt is assigned to the number according to the tariff scale. Then, within a few days, you can pay off this debt. And if you do not pay it off, a fine will come.
There may be other payment options, but in any case, it is seamless. You do not need to enter a physical toll point and wait for the barrier to open. This means that the traffic flow does not slow down. Personally, I find it annoying (and I think the drivers of the cars behind me too) when you enter a toll point, and the transponder does not work, the barrier does not open, and you have to crawl back and try to drive through another checkpoint. And if there is no toll point and barrier, such problems are basically impossible.
This illustration shows a frame with cameras that analyze the flow: they check the number, type of vehicle and determine the tariff accordingly. The cost on such roads is often also related to the number of axles of the car. The more wheels, the more expensive the ride, so video analytics systems also count the number of axles of the vehicle.
That's all for today. In the next post, I'll tell you about the vehicle recognition pipeline — it's used for all the cases I listed today. In the meantime, ask questions. I'll try to answer them in detail!
