“Pretend it”: how unmanned vehicles “surrender to rights”

Nvidia Simulator

Roughly speaking, there are two areas in car certification. Self-certification is popular in the United States. A safety standard is published, and manufacturers claim that their product satisfies it. If a problem arises, and it turns out that they lied, then they are faced with additional legal liability.

Europeans are leaning towards a different approach, known as homologation, in which the government or its approved third-party testing agency certifies that the vehicle meets the standards.

In real life, we often come across a combination of these approaches. For example, in the United States, external crash tests are conducted by both the NHTSA (government agency) and the Institute for Insurance and Road Safety. Thus, an NCAP rating system was obtained. Europe has adopted this rating system and it has become a key way to certify security. NCAP compliance is voluntary, but car manufacturers believe that if they don’t earn NCAP stars, it will have a big impact on sales, so it’s actually not so voluntary.

Proponents of each method consider the other approach insane. This is especially true when it comes to unmanned vehicles that do not obey the many old rules.

At an Nvidia graphics accelerator conference on Wednesday, representatives from AVL, a major European private testing laboratory and the largest private company in the automotive industry, put forward their ideas on how they can test and certify unmanned vehicles.

It was a test plan that many could have imagined, and it included the following points:

• Private testing live
• Software and hardware modulation testing with the addition of a random selection of scripts from a rich library of already known tests, including testing extreme situations that are not easy to verify in the real world.
• Some tests on public roads

Europeans expressed the view that an older American approach, which includes self-certification according to the ISO 26262 functional safety standard, as well as mandatory compliance with the federal vehicle safety standard and the NCAP voluntary rating, is incorrect. All other modes of transport are tested and certified by the state supervisory authority. They also stated that the recent 737 crashes show what happens when the wrong approach, so they want more certification with more stringent standards.

This is not the only approach proposed in Europe. Netherlands Certification Agency RWB offered a kind of driver license software, which will be more like a driving test than certification on a list of options.

Human driving checks are terrible

Any set of standardized tests will cover only a small fraction of what needs to be verified. It is hard to imagine what could be different. An example is obtaining a driver’s license, especially in the United States. This test hardly covers anything, and teen drivers with a terrible level of knowledge pass it routinely. We agree with this because we believe that people inherently know how to do certain things without proving it in tests, and because we don’t want to make the test too complicated. But in some countries, things are much more complicated.

Consider the industry leader, Waymo. Waymo cars have already traveled 15 million test miles. They also recorded about 10 billion “miles” in simulations. (In addition, unlike real miles, which in 99.9% of cases are just boring driving, miles in a simulation are special situations designed to load software).

But despite all this, Waymo is hardly ready to safely release their cars on the road. Last year, they announced that they would operate in real time by the end of the year, and that they were already operating vehicles that did not have a driver, or a device with a security monitor that could not take control.
In fact, they implemented only a small part of their autonomous operations, and their commercial services are very limited (my theory is that the accident with the Uber car reduced public tolerance and made Waymo behave more conservatively).

However, the fact is that even after all these tests (which were carried out thousands of times more than in any research laboratory) Waymo is still not sure that they are ready for release, although there is a feeling that they are already close. I repeat, a thousand times more tests.

A certification laboratory would at best confirm that the vehicle under test meets all basic requirements. This will mean that the development team is not negligent or lacks expertise. This is important and valuable, but does not confirm that the car is ready to leave the roads without supervision. A Uber car might not have passed such a test, but they did not attempt to certify it for unmanned or commercial use. It was just a prototype that might have been driven by a careless driver.

Thus, the certification process will be both expensive and time-consuming, and hardly useful. Too much bureaucracy and a lack of a real security check is not a good set.

Simulation for certification

Standardized tests, especially simulations, are not well suited for certification. This is because manufacturers will want and access simulation scenarios. Manufacturers want to test their cars in these scenarios in advance, and if any of the tests fails, they will make corrections to the machine. Manufacturers will not pass the car for certification until it receives an excellent mark in all tests. This forces manufacturers to improve their cars, but does not provide information about the quality of these cars. We only know that they pass the tests that must pass.

Simulated street on a rainy day.

You will not know how well a car is made if you test it on tests that it has already seen. You will know how good it is when you spend a ton of tests that this car sees for the first time, and when you see how well it copes with them. Only in this way you can find out how well this car can cope with an infinite number of new situations that it will face in the real world.

The only way to really experience the car is to test it in difficult situations that he had never seen before. As soon as the situation appears in any of the tests, the providers find out about this and put it in their own test collections. This means that the certification body must constantly propose a large number of new, meaningful and realistic tests. There are a great many different options for test situations, and in the case of unmanned vehicles, edge cases can be found even in the middle of the street, but there are some limitations.

Regulators should update tests on a regular basis. Many car manufacturers will often release new versions of software. During testing, they do release new versions every day. When the machines are put into production, updates will still be released approximately once a month, and if security errors are detected, even more often. Performing new tests on a monthly basis for each company is impractical. You can easily create simulation scenarios on a regular basis that are variations of existing ones, but you need realistic tests that are similar to what can happen in the real world. You do not want to make a mistake or transfer a machine to production based solely on how it handles situations that will never happen.

It is also worth noting that the billions of miles in the simulation that Waymo cars drove are not at all similar to those from AVL or those that were shown by Nvidia. They are full-fledged simulators that require recognition of visual information that try to create a virtual world in the style of video games, then provide the car software with an artificial image from cameras, lidar scans and radar data, after which they give the computer the opportunity to try to control the virtual machine. There are three different levels of testing that try to simulate as much data as possible in order to maximize the machine’s performance.

These simulations are useful, but much faster to do simulations that work with information that does not need to be recognized. In this approach, you are not creating a visible virtual world; instead, you are creating its basic representation. The simulator knows where all the objects are at the abstract level, but does not display them for recognition by the program. Instead, the tests go down one level and replace the perceptual system with a module that reports the same information as the replaced system. For example, he can report that with a probability of 85% a car is traveling in a certain direction, or that a pedestrian is in a certain area with a probability of 90%. In such simulations, it is checked that the machine receives all this information. This is only partial testing, but it is so faster that in this way you can conduct much more tests. Combinations of different methods must be used, but most of the test miles will be covered in post-perception tests, and as I understand it, billions of test miles for Waymo cars have been obtained in this way.

Tests that require recognition, or simulation with visualization, look much better as demos, because they offer realistic video game style pictures that people like. They are good for testing all aspects of the car, including the perception system, but the problem is that they test the car in an artificial, not real world. It is possible that the system will very well recognize virtual gaming pedestrians, but not very well cope with the real ones in the real world. This situation will mean that you received incorrect information from this type of simulation.

Inspectors from government agencies go even further. They use 3 types of simulations with visual display:

• A basic simulation in which you simply run car software on a computer, the computer generates artificial images from sensors and allows you to control a virtual machine using software commands.
• Software and hardware modulation, in which a real car is connected to a computer with a simulation that tries to behave like real sensors and a car for the on-board computer of a real car.
• Hardware simulation of a real ride, where the car is placed on a roller stand, on which the car can scroll wheels. This stand is located in a large room in which robots are redone to look like cars and pedestrians and move around the stand, simulating the behavior of objects from the real world. Ideally, real sensors should be used, although it may be necessary to use an artificial camera and radar. In the case of cameras, you can just point them at the locked screen.

All of these options sound good, although number 3 is not possible in some cases (for example, for a Waymo car, which determines its position using a laser-lit road texture that scales under its wheels and uses a number of other complex sensors). However, all of these options are slow, expensive, and suffer from the problems described above. They should work in real time, which in fact can be a big limitation compared to conventional testing or testing without display, which can be scaled to perform serious amounts of necessary testing.

Outsiders do not know how to test

In the case of an external laboratory, testing without visualization is very different. They can only be carried out in close collaboration with the vehicle development team. The truth is that the team knows how to test their car better than anyone else. In addition, since these teams are developing completely new ways to ensure safety that did not exist before, outsiders may not be able to fully understand the security systems of the vehicle under test. Standards can only define common truths and existing best practices. They do not imply innovation. There is no clear test to test the quality of security innovation. To do this, you have to come up with a new test.

All this does not give a clear idea of ​​the importance of the laboratory approach to testing, but this does not mean that it is not important. This approach can set the minimum test requirements, and unmanned vehicle developers will contract with testing companies and use their services to understand that their cars meet these very minimum requirements. And also contracts can be concluded in order to raise the level of quality above the minimum requirements.

The test proposed by AVL, however, would not provide enough information. An unmanned vehicle can pass these tests and still have serious problems and pose a big risk. The only way to minimize this risk is to carry out tests that were developed by the car manufacturer, as well as testing on real roads in various situations. You must make sure that the interests of the manufacturer coincide with the interests of society, and that there are strong reasons not to cheat, not to lie and not to neglect. This is still an unsolved problem, but a lot of attention has been paid to it.