Creating an optimal algorithm for switching traffic lights at an intersection

Introduction to the field

Each of us at least once in our lives has encountered a poorly working traffic light at an intersection: uneven traffic flow, too long switching intervals, etc. All this is due to a naive way of switching between “green” and “red”: traffic lights simply work according to a schedule, and in some places people even look after them to switch on time. It looks like a problem that needs to be solved. Therefore, our team set a goal to develop a “brain” for a traffic light, where effective decisions will be made about switching its signals.

Primitive Algorithms and the Idea with ML

Essentially, a traffic light requires a “box” that receives some data about the state of the intersection and sends an optimal signal. Let's gather together some ideas that could be inside the “box”:

• A standard traffic light that cycles through its signals. Designed for a specific traffic load ratio, ineffective when traffic load changes.

• An algorithm that turns on the green light for the busier road. At first glance, this is a good algorithm, but with equal traffic congestion, the traffic light will constantly switch the signal, which will lead to a large number of delays (when switching the traffic light, a delay is needed so that cars have time to leave the intersection)

• An algorithm that turns on the green light for cars that have been standing at an intersection the longest. Ineffective with different traffic congestion.

It is clear that a good algorithm should be able to classify the situation at the intersection depending on the traffic congestion, the length of the delay and the waiting time. Since there can be many different congestion ratios even at the simplest intersection (congestion 1:0, 1:1, 1:2, 1:10), it is impractical to create a primitive algorithm with a bunch of conditions, therefore, as is often done in such cases, we decided to train a neural network to determine the optimal signal.

A digression about your own emulator

The “starter pack” for training the neural network is almost ready – all that remains is to get a good environment (aka emulator). And we want the environment to be as efficient as possible, because neural networks do not learn quickly. Unfortunately, there was simply no emulator in the wild with only one traffic light and a convenient API. The most suitable option for our needs is Carla. This is a heavyweight realistic traffic simulator developed by a huge team. However, its use at an early stage was irrational due to excessive scale and complexity (but we will return to it later). Therefore, it was decided to make our own simple emulator in Python with rendering on PyGame. Now the model (neural network) can be “fed” the image of the intersection itself, the number of cars on the roads, etc. This is what we will do

To train the model, we will use reinforcement learning. In simple terms, the training process consists of a continuous sequence of two phases: the model's action and the environment's reaction. In the first, the model receives information about the environment (the state of the intersection in some form) and performs an action (changes the traffic light signal), in the second, the environment processes the completed action (the cars are rolling) and transmits new information about itself to the model, including a reward for the previous action. You probably guessed that the most creative and difficult task here is to invent the principle of this reward. Now, finally, let's try to reason with the traffic light itself.

Among the many reinforcement learning (RL) algorithms, we decided to choose the DQN (Deep Q Network) algorithm, since it is one of the classic RL algorithms, it is well suited for problems with an unlimited or very large number of possible input parameters (in our case, this will be the congestion of each road and the current traffic light signal, later we will transmit the processed image of the intersection instead of the congestion)

Model training through the number of machines

How does a traffic light know whether it needs to change its light? The first idea is to transmit the number of cars on the roads at the intersection and draw conclusions from this. It is clear that this cannot be tied to reality (for example, where can we get the exact coordinates of the cars?), but as a first try, we decided to implement this idea. After that, we faced a new task – to come up with a reward function. We conducted many experiments, figuring out how best to “reward” our model in order to understand which factors really “play a role”. As a result, we got a rather complex formula depending on the traffic congestion, changes in this congestion and how long the signal on the traffic light has not changed (surprisingly, this is a really important parameter that we did not immediately guess), with some coefficients. This formula showed itself the best during the experiments.

To implement the DQN algorithm, we decided to use the torch library, and we got the following neural network:

class QNetwork(nn.Module):
    def __init__(self, env):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(np.array(env.single_observation_space.shape).prod(), 120),
            nn.ReLU(),
            nn.Linear(120, 84),
            nn.ReLU(),
            nn.Linear(84, env.single_action_space.n),
        )

    def forward(self, x):
        return self.network(x)

The model takes 5 numbers as input — 4 of them represent the number of waiting cars on different roads, and the fifth number represents the current traffic light signal. The neural network returns 1 number (env.single_action_space.n = 1 from the environment configuration) — the optimal traffic light signal in this situation.

The results of this model were noticeably better than primitive algorithms in most road situations, so we decided to bring our developments closer to real life – to connect image processing so that it would be possible to monitor the road situation through CCTV cameras.

Training a model through an image

Now, instead of traffic congestion, we decided to feed the neural network images of two sections of the intersection, which would contain all the information about it. Photos of the intersection, as if taken from different angles (as divided in the picture), are well suited for this role.

The number of pixels in the image is too large to use each pixel as an input parameter for the neural network, so before passing the data to the input of the neural network, it must be processed. Let's do this using the Numpy and PIL libraries:

pil_string_image = pygame.image.tostring(self.painter.canvas, "RGB", False)
pil_image = Image.frombytes("RGB", (1_200, 1_200), pil_string_image)

width = 64
height = 64

resized_image = pil_image.resize((width, height))
cam1 = np.array(resized_image.crop((0, 0, 44, 44)))  # high left angle
cam2 = np.array(resized_image.crop((20, 20, 64, 64)))  # down right angle
cam1 = np.transpose(cam1, (2, 0, 1)) 
cam2 = np.transpose(cam2, (2, 0, 1))

observation = np.array([cam1, cam2]) # result

The input for the neural network is ready. Let's talk about its structure. It is clear that the neural network needs to understand what to do with the traffic light using only images of the area, which for it are just three-dimensional arrays of numbers-pixels. To make its task easier, we will add another node to this “path of understanding”, where it will understand for itself (no matter in what form) what situation has developed at the intersection in the language of numbers. In this half of the path, the model will “encode” the state of the intersection for itself. And in the next half, it will “decode” it into a form more or less understandable for making a decision to change the traffic light. So, the neural network is essentially a gluing together of these two paths: Encoder and Decoder. And since there are 2 images, there are also 2 Encoders. They will both “encrypt” their images into arrays of 32 numbers, which we will pass to the Decoder, adding the current traffic light signal. Finally, Decoder will give 2 numbers – on which road to turn on the green light.

And here is the implementation of the above

class Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.seq = nn.Sequential(
            nn.Conv2d(3, 16, 3),
            nn.ReLU(),
            nn.MaxPool2d(3, 3),
            nn.Conv2d(16, 64, 3),
            nn.ReLU(),
            nn.MaxPool2d(3, 3),
            nn.Conv2d(64, 8, 3)
        )

    def forward(self, x):
        x = self.seq(x)
        return x.view(x.shape[0], -1)


class Decoder(nn.Module):
    def __init__(self, n):
        super().__init__()
        self.seq = nn.Sequential(
            nn.Linear(n, 2 * n),  # nn.Linear(n + 1, 2 * n),
            nn.ReLU(),
            nn.Linear(2 * n, 8 * n),
            nn.ReLU(),
            nn.Linear(8 * n, 2)
        )

    def forward(self, x):
        # input shape: torch.Size([2, 2, 2])
        # output shape: torch.Size([2, 2, 1])
        x = self.seq(x)
        return x


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.enc1 = Encoder()
        self.enc2 = Encoder()
        self.dec = Decoder(n=32+1)  # +1 - signal

    def forward(self, img1, img2, current_signal):
        if len(img1.shape) == 3:
            img1 = torch.unsqueeze(img1, 0)
            img2 = torch.unsqueeze(img2, 0)
            current_signal = torch.tensor([[current_signal]])
        else:
            current_signal = torch.unsqueeze(current_signal, 1)

        x = self.enc1(img1) + self.enc2(img2)
        x = torch.concat((x, current_signal), dim=1)
        x = self.dec(x)
        return x  # logits

This neural network takes much longer to learn than the previous one, but it shows better test results, which we wrote about at the end of the article.

Traffic Generator

During the tests, we noticed that our model performed poorly on tests with very different road congestion (for example, when one road was completely empty). The traffic light started to “go crazy” and chaotically switched the traffic light signal. At first, we looked for an error in the reward function formulas, but no matter how we changed it, nothing helped. Then we realized that the problem was not in it, but in what our model learned from. We generated cars on each road with equal probability, which led to the fact that the road congestion was always approximately the same, and the model, learning only on such cases, could not cope with any others. After we improved the generator (added cases with empty roads, with cars generated on the roads according to a sine wave, and several more degenerate cases), our traffic light stopped behaving unpredictably in non-trivial road situations and began to cope perfectly with all of them, which suited us quite well.

Implementation of the adjusted algorithm for training with Carla

Let's change the environment from custom to Carla. The training principle will remain the same, but it is worth increasing the size of the intersection images, because cars are much less visible on a city background than purple circles on a black canvas. The downside of this solution is a strong increase in the training process of such a neural network in time. If the previous model was able to successfully train in our “sandbox” in half an hour, then the current one did not have enough time.

Results

It's time for the promised experiments and comparisons of teaching methods!

The test results show that the machine learning algorithm we obtained works more efficiently than primitive algorithms in various situations at intersections, which is good news!