Hello, Habr! We are employees of the Yekaterinburg office of NAUMEN. We are sharing an interesting project – an interactive tree map based on neural network algorithms. UrFU students and volunteers of our city also participate in its creation.
Many modern megacities face the problem of a lack of green spaces, and Yekaterinburg is no exception. Infrastructure development in the central regions and, as a result, illegal logging, reduction of park areas do not help to improve the situation. We think this problem is familiar to residents of many Russian cities. One solution is to create a regularly updated interactive tree map that is available to everyone. With its help, it will be possible to find out how many trees are already growing in the city, how many there should be and in which areas additional planting is needed.
To carry out a manual inventory of green spaces is long and ineffective. Modern technologies come to the rescue: aerial photography, Street View images, LiDAR systems and drones shooting. Among the variety of data sources for obtaining information about objects of the urban environment, the most accessible and relatively cheap are panoramic photographs from Street View. With their help, you can train a neural network to recognize trees in images, determine their geolocation and plot them on a city map. How this process went and what results were achieved – read the article.
Why does Yekaterinburg need an interactive map of trees
In 2019 by the urban environment quality index Yekaterinburg scored less than half of the maximum number of points in terms of landscaping. However, now there is no exact data on how many trees grow in the city, how many there should be, and in which areas they are lacking.
In assessing the quality of the urban environment, Yekaterinburg scored the least points for landscaping
The practice of counting trees has been implemented in many cities around the world. There are open registers of green spaces in Vienna, Amsterdam, Prague. The most successful example is central New York City tree map… Over 2,200 municipality employees and volunteers in one year applied about 700,000 trees to it and indicated their type. Now, having confirmed the skills of caring for plantings, anyone can take custody of the tree. Every New Yorker can complain about the condition of the tree or problems with it.
The creation of such a service in Yekaterinburg has long been thought of in the Parks and Squares volunteer community, which includes Ilya Kotelnikov and Boris Ovodov, employees of the NAUMEN company. The guys decided to help create an interactive map of Yekaterinburg trees, based on the data of which residents will be able to influence the greening of the city and prevent illegal logging.
As planned, all the trees of Yekaterinburg can be seen on an interactive map
In January 2020, a community of volunteers completed the initial setup of the system on the resource ekbtrees.ru… In two weeks, they mapped 150 trees in Weiner’s garden. It turned out to be a very long time to collect data manually, and it was impossible to update information promptly in the future. Therefore, we came to the decision to use machine learning technologies.
“Using intelligent algorithms, we want to create a system that will count how many trees there are in Yekaterinburg. The basic task is to protect what is, and then take an inventory every 3-5 years.
Ideally, create such a mechanism through which citizens can receive useful services: order planting of green spaces or care for them, complain about fallen and broken trees, report illegal actions. “
In addition, such a map can be used as an evidence base in all civil processes that take place in the city, for example, when they want to organize a construction site on the site of a forest.
Creating an interactive map using a neural network
To speed up the process of collecting information about the trees of the city, Ilya turned to Tatyana Zobnina, who specializes in the development of machine learning systems. He suggested that she try to implement an interactive map using neural network algorithms. Having studied the accumulated experience in the field of tree recognition and determination of the geo-position of objects according to Street View and other sources, Tatyana was convinced that the problem can be solved. In January 2020, she posted the project on the UrFU Project Workshop website to involve students in the development.
“I try to organize my work with students within the framework of the Project Workshop so that they have time to come to some result in a semester. It is important for me that they are interested, that they know how to interact in a team, distribute roles, fit into timing: in tasks related to machine learning, this can cause problems. It turns out such a game educational practice. Also, the implementation of the project using neural networks will give students an undoubted professional advantage when looking for a job after completing their studies at the university. “
4 students of the Faculty of Radio Engineering became interested in the creation of an interactive tree map using machine learning algorithms: Egor Demin, Egor Kipelov, Alexander Chernykh and Georgy Shishkin. All of them in the last semester had already taken the course “Recognizing emotions by sound” with Tatiana and had a sufficiently high level of technological knowledge and skills to implement this project.
Students received installation materials, found out what ready-made solutions are available for creating an interactive map using a neural network, and chose two of them that are most suitable:
- tree detection by aerial photographs;
- use of photos from Street View.
To test both options in parallel, the guys were divided into groups of two. The work was divided into stages and an approximate timing was set when the intermediate results are ready.
According to the results of the study, it turned out that despite all the advantages of aerial photography, there is no high-quality data for Yekaterinburg in the public domain. At this stage of the project development, this decision had to be abandoned. Then the two groups together began to implement the version with Street View.
The guys had to:
- Write code to download data from Street View.
- Mark up the uploaded photos – select trees on them.
- Train the neural network to recognize trees in photos.
- Write a code for converting photos from Street View, on which a tree is selected, into coordinates, evaluate their accuracy.
- Upload data to an interactive map of Yekaterinburg trees.
- Check the accuracy of the location of the trees on the map with the help of volunteers.
First hypothesis: train a neural network using a NY tree map and Street View images
Google and Yandex provide access to a huge number of city panoramas. At the same time, there is an extensive database of New York trees with the geo-position of each plant, but no images. A hypothesis appeared that it is possible to enrich the database with photographs and, with its help, train a neural network to determine coordinates from a picture, as well as other parameters of trees: height, diameter, species. After that, it will be possible to use the trained neural network to recognize trees in the panoramas of Yekaterinburg and indicate their characteristics, including geolocation.
However, this method has its drawbacks. The fact is that the tree map of New York was created in 2015, and the Street View survey contains more relevant information. Accordingly, the data on the card and in the photos may not match, which may cause errors in the system. Nevertheless, this was the best solution for training a neural network at this stage, and the guys started to implement it.
To work on the project, NAUMEN provided computing power on one of its servers. The students wrote code to download data from Street View, optimized it on their laptops, and pushed it to the server.
Then it was necessary to decide on a neural network suitable for the project. The guys opted for an Open Source solution – the YOLOv3 convolutional neural network. The main task that the students set before her was to classify the annotated data and recognize trees in Street View images.
YOLOv3 is trained on large amounts of data and can recognize over 100 different objects, so in theory it should have dealt with it. But trees are a rather complex object with many varieties, so in practice it turned out that it would be nice to retrain the network to better solve the problem.
It was necessary to take as many marked-up pictures of the trees as possible and upload them to the neural network. For this purpose, students have made telegram bot, in which anyone can help sort the photos by noting whether there is a tree in the selected frame and on which side it is located. Volunteers were also involved in the layout of the pictures. When part of the data was ready, they began to train the neural network on it.
With the help of the telegram bot, everyone can participate in the training of the neural network
In addition, MSTU named after M.V. N.E. Bauman. It turned out that Mikhail Ukhov, who participated in the collection of data for the New York tree map as a volunteer, is implementing a similar project in Moscow. To help colleagues, Tatiana shared her retrained YOLOv3.
Meanwhile, the students wrote a code that, using a neural network, recognizes and highlights trees in panoramas from Street View and allows them to get their coordinates. However, during the tests it turned out that this approach does not give the required accuracy in determining the geolocation of the tree. The irregular object and the distortion that is present in the Street View images prevents the correct latitude and longitude of the green space from being obtained.
Distortion in images from Street View prevents the neural network from accurately determining the geolocation of trees
The second hypothesis: determination of geo-position from Street View images of Yekaterinburg
In order to understand how to more accurately determine the coordinates of trees from photographs and in which direction to move further with the students, Tatiana again began research. It turned out that there is a ready-made solution described by our compatriot who teaches at the University of Dublin, Vladimir Krylov… IN article he described an algorithm that allows you to accurately recognize the geolocation of power poles and traffic lights on Street View, and posted a link to GitHub with Algorithm Code…
The algorithm hypothetically described by Vladimir could be suitable for recognizing trees. During the summer internship at NAUMEN, students were engaged in the refinement and linking of code and settings. We also connected the retrained YOLOv3 neural network to work.
The result is a pipeline of three neural networks and an algorithm that works as follows:
- The system collects images from Street View.
- The first neural network segments trees, highlighting their silhouette in the photo.
- YOLOv3 refines the segment by placing the tree in a square frame.
- The third neural network determines the distance from the car from which the Street View was filmed to the desired object – in our case to the tree.
- Next, the geotagging algorithm determines the coordinates of the tree.
This method allows you to immediately determine the coordinates of trees on images of Yekaterinburg, skipping the stage of training the neural network using the New York tree map.
As a result of the first test in this way, it was possible to recognize 68 trees along Surikov Street from the Street View images and determine their coordinates.
This is how the marks of recognized trees on one of the streets of Yekaterinburg are indicated on the map
The next step is to involve volunteers and assess the accuracy of the data received. Based on the results of the check, UrFU students and NAUMEN employees will continue to work on the project in the next semester.
Note that the training of neural network algorithms continues, so anyone can join the project and help. For this you need to download telegram bot and sort the tagged photos. A more significant contribution can be made by marking up the selected photos yourself in the YoloLabel program. If you want to participate, write in the comments.