Creating a 3D terrain model from a drone video
This article will discuss videogrammetry – the creation of 3D models of objects based on video. Videogrammetry has been around for a long time, but is still rarely used due to the need for more powerful hardware than when building 3D models of terrain/objects using lidars. In 2024, PC performance has grown so much that it is now possible to build models on home computers and even laptops within a reasonable time. First, I will show you what models I got, then I will share life hacks that will help you build models right away, bypassing technical difficulties, and at the end I will tell you how videogrammetry can be useful.
Here are the same models, but you can rotate them with a mouse on a PC or a finger on a smartphone
All these models are built in the free, open source photogrammetry program Meshroom (or its simplified version MeshroomCL community in vk). There are many other paid and free programs: https://rec3d.ru/rec-wiki/vosem-besplatnykh-programm-dlya-fotogrammetrii/
About the hardware:
Running photo/videogrammetry programs on laptops is quite realistic only for simple projects (150-200 frames extracted from 1-2 minute videos). To build good models, 500-1000 frames are often used. Therefore, it is better to use a home PC with a video card no worse than 4060ti.
About cameras:
The videogrammetry result strongly depends on the quality of the camera. Drones usually have high-quality large photo matrices, but on smartphones it can be different. Now I use a smartphone with a large matrix of 1/1.4 format. I also tested other smartphones with smaller matrices (1/2 inch, 1/1.66 inch and 1/1.5 inch), there Meshroom rejects much more frames due to insufficient clarity, so the quality of the final 3D model is noticeably worse)
About the quality of 3D models
Over the course of a year, I made about 100 3D models using videogrammetry. About half from drone video, half from smartphone video. About 30 models were made publicly available:
Perhaps some people will find the quality of these models not very good, so I will provide links to much higher quality results obtained on much more powerful hardware:
Videogrammetry from Nvidia
Videogrammetry by startup Mosaic
Photo/Videogrammetry from Google
From the Nvidia video it is clear that neural networks are now beginning to be used to build 3D terrain models. These are not quite the same classic photogrammetry methods used by Meshroom and other programs.
I would highlight three main methods for constructing a 3D terrain model:
Classic photo/videogrammetry (Meshroom and other similar programs)
Photo/videogrammetry using neural networks (Nvidia and others)
Building a 3D terrain model based on object recognition (Tesla and other autopilot systems)
this is how Tesla's autopilot builds a 3D terrain model
In fact, the autopilot recognizes nearby 3D objects (neighboring cars) and on this basis builds a model of the surrounding space. It is unknown how exactly the Tesla autopilot works, but I described one of the methods for recognizing 3D objects in the previous article (there the process of recognizing 3D objects is shown using the example of faces, but the method itself does not use any anthropological features, so it can be used to recognize any 3D objects, including cars)
To create a system for recognizing 3D objects (cars), a database of these 3D objects is needed. Therefore, my friend and I have compiled an open car scan database:
Instead of a conclusion
It may seem illogical to some that this article began with “Building a 3D terrain model with a drone” and ended with “Building a 3D terrain model with a Tesla drone”.
The answer is simple: because I believe that building a 3D terrain model is an important and interesting task. And how it is solved, with the help of classic photo/videogrammetry (meshroom), neural network photo/videogrammetry (nvidia) or with the help of 3D object recognition (tesla) is a personal matter. Everyone goes their own way. Time will tell who will be right.
