Practice of implementing Web GIS applications and services based on open resources. Start

The idea of ​​saving the world and earning some shekels at the same time has been in my head for a long time. Having good accumulated experience in the field of geographic information systems and having defended a dissertation using them at one time, I lacked the knowledge of a developer. Having completed IT courses and having gained access to the “Holy Grail of knowledge”, I realized that it was time, and things started to happen!

In the summer of 2024, we, as part of the Arrow team, won, taking third place in the “Leaders of Digital Transformation” hackathon and entered the top 100 with our project, becoming residents of the “Academy of Innovators” and we had our own startup.

The general idea is this. Arrow is a platform for analyzing and processing satellite images, using machine learning and neural network technologies for monitoring the environment, construction and natural resource management. Our product helps businesses and government agencies automate the detection of environmental violations and illegal structures, providing more accurate and timely responses. This is in the “rosy” future, but for now it’s just a project “Mobile application for managing anthropogenic load in specially protected natural areas of the Kamchatka Territory”which took a prize, although this result, too, was once only in dreams.

I want to open a whole series of articles in which I will try to highlight the history of the life cycle of our Arrow project, which will be written before your eyes. Everything will be here: front and back, mobile development, and deployment to the cloud. In these articles, which by the way will be written not only by me, but also by the guys from my team, we want to cover everything from creating an MVP (minimum viable product) to bringing the project into production, analyzing the target audience and finding the first clients, attracting the first investments, team selection, in general, all the stages we have to go through to achieve our goal – obtaining an interesting and in-demand product. Let's get started…

In the modern realities of blocking access to many software products, the question of moving to the use of domestic or, replacing them and openly distributed, resources for building elements of spatial data infrastructure is acute.

This article highlights a practical approach to solving the problem of building a Web GIS application and services based on open resources and using the example of our project. The main attention will be paid to creating the general structure of the project and highlighting the resources on which it operates.

The project was implemented using client-server architecture (Figure 1). The server part includes a DBMS for storing vector and raster data sets, file storage of raster tiles, the Gunicorn application server serving the project itself, and the Nginx web server for interaction, via the REST full protocol, with application services, client programs, which can be desktop and/or mobile and/or web browsers. Thus, the entire project will be hosted in the Reg.ru cloud, which will be discussed in a separate article in the future, but for now we are working on our local machine (localhost), organizing tunneled access using Ngrok.

The project's technology stack includes the following freely distributed resources:

• Postgresql DBMS with the PostGIS add-on, which provides additional capabilities for working with geospatial data, turning the database into a georelational one;

• Widely used scripting programming language Python;

• An equally well-known framework designed for creating Django web applications, interacting with the DBMS using ORM (Object Relations Mapping) technology. This technology allows you to work with the selected DBMS through programming language classes, changing only a special driver, while avoiding the need to write SQL queries and, most importantly, without changing the source code of the application;

• The Django REST framework is designed for client interaction with the server part. DRF is one of the modern technologies for creating an API (application programming interface), the main advantage of which is the ability to organize the collaboration of client and server programs written in various programming languages;

• The GDAL library contains the full functionality of modern general-purpose GIS (geographic information systems), combining two libraries – gdal of the same name, for working with raster data sets, and ogr, for working with vector data formats (shape, kml, geojson, etc.) . Activating this library turns the Django framework into GeoDjango;

• – To analyze satellite images (obtained through the Sentinel Hub API), the Numpy and Pandas libraries are used, Jupiter Notebook is additionally used, including for working with the U-Net convolutional neural network for image segmentation;

• The Leaflet library (or OpenLayer 3), through java scripts and in interaction with html templates and CSS style sheets, visualizes geodata in a web browser window and provides the user with interface elements for working with the server application;

• An OSM card is used as a base card.

Creating a project involves installing a programming language, a module for working with virtual environments venv, a package manager pip, a Postgrsql DBMS with the PostGIS extension, I use VSCode as a code editor. Depending on the operating system, various managers are used to install all this stuff, for example, in MacOS it is brew, in Ubuntu it is apt, in Mandjaro it is pacman, etc. I also work on Windows, but I use standard .exe files there. Here and further I will work with Ubuntu, since in the future, when deploying to a remote Reg.ru server, this Linux distribution will also be used.

sudo apt install python3-venv python3-pip postgresql postgis

Next, the database is created on behalf of the postgres user. Switch to this user and go to the psql utility.

sudo su postgres
psql

ALTER USER postgres WITH PASSWORD `admin`;

Now let's create the database.

CREATE DATABASE test;

Next, you need to activate the postgis extension for this database. We connect to the database.

\c test

And activate the extension.

CREATE EXTENSION postgis;

For a list of all available extensions for working with spatial data and for activating them, I recommend that you read the help on the PostGIS website.

Let's go out.

\q

Switch from the “postgres” user to the operating system user.

exit

Now, when creating fields in database models, in addition to standard data types, we will also have access to data fields of the “geometry” type.

The next step is to create a virtual environment to host our project and install the necessary dependencies into it, the main ones of which are the django and djangorest frameworks, as well as the driver for working with the Postgresql database – psycopg2-binary.

mkdir geodjango

Let's switch to this folder, this will be our working directory. Create a folder to house the project.

cd geodjango 

We create an environment, where env are the names of the environment.

python –m venv env

And activate it (to exit the virtual environment you need to run the deactivate file).

source env/bin/activate

Now let’s install the dependencies necessary for the project to work in the virtual environment. A list of these dependencies is contained in the requirements.txt file.

pip install –r requirements.txt

You can check whether all the necessary dependencies are installed with the pip freeze command (with their versions).

Being in the working directory “geodjango”, we begin to create a project and an application in it.

Creating a project is done by a command, where geodjango is the name of the project.

django-admin startproject geodjango

Creating an application is done with a command, where geoapp is the name of the application.

python manage.py startapp geoapp

When you create a project and application using the Django framework, an initial file structure is formed. Each of these files has its own strict purpose. For example, the settings.py file is intended, among other things, to connect to a database previously created based on the PostGIS template.

To configure the connection, use the DATABASES section.

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'test',
        'USER': 'postgres',
        'PASSWORD': 'admin',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

The database structure is created in the models.py file, where the necessary models (tables) with attribute fields storing data of certain types, as well as a field for storing the geometry of spatial objects are specified.

After the models are created, they are migrated to the database and registered in the admin.py file for access to the data through a special administrative panel.

Every change to a database model requires the creation of migrations.

python manage.py makemigrations

The command below creates models in the database.

python manage.py migrate

The views.py file is designed to interact with database data. It contains functions and view classes that describe the program logic of the application.

Setting up the client's interaction with these views is carried out as a result of the routing procedure – specifying access url addresses to certain functions, performed in the urls.py file.

The functions generate content that is rendered in an HTML template located in the templates folder and rendered in a web browser window using CSS style sheets.

If you “wrap” such functions in a special decorator (or instead of creating functions, create classes that inherit the attributes of Django’s built-in generic and mixin classes, which is a more modern approach), they will be able to process REST requests from the client.

Such requests (or responses to them) may contain a body of data written (or retrieved) to the database through serialization (converting an object (data dictionary) into a byte string for transmission over communication lines), deserialization (reverse serialization), and validation ( checking the data to ensure it meets the expected structure).

For understanding, an elementary example of such interaction implemented in the application is presented below.

The user selects a control placed in the browser window. This element is connected to a java script for generating, for example, some conditional POST request.

// Добавляем маркер
function saveMarkerToDatabase(coordinates, markerName) {
            fetch('https://swan-decent-shrew.ngrok-free.app/api/create_point/', {
                // Задаем метод REST-запроса
                method: 'POST',
                // Формируем хедер запроса
                headers: {
                    'Content-Type': 'application/json',
                    'X-CSRFToken': getCookie('csrftoken'), // Добавляем CSRF-токен в заголовок запроса
                },
                // Формируем тело запроса
                body: JSON.stringify({ name: markerName, location: 'SRID=4326;POINT(' + coordinates.lng + ' ' + coordinates.lat + ')' })
            });
}

At the moment the mouse clicks on a specific url, this request is sent to the server, containing in its body the content: event name and location data in WKT format (coordinate system and projection code and point coordinates). Such a request, in addition to the body, must contain service information in its header that protects the application from external attacks – a CSRF token.

The function associated with this url, also called the endpoint, executes successfully, as evidenced by the response status of 200, if no exceptions occur.

The return result of this function – the view – will be some content. This can be a GeoJSON file containing values ​​somehow processed and written (extracted, updated, deleted) into the database, as well as placed according to certain keys in GeoJSON itself.

For example, using the “objects” key, the geometry of the object is placed in a nested list along with its type and, using the “properties” key, the parameters of the attributes of spatial objects, in this example this is the name of the incident.

{ "context_point_trek": { "type": "FeatureCollection", "crs": { "type": "name", "properties": { "name": "EPSG:4326" } }, 
"features": [ 
{ "type": "Feature", "id": 1, "properties": { "name": "Точка: 1" }, "geometry": { "type": "Point", "coordinates": [ 158.839538, 53.572238 ] } }, 
{ "type": "Feature", "id": 2, "properties": { "name": "Точка: 2" }, "geometry": { "type": "Point", "coordinates": [ 158.83972, 53.572024 ] } }, (и так далее...)

Such content in the form of variables is transferred back to the html template, where it is processed through java scripts and displayed in a stylized manner in the web browser window in the form of map layers, tables, graphs, charts, etc., together with user interface elements (Figure 3). Also, such content can be sent in the body of the response to a generated POST request; to do this, you just need to adjust the endpoint.

To administer the application, there is a special Django panel that allows users who have passed the identification and authentication procedures to create and delete records from the database (Figure 4).

To create a user (at the initial stage), use the command

python manage.py createsuperuser

In the future, this functionality will be expanded to register new users of the application.

I would like to summarize the material of this article as follows. The logic of the web GIS application will implement tools for obtaining satellite images from open resources, such as Sentinel Hub, and processing the resulting rasters using special algorithms. The essence of this processing is to combine spectral channels of satellite images and calculate spectral indices to solve problems based on machine learning technologies: decoding, classification and clustering of objects and terrain phenomena. In the future, it is also planned to introduce pre-trained neural networks into the project based on these developments. Of course, these are materials for writing several more articles in the future. Stay in touch, if you are interested, subscribe.

In general, this approach will make it possible to implement a number of services hosted in the cloud, access to which will be provided by subscription. Customers will use such services to build their own applications, like Notion.

It is also planned that it will be possible to organize an entire infrastructure (as you probably already understood, this has already been implemented in a minimally workable version), when based on the services it will be possible to create monitoring panels, for example, places where solid waste accumulates, illegal buildings, fire and burn zones, floods and much more. Such dashboards can be constructed by users themselves based on pre-created front-end tools (maps, tables, graphs, charts, etc.). A kind of nocode application for untrained users, or with coding elements, for more prepared ones, by using our API. In addition, we ourselves will be able to provide the entire chain of creation, deployment and support of the application for more specific user requests.

Another element of this infrastructure will be the ability not only to monitor the situation, but also to give orders and control their execution to various response services (Figure 5), by implementing an additional mobile application associated with the database (Android, IOS, etc. our application is platform independent, This is its beauty!).

A video presentation of the full cycle of the application is available at link.