Development of a cross-platform application on Qt using neural networks trained on tensorflow

Since you are interested in this article, it is expected that you can program in C ++ using the Qt library and develop neural networks in Python using the tensorflow library.
Accordingly, it remains only to understand how to use the neural network models you trained in Qt projects.

And you need to do the following:

  • Download and compile for the required tensorflow platform

  • Download and compile for the desired opencv platform

  • Download android SDK and configure Qt to work with it

  • In the Qt pro file, specify the location of the directory with the header files and the compiled libraries themselves

  • Write a program

Download and compile for the required tensorflow platform

You can download the source codes from the official website.
To do this, in the folder where you plan to store projects, run (in the console, of course) git clone https://github.com/tensorflow/tensorflow.git and team
git checkout branch_name select the desired version of tensorflow.

To compile tensorflow you will need bazel. But each version of tensorflow requires a different version of basel. In order not to bathe yourself with versions, you should install bazelisk. Why download the binary from https://github.com/bazelbuild/bazelisk/releasesrename it to bazelisk and put it in the system folder with programs, for example, in /usr/local/bin (for linux), after which we write bazelisk instead of bazel in the compilation commands.

There are two options for using tensorflow:

original tensorflow

Pros:

Minuses:

  • Complex interface. Here it is tensorflow without keras, that is, you operate not with the concepts of model, layer, but with the concepts of computational graph, computational operation

  • Huge library size of about 300 MB

  • No settings for compiling for mobile OS

Tensorflow lite

Pros:

  • Simple interface, although not the same as keras

  • Small library size – a few megabytes

  • There are compilation settings for mobile OS

Minuses:

  • The need to convert the “.h5” format to the “.tflite” format (done with one command in python).

  • Not all operations may be supported (there were none in my model).

Compiling the library for the original tensorflow

To compile, run the following commands:
Install protobuf:

git clone https://github.com/protocolbuffers/protobuf.git 
cd protobufgit 
checkout 3.9.x
./autogen.sh
./confugure
make -j$(nproc)
sudo make install
sudo ldconfig

git clone https://github.com/tensorflow/tensorflowcd tensorflow
git checkout r2.7
git clone https://github.com/abseil/abseil-cpp.git
ln -s abseil-cpp/absl ./absl/

Add googleprotobuf*; in tensorflow/tensorflow/tf_version_script.lds
Then:

./confugure
bazelisk build --jobs=10 --verbose_failures -c opt --config=monolithic //tensorflow:libtensorflow_cc.so

It should be noted that the protobuf version must match the tensorflow version. I found a match for the tensorflow compilation error.

Compiling a library for tensorflow lite

To compile, run the following commands:

git clone https://github.com/tensorflow/tensorflow
cd tensorflow
git checkout r2.7
./confugure
bazelisk build -c opt --config=android_arm64 --config=monolithic //tensorflow/lite:libtensorflowlite.so

./configure will interactively configure the build, if the setting is to compile for android, you will need to specify the location of the Android NDK that you downloaded as part of Android studio.
Here android_arm64 – settings for compiling under 64 bit version of android.
Replace with android_arm to compile under 32 bit version of android.
Remove –config=android_arm64 to compile under the OS you’re developing on.
Related link to the official site https://www.tensorflow.org/lite/android/development

After compilation, the bazel-bin directory will appear, in which the compiled library will be located in the tensorflow/lite directory

In the pro project file on Qt for android, add the following lines:
INCLUDEPATH += “Path to folder with tensorflow”
INCLUDEPATH += “Path to folder with tensorflow”/bazel-bin/
INCLUDEPATH += “Path to folder with tensorflow”/bazel-tensorflow/external
INCLUDEPATH += “Path to folder with tensorflow”/bazel-bin/external/flatbuffers/_virtual_includes/flatbuffers

LIBS += -L “Path to folder with tensorflow”/bazel-bin/tensorflow/lite -ltensorflowlite

Download and compile for the desired opencv platform

Run git clone https://github.com/opencv/opencv.git
Go to the desired version git checkout “Branch”, there is a branch for each version of opencv.
Next to the opencv directory, create a directory, for example, opencv_build, in which create a directory for each platform.
In the opencv_build directory for the Android 64 platform, create scripts of the following content, replacing
“Folder where compilation will be carried out for a specific OS”
“Folder with specific android NDK version”
“Folder with build result”,
your folder names.

# !/bin/bash  
cd "Папка куда будет вестись компиляция под конкретную ОС"  
rm -R *  
PATH=$PATH:"Папка с конкретной версией android NDK"/toolchains/llvm/prebuilt/linux-x86_64/bin  
export ANDROID_HOME="Папка с  SDK"  
export ANDROID_SDK_ROOT="Папка с  SDK"  
export CMAKE_CONFIG_GENERATOR="Unix Makefiles"  
cmake -DCMAKE_BUILD_TYPE=Debug -DANDROID_NATIVE_API_LEVEL=lastest -DANDROID_ABI=arm64-v8a -DCMAKE_BUILD_TYPE=Debug -G"$CMAKE_CONFIG_GENERATOR" -DANDROID_ARM_NEON=ON -DANDROID_STL=c++_static -DBUILD_ANDROID_PROJECTS:BOOL=ON -DBUILD_opencv_world:BOOL=OFF -DBUILD_PERF_TESTS:BOOL=OFF -DBUILD_TESTS:BOOL=OFF -DBUILD_DOCS:BOOL=OFF -DWITH_CUDA:BOOL=ON -DBUILD_EXAMPLES:BOOL=OFF -DENABLE_PRECOMPILED_HEADERS=OFF -DWITH_IPP=ON -DWITH_MSMF=ON -DOPENCV_ENABLE_NONFREE:BOOL=ON -DWITH_OPENEXR=OFF -DWITH_CAROTENE=ON \-DINSTALL_CREATE_DISTRIB=ON -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -DCMAKE_TOOLCHAIN_FILE="Папка с конкретной версией android NDK"/build/cmake/android.toolchain.cmake ../../opencv  
make -j16  
cmake --install . --prefix "Папка с результатом сборки"

For 32-bit Android platform, you need to replace arm64-v8a with armeabi-v7a

For the desktop platform, the script will be as follows:

#!/bin/bash
cd "Папка куда будет вестись компиляция под конкретную ОС"
rm -R *
cmake ../../opencv
make -j16
cmake --install . --prefix "Папка с результатом сборки"

In the pro project file, add the following for desktop:

INCLUDEPATH += "Папка с результатом сборки"/include/opencv4    
LIBS += -L"Папка с результатом сборки"/lib \    
-lopencv_dnn \    
-lopencv_videoio \    
-lopencv_objdetect \    
-lopencv_calib3d \    
-lopencv_imgcodecs \    
-lopencv_features2d \    
-lopencv_flann \    
-lopencv_imgproc \    
-lopencv_core

In the pro project file, add the following for android:

OPENCV_ANDROID = "Папка с результатом сборки"
INCLUDEPATH += "$$OPENCV_ANDROID/sdk/native/jni/include"
LIBS += -lmediandkcontains(ANDROID_TARGET_ARCH,armeabi-v7a){   
LIBS += \           
-L"$$OPENCV_ANDROID/sdk/native/3rdparty/libs/$$ANDROID_TARGET_ARCH" \                
-ltbb \                
-lIlmImf        }        
LIBS += \            
-L"$$OPENCV_ANDROID/sdk/native/libs/$$ANDROID_TARGET_ARCH" \            
-L"$$OPENCV_ANDROID/sdk/native/staticlibs/$$ANDROID_TARGET_ARCH" \            
-L"$$OPENCV_ANDROID/sdk/native/3rdparty/libs/$$ANDROID_TARGET_ARCH" \            
-lade \            
-littnotify \            
-llibjpeg-turbo \            
-llibwebp \            
-llibpng \            
-llibtiff \            
-llibopenjp2 \            
-lquirc \            
-ltegra_hal \            
-lopencv_dnn \            
-lopencv_objdetect \            
-lopencv_calib3d \            
-lopencv_imgcodecs \            
-lopencv_features2d \            
-lopencv_flann \            
-lopencv_imgproc \            
-lopencv_core \            
-lopencv_videoio \            
-lcpufeatures \            
-llibprotobuf \        
ANDROID_EXTRA_LIBS = $$OPENCV_ANDROID/sdk/native/libs/arm64-v8a/libopencv_java4.so

You also need flatbuffers to compile under desktop tensorflow

Install it globally with the following commands:

git clone https://github.com/google/flatbuffers.git
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
make
sudo make install
sudo ldconfig

You can install flatbuffers locally by replacing
sudo make install
cmake –install . –prefix “Folder with build result”,
and then with INCLUDEPATH += and LIBS += add to the Qt project

Software development

Here I will describe the work with tensorflow lite

To work with tensorflow lite, let’s convert the model to its format:

pred_center_model.save('pred_center_model_full')
converter = tf.lite.TFLiteConverter.from_saved_model('pred_center_model_full') # path to the SavedModel directory
tflite_model = converter.convert()

# Save the model.
with open('pred_center_model2.tflite', 'wb') as f:
  f.write(tflite_model)

Now let’s include the necessary libraries:

#include <opencv2/opencv.hpp>
#include "tensorflow/lite/interpreter.h"
#include "tensorflow/lite/kernels/register.h"
#include "tensorflow/lite/c/c_api_types.h"

Now let’s load the model:

std::unique_ptr<tflite::FlatBufferModel> m_model;
tflite::ops::builtin::BuiltinOpResolver resolver;
std::unique_ptr<tflite::Interpreter> interpreter;
m_model = tflite::FlatBufferModel::BuildFromFile("путь к модели");
tflite::InterpreterBuilder builder(*m_model, resolver);
TfLiteStatus tatus = builder(&interpreter);
interpreter->AllocateTensors();

If status == kTfLiteOk, then we can perform model inference.
Unfortunately, I did not find how to get the dimensions of the input and output layers from the model itself, so you just need to know them. In this example, the input is a cv:Mat video frame, and the output is an array of 9 numbers (the probabilities of a particular class).

// загружаем данные на входной слой
сonst size_t DATA_SIZE 224*224*3
float* input = interpreter->typed_input_tensor<float>(0);
auto *from_data = (uint8_t*)frame.data;.
copy(from_data, from_data + DATA_SIZE, input);

// делаем инференс
auto status = interpreter->Invoke();

// разбираем данные с выходного слоя
float* output = interpreter->typed_output_tensor<float>(0);
if (status == kTfLiteOk)
  {
    auto size = 9;
    int max_idx {0};
    float max = output[0];
    static const vector<string> emo_names = {"злость", "презрение", "отвращение", "страх", "радость", "норма", "печаль",
                                      "удивление", "неуверенность"};
    vector<string> emotions;
    for (int i = 0; i < size; ++i)
      {
        float curr_val = output[i];
        if (curr_val > 0.2)
          emotions.push_back(emo_names[i]);
        if (curr_val > max)
          {
            max_idx = i;
            max = curr_val;
          }
      }

    return emotions;
  }
else
  return {"predict error"};

Very often it is required to recognize frames from the video stream of a webcam or a smartphone camera, which is the same programmatically. Capturing can be done either using opencv or Qt. Capturing using opencv is tempting, since you can conveniently do many operations with the received frame, for example, cut out a given area, but I still couldn’t get capture through opencv under android to work. Therefore, I made the capture and output of the video stream using Qt, and the frame conversion using opencv. This works for all platforms.

To capture a camera, you need to create 3 objects:

QScopedPointer<QCamera> m_camera;
QVideoSink *m_video_sink{new QVideoSink{this}};
QMediaCaptureSession m_captureSession;

Then select a camera (for example, the default camera) and link these objects:

m_camera.reset(QMediaDevices::defaultVideoInput());
m_captureSession.setCamera(m_camera.data());
m_camera->start();
m_captureSession.setVideoSink(m_video_sink);

Then periodically poll the video stream and convert the image to cv:Mat:

m_curr_image = m_video_sink->videoFrame().toImage();
m_frame = QImage2Mat(m_curr_image);

Here are the conversion functions from QImage to cv::Mat and vice versa:

using namespace cv;

QImage Mat2QImage(cv::Mat const& src)
{
     cv::Mat temp; // make the same cv::Mat
     cvtColor(src, temp, COLOR_BGR2RGBA); // cvtColor Makes a copt, that what i need
     QImage dest((const uchar *) temp.data, temp.cols, temp.rows, temp.step, QImage::Format_RGB32);
     dest.bits(); // enforce deep copy, see documentation
     // of QImage::QImage ( const uchar * data, int width, int height, Format format )
     return dest;
}

cv::Mat QImage2Mat(QImage const& src)
{
     cv::Mat tmp(src.height(),src.width(),CV_8UC4,(uchar*)src.bits(),src.bytesPerLine());
     cv::Mat result; // deep copy just in case (my lack of knowledge with open cv)
     cvtColor(tmp, result, COLOR_RGBA2BGR);
     return result;
}

In addition to your own models, it is useful to use someone else’s, already trained, for example, opencv has a built-in face detection model cv::dnn::Net, here is an example:

auto prepared_frame = cv::dnn::blobFromImage(frame, 1.0, Size(300,300), Scalar(104.0, 177.0, 123.0));
m_face_detect_model.setInput(prepared_frame);
Mat output = m_face_detect_model.forward();
const int SHIFT = 7;
using currTp = Vec<float,SHIFT>;
auto it = output.begin<currTp>();
while(it != output.end<currTp>())
  {
    currTp pred = *it;
    if (pred[2] < 0.5)
      break;

    int x = pred[3]*m_img_width;
    int y = pred[4]*m_img_height;
    int width = (pred[5] - pred[3])*m_img_width;
    int height = (pred[6] - pred[4])*m_img_height;
    coords.push_back(Rect{x, y, width, height});
    it+=SHIFT;
  }

True, under Android, the opencv models worked for me extremely inefficiently, 40 times worse than the tensorflow models I trained. If someone knows how to fix this write, I will be glad.
Tensorflow provides a bunch of ready-made, trained computer vision models in the mediapipe project and under python they can be conveniently used, but under c++ it is planned that you integrate into mediapipe, and not vice versa, since it is planned that if you use c++, then You need a minimum application size.
The idea of ​​mediapipe is that a text file is written in a special format that describes the data path from input, for example, from a camera, to output to the device screen, this file is fed to the input of a program that implements the general application code. If some transformation does not exist in mediapipe, then a class inherited from mediapipe::CalculatorBase is written, in which this transformation is implemented. You can read more at the link. But this is a topic for a separate article.

Here is a link to the finished project
It has branches with different implementations.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *