Development of a cross-platform application on Qt using neural networks trained on tensorflow
Since you are interested in this article, it is expected that you can program in C ++ using the Qt library and develop neural networks in Python using the tensorflow library.
Accordingly, it remains only to understand how to use the neural network models you trained in Qt projects.
And you need to do the following:
Download and compile for the required tensorflow platform
Download and compile for the desired opencv platform
Download android SDK and configure Qt to work with it
In the Qt pro file, specify the location of the directory with the header files and the compiled libraries themselves
Write a program
Download and compile for the required tensorflow platform
You can download the source codes from the official website.
To do this, in the folder where you plan to store projects, run (in the console, of course) git clone https://github.com/tensorflow/tensorflow.git and team
git checkout branch_name select the desired version of tensorflow.
To compile tensorflow you will need bazel. But each version of tensorflow requires a different version of basel. In order not to bathe yourself with versions, you should install bazelisk. Why download the binary from https://github.com/bazelbuild/bazelisk/releasesrename it to bazelisk and put it in the system folder with programs, for example, in /usr/local/bin (for linux), after which we write bazelisk instead of bazel in the compilation commands.
There are two options for using tensorflow:
original tensorflow
Pros:
Minuses:
Complex interface. Here it is tensorflow without keras, that is, you operate not with the concepts of model, layer, but with the concepts of computational graph, computational operation
Huge library size of about 300 MB
No settings for compiling for mobile OS
Tensorflow lite
Pros:
Simple interface, although not the same as keras
Small library size – a few megabytes
There are compilation settings for mobile OS
Minuses:
The need to convert the “.h5” format to the “.tflite” format (done with one command in python).
Not all operations may be supported (there were none in my model).
Compiling the library for the original tensorflow
To compile, run the following commands:
Install protobuf:
git clone https://github.com/protocolbuffers/protobuf.git
cd protobufgit
checkout 3.9.x
./autogen.sh
./confugure
make -j$(nproc)
sudo make install
sudo ldconfig
git clone https://github.com/tensorflow/tensorflowcd tensorflow
git checkout r2.7
git clone https://github.com/abseil/abseil-cpp.git
ln -s abseil-cpp/absl ./absl/
Add googleprotobuf*; in tensorflow/tensorflow/tf_version_script.lds
Then:
./confugure
bazelisk build --jobs=10 --verbose_failures -c opt --config=monolithic //tensorflow:libtensorflow_cc.so
It should be noted that the protobuf version must match the tensorflow version. I found a match for the tensorflow compilation error.
Compiling a library for tensorflow lite
To compile, run the following commands:
git clone https://github.com/tensorflow/tensorflow
cd tensorflow
git checkout r2.7
./confugure
bazelisk build -c opt --config=android_arm64 --config=monolithic //tensorflow/lite:libtensorflowlite.so
./configure will interactively configure the build, if the setting is to compile for android, you will need to specify the location of the Android NDK that you downloaded as part of Android studio.
Here android_arm64 – settings for compiling under 64 bit version of android.
Replace with android_arm to compile under 32 bit version of android.
Remove –config=android_arm64 to compile under the OS you’re developing on.
Related link to the official site https://www.tensorflow.org/lite/android/development
After compilation, the bazel-bin directory will appear, in which the compiled library will be located in the tensorflow/lite directory
In the pro project file on Qt for android, add the following lines:
INCLUDEPATH += “Path to folder with tensorflow”
INCLUDEPATH += “Path to folder with tensorflow”/bazel-bin/
INCLUDEPATH += “Path to folder with tensorflow”/bazel-tensorflow/external
INCLUDEPATH += “Path to folder with tensorflow”/bazel-bin/external/flatbuffers/_virtual_includes/flatbuffers
LIBS += -L “Path to folder with tensorflow”/bazel-bin/tensorflow/lite -ltensorflowlite
Download and compile for the desired opencv platform
Run git clone https://github.com/opencv/opencv.git
Go to the desired version git checkout “Branch”, there is a branch for each version of opencv.
Next to the opencv directory, create a directory, for example, opencv_build, in which create a directory for each platform.
In the opencv_build directory for the Android 64 platform, create scripts of the following content, replacing
“Folder where compilation will be carried out for a specific OS”
“Folder with specific android NDK version”
“Folder with build result”,
your folder names.
# !/bin/bash
cd "Папка куда будет вестись компиляция под конкретную ОС"
rm -R *
PATH=$PATH:"Папка с конкретной версией android NDK"/toolchains/llvm/prebuilt/linux-x86_64/bin
export ANDROID_HOME="Папка с SDK"
export ANDROID_SDK_ROOT="Папка с SDK"
export CMAKE_CONFIG_GENERATOR="Unix Makefiles"
cmake -DCMAKE_BUILD_TYPE=Debug -DANDROID_NATIVE_API_LEVEL=lastest -DANDROID_ABI=arm64-v8a -DCMAKE_BUILD_TYPE=Debug -G"$CMAKE_CONFIG_GENERATOR" -DANDROID_ARM_NEON=ON -DANDROID_STL=c++_static -DBUILD_ANDROID_PROJECTS:BOOL=ON -DBUILD_opencv_world:BOOL=OFF -DBUILD_PERF_TESTS:BOOL=OFF -DBUILD_TESTS:BOOL=OFF -DBUILD_DOCS:BOOL=OFF -DWITH_CUDA:BOOL=ON -DBUILD_EXAMPLES:BOOL=OFF -DENABLE_PRECOMPILED_HEADERS=OFF -DWITH_IPP=ON -DWITH_MSMF=ON -DOPENCV_ENABLE_NONFREE:BOOL=ON -DWITH_OPENEXR=OFF -DWITH_CAROTENE=ON \-DINSTALL_CREATE_DISTRIB=ON -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -DCMAKE_TOOLCHAIN_FILE="Папка с конкретной версией android NDK"/build/cmake/android.toolchain.cmake ../../opencv
make -j16
cmake --install . --prefix "Папка с результатом сборки"
For 32-bit Android platform, you need to replace arm64-v8a with armeabi-v7a
For the desktop platform, the script will be as follows:
#!/bin/bash
cd "Папка куда будет вестись компиляция под конкретную ОС"
rm -R *
cmake ../../opencv
make -j16
cmake --install . --prefix "Папка с результатом сборки"
In the pro project file, add the following for desktop:
INCLUDEPATH += "Папка с результатом сборки"/include/opencv4
LIBS += -L"Папка с результатом сборки"/lib \
-lopencv_dnn \
-lopencv_videoio \
-lopencv_objdetect \
-lopencv_calib3d \
-lopencv_imgcodecs \
-lopencv_features2d \
-lopencv_flann \
-lopencv_imgproc \
-lopencv_core
In the pro project file, add the following for android:
OPENCV_ANDROID = "Папка с результатом сборки"
INCLUDEPATH += "$$OPENCV_ANDROID/sdk/native/jni/include"
LIBS += -lmediandkcontains(ANDROID_TARGET_ARCH,armeabi-v7a){
LIBS += \
-L"$$OPENCV_ANDROID/sdk/native/3rdparty/libs/$$ANDROID_TARGET_ARCH" \
-ltbb \
-lIlmImf }
LIBS += \
-L"$$OPENCV_ANDROID/sdk/native/libs/$$ANDROID_TARGET_ARCH" \
-L"$$OPENCV_ANDROID/sdk/native/staticlibs/$$ANDROID_TARGET_ARCH" \
-L"$$OPENCV_ANDROID/sdk/native/3rdparty/libs/$$ANDROID_TARGET_ARCH" \
-lade \
-littnotify \
-llibjpeg-turbo \
-llibwebp \
-llibpng \
-llibtiff \
-llibopenjp2 \
-lquirc \
-ltegra_hal \
-lopencv_dnn \
-lopencv_objdetect \
-lopencv_calib3d \
-lopencv_imgcodecs \
-lopencv_features2d \
-lopencv_flann \
-lopencv_imgproc \
-lopencv_core \
-lopencv_videoio \
-lcpufeatures \
-llibprotobuf \
ANDROID_EXTRA_LIBS = $$OPENCV_ANDROID/sdk/native/libs/arm64-v8a/libopencv_java4.so
You also need flatbuffers to compile under desktop tensorflow
Install it globally with the following commands:
git clone https://github.com/google/flatbuffers.git
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
make
sudo make install
sudo ldconfig
You can install flatbuffers locally by replacing
sudo make install
cmake –install . –prefix “Folder with build result”,
and then with INCLUDEPATH += and LIBS += add to the Qt project
Software development
Here I will describe the work with tensorflow lite
To work with tensorflow lite, let’s convert the model to its format:
pred_center_model.save('pred_center_model_full')
converter = tf.lite.TFLiteConverter.from_saved_model('pred_center_model_full') # path to the SavedModel directory
tflite_model = converter.convert()
# Save the model.
with open('pred_center_model2.tflite', 'wb') as f:
f.write(tflite_model)
Now let’s include the necessary libraries:
#include <opencv2/opencv.hpp>
#include "tensorflow/lite/interpreter.h"
#include "tensorflow/lite/kernels/register.h"
#include "tensorflow/lite/c/c_api_types.h"
Now let’s load the model:
std::unique_ptr<tflite::FlatBufferModel> m_model;
tflite::ops::builtin::BuiltinOpResolver resolver;
std::unique_ptr<tflite::Interpreter> interpreter;
m_model = tflite::FlatBufferModel::BuildFromFile("путь к модели");
tflite::InterpreterBuilder builder(*m_model, resolver);
TfLiteStatus tatus = builder(&interpreter);
interpreter->AllocateTensors();
If status == kTfLiteOk, then we can perform model inference.
Unfortunately, I did not find how to get the dimensions of the input and output layers from the model itself, so you just need to know them. In this example, the input is a cv:Mat video frame, and the output is an array of 9 numbers (the probabilities of a particular class).
// загружаем данные на входной слой
сonst size_t DATA_SIZE 224*224*3
float* input = interpreter->typed_input_tensor<float>(0);
auto *from_data = (uint8_t*)frame.data;.
copy(from_data, from_data + DATA_SIZE, input);
// делаем инференс
auto status = interpreter->Invoke();
// разбираем данные с выходного слоя
float* output = interpreter->typed_output_tensor<float>(0);
if (status == kTfLiteOk)
{
auto size = 9;
int max_idx {0};
float max = output[0];
static const vector<string> emo_names = {"злость", "презрение", "отвращение", "страх", "радость", "норма", "печаль",
"удивление", "неуверенность"};
vector<string> emotions;
for (int i = 0; i < size; ++i)
{
float curr_val = output[i];
if (curr_val > 0.2)
emotions.push_back(emo_names[i]);
if (curr_val > max)
{
max_idx = i;
max = curr_val;
}
}
return emotions;
}
else
return {"predict error"};
Very often it is required to recognize frames from the video stream of a webcam or a smartphone camera, which is the same programmatically. Capturing can be done either using opencv or Qt. Capturing using opencv is tempting, since you can conveniently do many operations with the received frame, for example, cut out a given area, but I still couldn’t get capture through opencv under android to work. Therefore, I made the capture and output of the video stream using Qt, and the frame conversion using opencv. This works for all platforms.
To capture a camera, you need to create 3 objects:
QScopedPointer<QCamera> m_camera;
QVideoSink *m_video_sink{new QVideoSink{this}};
QMediaCaptureSession m_captureSession;
Then select a camera (for example, the default camera) and link these objects:
m_camera.reset(QMediaDevices::defaultVideoInput());
m_captureSession.setCamera(m_camera.data());
m_camera->start();
m_captureSession.setVideoSink(m_video_sink);
Then periodically poll the video stream and convert the image to cv:Mat:
m_curr_image = m_video_sink->videoFrame().toImage();
m_frame = QImage2Mat(m_curr_image);
Here are the conversion functions from QImage to cv::Mat and vice versa:
using namespace cv;
QImage Mat2QImage(cv::Mat const& src)
{
cv::Mat temp; // make the same cv::Mat
cvtColor(src, temp, COLOR_BGR2RGBA); // cvtColor Makes a copt, that what i need
QImage dest((const uchar *) temp.data, temp.cols, temp.rows, temp.step, QImage::Format_RGB32);
dest.bits(); // enforce deep copy, see documentation
// of QImage::QImage ( const uchar * data, int width, int height, Format format )
return dest;
}
cv::Mat QImage2Mat(QImage const& src)
{
cv::Mat tmp(src.height(),src.width(),CV_8UC4,(uchar*)src.bits(),src.bytesPerLine());
cv::Mat result; // deep copy just in case (my lack of knowledge with open cv)
cvtColor(tmp, result, COLOR_RGBA2BGR);
return result;
}
In addition to your own models, it is useful to use someone else’s, already trained, for example, opencv has a built-in face detection model cv::dnn::Net, here is an example:
auto prepared_frame = cv::dnn::blobFromImage(frame, 1.0, Size(300,300), Scalar(104.0, 177.0, 123.0));
m_face_detect_model.setInput(prepared_frame);
Mat output = m_face_detect_model.forward();
const int SHIFT = 7;
using currTp = Vec<float,SHIFT>;
auto it = output.begin<currTp>();
while(it != output.end<currTp>())
{
currTp pred = *it;
if (pred[2] < 0.5)
break;
int x = pred[3]*m_img_width;
int y = pred[4]*m_img_height;
int width = (pred[5] - pred[3])*m_img_width;
int height = (pred[6] - pred[4])*m_img_height;
coords.push_back(Rect{x, y, width, height});
it+=SHIFT;
}
True, under Android, the opencv models worked for me extremely inefficiently, 40 times worse than the tensorflow models I trained. If someone knows how to fix this write, I will be glad.
Tensorflow provides a bunch of ready-made, trained computer vision models in the mediapipe project and under python they can be conveniently used, but under c++ it is planned that you integrate into mediapipe, and not vice versa, since it is planned that if you use c++, then You need a minimum application size.
The idea of mediapipe is that a text file is written in a special format that describes the data path from input, for example, from a camera, to output to the device screen, this file is fed to the input of a program that implements the general application code. If some transformation does not exist in mediapipe, then a class inherited from mediapipe::CalculatorBase is written, in which this transformation is implemented. You can read more at the link. But this is a topic for a separate article.
Here is a link to the finished project
It has branches with different implementations.