How to deploy a virtual machine learning model environment on any machine?
Sometimes problems arise with deploying a development environment in the cloud, because there are almost no free services with large cloud computing powers. The same Google Collab has its own limits on GPU use; after using up all the memory, you need to wait a day. And the price of the paid version is sometimes not entirely justified… If you have your own good video card, you can always abandon cloud development and switch to the home version.
We remind you that the GPU performs computational work faster due to the ability to execute processes in parallel. If you want to use many video cards, you should connect it to one system, forming a kind of farm.
So how do you containerize your own virtual environment and deploy it using your GPU?
You will need:
Pre-installed cuDNN library for TensorFlow. You need it to gain access to the cores of your card. By the way, Pytorch does not require any libraries and you can easily use the GPU there.
Package your Jupyter work into .py files.
Create a virtual environment by first specifying all the libraries used in requirements.txt. for fast deployment.
Create a Docker image, if desired, and deploy the environment wherever your heart desires using CUDA.
Development environment with cuDNN library and CUDA connection
To work with your GPU, the easiest way is to use the CUDA Toolkit, otherwise CUDA – an interface for parallel computing of the same type of calculations. Nvidia has its own library for efficient use of cores – cuDNN. We also remind you that the library is downloaded along with some frameworks.
How to deploy a development environment with cuDNN?
First you need to update the card drivers. You can do this on the Geforce Experience website. And download the Toolkit itself here: Toolkit and CuDNN library.
Installing the latter is simply copying files from a folder without the usual installation, so you can check the library’s operation as follows, for example, in Visual Studio:
import tensorflow as tf
from tensorflow.python.client import device_lib
print(tf.test.is_built_with_cuda()) #проверка работы CUDA
print(tf.sysconfig.get_build_info(), "\n") #static numbers
sys_details = tf.sysconfig.get_build_info()
print('Prescribed CUDA version:', sys_details["cuda_version"]) #версия CUDA
print('Prescribed cuDNN version:', sys_details["cudnn_version"], "\n") #версия CUDNN
print(tf.reduce_sum(tf.random.normal([1000, 1000]))) #проверка тензоров
print(tf.config.list_physical_devices('GPU'), "\n") #проверка доступных видеокарт/GPU
print(device_lib.list_local_devices())You need to install the ML framework itself. We'll take TensorFlow as an example. We remind you that not all versions of TensorFlow work with video cards or simply GPUs, so be careful. The maximum supported TF version for running video cards on Windows with Python version from 3.7 to 3.10 is 2.10.0
Everything can be installed via pip. Usually Pip can be downloaded directly from the Python installer for Windows.
Install additional libraries: Pandas, Matplotlib and Numpy.
pip install tensorflow-gpu==2.10.0
pip install numpy pandas matplotlib Below is a simple example of deploying a machine learning model using GPU. In our case, based on MNIST data.
import keras
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropout
from keras.datasets import mnist
import sys
# import tensorflow as tf
# config = tf.compat.v1.ConfigProto(gpu_options =
# tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.8)
# # device_count = {'GPU': 1}
# )
# config.gpu_options.allow_growth = True
# session = tf.compat.v1.Session(config=config)
# tf.compat.v1.keras.backend.set_session(session)
def one_hot(data, num_categories):
oh = np.zeros((len(data),num_categories))
for i,entry in enumerate(data):
oh[i, entry] = 1
return oh
# импортируем данные
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Делаем препроцессинг данных
x_train = x_train.reshape( (60000,28,28,1) ) / 256
x_test = x_test.reshape( (10000,28,28,1) ) / 256
y_train = one_hot(y_train, 10)
y_test = one_hot(y_test, 10)
# Строим саму модель
model = Sequential()
input_shape=(28,28,1)
model.add(Conv2D(filters=32,
kernel_size=(3,3),
activation='relu',
input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2,2),
strides=(2,2)))
model.add(Conv2D(filters=32,
kernel_size=(3,3),
activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2),
strides=(2,2)))
model.add(Flatten())
model.add(Dropout(0.5))
model.add(Dense(units=256,
activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(units=10,
activation='softmax'))
# load model weight
# Компилируем
model.compile(loss="categorical_crossentropy",
optimizer="adam",
metrics=['accuracy'])
model.summary()
# Обучаем модель
num_epochs = 1
if num_epochs != 0:
model.fit(x_train, y_train,
batch_size=4,
epochs=num_epochs)
# Делаем оценку
score = model.evaluate(x_test, y_test)
print('\nScore: ', score)Ready. We remind you that if you have a whole render farm at hand, change the unit to the desired number in the line: “# device_count = {'GPU': 1}”.
In the line above you can determine the coefficient of video memory used.
Let our model be previously called in the .py file model – that’s what we will call it in the future.
Congratulations, you have installed a global library for working with graphics cores. To restart the application on a third-party PC, we should create a Docker file. But before that we need to create a virtual environment, which we will put in our “container”.
Creating a Virtual Environment in Python
The virtual environment is one of the most beautiful features of this programming language. You’ve “imported” kilotons of libraries into your project, but how can you launch it from another PC? This is where a virtual environment allows you to create local dependencies for individual projects without clogging up the environment, and solve the problem of incompatibility between operating systems (the environment can be run on any OS, be it Linux or Windows).
To put it even more simply, a virtual environment is a set of tools that will be included with your program. It can be activated/deactivated for the interpreter.
But that's not all. Libraries come in completely different versions… TensorFlow may not be compatible with video cards. The virtual environment allows the interpreter to use the libraries of the required versions. If you install libraries globally, then new versions will replace old ones, and previous projects may not work.
The libraries we need are listed above.
Creating a virtual environment is easy. We will describe the creation for Windows.
1. To create a virtual environment, enter the following code to the console.
python3 -m venv myenv
Myenv is the name of our virtual. environment. Naturally, it can be changed. Using this command, a special directory is created in which the libraries we need will be stored.
2. Activation of the virtual environment. Environments can be activated and deactivated depending on the project.
myenv\Scripts\activate
3. Next, using pip, which was described above, you should install all the dependencies, if necessary, with their versions according to the specification. For example:
(myenv) user@host:~$ pip install tensorflow-gpu==2.10.0
4. So we install all the necessary libraries of our versions. Don't forget to create a requirements file that lists all the dependencies so that all the libraries from the virtual environment on another PC can be easily installed.
pip freeze > requirements.txt
Creating requirements is not necessary, but it is good form in the coding community. Go to any GitHub repository and you will see that every project has such a text box built into it.
A file with dependencies allows you not to transfer the directory to, say, your friend, but simply send him ready-made .py scripts and the file itself with the requirements so that he can reinstall all the necessary libraries/frameworks in his virtual environment.
In our case, the entire list of necessary libraries for our model can be found in the script code itself, which we provided in the first chapter. And don't forget that all your functions from Jupyter need to be translated into python .py files and moved to the created directory.
For Linux, the installation scheme looks similar, but using slightly modified commands.
Docker and containerization
The container allows you to deploy your virtual environment with ready-made scripts on any machine… It is also called a Docker file. These same dockers are easily portable from one runtime environment to another and use host resources locally. Let us add that, due to its lightness, the file allows for greater hardware performance.
Otherwise, Docker is a kind of virtual machine.
If you don't already have Docker, start by installing it on your operating system. You can download and install Docker Desktop from the official Docker website. Next you need to check your CUDA version using the command nvcc - version (on the command line).
And evaluate your version of cuDNN. This is important, otherwise when writing code you can make a mistake with the version and Docker will not use the video card. However, the host of the container is your PC. Next, you need to select the version of the Docker image in accordance with your cuDNN and CUDA versions: nvidia/cuda:10.1-cudnn7-runtime
The Dockerfile itself is a set of sequential instructions for building a container. The docker file executes all instructions sequentially.
In our case, we need to wrap our virtual environment in a container with the installation of all dependencies (libraries specified in requirements.txt) and scripts in the .py editor for working with the machine learning model itself.
In our case, we need to create instructions to run the Model.py script and install all the dependencies.
FROM – specifies the base image from which your Docker image will be built. You can choose a pre-built image from Docker Hub or create your own.
Next, let's assume we don't have python pre-installed in the container. Accordingly, we need to sew it into a container. This is done using the command RUN.
RUN apt-get update && \
apt-get install -y python3
Next you need to add all folders and files (applications) to the container.
In our case, the script and list of dependencies using the command COPY.
COPY model.py / home/ Model.py
COPY requirements.txt / home / requirements.txt
Next, you need to install dependencies (in our case, libraries).
You can do this manually by listing all the libraries using the RUN command via pip.
RUN pip install package1 package2 package3
Or use our created requirements.txt file.
RUN pip install -r requirements.txt
Thus, we loaded our Python script with the finished model and all the dependencies from the file requirements.txt
Since our docker file uses only one Python script, we just need to enter:
ENTRYPOINT ["/usr/bin/python3", "./model.py"]
The command can also be used CMD. Not surprisingly, our command is called ENTRYPOINT, which is where our container application will start executing.
Next, to build the image, you need to write the following line and build Docker itself.
docker build -t model
After model separated by a space, you need to indicate the location where Docker will place the container.
The container must be run using GPU resources, this is written in the command:
docker run --gpus=all -it model
Where model is the name of the container or its id. And the –gpus=all indication is to indicate that GPU resources will be used during startup.
Docker and transferring a virtual environment with scripts and requirements.txt?
Pre-installed cuDNN and the correct choice of TensorFlow version allows us to simply run your machine learning model on a PC using the GPU, but transferring such a virtual environment to another PC is a somewhat tedious task. On the other hand, the first part of the instructions is enough to use the accelerators of your project locally from home if you do not want to transfer the scripts to another machine.
But the container, as we have already written, has a number of advantages. Especially when working with large projects, we can at least use several scripts in a certain logic. Containers are lightweight and allow you to get the most out of your GPU and easily deploy a virtual environment in any runtime environment.
