Distribution of computing power among several PCs

In the course of the work, the task of processing a huge volume (~ 500 GB) of audio recordings arose in a limited time. On one PC, processing would have taken at least a month, which did not fit into the deadlines set by the customer. There was an idea to connect the computers of colleagues whose PCs “rest” at night.

The whole process can be broken down into several steps:

1. Installing Python on a working computer.

2. Downloading to the computer an archive consisting of 2 bat-files (one for setting up the environment, the second for running the script), a list of required libraries, a script for processing audio recordings.

3. Setting up a virtual environment and running the script.

Let’s take a look at the configuration code.

Bat-file of environment settings, which is launched in the folder for the script:

Python –m venv work_env
Work_env\Scripts\pip.exe install -r requirements.txt

With this action, we set up the environment to run the second bat file with the code that runs the script itself.

Code of the 2nd bat-file.

Work_env\Scripts\python.exe script_name.py

It is important to specify the full path from the virtual environment for pip and python to run them correctly.

Thus, it is possible to connect the computers of colleagues, even those who are not familiar with programming and located in different time zones, to processing by writing a small instruction for them.

Let’s complicate the task. Imagine that we have processing in several stages. We carry out the first stage on our PC, send the result to a network folder, from where the second PC should take it for processing at night. When transferring files, we will make an email notification to ourselves in Outlook so that you can track the progress of the work. One of the options is to write files to a test document, but this is not convenient for viewing, and in Outlook you can put interested parties in a copy so that they are also notified about the processing steps. For the convenience of viewing messages, it is possible to create a rule for moving generated messages to a separate folder according to the subject of the message.

The code for solving such a problem on a colleague’s PC might look like this:

#Импорт библиотек
import os 
import time
import shutil
import win32com.client as win32
#Прописываем путь до сетевой папки и папки, в которую необходимо сбрасывать результаты. 
directory = r’Folder_path’ 
destination_path= r’Destination_folder_path’
#Запускаем проверку на наличие файлов каждую минуту
While True: 
 	Time.sleep(60)
#Получаем список файлов
files = os.listdir(directory)
#Проверяем, появились ли файлы для обработки.
if  len(files)>0:
	#При наличии файлов запускаем их в обработку.
		for file in files:
			#Для примера они просто переносятся в другую папку.
			shutil.move(file, file_destination)
outlook=win32.Dispatch('outlook.application')
mail=outlook.CreateItem(0)
mail.To=’my_mail@mail.ru'  #e-mail
mail.Subject="Topic"            #Тема
mail.Body=’Перенесено '+ str(len(files)) + ‘ в папку ’ + destination_path         #Пишет сколько файлов перенесено и в какую папку
mail.Send()                          #Отправка
	#Если файлов нет, возвращаемся в начало цикла.
elif len(files)==0:
		continue

Improvement of the program and the principle of operation of the ceiling has no. If necessary, it is possible to create a trigger for incoming mail and run the script on the trigger.

By the way, in my case, using 5 computers, the task of processing audio recordings was solved in 6 days.

Thank you for your attention!