PyInstaller and Nuitka
Recently, there was a need to provide our QA department with one of the Python modules in the form of a standalone binary that would not require installation and environment configuration. Following the need, interest in existing tools for this was formed.
One of the options was to use Docker, but I rejected it because the environment for Docker would also need to be prepared. Then this image would need to be launched correctly and interacted with correctly. Of course, docker compose can be used to simplify things, but this does not greatly reduce the complexity for the end user. In addition, the image will be quite large.
So after some thought I turned to tools like Python Compilers, namely – Nuitka And PyInstaller and did a little research to see if they would suit my needs.
Both tools package a Python application with all its dependencies into a single package so that the end user of the application does not need to install Python on their machine.
There are two options for what we get from their work as a result (other than emotional feelings):
A Python application is represented by a single directory with a binary for launching and all dependencies as separate files.
A Python application and all its dependencies are packaged into a single binary.
PyInstaller
The experiment was conducted on version 6.10.0
% pyinstaller -version
6.10.0Application in the catalog
% time pyinstaller generator/main.py
pyinstaller generator/main.py 18.63s user 2.72s system 95% cpu 22.376 totalThe output was two catalogues – build And dist
% du -sh build dist
56M build
75M dist% ls -l build/main dist/main/*
-rwxr-xr-x 1 max staff 17177744 Aug 14 12:00 dist/main/main
build/main:
total 113656
-rw-r--r-- 1 max staff 999486 Aug 14 12:00 Analysis-00.toc
-rw-r--r-- 1 max staff 562354 Aug 14 12:00 COLLECT-00.toc
-rw-r--r-- 1 max staff 2974 Aug 14 12:00 EXE-00.toc
-rw-r--r-- 1 max staff 2780 Aug 14 12:00 PKG-00.toc
-rw-r--r-- 1 max staff 16906333 Aug 14 12:00 PYZ-00.pyz
-rw-r--r-- 1 max staff 435304 Aug 14 12:00 PYZ-00.toc
-rw-r--r-- 1 max staff 1443565 Aug 14 11:59 base_library.zip
drwxr-xr-x 6 max staff 192 Aug 14 12:00 localpycs
-rwxr-xr-x 1 max staff 17177744 Aug 14 12:00 main
-rw-r--r-- 1 max staff 16937430 Aug 14 12:00 main.pkg
-rw-r--r-- 1 max staff 17259 Aug 14 12:00 warn-main.txt
-rw-r--r-- 1 max staff 3680264 Aug 14 12:00 xref-main.html
dist/main/_internal:
total 15976
drwxr-xr-x 7 max staff 224 Aug 14 12:00 IPython
drwxr-xr-x 9 max staff 288 Aug 14 12:00 PIL
lrwxr-xr-x 1 max staff 37 Aug 14 12:00 Python -> Python.framework/Versions/3.11/Python
drwxr-xr-x 5 max staff 160 Aug 14 12:00 Python.framework
-rwxr-xr-x 1 max staff 234176 Aug 14 12:00 _cffi_backend.cpython-311-darwin.so
-rw-r--r-- 1 max staff 1443565 Aug 14 12:00 base_library.zip
drwxr-xr-x 3 max staff 96 Aug 14 12:00 cryptography
drwxr-xr-x 10 max staff 320 Aug 14 12:00 cryptography-41.0.1.dist-info
drwxr-xr-x 10 max staff 320 Aug 14 12:00 email_validator-2.2.0.dist-info
drwxr-xr-x 9 max staff 288 Aug 14 12:00 factory_boy-3.3.0.dist-info
drwxr-xr-x 3 max staff 96 Aug 14 12:00 faker
drwxr-xr-x 3 max staff 96 Aug 14 12:00 jedi
drwxr-xr-x 60 max staff 1920 Aug 14 12:00 lib-dynload
drwxr-xr-x 7 max staff 224 Aug 14 12:00 lib2to3
lrwxr-xr-x 1 max staff 30 Aug 14 12:00 libXau.6.0.0.dylib -> PIL/.dylibs/libXau.6.0.0.dylib
lrwxr-xr-x 1 max staff 39 Aug 14 12:00 libbrotlicommon.1.1.0.dylib -> PIL/.dylibs/libbrotlicommon.1.1.0.dylib
lrwxr-xr-x 1 max staff 36 Aug 14 12:00 libbrotlidec.1.1.0.dylib -> PIL/.dylibs/libbrotlidec.1.1.0.dylib
-rwxr-xr-x 1 max staff 4222928 Aug 14 12:00 libcrypto.3.dylib
lrwxr-xr-x 1 max staff 31 Aug 14 12:00 libfreetype.6.dylib -> PIL/.dylibs/libfreetype.6.dylib
lrwxr-xr-x 1 max staff 31 Aug 14 12:00 libharfbuzz.0.dylib -> PIL/.dylibs/libharfbuzz.0.dylib
lrwxr-xr-x 1 max staff 32 Aug 14 12:00 libjpeg.62.4.0.dylib -> PIL/.dylibs/libjpeg.62.4.0.dylib
lrwxr-xr-x 1 max staff 28 Aug 14 12:00 liblcms2.2.dylib -> PIL/.dylibs/liblcms2.2.dylib
lrwxr-xr-x 1 max staff 27 Aug 14 12:00 liblzma.5.dylib -> PIL/.dylibs/liblzma.5.dylib
-rwxr-xr-x 1 max staff 189360 Aug 14 12:00 libmpdec.4.dylib
lrwxr-xr-x 1 max staff 34 Aug 14 12:00 libopenjp2.2.5.2.dylib -> PIL/.dylibs/libopenjp2.2.5.2.dylib
lrwxr-xr-x 1 max staff 29 Aug 14 12:00 libpng16.16.dylib -> PIL/.dylibs/libpng16.16.dylib
lrwxr-xr-x 1 max staff 31 Aug 14 12:00 libsharpyuv.0.dylib -> PIL/.dylibs/libsharpyuv.0.dylib
-rwxr-xr-x 1 max staff 1240816 Aug 14 12:00 libsqlite3.0.dylib
-rwxr-xr-x 1 max staff 838736 Aug 14 12:00 libssl.3.dylib
lrwxr-xr-x 1 max staff 27 Aug 14 12:00 libtiff.6.dylib -> PIL/.dylibs/libtiff.6.dylib
lrwxr-xr-x 1 max staff 27 Aug 14 12:00 libwebp.7.dylib -> PIL/.dylibs/libwebp.7.dylib
lrwxr-xr-x 1 max staff 32 Aug 14 12:00 libwebpdemux.2.dylib -> PIL/.dylibs/libwebpdemux.2.dylib
lrwxr-xr-x 1 max staff 30 Aug 14 12:00 libwebpmux.3.dylib -> PIL/.dylibs/libwebpmux.3.dylib
lrwxr-xr-x 1 max staff 30 Aug 14 12:00 libxcb.1.1.0.dylib -> PIL/.dylibs/libxcb.1.1.0.dylib
lrwxr-xr-x 1 max staff 28 Aug 14 12:00 libz.1.3.1.dylib -> PIL/.dylibs/libz.1.3.1.dylib
drwxr-xr-x 3 max staff 96 Aug 14 12:00 markupsafe
drwxr-xr-x 3 max staff 96 Aug 14 12:00 ossl-modules
drwxr-xr-x 4 max staff 128 Aug 14 12:00 parso
drwxr-xr-x 3 max staff 96 Aug 14 12:00 pydantic_core
drwxr-xr-x 3 max staff 96 Aug 14 12:00 setuptools
drwxr-xr-x 3 max staff 96 Aug 14 12:00 text_unidecode
drwxr-xr-x 9 max staff 288 Aug 14 12:00 typeguard-4.3.0.dist-info
drwxr-xr-x 9 max staff 288 Aug 14 12:00 wheel-0.43.0.dist-infoAs you can see, the directory contains all the application dependencies and also the Python interpreter itself. All of them, including Python, are represented by shared libraries.
% file dist.folder/main/_internal/Python.framework/Versions/3.11/Python
dist.folder/main/_internal/Python.framework/Versions/3.11/Python: Mach-O 64-bit dynamically linked shared library arm64During startup, the bootloader sets environment variables, signal handlers, etc., and then starts a child process. The child process, in turn, is the Python interpreter, which begins executing our application.
Details: https://pyinstaller.org/en/stable/advanced-topics.html#the-bootstrap-process-in-detail
Application in one file
In this case, the bootloader unpacks the contents of the binary into a temporary directory, from which the application itself is then launched. When the work is finished, the temporary directory is deleted. Otherwise, the process looks identical to launching the application in a directory.
For this reason, a single file application takes longer to launch than a directory application.
% time pyinstaller --onefile generator/main.py
pyinstaller --onefile generator/main.py 22.72s user 2.49s system 96% cpu 26.063 total% du -sh build dist
56M build
33M dist% ls -l build/main dist/main
-rwxr-xr-x 1 max staff 34381088 Aug 15 08:45 dist/main
build/main:
total 114872
-rw-r--r-- 1 max staff 1000578 Aug 15 08:45 Analysis-00.toc
-rw-r--r-- 1 max staff 564818 Aug 15 08:45 EXE-00.toc
-rw-r--r-- 1 max staff 564630 Aug 15 08:45 PKG-00.toc
-rw-r--r-- 1 max staff 17070667 Aug 15 08:45 PYZ-00.pyz
-rw-r--r-- 1 max staff 436396 Aug 15 08:45 PYZ-00.toc
-rw-r--r-- 1 max staff 1443565 Aug 15 08:45 base_library.zip
drwxr-xr-x 6 max staff 192 Aug 15 08:45 localpycs
-rw-r--r-- 1 max staff 34007396 Aug 15 08:45 main.pkg
-rw-r--r-- 1 max staff 18491 Aug 15 08:45 warn-main.txt
-rw-r--r-- 1 max staff 3693075 Aug 15 08:45 xref-main.htmlHere you can see that at the output we received a single binary file.
Launching what happened
% time dist/main/main --no-serve-files
FileNotFoundError: [Errno 2] No such file or directory: '/Users/max/work/wectory/qa-automation/dist/main/_internal/generator/data.json'
[PYI-4129:ERROR] Failed to execute script 'main' due to unhandled exception!
dist/main/main --no-serve-files 0.35s user 0.04s system 97% cpu 0.406 totalApparently the distribution needs to be supplemented with some files
% cp generator/data.json dist/main/_internal/generator/Let's try again and get
FileNotFoundError: [Errno 2] No such file or directory: '/Users/max/work/wectory/qa-automation/dist/main/_internal/generator/templates/default'
[PYI-4343:ERROR] Failed to execute script 'main' due to unhandled exception!Well, let's move this too.
% cp -r generator/templates dist/main/_internal/generatorThis time everything went well:
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Landlord ┃ Katherine Francesca Mills (Company) ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ birthdate │ 30.05.1969 │
│ email │ developers+l240815084019@wectory.com │
│ phone │ +447181970103 │
│ address │ 3046 Powell Union Suite 769, North Rita, NH 65169 │
└───────────┴───────────────────────────────────────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Agent ┃ Chelsea Hazel Williams ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ birthdate │ 22.11.1988 │
│ email │ developers+a240815084019@wectory.com │
│ phone │ +447197971413 │
│ address │ 2310 Bolton Lodge Apt. 402, Jonesstad, WA 61453 │
└───────────┴─────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Agency ┃ Jones, Bradley and Murphy ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ email │ developers+a240815084019@wectory.com │
│ sort code │ 846683 │
│ account number │ 73977208 │
│ address │ 577 Hull Drives, Curtisberg, WY 87473 │
└────────────────┴───────────────────────────────────────┘
Results 1: /Users/max/work/wectory/qa-automation/dist/main/_internal/generator/generated/2024-08-15-08-40-19In the case of launching the application as a single file, we see the same errors
% time dist/main --no-serve-files
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/dw/vk1bw2wd18zdpgs2m8lfzw4r0000gn/T/_MEIzYrZXM/generator/data.json'
[PYI-5126:ERROR] Failed to execute script 'main' due to unhandled exception!
dist/main --no-serve-files 0.93s user 0.87s system 14% cpu 12.197 totalIt should be noted that the launch time has increased significantly and does not change much on repeated launches, as was the case with the application in the catalog.
But let's get back to the error. Because a temporary directory is created each time, there is no way to just put our statics somewhere. Fortunately, the developers have foreseen this and allowed us to set a temporary directory in several ways: via an environment variable or via the command line.
The difference is that through the environment variable we can set the path to the temporary directory during the execution of our application, and we can use the command line only at the build stage.
Let's try with the command line:
% pyinstaller --runtime-tmpdir main.tmp --onefile generator/main.py% dist/main --no-serve-files
FileNotFoundError: [Errno 2] No such file or directory: '/Users/max/work/wectory/qa-automation/main.tmp/_MEIXIoz7Y/generator/data.json'
[PYI-7163:ERROR] Failed to execute script 'main' due to unhandled exception!As you can see, the previously specified directory was actually used as the temporary directory. main.tmpbut with a nuance – the application itself was unpacked into a subdirectory _MEIXIoz7Ywhich was automatically deleted after the execution was completed.
% ls -l main.tmp
total 0The same thing if you specify to sing via an environment variable (I had to rebuild the application without the key) --runtime-tmpdir). The path must be created in advance.
% mkdir main.tmp2
% env TMPDIR=main.tmp2 dist/main --no-serve-files
FileNotFoundError: [Errno 2] No such file or directory: '/Users/max/work/wectory/qa-automation/main.tmp2/_MEIbatKsa/generator/data.json'
[PYI-7744:ERROR] Failed to execute script 'main' due to unhandled exception!It turns out that there is no easy way to place our statics somewhere. I was too lazy to figure it out further.
The conclusion from this is this: at the design and development stage, you need to decide where you will store the statics.
Nuitka
Nuitka differs from PyInstaller in that it transpiles Python code to C code and then compiles it into a native executable file. But to make the application fully portable, you need to use the option --standaloneotherwise the application will depend on libraries that will have to be installed on the target machine.
Version
% nuitka --version
2.4.5
Commercial: None
Python: 3.11.9 (main, Apr 2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)]
Flavor: Homebrew Python
Executable: /Users/max/work/wectory/qa-automation/env/bin/python3.11
OS: Darwin
Arch: arm64
macOSRelease: 14.6
Version C compiler: /usr/bin/clang (clang 15.0.0).Application in the form of a directory
% time nuitka --standalone generator
nuitka --standalone generator 1616.68s user 210.46s system 473% cpu 6:26.05 totalThe build took noticeably longer (even my feet got warm during the process) than PyInstaller did. But that's understandable – the process involves a full compilation of not only our application code, but also all dependencies.
% du -sh generator.build generator.dist
442M generator.build
104M generator.distI will not give the conclusion here. ls -ltr generator.build generator.dist – it turns out to be very large due to the number of modules.
Application in one file
% time nuitka --standalone --onefile generator
nuitka --standalone --onefile generator 301.93s user 106.49s system 202% cpu 3:21.68 totalWhat do we get?
% du -sh generator.build generator.dist generator.onefile-build generator.bin
440M generator.build
104M generator.dist
23M generator.onefile-build
23M generator.binLaunching what happened
% time generator.dist/generator.bin --no-serve-files
FileNotFoundError: [Errno 2] No such file or directory: '/Users/max/work/wectory/qa-automation/generator.dist/generator/data.json'
generator.dist/generator.bin 0.23s user 0.03s system 97% cpu 0.274 totalAnd we see an error that is similar to the one we saw with PyInstaller. Fortunately, it is clear what to do
% cp -r generator/data.json generator/templates generator.dist/generator/Attempt #2
% time generator.dist/generator.bin --no-serve-filesAnd inevitable success
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Landlord ┃ Carly Ashley Roberts (Individual) ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ birthdate │ 26.08.1988 │
│ email │ developers+l240815105459@wectory.com │
│ phone │ +447417436058 │
│ address │ 597 Allison Shoal, North James, TN 27518 │
└───────────┴──────────────────────────────────────────┘
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Agent ┃ Jenna Lorraine Humphreys ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ birthdate │ 17.05.1975 │
│ email │ developers+a240815105459@wectory.com │
│ phone │ +447934521633 │
│ address │ 551 Justin Light Apt. 663, Hessberg, HI 69859 │
└───────────┴───────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Agency ┃ Smith Group ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ email │ developers+a240815105459@wectory.com │
│ sort code │ 563919 │
│ account number │ 04717704 │
│ address │ 0237 Haley Mountain Suite 776, Meredithmouth, ME 27066 │
└────────────────┴────────────────────────────────────────────────────────┘
Results 1: /Users/max/work/wectory/qa-automation/generator.dist/generator/generated/2024-08-15-10-54-59
generator.dist/generator.bin --no-serve-files 1.18s user 0.33s system 94% cpu 1.590 totalNow let's try the single-file option
% time ./generator.bin
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/dw/vk1bw2wd18zdpgs2m8lfzw4r0000gn/T/onefile_20785_1723701790_311895/generator/data.json'
./generator.bin 0.77s user 0.24s system 15% cpu 6.668 totalThe problem is similar to the one with PyInstaller – statics are searched in the temporary directory. At this point I finished trying the single-file option, because I was too lazy to figure out how to get around it.
The conclusion is the same: decide in advance where we will store the static.
Conclusion
My conclusions after getting to know PyInstaller and Nuitka are as follows:
both solutions work without any special dancing with a tambourine, which is good. True, there were difficulties with the statics of the application, but this is a flaw on the part of the application itself, and not PyInstaller or Nuitka
At the application development stage, it is imperative to make the right decision about where data files and other statics will be stored, and how we will pack them or transfer them to users
As for the speed of assembly, PyInstaller significantly outperforms Nuitka in this matter. It is understandable – PyInstaller does not recompile the source code of the entire application and its dependencies, but only transfers it to the target directory. Assembly time is important to consider when we build pipelines
PyInstaller also outperforms Nuitka in terms of the size of the resulting directories/files, although not significantly. Perhaps, on larger projects, this difference will not be so noticeable
For myself, I would choose Nuitka. And what instrument will you choose today?
