9. Automatic runtime dependencies

Welcome to the ninth Nix pill.
In the previous eighth pill we developed a universal build script for projects autotools.
We downloaded the dependencies and sources, and got a Nix derivation as a result.

Today we will turn to the GNU program helloto examine build-time and run-time dependencies. We then refine our script to eliminate unnecessary dependencies.

Build Dependencies

Let's start analyzing the build dependencies for the GNU package hello:

$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.10.tar.gz
/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv
/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv
/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv
/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh
/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv
/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv
/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv
/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv
/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv
/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv
/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv

Considering that our universal function mkDerivation always extracts such dependencies (compare with package build-essential from Debian), they will be in the Nix repository before they are needed by any package to build.

Why do we look at files? .drv?
Because hello.drv represents the action that compiles the program hello on the way out.
As such, it contains the input derivations needed for assembly. hello.

A little bit about NAR files

Format NAR stands for “Nix ARchive”.
It was developed because existing archive formats such as tardo not meet some important requirements.
Nix requires deterministic build tools, but regular archivers are not deterministic: they align data, they don't sort files, they add timestamps, and so on.
As a result, directories containing bit-identical files are converted into bit-non-identical archives, which leads to different hashes.

Unlike tar, NAR was designed as a simple, deterministic archive format.
Below we will see that it is widely used in Nix.

For detailed justification and implementation details NAR contact us Dolstra's doctoral dissertation.

To create archives NAR from the storage paths, we can use utilities nix-store --dump And nix-store --restore.

Runtime dependencies

Note that Nix automatically recognized the build dependencies because they were referenced by the function derivationbut we never specified runtime dependencies.

Nix detects runtime dependencies automatically.
The technology he uses may seem fragile at first glance, but it works so well that the NixOS operating system is built on it.
The underlying mechanism uses storage path hashes.
It is completed in three steps.

Creates an archive NAR from the derivation. Note that this step serializes the output of the derivation – it works well both when the derivation is a single file and when it is an entire directory.
For each file .drvon which the assembly depends and its relative output path, searches this path for the NAR archive.
If the archive is found, the path is a runtime dependency.

The given fragment shows the dependencies hello.

$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello

We see that glibc And gcc are runtime dependencies.
Intuitively, there shouldn't be any here. gcc! But outputting lines from a binary file to the screen hello shows that gcc it really does occur there:

$ strings result/bin/hello|grep gcc
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64

That's why Nix added gcc.
But where did this path come from in the first place?
The thing is, it is in ld rpath: list of directories containing runtime libraries.
Other distributions don't usually abuse this.
But in Nix we have to reference specific versions of libraries, so rpath plays an important role.

The build process adds the path to the libraries gccassuming that it will be required at runtime. However, this is not necessarily the case.
To solve this problem, Nix provides a tool called patchelfwhich brings together rpath to the paths that are actually needed when executing the program.

Even after the cuts rpathexecutable file hello still depends on gcc due to debug information.
In the next section we will explore how to use strip completely get rid of this addiction.

Another phase of assembly

Let's add one more phase to the build script autotools.
Currently the collector has six phases:

“Environment setup” phase
The “unpack” phase: we unpack the sources into the current directory (remember that Nix runs the build in a temporary directory)
“Change directory” phase: the temporary directory becomes the root of the source tree
“Configuration” phase: ./configure
Assembly phase: make
The “installation” phase: make install

Let's add a new phase right after the “installation”, this will be the “correction” phase.
Let's add it to the end builder.sh:

find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null

That is, for each file we perform patchelf --shrink-rpath And strip.
Note that since we used two new commands find And patchelfwe need to add them to the derivation.

Exercise: Add findutils And patchelf To baseInputs script autotools.nix.

Now let's put it back together hello.nix…

$ nix-build hello.nix
[...]
$ nix-store -q --references result
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello

and we will see that only the library remains in the list of runtime dependencies glibc.
This is exactly what we were trying to achieve.

We have made a stand-alone (self-sufficient) package.
This means that we can copy it to another machine, where it will compile exactly the same program. hello.
Please note that it requires some components from to run it. /nix/storeso it will be necessary run nix.
Executable file hello it starts with exactly that version of the library glibc and the interpreter that are specified in it, and not with the system versions.

(Here we are talking about the ELF loader, which sometimes called an interpreter — translator's note).

$ ldd result/bin/hello
 linux-vdso.so.1 (0x00007fff11294000)
 libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000)
 /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000)

Of course, the executable file will run fine as long as all the necessary packages are in the directory. /nix/store.

Conclusion

We got acquainted with several Nix tools and their capabilities.
In particular, we learned how Nix recognizes runtime dependencies.
And we're not just talking about shared libraries, but also about executables, scripts, Python libraries, and so on.

This approach to building allows packages to be self-contained, ensuring that copying the package to another machine is enough to run the program.
Thanks to this, we can run programs without installation using nix-shell and use Nix for reliable deployment in the cloud.

In the next pill

The next pill will tell you about nix-shell.
Previously, we built derivations from scratch using nix-build: unpacked the source codes, configured, compiled and installed.
Deploying large packages can take a long time.
We could make small changes and use incremental compilation, while still benefiting from a self-contained environment, as in nix-build. For this we will need nix-shell.