Universal build scripts

Welcome to the eighth Nix pill. In the previous seventh pill, we successfully collected the derivation. We wrote a build script that compiled the C program and installed the binary image into the Nix repository.

In this post we will summarize the build script by writing a Nix expression for GNU hello world and create a wrapper over the built-in function derivation.

Packaging GNU hello world

In the previous pill we packed a simple file .cwhich was compiled using the normal call gcc. This is not the most successful example of a project. Many people use autotools and since we want to generalize our script, it is better to focus on the most popular build system.

GNU hello world despite its name, this is still a simple project, assembled using autotools. Download the latest archive from here: https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz.

Let's create a build script for GNU hello world, let's call it hello_builder.sh:

export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzip/bin:$gnugrep/bin:$gnused/bin:$bintools/bin"
tar -xzf $src
cd hello-2.12.1
./configure --prefix=$out
make
make install

Derivation hello.nix:

let
  pkgs = import <nixpkgs> { };
in
derivation {
  name = "hello";
  builder = "${pkgs.bash}/bin/bash";
  args = [ ./hello_builder.sh ];
  inherit (pkgs)
    gnutar
    gzip
    gnumake
    gcc
    coreutils
    gawk
    gnused
    gnugrep
    ;
  bintools = pkgs.binutils.bintools;
  src = ./hello-2.12.1.tar.gz;
  system = builtins.currentSystem;
}

Nix in Darwin

Building in Darwin (i.e. macOS) traditionally uses C as the compiler clang instead of gcc. To adapt our example for Darwin, let's write the following modified version: hello.nix:

let pkgs = import <nixpkgs> { }; in derivation { name = "hello"; builder = "${pkgs.bash}/bin/bash"; args = [ ./hello_builder.sh ]; inherit (pkgs) gnutar gzip gnumake coreutils gawk gnused gnugrep ; gcc = pkgs.clang; bintools = pkgs.clang.bintools.bintools_bin; src = ./hello-2.12.1.tar.gz; system = builtins.currentSystem; }

Later we will show how to handle these differences automatically. For now, just keep in mind that other scripts may also require changes similar to those described above.

Let's build the program by running nix build hello.nix. Now you can do result/bin/hello. Everything is quite simple, but is it necessary to write builder.sh for each package? Should I always pass dependencies to a function? derivation?

Please pay attention to the parameter --prefix=$outwhich we discussed in the previous pill.

Universal script

Let's summarize builder.sh for all projects autotools:

set -e
unset PATH
for p in $buildInputs; do
    export PATH=$p/bin${PATH:+:}$PATH
done

tar -xf $src

for d in *; do
    if [ -d "$d" ]; then
        cd "$d"
        break
    fi
done

./configure --prefix=$out
make
make install

What are we doing?

  1. By using set -e we ask the shell to abort execution of the script in case of any error.

  2. First we clean PATH`` (unset PATH`), because in this place the variable contains non-existent paths.

  3. To the end of every path from $buildInputs we add bin and add everything together to PATH. We'll discuss the details a little later.

  4. Let's unpack the sources.

  5. We look for the directory where the sources were unpacked and go to it by running the command cd.

  6. Finally, we configure, compile and install the project.

As you can see, there are no longer any references to “hello” in the build script. The script still relies on several conventions, but this version is certainly more universal.

Now let's rewrite hello.nix:

let
  pkgs = import <nixpkgs> { };
in
derivation {
  name = "hello";
  builder = "${pkgs.bash}/bin/bash";
  args = [ ./builder.sh ];
  buildInputs = with pkgs; [
    gnutar
    gzip
    gnumake
    gcc
    coreutils
    gawk
    gnused
    gnugrep
    binutils.bintools
  ];
  src = ./hello-2.12.1.tar.gz;
  system = builtins.currentSystem;
}```

Тут всё ясно, за исключением, может быть, `buildInputs`. Но и в `buildInputs` нет никакой чёрной магии.

Nix умеет конвертировать списки в строку. Сначала он конвертирует в строки каждый отдельный элемент, а затем склеивает их, разделяя пробелом:

```text
nix-repl> builtins.toString 123
"123"

nix-repl> builtins.toString [ 123 456 ]
"123 456"

Let us remember that derivations can also be converted to a string, therefore:

nix-repl> :l <nixpkgs>
Added 3950 variables.

nix-repl> builtins.toString gnugrep
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14"

nix-repl> builtins.toString [ gnugrep gnused ]
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14 /nix/store/krgdc4sknzpw8iyk9p20lhqfd52kjmg0-gnused-4.2.2"

It's that simple! Variable buildInputs will ultimately contain the paths we need, separated by a space. There's nothing better to use in a loop for interpreter bash.

Convenient version of the derivation function

We managed to write a script that can be used for different projects autotools.
But in expression hello.nix we identify all the programs that may be required, including those that are not needed to build a particular project.

We can write a function that is the same as derivationtakes a set of attributes, and merges it with another set of attributes common to all projects.

autotools.nix:

pkgs: attrs:
let
  defaultAttrs = {
    builder = "${pkgs.bash}/bin/bash";
    args = [ ./builder.sh ];
    baseInputs = with pkgs; [
      gnutar
      gzip
      gnumake
      gcc
      coreutils
      gawk
      gnused
      gnugrep
      binutils.bintools
    ];
    buildInputs = [ ];
    system = builtins.currentSystem;
  };
in
derivation (defaultAttrs // attrs)

To understand how this code works, let's remember something about Nix features. Entire Nix expression from file autotools.nix turns into a function.
This function takes a parameter pkgs and returns a function that takes a parameter attrs.

There's nothing complicated going on inside the function, but when we first look at it we may have to spend some time understanding how it works.

  1. First, add a magic set of attributes to the visibility area pkgs.

  2. Using the expression let define an auxiliary variable defaultAttrswhere we add several attributes needed for derivation.

  3. At the end we create and call derivationpassing a strange expression as a parameter (defaultAttrs // attrs).

Operator // — accepts two sets as input. The result is their unification. In case of attribute name conflict, the value from the right set is used.

So we use defaultAttrs as a basis, and add (override) attributes from attrs.

A couple of examples will clarify the operator's work:

nix-repl> { a = "b"; } // { c = "d"; }
{ a = "b"; c = "d"; }

nix-repl> { a = "b"; } // { a = "c"; }
{ a = "c"; }

Exercise: End the new script builder.sh adding $baseInputs into a cycle for together with $buildInputs.

Result operator // we pass to the function derivation. Attribute buildInputs empty, so it will have exactly the value specified in the set attrs.

Let's rewrite hello.nix:

let
  pkgs = import <nixpkgs> { };
  mkDerivation = import ./autotools.nix pkgs;
in
mkDerivation {
  name = "hello";
  src = ./hello-2.12.1.tar.gz;
}

The final! We have received the simplest description of the package! A few comments to help you understand the Nix language better.

  • We put in a variable pkgs import, which in previous expressions was placed in the “with” statement. This is a common practice and should not be taken for granted.

  • Variable mkDerivation is a perfect example of partial application. You can look at it as '(import ./autotools.nix) pkgs'.

  • First we import the expression, then apply it to the parameter pkgs (in functional languages, calling a function with a parameter is often called applying a function to a parameter – *translator's note).

  • This gives us a function that accepts a set of attributes attrs.

  • We create a derivation by specifying only attributes name And src. If the project needs other dependencies in PATHthey can be added to buildInputsbut in the example with hello.nix we didn't need it.

Please note that we do not use any other libraries. We may need C compiler flags to look for include files of other libraries.
Also, we may need linker flags to look for static library ones.

Conclusion

Nix gives us the basic tools for creating derivations, preparing the build environment, and storing the result in Nix storage.

In this pill we wrote a universal project build script autotools and function mkDerivation. The latter combines the main components used in projects autotools with default settings, and saves us from duplicating code in different projects.

We learned how to extend the Nix system by writing and merging new derivations.

Analogy: In C, you create objects that are on the heap and then create new objects based on them. Pointers are used to refer to other objects.

In Nix, you create derivations that are in the Nix repository, and then create new derivations from them. Output paths are used to link to other derivations.

In the next pill

…we'll talk about runtime dependencies. Is the GNU hello world package standalone? What are its runtime dependencies? So far we have defined the dependencies for the assembly by using other derivations in the “hello” derivation.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *