Universal build scripts
Welcome to the eighth Nix pill. In the previous seventh pill, we successfully collected the derivation. We wrote a build script that compiled the C program and installed the binary image into the Nix repository.
In this post we will summarize the build script by writing a Nix expression for GNU hello world and create a wrapper over the built-in function derivation
.
Packaging GNU hello world
In the previous pill we packed a simple file .c
which was compiled using the normal call gcc
. This is not the most successful example of a project. Many people use autotools
and since we want to generalize our script, it is better to focus on the most popular build system.
GNU hello world despite its name, this is still a simple project, assembled using autotools
. Download the latest archive from here: https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz.
Let's create a build script for GNU hello world, let's call it hello_builder.sh
:
export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzip/bin:$gnugrep/bin:$gnused/bin:$bintools/bin"
tar -xzf $src
cd hello-2.12.1
./configure --prefix=$out
make
make install
Derivation hello.nix
:
let
pkgs = import <nixpkgs> { };
in
derivation {
name = "hello";
builder = "${pkgs.bash}/bin/bash";
args = [ ./hello_builder.sh ];
inherit (pkgs)
gnutar
gzip
gnumake
gcc
coreutils
gawk
gnused
gnugrep
;
bintools = pkgs.binutils.bintools;
src = ./hello-2.12.1.tar.gz;
system = builtins.currentSystem;
}
Nix in Darwin
Building in Darwin (i.e. macOS) traditionally uses C as the compiler
clang
instead ofgcc
. To adapt our example for Darwin, let's write the following modified version:hello.nix
:
let pkgs = import <nixpkgs> { }; in derivation { name = "hello"; builder = "${pkgs.bash}/bin/bash"; args = [ ./hello_builder.sh ]; inherit (pkgs) gnutar gzip gnumake coreutils gawk gnused gnugrep ; gcc = pkgs.clang; bintools = pkgs.clang.bintools.bintools_bin; src = ./hello-2.12.1.tar.gz; system = builtins.currentSystem; }
Later we will show how to handle these differences automatically. For now, just keep in mind that other scripts may also require changes similar to those described above.
Let's build the program by running nix build hello.nix
. Now you can do result/bin/hello
. Everything is quite simple, but is it necessary to write builder.sh
for each package? Should I always pass dependencies to a function? derivation
?
Please pay attention to the parameter --prefix=$out
which we discussed in the previous pill.
Universal script
Let's summarize builder.sh
for all projects autotools
:
set -e
unset PATH
for p in $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done
tar -xf $src
for d in *; do
if [ -d "$d" ]; then
cd "$d"
break
fi
done
./configure --prefix=$out
make
make install
What are we doing?
By using
set -e
we ask the shell to abort execution of the script in case of any error.First we clean
PATH`` (
unset PATH`), because in this place the variable contains non-existent paths.To the end of every path from
$buildInputs
we addbin
and add everything together toPATH
. We'll discuss the details a little later.Let's unpack the sources.
We look for the directory where the sources were unpacked and go to it by running the command
cd
.Finally, we configure, compile and install the project.
As you can see, there are no longer any references to “hello” in the build script. The script still relies on several conventions, but this version is certainly more universal.
Now let's rewrite hello.nix
:
let
pkgs = import <nixpkgs> { };
in
derivation {
name = "hello";
builder = "${pkgs.bash}/bin/bash";
args = [ ./builder.sh ];
buildInputs = with pkgs; [
gnutar
gzip
gnumake
gcc
coreutils
gawk
gnused
gnugrep
binutils.bintools
];
src = ./hello-2.12.1.tar.gz;
system = builtins.currentSystem;
}```
Тут всё ясно, за исключением, может быть, `buildInputs`. Но и в `buildInputs` нет никакой чёрной магии.
Nix умеет конвертировать списки в строку. Сначала он конвертирует в строки каждый отдельный элемент, а затем склеивает их, разделяя пробелом:
```text
nix-repl> builtins.toString 123
"123"
nix-repl> builtins.toString [ 123 456 ]
"123 456"
Let us remember that derivations can also be converted to a string, therefore:
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> builtins.toString gnugrep
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14"
nix-repl> builtins.toString [ gnugrep gnused ]
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14 /nix/store/krgdc4sknzpw8iyk9p20lhqfd52kjmg0-gnused-4.2.2"
It's that simple! Variable buildInputs
will ultimately contain the paths we need, separated by a space. There's nothing better to use in a loop for
interpreter bash
.
Convenient version of the derivation function
We managed to write a script that can be used for different projects autotools
.
But in expression hello.nix
we identify all the programs that may be required, including those that are not needed to build a particular project.
We can write a function that is the same as derivation
takes a set of attributes, and merges it with another set of attributes common to all projects.
autotools.nix
:
pkgs: attrs:
let
defaultAttrs = {
builder = "${pkgs.bash}/bin/bash";
args = [ ./builder.sh ];
baseInputs = with pkgs; [
gnutar
gzip
gnumake
gcc
coreutils
gawk
gnused
gnugrep
binutils.bintools
];
buildInputs = [ ];
system = builtins.currentSystem;
};
in
derivation (defaultAttrs // attrs)
To understand how this code works, let's remember something about Nix features. Entire Nix expression from file autotools.nix
turns into a function.
This function takes a parameter pkgs
and returns a function that takes a parameter attrs
.
There's nothing complicated going on inside the function, but when we first look at it we may have to spend some time understanding how it works.
First, add a magic set of attributes to the visibility area
pkgs
.Using the expression
let
define an auxiliary variabledefaultAttrs
where we add several attributes needed for derivation.At the end we create and call
derivation
passing a strange expression as a parameter (defaultAttrs // attrs
).
Operator // — accepts two sets as input. The result is their unification. In case of attribute name conflict, the value from the right set is used.
So we use defaultAttrs
as a basis, and add (override) attributes from attrs
.
A couple of examples will clarify the operator's work:
nix-repl> { a = "b"; } // { c = "d"; }
{ a = "b"; c = "d"; }
nix-repl> { a = "b"; } // { a = "c"; }
{ a = "c"; }
Exercise: End the new script builder.sh
adding $baseInputs
into a cycle for
together with $buildInputs
.
Result operator // we pass to the function derivation
. Attribute buildInputs
empty, so it will have exactly the value specified in the set attrs
.
Let's rewrite hello.nix
:
let
pkgs = import <nixpkgs> { };
mkDerivation = import ./autotools.nix pkgs;
in
mkDerivation {
name = "hello";
src = ./hello-2.12.1.tar.gz;
}
The final! We have received the simplest description of the package! A few comments to help you understand the Nix language better.
We put in a variable
pkgs
import, which in previous expressions was placed in the “with” statement. This is a common practice and should not be taken for granted.Variable
mkDerivation
is a perfect example of partial application. You can look at it as '(import ./autotools.nix) pkgs'.First we import the expression, then apply it to the parameter
pkgs
(in functional languages, calling a function with a parameter is often called applying a function to a parameter – *translator's note).This gives us a function that accepts a set of attributes
attrs
.We create a derivation by specifying only attributes
name
Andsrc
. If the project needs other dependencies inPATH
they can be added tobuildInputs
but in the example withhello.nix
we didn't need it.
Please note that we do not use any other libraries. We may need C compiler flags to look for include files of other libraries.
Also, we may need linker flags to look for static library ones.
Conclusion
Nix gives us the basic tools for creating derivations, preparing the build environment, and storing the result in Nix storage.
In this pill we wrote a universal project build script autotools
and function mkDerivation
. The latter combines the main components used in projects autotools
with default settings, and saves us from duplicating code in different projects.
We learned how to extend the Nix system by writing and merging new derivations.
Analogy: In C, you create objects that are on the heap and then create new objects based on them. Pointers are used to refer to other objects.
In Nix, you create derivations that are in the Nix repository, and then create new derivations from them. Output paths are used to link to other derivations.
In the next pill
…we'll talk about runtime dependencies. Is the GNU hello world package standalone? What are its runtime dependencies? So far we have defined the dependencies for the assembly by using other derivations in the “hello” derivation.