Log collector on old hardware or meet Vector 32bit

In any large company, and X5 is no exception, there is “old hardware” that works and works well, but the time comes and it is also necessary to start collecting logs and metrics from it. In our case, these are machines with x86_32 Debian 9.5 and 512MB of RAM.

Small Disclaimer

The author of the article is not an expert on the Rust language and accordingly any improvements and specific recommendations from the guru are welcome. This article is not an advertisement for Vector and the author is not affiliated with Timber or its contributors. Everything described below is a pure DIY experiment to show that there are no insurmountable barriers, if you really want to.

So, let’s begin

We will talk about Vector version 0.10.0, although the trendy and hype log collector has only recently been updated to version 0.11.0, perhaps this method will work for it as well.

We at X5 love Vector and wanted to try it out on the x86_32 architecture. Whereas out of the box, it is available for almost any operating system in two architectures x86_64 and ARM7.

We will collect in Docker based on the recommendations, set out here…

The description from the link above is only partially suitable, since we need some kind of i686-unknown-linux-gnu configuration for Rust, as the great Google shows. We will compile the “target” system in docker (x86_32 Debian 9.5) because I realized that compiling 32bit Rust in a 64bit environment is beyond my power.

FYI: Yes, before that I tried i686-unknown-linux-musl and I couldn’t get it to work.

Docker magic

Copy the files from the repository on github:

mkdir -p vector && 
  curl -sSfL --proto '=https' --tlsv1.2 https://api.github.com/repos/timberio/vector/tarball/v0.10.0 | 
  tar xzf - -C vector --strip-components=1

In the vector / scripts / ci-docker-images directory, create the builder-i686-unknown-linux-gnu folder, take the base Dockerfile from vector / scripts / ci-docker-images / builder-x86_64-unknown-linux-gnu and start editing.

After several hours of experimentation, googling and agony of choice, the following came out:

FROM i386/debian:9.5 as builder
# This is formatted "$UID:$GID" by the docker-compose/scripts.
ARG USER

RUN apt-get update && 
    apt-get -y upgrade

RUN apt-get install -y 
        make libssl-dev cmake git 
        build-essential sudo curl
        
RUN curl -L https://cpanmin.us | perl - App::cpanminus

RUN cpanm File::Rename 
 && rename --version

RUN cd /tmp && 
  git clone https://github.com/github/cmark-gfm && 
  cd cmark-gfm && 
  git checkout 0.29.0.gfm.0 && 
  make install INSTALL_PREFIX=/usr && 
  ldconfig && 
  cd .. && 
  rm -rf cmark-gfm && 
  cmark-gfm --version

RUN sudo adduser runner
RUN sudo usermod -aG sudo runner

USER runner
RUN curl https://sh.rustup.rs -sSf | sh -s -- --no-modify-path --default-host i686-unknown-linux-gnu -y
ENV PATH=/home/runner/.cargo/bin:$PATH
RUN echo "export PATH=/home/runner/.cargo/bin:$PATH" >> ~/bashrc
ENV LIBRARY_PATH /usr/local/lib:$LIBRARY_PATH
ENV LD_LIBRARY_PATH /usr/local/lib:$LD_LIBRARY_PATH

RUN rustup update stable
RUN rustup run stable cargo install cargo-deb --target=i686-unknown-linux-gnu --version '^1.24.0'

CMD ["bash"]

I also had to fix vector / Makefile.

By adding and describing build-i686-unknown-linux-gnu in several places and making it the only build option.

A snippet of vector / Makefile for example:

#build-all: build-x86_64-unknown-linux-musl build-armv7-unknown-linux-musleabihf build-aarch64-unknown-linux-musl  ## Build the project in release mode for all supported platforms
build-all: build-i686-unknown-linux-gnu

….

package-archive-i686-unknown-linux-gnu: build-i686-unknown-linux-gnu ## Build the x86_32 archive
$(RUN) package-archive-i686-unknown-linux-gnu

Rust dependencies are configured in vector / Cargo.toml. The profile.release section has been added there:

[profile.release]
opt-level="z"  # Optimize for size.
debug = false
debug-assertions = false
lto = true
codegen-units = 1

This was done for optimization reasons. the first assembled binaries were the size of a cast-iron bridge ~ 60MB, which is not acceptable. The size of the original binaries from the site is about 7MB.

Inspiration was inspired in my opinion by @johnthagen’s excellent description of techniques for reducing the size of Rust binaries here: https://github.com/johnthagen/min-sized-rust

In an effort to save on size in sources sections, transforms are commented out
logplex | splunk_hec and aws_ec2_metadata | lua, respectively, which allowed saving about 10MB in the compiled state without the profile.release section.

# Sources
sources = [
  "sources-docker",
  "sources-file",
  "sources-generator",
  "sources-http",
  "sources-internal_metrics",
  "sources-journald",
  "sources-kafka",
  #"sources-logplex",
  "sources-prometheus",
  "sources-socket",
  #"sources-splunk_hec",
  "sources-statsd",
  "sources-stdin",
  "sources-syslog",
  "sources-tls",
  "sources-vector",
]

# Transforms
transforms = [
  "transforms-add_fields",
  "transforms-add_tags",
  "transforms-ansi_stripper",
  #"transforms-aws_ec2_metadata",
  "transforms-coercer",
  "transforms-concat",
  "transforms-dedupe",
  "transforms-field_filter",
  "transforms-filter",
  "transforms-geoip",
  "transforms-grok_parser",
  "transforms-json_parser",
  "transforms-log_to_metric",
  "transforms-logfmt_parser",
  #"transforms-lua",
  "transforms-merge",
  "transforms-regex_parser",
  "transforms-remove_fields",
  "transforms-remove_tags",
  "transforms-rename_fields",
  "transforms-sampler",
  "transforms-split",
  "transforms-swimlanes",
  "transforms-tag_cardinality_limit",
  "transforms-tokenizer",
  "transforms-reduce",
]

Assembly

We start the whole thing with the command:

PASS_FEATURES=default-cmake ./scripts/docker-run.sh builder-i686-unknown-linux-gnu make build

During the build process, errors like this may fall out:

error: Input/output error (os error 5)
warning: build failed, waiting for other jobs to finish...
    Building [===>                                                    ] 40/537: regex-syntax                                                                                                             
error: build failed
Makefile:156: recipe for target 'build' failed
make: *** [build] Error 101

I do not advise you to get discouraged, we restart make and after an “endless” cargo downloading component, Updating crates.io index and Updating git repository, the build continues from the point where it was interrupted. By the way, if someone knows how to stop this and force cargo and crates to use local cache it would be super)

On a macbook pro i5 8gb, this process lasts about 2 hours, of which the assembly itself is about 50 minutes.

After assembly, the file can be compressed with UPX:

upx --best --lzma target/release/vector

That’s all.

Github repository with patched files and Vector 0.10.0 here

Compiled binary 8.7MB, compressed UPX here