Automatic Archiving of Artifacts

When programming microcontrollers on electronic boards, the output always contains quite a lot of artifacts. This is firmware, documentation, debugging info.

If possible, all this stuff should be somehow linked and locked into one archive, so that you can always associate .hex with the .map and .elf you need. Also, one archive is very convenient when transporting software.

The logical step would be to archive all these files in a *.tar archive so that all this does not scatter across the hard drive.

Terminology

Artifacts In the context of microcontroller programming, artifacts are what crystallize at the output of the tool chain. In the most general case, this is

Formulation of the problem

It is necessary to implement automatic archiving of firmware artifacts. Write a make script for packing artifacts. Embed this action in the general build script of any project from the repository.

Implementation

For this you can use the classic utility tar. This is a console utility for archiving files. Here is its help.

tar -h
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
  -c Create  -r Add/Replace  -t List  -u Update  -x Extract
Common Options:
  -b #  Use # 512-byte records per I/O block
  -f <filename>  Location of archive (default \\.\tape0)
  -v    Verbose
  -w    Interactive
Create: tar -c [options] [<file> | <dir> | @<archive> | -C <dir> ]
  <file>, <dir>  add these items to archive
  -z, -j, -J, --lzma  Compress archive with gzip/bzip2/xz/lzma
  --format {ustar|pax|cpio|shar}  Select archive format
  --exclude <pattern>  Skip files that match pattern
  -C <dir>  Change to <dir> before processing remaining files
  @<archive>  Add entries from <archive> to output
List: tar -t [options] [<patterns>]
  <patterns>  If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
  <patterns>  If specified, extract only entries that match
  -k    Keep (don't overwrite) existing files
  -m    Don't restore modification times
  -O    Write entries to stdout, don't restore to disk
  -p    Restore permissions (including ACLs, owner, file flags)
bsdtar 3.5.2 - libarchive 3.5.2 zlib/1.2.5.f-ipp

We only need these options of the tar utility

We need to organize a file processing pipeline like this

First, let's experiment in Windows cmd. This script can basically solve the problem.

echo off
cls

set project_name=boardname_appname_m
set project_dir=%cd%
set artefact_dir=%project_dir%\build
echo project_dir=%project_dir%

set FILES_TO_PACK=%artefact_dir%\%project_name%.map
set FILES_TO_PACK=%artefact_dir%\%project_name%.bin %FILES_TO_PACK%
set FILES_TO_PACK=%artefact_dir%\%project_name%.elf %FILES_TO_PACK%
set FILES_TO_PACK=%artefact_dir%\%project_name%.hex %FILES_TO_PACK%
set FILES_TO_PACK=%artefact_dir%\%project_name%.jpeg %FILES_TO_PACK%
set FILES_TO_PACK=%artefact_dir%\%project_name%.pdf %FILES_TO_PACK%
set FILES_TO_PACK=%artefact_dir%\%project_name%.svg %FILES_TO_PACK%
set FILES_TO_PACK=%artefact_dir%\%project_name%_dep.gv %FILES_TO_PACK%

echo artefact_dir=%artefact_dir%
echo FILES_TO_PACK=%FILES_TO_PACK%

tar.exe -v -f %artefact_dir%\%project_name%.tar -c %FILES_TO_PACK%  --

However, my build system is Make, so I need to write a common script for all builds to pack files. This is what the make script might look like archive_artifacts.mk for packing binaries into an archive

$(info ArchiveArtifactsScript)

FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).pdf
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).svg
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).jpeg
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET)_dep.gv
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).map
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).elf
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).hex
FILES_TO_PACK += $(BUILD_DIR)/$(TARGET).bin

ARCHIVE_FILE :=$(BUILD_DIR)\$(TARGET).tar

.PHONY: archive_artifacts
archive_artifacts: generate_dep auto_version_target
	$(info BUILD_DIR=$(BUILD_DIR))
	$(info FILES_TO_PACK=$(FILES_TO_PACK))
	$(info ARCHIVE_FILE=$(ARCHIVE_FILE))
	$(info Archive Artifacts...)
	tar.exe -v -f $(ARCHIVE_FILE) -c $(FILES_TO_PACK) --

After running this script, a *.tar archive with binaries appears in the build folder (BUILD_DIR) of the project root.

Results

Managed to add automatic archiving into the overall project build process.

Links