Another tool for updating dependencies

In brief

This article is a story about a small and useful script for updating go-mod-bump dependencies. Here you may be interested in the problem, solution or history of writing the script. If you want to touch the script with your hands, you can find it here in the public repository.

Disclaimer

• The script has not been used in CI and may not be suitable for such use.

• The story is described post factum, after a significant period of time, and therefore may contain obvious inconsistencies or distortions.

• The script modifies files go.mod And go.sum in the git working directory, and also changes the list of staged files; you can lose your changes if you use it thoughtlessly.

• This tool is not a competitor to popular solutions for automating dependency updates and is not intended to take their place.

Problem

I work with many repositories both at work and in personal projects, and they regularly need to update dependencies. They are stored in different project hosting services, in private and public repositories. And, unfortunately, not all of these projects have automation set up for updating dependencies, and for some of them there are no suitable ready-made solutions.

I also need to somehow find out about updating a dependency, monitor hundreds of repositories – this is an overwhelming task for me, and I wouldn’t want to do it anyway.

Restrictions

• Updating dependencies is done with one command.

• Each module is updated in a separate commit.

• The commit messages are similar to those generated by popular dependency update solutions.

Solution

Automate a set of routine operations so that I have to do them minimally or not at all.
Essentially, there are several steps that need to be automated:

1. Find out if a new version of the module is available.

2. go get github.com/xorcare/pointer@v1.2.2 – update the module.

3. go mod tidy – synchronize dependencies in go.mod and go.sum, and at the same time check if any problems have appeared.

4. go build ./... – make minimal checks, in this case check that all source code compiles.

5. git commit -m 'Bump github.com/xorcare/pointer from v1.2.1 to v1.2.2' – create a commit with a clear name.

6. Repeat these steps for all modules on which the module we are servicing depends.

History of the solution

Okay, let's dive into history a little. The task is to make a small script for yourself – a dependency update assistant.

Step one

We need to know somehow that new versions of dependencies have appeared. Luckily, the Go toolkit already has a suitable command go get -u allit copes well with updating everything and everyone. Based on the presence of this command, we can draw 2 conclusions: Go can check for new versions of packages, and when using this command, we cannot control the process in any way.

By the time I created the script, I was already familiar with the command go listand instead go get -u all you can dig into it. Having dug a little into go help listtwo important flags can be found:

• -mwhich allows you to get a list of modules using the command go list -m all.

• -uwhich is the same as in go get -uwill allow you to check for new versions.

And now we have a more manageable team. go list -u -m allwhich gives us a list of modules, the version currently used in the project, and the newest version if it differs from the installed one. Example of command output:

github.com/xorcare/tornado
golang.org/x/mod v0.21.0
golang.org/x/net v0.22.0 [v0.29.0]
golang.org/x/sys v0.21.0 [v0.25.0]

From it we see that for the module golang.org/x/mod current version – v0.21.0and there is no newer version, but, for example, for golang.org/x/net current version – v0.21.0and there is a new version [v0.29.0].

To summarize: a way was found to get a list of modules and at the same time find out about the availability of new versions.

Step two

There is a list of modules and versions, all that remains is to iterate through it. In bash this is not a problem, but there are nuances.

• Each line looks like this: golang.org/x/net v0.22.0 [v0.29.0]; we need to get 3 values ​​from it, and I don’t really like parsing strings.

• Only direct modules are needed.

• Only those modules are needed for which there is a new version.

In the previous step in go help list you can apply the parameter -fwhich sets the output format. I used it and made something like csv with a separator @ (at the time of creating the script I thought that this symbol would never appear in the string, but now I'm not sure).

go list -u -m -f '{{.Path}}@{{.Version}}@{{if .Update}}{{.Update.Version}}{{end}}{{if not .Update}}<SKIP>{{end}}{{if .Indirect}}<SKIP>{{end}}' all | grep -v '<SKIP>'

We get a list of modules that are definitely imported by our module and definitely need to be updated:

golang.org/x/net@v0.22.0@v0.29.0

This format is easy to parse using cut:

module=$(echo "$mdl" | cut -f1 -d@)  
current_version=$(echo "$mdl" | cut -f2 -d@)  
latest_version=$(echo "$mdl" | cut -f3 -d@)  

Step three

We got a list of dependencies that we successfully updated. At that point, the script looked something like this:

#!/usr/bin/env bash  
  
set -e  
  
MODULES_FOR_UPDATE=$(go list -m -u -f "{{.Path}}@{{.Version}}@{{if .Update}}{{.Update.Version}}{{end}}{{if not .Update}}<SKIP>{{end}}" all | grep -v '<SKIP>')  
  
for mdl in $MODULES_FOR_UPDATE; do  
    module=$(echo "$mdl" | cut -f1 -d@)  
    current_version=$(echo "$mdl" | cut -f2 -d@)  
    latest_version=$(echo "$mdl" | cut -f3 -d@)  
  
    go get "$module@${latest_version}"  
    go mod tidy  
    go build ./...  
  
    git reset HEAD -- . >/dev/null  
    git add go.mod go.sum >/dev/null  
    git cm -a -m "${PREFIX}Bump ${module} from ${current_version} to ${latest_version}" >/dev/null  
  
    echoerr "go-mod-bump: upgraded ${module} ${current_version} => [${latest_version}]"  
done

At this point we have a working tool that we can use. But there are still some nuances.

Step four

On projects with a large number of dependencies, the script works very slowly. This happens because go list -u checks for updates for all modules in general, regardless of whether they are direct or indirect.

To speed up the script, you can divide the script's work into 2 stages.

Step one: we collect a list of only direct modules; similar actions were done in the previous steps, so the idea of ​​the following command is born quite easily:

go list -m -f '{{.Path}}{{if .Indirect}}<SKIP>{{end}}' all | grep -v '<SKIP>'

github.com/xorcare/tornado
golang.org/x/net

Step two: storing this list in a variable $DIRECT_MODULESwe can request a check for updates only for direct modules:

go list -u -m -f '{{.Path}}@{{.Version}}@{{if .Update}}{{.Update.Version}}{{end}}{{if not .Update}}<SKIP>{{end}}' $DIRECT_MODULES | grep -v '<SKIP>'

This way we will get a list of modules that need to be updated without having to go through all the modules.

Step five

Already during use, it turns out that modules cannot always be updated, and it is necessary to somehow skip modules whose update leads to errors. This can be implemented by combining the commands responsible for checking and updating dependencies into one function:

function update_module() {  
    go get "$1"  
  
    go mod tidy  
    go build ./...  
}

And then, in case of failure, simply skip the commit, rolling back the changes in go.mod and go.sum:

if ! update_module "${module}@${latest_version}" >/dev/null 2>&1 >/dev/null; then  
    git checkout -f HEAD -- go.mod go.sum  
fi

After that, we proceed to update the next module from the list until the list is complete.

At this stage, the basic mechanics of the script are already similar to published version.

The last step, but not the least important

At first I used the script only myself, and there was no point in decorating it. But later I shared the script in a public repository, having previously refined it.

• Removed the hardcoded one all and gave the developer the ability to choose the list of modules themselves. Implicitly, this gave the ability to update Go in go.mod.

• Added flags -h And -help with usage examples. And also added a flag -pso that you can add a prefix to the commit title if you follow some commit naming conventions.

• For cases when the module update ends with an error, I added the output of a detailed message and commands that can be copied and run for manual diagnosis of the problem.

Conclusion

As a result of the work, I got a convenient script go-mod-bump, which solves the task and does it well. It is relatively cross-platform, as it uses a small number of commands and I tested it on macOS, Debian, and Ubuntu.

Perhaps it would have been worthwhile to make this tool in the spirit of Go and write it in Go. But, at that moment, for simplicity, I chose bash, and I have no inspiration to rewrite it in Go yet.

Afterword

If the problem described is close to you, then you may like to use a script. If your git repository hosts have ready-made tools for automating dependency updates, then use them. These tools will almost certainly solve the problem better, run themselves on schedule and make beautiful merge requests for you.