How I Saved $ 5,000 With a $ 5 Droplet
On November 20, 2020, Docker started
queries against his popular Docker Hub registry. This change affected all users, anonymous and free. After the change was implemented, the process of work of developers around the world slowed down dramatically. To solve the problem, many simply had to log in (for logged in accounts, the transfer restriction level is higher), but others had to pay for
… Under high loads, service accounts can be expensive.
There is nothing wrong with solving problems with money. Perhaps, in your situation, this will even be the right decision. For others, however, the reality may not be so pleasant.
While working at Earthly, I encountered these transfer restrictions. To create a containerized assembly, you have to pull up a bunch of containers, and do so often. In a couple of hours, we ran our test suite 2-3 times, which led to the activation of the transmission limitation … and with each new test the situation got worse. Perhaps this is familiar to you?
Therefore, instead of paying for a service account, I set up Pull-Through Cachethat serves as a broker for all requests to Docker Hub. After its creation, all failures caused by transmission restrictions disappeared. Plus, it’s cheaper than paying for a service account! To save you some time, I have documented what I did.
What is Pull-Through Cache?
Before going into the details of our system in Earthly, let’s understand what it is and what
is not
pull-through cache.
From a client’s perspective, the pull-through cache is just a regular registry. Well, almost. For example, images cannot be pushed into it… But you can pull from it. When an image (or its metadata) is first requested from the cache, it is retrieved transparently from the repository upstream. Subsequent requests will use the cached version.
This scheme works especially well when the retrieved images do not change frequently. Although to ensure repeatability Usually it is recommended to use certain tags, applying this practice when working with the cache will also lead to a decrease in the number of round-trip cycles. Do we need another reason not to use :latest?
There are, of course, additional methods and tools that can be used to cache images. In particular, there is a convenient rpardini/docker-registry-proxyusing proxy nginx and caching requests; the working principle is similar to MITM Proxy… Other registries have caching modes like Artifactory and Gcp…
In this article, I will take a look at the standard Docker registry located in (distribution/distribution) because it is simple and well documented. If you want to get straight to the point, then all our work is posted on GitHub…
Getting the register
The canonical registry is
… You can get it simply by doing
docker pull registry:2
… However, there are some subtleties outlined in the HTTPS section below.
Configuring the Registry
In the process of parsing the options that you may need to configure for your pull-through cache, I will share snippets from the finished
… If some of the information does not suit you completely, then there is a fairly comprehensive
registry configuration documentation
…
Proxy mode
To use the Distribution registry as a pull-through cache, you need to tell it to act as a cache, which is not surprising. This can be done using the top-level key
…
proxy:
remoteurl: https://registry-1.docker.io
username: my_dockerhub_user
password: my_dockerhub_password
It is worth noting that
username
and
password
in this section
not
are the credentials that you will use to login to the cache; this is the credential that the cache will use to pull upstream in
remoteurl
…
Default cache not will authenticate users. This means that without setting up authentication for the mirror (see below), any private repos available in my_dockerhub_user, in fact, will become public… Make sure you are doing everything right to avoid leaking sensitive information!
Cache Authentication
To prevent third-party people from pulling your private images or wasting precious traffic, the mirror must be protected by some kind of authentication. It can be implemented with a top-level key
:
auth:
htpasswd:
realm: basic-realm
path: /auth/htpasswd
Since I am working with a relatively small team, it is sufficient to use a static username / password in the standard file
htpasswd
… If you need help generating the file
htpasswd
then read
…
Not use authentication type sillysince it is for development only. It should be clear from the title (well, hopefully).
System token should allow you to connect it to your company’s existing authentication framework. Usually it is present in large organizations and is more suitable for such an environment.
HTTPS
The infrastructure of our company is located in the domain
.dev
… The whole
.dev
uses
… This means that I can’t just leave our cache on HTTP. Also, in the modern era
it’s all pretty easy to set up, isn’t it?
Something like that. At the time of this writing, there is default image problem and a similar problem upstream for the registry program itself… Since Let’s Encrypt has disabled the respective APIs, and the default image is very old, you will have to use one of three solutions:
Compile the registry
… You can just wrap an existing image
registry
into another Dockerfile using
FROM registry:2
(or use Earthfile) and replace the binary with another one built from source
distribution/registry
…
After that, it will be enough to configure it as follows using the key http.letsencrypt:
tls:
letsencrypt:
cachefile: /certs/cachefile
email: me@my_domain.dev
hosts: [mirror.my_domain.dev]
Thanks to this, Let’s Encrypt will issue a certificate for domains in the key
hosts
and will automatically keep it up to date.
Uploading certificates manually
You can upload your own certificates using the key
… However, you cannot use old or broken Let’s Encrypt libraries in the default image. If necessary, you can customize
certbot to process them manually
…
Reverse proxy
This was our second option after compiling our own version. Using something like
with built-in support for Let’s Encrypt is standard practice and is mentioned in the above problem descriptions as a possible solution.
Storage
Since the registry is just a cache, and it is not critical, I decided to save the cache simply to local disk space on the VPS in which I deployed the system. In addition, metadata for images is also not so critical, so I decided to place it in memory. You can read more about these options in the keys
and
…
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
There are other storage drivers as well. In the root key
the available drivers are described in detail.
Other minor improvements
I AM
already
added tons of customization options … so why not add some more? Here are some other improvements I’ve added:
Checking the state of the storage (in case the cache starts to run out of space or other strange things happen with the VPC):
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
Configuring the port that the registry will listen to. The commented out part configures the debug port; if you skip it, the debug port will be disabled. It is useful as an indication that cache hits are occurring.
You can get this information through the debug port (if enabled) by going to /debug/vars…
http:
addr: :5000
# Uncomment to add debug to the mirror.
# debug:
# addr: 0.0.0.0:6000
headers:
X-Content-Type-Options: [nosniff]
Enable logging at info level, which makes debugging and testing easier:
log:
level: info
fields:
service: registry
Hosting your cache
After all this, you can start the registry locally and even connect to it successfully! The command for this might look like this:
docker run -d -p 443:5000 --restart=always --name=through-cache -v ./auth:/auth -v ./certs:/certs -v ./config.yaml:/etc/docker/registry/config.yml registry:2
But a cache that is only on your machine is not as useful as a cache that is shared or between Docker Hub and your CI. Fortunately, it’s not that hard to get it up and running on a VPS.
Choosing a VPS for your cache is easy – you should probably just use the one your company uses. However, I decided to choose Digital Ocean because of its reasonable prices, easy setup and generous limits.
Most of the time, our cache performed reasonably well on a single $ 5 droplet, however, the additional load due to CI made us go up one level. If your needs are higher than the capabilities of one node, then there is always the possibility run multiple instances with a load balancer or use CDN…
While it’s entirely possible to create a VPS instance manually, let’s take it one step further and fully automate this process with Terraform and clout-init… If you want to get down to business right away, then check out the full example…
Let’s start by creating a VPS instance. Unlike the registry examples shown above, I’ll keep the variables I used in our Terraform module.
resource "digitalocean_droplet" "docker_cache" {
image = "ubuntu-20-04-x64"
name = "docker-cache-${var.repository_to_mirror}"
region = "sfo3"
size = "s-1cpu-1gb"
monitoring = true
ssh_keys = [var.ssh_key.fingerprint]
user_data = data.template_file.cloud-init.rendered
}
This is a pretty standard, simple configuration for running a droplet. But how to start our cache on a fresh droplet? With a key
user_data
cloud-init. We use Terraform to create a template with the same variables we specified in our module and put the result in this HCL section. Here’s a stripped down version of our cloud-init template:
#cloud-configpackage_update: true
package_upgrade: true
package_reboot_if_required: truegroups:
- dockerusers:
- name: griswoldthecat
lock_passwd: true
shell: /bin/bash
ssh_authorized_keys:
- ${init_ssh_public_key}
groups: docker
sudo: ALL=(ALL) NOPASSWD:ALLpackages:
- apt-transport-https
- ca-certificates
- curl
- gnupg-agent
- software-properties-common
- unattended-upgradeswrite_files:
- path: /auth/htpasswd
owner: root:root
permissions: 0644
content: ${init_htpasswd}- path: /config.yaml
owner: root:root
permissions: 0644
content: |
# A parameterized version of our registry config...runcmd:
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- apt-get update -y
- apt-get install -y docker-ce docker-ce-cli containerd.io
- systemctl start docker
- systemctl enable docker
- docker run -d -p 443:5000 --restart=always --name=through-cache -v /auth:/auth -v /certs:/certs -v /config.yaml:/etc/docker/registry/config.yml registry:2
This cloud-init template sets up Docker, configures our registry, and starts the container.
We use our cache
Using a mirror is very easy. Enough
, and if authentication is configured, execute
docker login
with the corresponding identification data (the above data
htpasswd
).
docker
should start automatically and already use the mirror.
If your mirror is not available, docker will go upstream without additional adjustment.
Conclusion
Setting up and maintaining our own pull-through cache saved us headaches and saved money. Our CI is no longer subject to transfer restrictions. This means our assemblies are more consistent, reliable, and faster.
