How I Saved $ 5,000 With a $ 5 Droplet

On November 20, 2020, Docker started

limit the number of gears

queries against his popular Docker Hub registry. This change affected all users, anonymous and free. After the change was implemented, the process of work of developers around the world slowed down dramatically. To solve the problem, many simply had to log in (for logged in accounts, the transfer restriction level is higher), but others had to pay for

service account

… Under high loads, service accounts can be expensive.

There is nothing wrong with solving problems with money. Perhaps, in your situation, this will even be the right decision. For others, however, the reality may not be so pleasant.

While working at Earthly, I encountered these transfer restrictions. To create a containerized assembly, you have to pull up a bunch of containers, and do so often. In a couple of hours, we ran our test suite 2-3 times, which led to the activation of the transmission limitation … and with each new test the situation got worse. Perhaps this is familiar to you?

Therefore, instead of paying for a service account, I set up Pull-Through Cachethat serves as a broker for all requests to Docker Hub. After its creation, all failures caused by transmission restrictions disappeared. Plus, it’s cheaper than paying for a service account! To save you some time, I have documented what I did.

What is Pull-Through Cache?

Before going into the details of our system in Earthly, let’s understand what it is and what

is not

pull-through cache.

From a client’s perspective, the pull-through cache is just a regular registry. Well, almost. For example, images cannot be pushed into it… But you can pull from it. When an image (or its metadata) is first requested from the cache, it is retrieved transparently from the repository upstream. Subsequent requests will use the cached version.

This scheme works especially well when the retrieved images do not change frequently. Although to ensure repeatability Usually it is recommended to use certain tags, applying this practice when working with the cache will also lead to a decrease in the number of round-trip cycles. Do we need another reason not to use :latest?

There are, of course, additional methods and tools that can be used to cache images. In particular, there is a convenient rpardini/docker-registry-proxyusing proxy nginx and caching requests; the working principle is similar to MITM Proxy… Other registries have caching modes like Artifactory and Gcp

In this article, I will take a look at the standard Docker registry located in (distribution/distribution) because it is simple and well documented. If you want to get straight to the point, then all our work is posted on GitHub

Getting the register

The canonical registry is


… You can get it simply by doing

docker pull registry:2

… However, there are some subtleties outlined in the HTTPS section below.

Configuring the Registry

In the process of parsing the options that you may need to configure for your pull-through cache, I will share snippets from the finished

example config file

… If some of the information does not suit you completely, then there is a fairly comprehensive

registry configuration documentation

Proxy mode

To use the Distribution registry as a pull-through cache, you need to tell it to act as a cache, which is not surprising. This can be done using the top-level key




username: my_dockerhub_user

password: my_dockerhub_password

It is worth noting that




in this section


are the credentials that you will use to login to the cache; this is the credential that the cache will use to pull upstream in


Default cache not will authenticate users. This means that without setting up authentication for the mirror (see below), any private repos available in my_dockerhub_user, in fact, will become public… Make sure you are doing everything right to avoid leaking sensitive information!

Cache Authentication

To prevent third-party people from pulling your private images or wasting precious traffic, the mirror must be protected by some kind of authentication. It can be implemented with a top-level key





realm: basic-realm

path: /auth/htpasswd

Since I am working with a relatively small team, it is sufficient to use a static username / password in the standard file


… If you need help generating the file


then read

Apache documentation

Not use authentication type sillysince it is for development only. It should be clear from the title (well, hopefully).

System token should allow you to connect it to your company’s existing authentication framework. Usually it is present in large organizations and is more suitable for such an environment.


The infrastructure of our company is located in the domain


… The whole




… This means that I can’t just leave our cache on HTTP. Also, in the modern era

Let’s Encrypt

it’s all pretty easy to set up, isn’t it?

Something like that. At the time of this writing, there is default image problem and a similar problem upstream for the registry program itself… Since Let’s Encrypt has disabled the respective APIs, and the default image is very old, you will have to use one of three solutions:

Compile the registry

I used this solution

… You can just wrap an existing image


into another Dockerfile using

FROM registry:2

(or use Earthfile) and replace the binary with another one built from source


After that, it will be enough to configure it as follows using the key http.letsencrypt:



cachefile: /certs/cachefile


hosts: []

Thanks to this, Let’s Encrypt will issue a certificate for domains in the key


and will automatically keep it up to date.

Uploading certificates manually

You can upload your own certificates using the key


… However, you cannot use old or broken Let’s Encrypt libraries in the default image. If necessary, you can customize

certbot to process them manually

Reverse proxy

This was our second option after compiling our own version. Using something like


with built-in support for Let’s Encrypt is standard practice and is mentioned in the above problem descriptions as a possible solution.


Since the registry is just a cache, and it is not critical, I decided to save the cache simply to local disk space on the VPS in which I deployed the system. In addition, metadata for images is also not so critical, so I decided to place it in memory. You can read more about these options in the keys






blobdescriptor: inmemory


rootdirectory: /var/lib/registry

There are other storage drivers as well. In the root key


the available drivers are described in detail.

Other minor improvements



added tons of customization options … so why not add some more? Here are some other improvements I’ve added:

Checking the state of the storage (in case the cache starts to run out of space or other strange things happen with the VPC):



enabled: true

interval: 10s

threshold: 3

Configuring the port that the registry will listen to. The commented out part configures the debug port; if you skip it, the debug port will be disabled. It is useful as an indication that cache hits are occurring.

You can get this information through the debug port (if enabled) by going to /debug/vars


addr: :5000

# Uncomment to add debug to the mirror.

# debug:

# addr:


X-Content-Type-Options: [nosniff]

Enable logging at info level, which makes debugging and testing easier:


level: info


service: registry

Hosting your cache

After all this, you can start the registry locally and even connect to it successfully! The command for this might look like this:

docker run -d -p 443:5000 --restart=always --name=through-cache -v ./auth:/auth -v ./certs:/certs -v ./config.yaml:/etc/docker/registry/config.yml registry:2

But a cache that is only on your machine is not as useful as a cache that is shared or between Docker Hub and your CI. Fortunately, it’s not that hard to get it up and running on a VPS.

Choosing a VPS for your cache is easy – you should probably just use the one your company uses. However, I decided to choose Digital Ocean because of its reasonable prices, easy setup and generous limits.

Most of the time, our cache performed reasonably well on a single $ 5 droplet, however, the additional load due to CI made us go up one level. If your needs are higher than the capabilities of one node, then there is always the possibility run multiple instances with a load balancer or use CDN

While it’s entirely possible to create a VPS instance manually, let’s take it one step further and fully automate this process with Terraform and clout-init… If you want to get down to business right away, then check out the full example

Let’s start by creating a VPS instance. Unlike the registry examples shown above, I’ll keep the variables I used in our Terraform module.

resource "digitalocean_droplet" "docker_cache" {

image = "ubuntu-20-04-x64"

name = "docker-cache-${var.repository_to_mirror}"

region = "sfo3"

size = "s-1cpu-1gb"

monitoring = true

ssh_keys = [var.ssh_key.fingerprint]

user_data =


This is a pretty standard, simple configuration for running a droplet. But how to start our cache on a fresh droplet? With a key


cloud-init. We use Terraform to create a template with the same variables we specified in our module and put the result in this HCL section. Here’s a stripped down version of our cloud-init template:


package_update: true

package_upgrade: true

package_reboot_if_required: true


- docker


- name: griswoldthecat

lock_passwd: true

shell: /bin/bash


- ${init_ssh_public_key}

groups: docker



- apt-transport-https

- ca-certificates

- curl

- gnupg-agent

- software-properties-common

- unattended-upgrades


- path: /auth/htpasswd

owner: root:root

permissions: 0644

content: ${init_htpasswd}

- path: /config.yaml

owner: root:root

permissions: 0644

content: |

# A parameterized version of our registry config...


- curl -fsSL | apt-key add -

- add-apt-repository "deb [arch=amd64] $(lsb_release -cs) stable"

- apt-get update -y

- apt-get install -y docker-ce docker-ce-cli

- systemctl start docker

- systemctl enable docker

- docker run -d -p 443:5000 --restart=always --name=through-cache -v /auth:/auth -v /certs:/certs -v /config.yaml:/etc/docker/registry/config.yml registry:2

This cloud-init template sets up Docker, configures our registry, and starts the container.

We use our cache

Using a mirror is very easy. Enough

add mirror to your list

, and if authentication is configured, execute

docker login

with the corresponding identification data (the above data




should start automatically and already use the mirror.

If your mirror is not available, docker will go upstream without additional adjustment.


Setting up and maintaining our own pull-through cache saved us headaches and saved money. Our CI is no longer subject to transfer restrictions. This means our assemblies are more consistent, reliable, and faster.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *