New restrictions on Docker Hub usage and how GitLab reacted to their introduction

It is no longer news to anyone that starting from November 2, 2020 Docker Hub introduced restrictions on downloading images: for anonymous users it will be equal to one hundred in six hours, and for authorized users it will depend on subscription level…

At GitLab, we followed these changes very closely, as they affected almost all of our users and customers. Today I would like to talk about what changes we made and how we adapted to the new rules.

What happened?

The public registry of Docker Hub containers is very widely and often used by the DevOps community for a variety of purposes: launching CI / CD tasks, standardizing processes, rolling out container applications both in the sandbox and in production. As soon as we learned about the introduction of new restrictions on the number of requests, we began to analyze the new rules to understand how they will affect the experience of our users, and how we can solve possible problems.

According to the new rules, after 100 requests in 6 hours from one client IP address, each new one docker pull will return an error 429 - too many requests, which will undoubtedly lead to broken CI / CD pipelines, the inability to roll out applications and a whole bunch of errors in your Kubernetes clusters. It is quite understandable that this limit can be reached very quickly, especially if all tasks are performed from the same GitLab Runner agent, or if a team of several engineers is working from the same public address.

Dependency Proxy Outsourcing

how noted by my colleague Tim Rizzi “We should urgently tell everyone about Dependency Proxy“, which was originally created for proxying and caching Docker Hub images. This functionality has existed in GitLab for a long time (starting with version 11.11), but until now it was only available for Enterprise Premium users. The product team was faced with a quite reasonable question: “Should we move Dependency Proxy to an open source version of the product, thus helping the wide user community to minimize problems due to new Docker Hub restrictions?“

Without going into details, the answer to this question was YES… When deciding what level of subscription a particular functionality should fall into, the GitLab product team is always guided by the question “Who is the target user?“According to this principle, the features that are most often requested by an individual contributor or developer fall into the Core or Open Source version of the product. Downloading images from the Docker Hub fits this description. Moreover, outsourcing the Dependency Proxy will help a large number of developers improve their reliability and performance. CI / CD pipelines.

As a result, starting from version 13.6 Released on November 22, 2020, image proxying and caching in GitLab is completely free for all our users!

What’s next?

Unfortunately, perfect solutions don’t appear instantly. We know like no one else that our products have room to grow, so we are actively working on improving the functionality of Dependency Proxy and in the next releases we plan to add new features / solve existing problems:

• 13.7 (Dec 22, 2020)

  • gitlab- # 11582 will make it possible to use Dependency Proxy for private groups in GitLab (currently only works for public projects)

  • gitlab- # 241639 will allow using the cached image even if Docker Hub is not available. Today this is impossible, since even if there is a cached image, its manifest is always downloaded from the final register

  • gitlab- # 21619 will add a new parameter pull_policy in the YAML description of CI / CD tasks, allowing developers to specify the container download policy themselves (always, if-not-present, never) instead of relying on GitLab Runner settings

  • gitlab-runner- # 26558 will allow configuring GitLab Runner with a set of policies for downloading images (always, if-not-present)

Monitoring restrictions

The solutions indicated above, as well as in the blog post of my colleague Steve Azzopardi, will help to simplify the work with the new restrictions, but they will not eliminate them 100%. Therefore, we have also developed a set of tips and tools that aim to help the wider community adapt to the new limits by monitoring them.

How can I check the current limit value?

Docker Documentation recommends using HTTP request to check the current value of request limits in Docker Hub.

Let’s start by defining the image for which we want to get the current limit and get the authorization token:

$ IMAGE="ratelimitpreview/test"
$ TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" | jq -r .token)

$ echo $TOKEN

The next step is to simulate the request docker pull… Instead of using the method GET send HEAD request (it is not counted when calculating limits). The response to this request contains parameters RateLimit-Limit and RateLimit-Remaining…

$ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/$IMAGE/manifests/latest

In the example below, the number of requests is limited 2500, of which 2495 still available. 21600 defines a six hour period (in seconds)

RateLimit-Limit: 2500;w=21600
RateLimit-Remaining: 2495;w=21600

Limit check automation

Michael Friedrich, one of the GitLab evangelists, shared ready-made solution in Python to check constraints. The project includes detailed installation and use documentation

$ python check_docker_hub_limit.py --help

usage: check_docker_hub_limit.py [-h] [-w WARNING] [-c CRITICAL] [-v] [-t TIMEOUT]

Version: 2.0.0

optional arguments:
  -h, --help            show this help message and exit
  -w WARNING, --warning WARNING
                        warning threshold for remaining
  -c CRITICAL, --critical CRITICAL
                        critical threshold for remaining
  -v, --verbose         increase output verbosity
  -t TIMEOUT, --timeout TIMEOUT
                        Timeout in seconds (default 10s)

The script returns the following exit codes depending on the specified parameters

• 0– OK

• 1– WARNING

• 2– CRITICAL

$ python3 check_docker_hub_limit.py
OK - Docker Hub: Limit is 5000 remaining 4997|'limit'=5000 'remaining'=4997

$ echo $?
0

$ python3 check_docker_hub_limit.py -w 10000 -c 3000
WARNING - Docker Hub: Limit is 5000 remaining 4999|'limit'=5000 'remaining'=4999

$ echo $?
1

$ python3 check_docker_hub_limit.py -w 10000 -c 5000
CRITICAL - Docker Hub: Limit is 5000 remaining 4998|'limit'=5000 'remaining'=4998

$ echo $?
2

Exporter Prometheus

In addition, we have prepared and released exporter for Prometheus, which will allow you to export metrics of restrictions and implement their full monitoring.

The project repository includes a demo container that includes an exporter, Prometheus, Grafana, and docker-compose instructions for rolling it out

$ cd example/docker-compose

$ docker-compose up -d

Go to address http://localhost:3000 to access the Grafana dashboard

I hope you find our experience and advice useful!