Securing Docker Compose Stacks with CrowdSec

This article shows you how to combine CrowdSec and Docker Compose to secure containerized applications. This will allow us:

  • automatically block compromised IP addresses from accessing our container services;

  • manually add / remove and check ban decisions;

  • track CrowdSec behavior (using cli and dashboards)

This article was produced by the CrowdSec technical team.

Target architecture

The diagram below shows our target architecture:

First, let’s create a Docker Compose file that defines the following settings:

  • reverse proxy server using Nginx;

  • a test application that outputs “hello world” to Apache2;

  • The CrowdSec container, which reads the reverse proxy logs to detect attacks on the HTTP service.

  • a Metabase container that will generate complex dashboards to keep track of what is happening.

We chose the simplest way to collect logs: by splitting volumes between containers. If you are working in a production environment then you are most likely using logging to centralize logs using rsyslog or some other mechanism. So don’t forget to debug the Docker Compose CrowdSec config to read the logs correctly.

File docker-compose.yml as follows:

version: '3'
 
services:
 #the application itself : static html served by apache2.
 #the html can be found in ./app/
 app:
   image: httpd:alpine
   restart: always
   volumes:
     - ./app/:/usr/local/apache2/htdocs/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.2
 
 #the reverse proxy that will serve the application
 #you can see nginx's config in ./reverse-proxy/nginx.conf
 reverse-proxy:
   image: nginx:alpine
   restart: always
   ports:
     - 8000:80
   depends_on:
     - 'app'
   volumes:
     - ./reverse-proxy/nginx.conf:/etc/nginx/nginx.conf
     - logs:/var/log/nginx
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.3
  #crowdsec : it will be fed nginx's logs
 #and later we're going to plug a firewall bouncer to it
 crowdsec:
   image: crowdsecurity/crowdsec:v1.0.8
   restart: always
   environment:
     #this is the list of collections we want to install
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/nginx"
     GID: "${GID-1000}"
   depends_on:
     - 'reverse-proxy'
   volumes:
     - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
     - logs:/var/log/nginx
     - crowdsec-db:/var/lib/crowdsec/data/
     - crowdsec-config:/etc/crowdsec/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.4
  #metabase, because security is cool, but dashboards are cooler
 dashboard:
   #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
   build: ./crowdsec/dashboard
   restart: always
   ports:
     - 3000:3000
   environment:
     MB_DB_FILE: /data/metabase.db
     MGID: "${GID-1000}"
   depends_on:
     - 'crowdsec'
   volumes:
     - crowdsec-db:/metabase-data/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.5
 
volumes:
 logs:
 crowdsec-db:
 crowdsec-config:
 
networks:
 crowdsec_test:
   ipam:
     driver: default
     config:
       - subnet: 172.20.0.0/24

The reverse-proxy container (nginx) writes its logs to the log volume set by the crowdsec container.

The SQLite CrowdSec database resides in the crowdsec-db volume installed by the (metabase) dashboard container.

Initial deployment

Mandatory requirements: availability Docker / Docker compose

We have put all the config files to this repositoryso that you can just clone them for deployment.

You can deploy from the Docker Compose directory using the docker-compose up -d command and then test it up with docker-compose ps.

> git clone https://github.com/crowdsecurity/example-docker-compose
> cd example-docker-compose
> sudo docker-compose up
> sudo docker-compose ps

# cd examples/docker-compose
# docker-compose up -d
...
# docker-compose ps     
             Name                           Command               State           Ports         
------------------------------------------------------------------------------------------------
docker-compose_app_1             httpd-foreground                 Up      80/tcp                
docker-compose_crowdsec_1        /bin/sh -c /bin/sh docker_ ...   Up                            
docker-compose_dashboard_1       /app/run_metabase.sh             Up      0.0.0.0:3000->3000/tcp
docker-compose_reverse-proxy_1   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:8000->80/tcp 

Let’s check that everything works!

Testing the demo application

With this command, we can check if access to our demo application is working correctly.

 curl http://localhost:8000/
Hello world !%

Examination

We need to check if CrowdSec is reading the logs correctly.

docker-compose exec crowdsec cscli metrics

sudo docker-compose exec crowdsec cscli metrics
INFO[25-02-2021 03:38:50 PM] Buckets Metrics:                             
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-crawl-non_statics | -             | -         |            2 |      2 |       2 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[25-02-2021 03:38:50 PM] Acquisition Metrics:                         
+-----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE               | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------+------------+--------------+----------------+------------------------+
| /var/log/nginx/example.access.log |          2 |            2 | -              |                      2 |
+-----------------------------------+------------+--------------+----------------+------------------------+
INFO[25-02-2021 03:38:50 PM] Parser Metrics:                              
+--------------------------------+------+--------+----------+
|            PARSERS             | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs  |    6 |      2 |        4 |
| child-crowdsecurity/nginx-logs |    2 |      2 | -        |
| crowdsecurity/dateparse-enrich |    2 |      2 | -        |
| crowdsecurity/geoip-enrich     |    2 |      2 | -        |
| crowdsecurity/http-logs        |    2 | -      |        2 |
| crowdsecurity/nginx-logs       |    2 |      2 | -        |
| crowdsecurity/non-syslog       |    2 |      2 | -        |
+--------------------------------+------+--------+----------+
INFO[25-02-2021 03:38:50 PM] Local Api Metrics:                           
+--------------------+--------+------+
|       ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /v1/watchers/login | POST   |    2 |
+--------------------+--------+------+

What happened and what’s what?

The cscli metrics command asks for metrics from Prometheusavailable locally by CrowdSec and presents them in this unusual way:

  • Acquisition metrics show us that our queries do indeed create logs that are read (LINES READ), parsed (LINES PARSED) and even mapped against installed scripts (LINES POURED TO BUCKET).

  • Buckets metrics and parser metrics tell us which parsers and scripts are being run.

Verifying CrowdSec Configuration

The cscli hub list command shows you which analyzers and scripts are deployed.

sudo docker-compose exec crowdsec cscli hub list
INFO[25-02-2021 03:46:41 PM] Loaded 14 collecs, 19 parsers, 23 scenarios, 3 post-overflow parsers 
INFO[25-02-2021 03:46:41 PM] unmanaged items : 20 local, 0 tainted        
INFO[25-02-2021 03:46:41 PM] PARSERS:                                     
-------------------------------------------------------------------------------------------------------------
 NAME                             STATUS   VERSION  LOCAL PATH                                             
-------------------------------------------------------------------------------------------------------------
 crowdsecurity/sshd-logs         ✔  enabled  0.1      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml         
 crowdsecurity/syslog-logs       ✔  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml         
 crowdsecurity/dateparse-enrich  ✔  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml 
 crowdsecurity/geoip-enrich      ✔  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml     
 crowdsecurity/nginx-logs        ✔  enabled  0.2      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml        
 crowdsecurity/http-logs         ✔  enabled  0.4      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml        
-------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] SCENARIOS:                                   
--------------------------------------------------------------------------------------------------------------------------
 NAME                                        STATUS   VERSION  LOCAL PATH                                               
--------------------------------------------------------------------------------------------------------------------------
 ltsich/http-w00tw00t                       ✔  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml               
 crowdsecurity/http-crawl-non_statics       ✔  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml      
 crowdsecurity/http-probing                 ✔  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml                
 crowdsecurity/http-path-traversal-probing  ✔  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml 
 crowdsecurity/http-xss-probing             ✔  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml            
 crowdsecurity/http-bad-user-agent          ✔  enabled  0.3      /etc/crowdsec/scenarios/http-bad-user-agent.yaml         
 crowdsecurity/ssh-bf                       ✔  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml                      
 crowdsecurity/http-backdoors-attempts      ✔  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml     
 crowdsecurity/http-sensitive-files         ✔  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml        
 crowdsecurity/http-sqli-probing            ✔  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml           
--------------------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] COLLECTIONS:                                 
------------------------------------------------------------------------------------------------------------
 NAME                                STATUS   VERSION  LOCAL PATH                                         
------------------------------------------------------------------------------------------------------------
 crowdsecurity/sshd                 ✔  enabled  0.1      /etc/crowdsec/collections/sshd.yaml                
 crowdsecurity/base-http-scenarios  ✔  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml 
 crowdsecurity/linux                ✔  enabled  0.2      /etc/crowdsec/collections/linux.yaml               
 crowdsecurity/nginx                ✔  enabled  0.1      /etc/crowdsec/collections/nginx.yaml               
------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] POSTOVERFLOWS:                               
--------------------------------------
 NAME   STATUS  VERSION  LOCAL PATH 
--------------------------------------
--------------------------------------

Metabase check

Metabase is one of the deployed components that allows you to create dashboards for more comfortable tracking of what is happening. You can go to http://127.0.0.1:3000/ and log in with crowdsec@crowdsec.net and password !! Cr0wdS3c_M3t4b4s3 ??

Metabase comes with a default password depending on how you deploy it. Remember to change the default password and restrict metabase access to only the required IP addresses or network ranges.

At first, the dashboards will be empty as no attacks have been detected yet. The main dashboard should look like this:

If any of the checks fail, view the container logs using the docker-compose logs crowdsec command (as an example).

Detection functions

Note… In real conditions, are used whitelisting to prevent banning private IP addresses.

After making sure everything is ready to go, you can try some of the detection functions. Since we are working with an open HTTP service, launch Nikto from another computer on the local network nikto -host http://192.168.2.227:8000

Note… The IP address depends on the LAN settings and the addressing plan.

We can also monitor the CrowdSec logs with the following command: docker-compose logs -f crowdsec

Here you can see that our client’s IP address (192.168.2.211) has been flagged as it ran several scripts:

We can see that our IP was blocked with docker-compose exec crowdsec cscli decisions list.

We can view and check for alerts with docker-compose exec crowdsec cscli alerts list and docker-compose exec crowdsec cscli alerts inspect -d XX:

Note… cscli alerts list shows a list of all triggered alerts.

sudo docker-compose exec crowdsec cscli alerts inspect -d 43

################################################################################################

 - ID         : 43
 - Date       : 2021-02-26T08:26:07Z
 - Machine    : ee0ebe5b529c4995964ff0b3e01b1801sxSpiCdYj9lpSD9W
 - Simulation : false
 - Reason     : crowdsecurity/http-sensitive-files
 - Events Count : 5
 - Scope:Value: Ip:192.168.2.211
 - Country    : 
 - AS         : 

 - Active Decisions  :
+-----+------------------+--------+--------------------+----------------------+
| ID  |   SCOPE:VALUE    | ACTION |     EXPIRATION     |      CREATED AT      |
+-----+------------------+--------+--------------------+----------------------+
| 802 | Ip:192.168.2.211 | ban    | 3h53m15.124782708s | 2021-02-26T08:26:07Z |
+-----+------------------+--------+--------------------+----------------------+

 - Events  :

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
|      KEY      |      VALUE      |
+---------------+-----------------+
| ASNNumber     |               0 |
| http_args_len |               0 |
| log_type      | http_access-log |
| service       | http            |
| source_ip     | 192.168.2.211   |
| http_status   |             404 |
| http_path     | /CyNqbugR.bak   |
| IsInEU        | false           |
+---------------+-----------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
|      KEY      |      VALUE      |
+---------------+-----------------+
| log_type      | http_access-log |
| service       | http            |
| source_ip     | 192.168.2.211   |
| http_status   |             404 |
| http_path     | /CyNqbugR.sql   |
| IsInEU        | false           |
| ASNNumber     |               0 |
| http_args_len |               0 |
+---------------+-----------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
|      KEY      |      VALUE      |
+---------------+-----------------+
| service       | http            |
| source_ip     | 192.168.2.211   |
| http_status   |             404 |
| http_path     | /CyNqbugR.exe   |
| IsInEU        | false           |
| ASNNumber     |               0 |
| http_args_len |               0 |
| log_type      | http_access-log |
+---------------+-----------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-------------------+
|      KEY      |       VALUE       |
+---------------+-------------------+
| IsInEU        | false             |
| ASNNumber     |                 0 |
| http_args_len |                 0 |
| log_type      | http_access-log   |
| service       | http              |
| source_ip     | 192.168.2.211     |
| http_status   |               404 |
| http_path     | /CyNqbugR.printer |
+---------------+-------------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+--------------------+
|      KEY      |       VALUE        |
+---------------+--------------------+
| ASNNumber     |                  0 |
| http_args_len |                  0 |
| log_type      | http_access-log    |
| service       | http               |
| source_ip     | 192.168.2.211      |
| http_status   |                404 |
| http_path     | /CyNqbugR.htaccess |
| IsInEU        | false              |
+---------------+--------------------+

Note. cscli alerts inspect -d allows you to get more detailed information about the alert.

Monitoring actions using dashboards

After running a few scripts, we can return to the Metabase dashboards (http://127.0.0.1:3000 with default setting) and check for activity.

If the traffic came from a public IP address (and not from a private one, as in this example), crowdsecurity / geoip-enrich would enrich the events with geolocation data and AS / range information.

Blocking attacks with bouncers

Now that we have a fully functional CrowdSec service, we can detect incoming attacks on our service using installed data collection scripts

After detecting attacks, our goal is to block them. For this we will use cs-firewall-bouncer… Start by installing it on the host and block malicious traffic directly in DOCKER-USER, the default path created by Docker to filter traffic destined for containers.

You can find firewall bouncer on CrowdSec Hub… Its latest version is v0.0.16.

wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz
tar xvzf cs-firewall-bouncer.tgz
cd cs-firewall-bouncer-v0.0.10
sudo ./install.sh

Since July, Debian, Ubuntu, CentOS, RHEL, and Amazon Linux include their own firewall bouncer packages. Check out our github repository for more information. Installation is very easy. It includes deploying the systemd module for the service and validating it for compliance. We didn’t have ipset running here, so it was installed for us.

Here we have installed a bouncer on a host that is not running CrowdSec. Therefore, the service is unhappy.

Now let’s configure the bouncer to interact with the local API on our CrowdSec container. Let’s start by creating an API token for our bouncer using cscli.

docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer

sudo docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer
Api key for 'HostFirewallBouncer':

   aaebb3708fe67eeeccbb52a21e5e7862

Please keep this key since you will not be able to retrive it!

Next, you need to configure the bouncer to use this token to authenticate with the local CrowdSec API. In /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml, edit api_url, api_key, and iptables_chains. In this case, IPv6 was also disabled using the disable_ipv6 command:

mode: iptables
piddir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
api_url: http://172.20.0.4:8080/
api_key: aaebb3708fe67eeeccbb52a21e5e7862
disable_ipv6: true
#if present, insert rule in those chains
iptables_chains:
 - DOCKER-USER

Note… We edited the paths to only use the DOCKER-USER path, and also set the api_url to match our docker-compose.yml file and specify the newly generated api token.

CrowdSec in action

We can now start the configured bouncer with sudo systemctl start cs-firewall-bouncer.service, then take a look at the new firewall configuration:

sudo iptables -L -n                                                                  
...
Chain DOCKER-USER (1 references)
target     prot opt source               destination         
DROP       all  --  0.0.0.0/0            0.0.0.0/0            match-set crowdsec-blacklists src
...

We can see that our DOCKER-USER path has received a rule to match incoming traffic to our ipset, and the ipset is filled with the appropriate information.

 sudo ipset -L crowdsec-blacklists            
Name: crowdsec-blacklists
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 300
Size in memory: 6016
References: 1
Number of entries: 61
Members:
178.20.157.98 timeout 80103
52.184.35.59 timeout 80103
...

As you may have noticed, ipset is filled with not only our local solutions, but also community solutions. However, we can see that our “local” solutions have ended up in the ipset list.

 sudo ipset -L crowdsec-blacklists  | grep 192
192.168.2.211 timeout 12859

Now we can look at the attacker’s machine and find that the blocking has been implemented and there is no access to the application.

$ curl -vv 192.168.2.227:8000
* Rebuilt URL to: 192.168.2.227:8000/
*   Trying 192.168.2.227...
* TCP_NODELAY set
^C

An attacker cannot access all Docker Compose applications. We deliberately limited the solution to DOCKER-USER, so local applications on the host will still be available. If we want to deny all incoming traffic, then we can add the INPUT path to the list in the same way as the default setting.

Conclusion

In this tutorial, we deployed the smallest possible yet complete application stack with Docker Compose. We then looked at how to secure it with CrowdSec. While most use CrowdSec as a host-based security mechanism, we’ve learned that it is also suitable for Docker environments. If you would like to chat with the team and share your opinion, you can contact us on our channel Gitter or chat on our website

about the author

Thibault Koechlin – CTO of CrowdSec, graduated from the EPITECH IT school with a degree in IT systems and network security. He began his career at NBS in 2004 as a Pen-Testing Expert and was later named Head of the Security Team. He then became Director of Information Security and deepened his knowledge of defensive security, before he began developing his own open source products and assembled a team of experts with rare skills. He reached the pinnacle of his career with the company as an Operational Partner and spearheaded the creation of the company’s flagship product: Cerberhost. In December 2019, he co-founded CrowdSec with Philippe Humeau and Laurent Soubrevilla. He is the CTO of the company.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *