SaltStack: configuration management

Salt Minion.

The Salt Master coordinates activities and distributes tasks among the Salt Stack nodes. Salt Minions, installed on managed servers and devices, listen to commands from the master and execute them, thereby ensuring that configuration changes and updates are instantly propagated throughout the entire infrastructure.

Let's install and configure

Salt Master manages Minions by sending them instructions. Installing Salt Master typically begins by adding the SaltStack repository to the source list of your package management system and running the installation through that system.

For systems based Debian/Ubuntu we use mands:

wget -O - https://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo "deb http://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest focal main" | sudo tee /etc/apt/sources.list.d/saltstack.list
sudo apt-get update
sudo apt-get install salt-master

For CentOS/RHEL:

sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el7.noarch.rpm
sudo yum clean expire-cache
sudo yum install salt-master

Salt Minion installed on managed nodes. Just like for Master, you can start by adding a repository:

On Debian/Ubuntu:

wget -O - https://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo "deb http://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest focal main" | sudo tee /etc/apt/sources.list.d/saltstack.list
sudo apt-get update
sudo apt-get install salt-minion

On CentOS/RHEL:

sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el7.noarch.rpm
sudo yum clean expire-cache
sudo yum install salt-minion

After installing Minion, you need to configure it to communicate with the Master. This is done by changing the file /etc/salt/minionwhere you need to specify the address, or the name of the Salt Master:

master: ip_address_or_hostname_of_salt_master

After changing the configuration file, restart the Minion service:

sudo systemctl restart salt-minion

After installation and configuration on Master we perform:

sudo salt-key -L

This will show a list of Minions that are trying to connect to the Master. To confirm the Minion key use:

sudo salt-key -A

Basics

Salt States allow you to define the desired state of the system in the form of declarative code. These definitions are stored in files with the extension .slsusually written in YAML for ease of readability and maintainability.

A simple example of a state file, webserver.slsmight look like this:

apache:
  pkg.installed: []

Here's the package apachemust be installed on the target system.

File top.sls defines which state files to apply to which minions. Sample content top.sls:

base:
  '*':
    - core

State core applies to all minions. The system allows the use of various targeting mechanisms, for example, through glob patterns, regular expressions or grain data.

For example, we use grains data to target minions based on their operating system. State core.ubuntu applies to minions running Ubuntu OS, and core.centos – to minions on CentOS:

base:
  'os:Ubuntu':
    - match: grain
    - core.ubuntu
  'os:CentOS':
    - match: grain
    - core.centos

Let's see what it might look like core.sls for minions on Ubuntu:

# файл: core.ubuntu.sls
update_system:
  pkg.uptodate:
    - refresh: true

install_basic_packages:
  pkg.installed:
    - names:
      - curl
      - git
      - vim

You can use regular expressions for targeting:

base:
  '^(web|db).*':
    - match: pcre
    - core.web_db_common

Here regular expressions are used for targeting and thus state can be applied core.web_db_common to minions whose IDs start with web or db.

States are organized into a hierarchical file structure starting from the root /srv/salt:

/srv/salt
├── core.sls
├── httpd
│   ├── files
│   │   ├── apache2.conf
│   │   └── httpd.conf
│   └── init.sls
└── top.sls

Salt allows you to define different environments, such as development, testing, and production, using separate file_roots for each environment and corresponding sections in the file top.sls:

base:
  '*':
    - core
dev:
  'webserver*dev*':
    - webserver
prod:
  'webserver*prod*':
    - webserver

Here, different sets of states are specified for different environments and corresponding minions.

For example, you need to make sure that the vim editor is installed, the Salt package is installed and updated to the latest version, the salt-master and salt-minion services are running and the minion configuration file is in place:

vim:
  pkg.installed: []

salt:
  pkg.latest:
    - name: salt
  service.running:
    - names:
      - salt-master
      - salt

-minion
    - require:
      - pkg: salt
    - watch:
      - file: /etc/salt/minion

/etc/salt/minion:
  file.managed:
    - source: salt://salt/minion
    - user: root
    - group: root
    - mode: 644
    - require:
      - pkg: salt

Installing multiple packages is simple:

common_packages:
  pkg.installed:
    - pkgs:
      - git
      - htop
      - vim

Managing state services is also quite simple. For example, to ensure that the Apache service is running:

apache_service:
  service.running:
    - name: apache2
    - enable: True
    - reload: True
    - require:
      - pkg: apache2

Here in addition to starting the service apache2the parameter is also used requireto make sure the package apache2 installed before attempting to start the service.

To control the contents of a file on the target system, you can use the following example, which ensures that certain text is present in the file:

motd_custom_message:
  file.managed:
    - name: /etc/motd
    - source: salt://files/motd.txt
    - user: root
    - group: root
    - mode: 644

File /etc/motd on the target node will have content matching the file motd.txtstored on master in the directory salt://files/.

You can overwrite the configuration file using the Jinja2 template, and update the service if the file has been changed:

nginx_config:
  file.managed:
    - name: /etc/nginx/nginx.conf
    - source: salt://nginx/nginx.conf.jinja
    - template: jinja
    - user: root
    - group: root
    - mode: 644
    - require:
      - pkg: nginx

nginx_service:
  service.running:
    - name: nginx
    - watch:
      - file: nginx_config

File /etc/nginx/nginx.conf is generated from the Jinja2 template and if it changes, the nginx service will be restarted.

SaltStack for task automation and orchestration

Salt Reactor monitors the Salt event bus and executes reactions to events that meet certain criteria. These reactions are defined in the reactor SLS files and can run various functions including deleting minion keys or running tasks on the minion using the minion side Reactor system.

Example Reactor SLS file:

remove_key:
  wheel.key.delete:
    - match: {{ data['id'] }}

Here the minion key will be deleted upon a certain event. You can also use Reactor to run tasks at the minion level:

touch_file:
  caller.file.touch:
    - args:
      - name: /tmp/foo

When writing Reactor SLS files, it's worth remembering that reactions should be kept simple! Eliminate complex calculations and Jinja templates that can slow down processing. For more complex tasks, it is better to use the Salt orchestration system.

The Salt orchestration system provides a framework for coordinating and executing tasks across multiple nodes. Using Runner state.orchestrate You can define states that control the execution of modules, states, and other Runners on master and minion.

Orchestration example:

# /srv/salt/orch/deploy.sls
create_instance:
  salt.runner:
    - name: cloud.profile
    - prof: cloud-centos
    - provider: cloud
    - instances:
      - server1
    - opts:
        minion:
          master: master1

You can also use Jinja to dynamically pass data through Pillar when performing orchestration.

With orchestration, you can define complex sequences of tasks, taking into account the dependencies between them, using the full set of requisites available in Salt States. For example:

bootstrap_servers:
  salt.function:
    - name: cmd.run
    - tgt: 10.0.0.0/24
    - tgt_type: ipcidr
    - arg:
      - bootstrap

Team bootstrap runs on all nodes in the 10.0.0.0/24 subnet. You can then run highstate on specific minion groups, apply storage settings, and so on, using dependencies and other Salt features to control the order in which tasks are executed.

---

Other relevant tools for interacting with various levels of infrastructures can be explored on courses at OTUS under the guidance of expert practitioners.