Using Foreman and Puppet in an IaC approach

Introduction

This is an introductory publication, an overview of some of the platform’s functions Foremanregarding IaC Puppet.

Foreman provides a range of tools for monitoring your IT environment, but this article describes specific capabilities, such as:

• using Puppet modules

• setting options for Puppet

• module delivery process

• writing scripts for collecting the required facts

• using templates for uploading reports

• also affected by the use of the software interface and database

Foreman’s capabilities are described from the platform user’s point of view. The technical implementation of the demonstration stand will not be considered.

If you are interested in reading about installing Foreman, the Bare Metal Controls plugin, setting up GIthub Actions for Puppet CI code, using r10k, Puppet linters and Puppet code autotests, as well as working with Smart-proxy (PXE, DHCP, DNS), please write to us . Contact information is provided at the conclusion of the publication.

Stand components:

We use the Ubuntu 20.04 LTS operating system and Foreman 3.7 version without additional plugins.

Puppet manifests in Foreman Web Interface

Task:
Install Zabbix Agent, specify the address Server=8.8.8.8 in the agent config.

Possible solution using the Foreman interface:

The last report displays manifest application graphs.

That’s it, task completed! Next, Puppet Agent will check and maintain the specified configuration every time it accesses Puppet Server.

In this example and later in the article, we use the wonderful Zabbix Module which is taken from forge puppetbut you can create Puppet modules at your discretion and for your ideas.

In the following chapters we will look at overriding the parameters used in Puppet code.

Puppet Code and Puppet ENC

Developing code for Puppet is no different from any other development and can be customized using version control systems and your favorite IDE. This post uses Pycharm and GitHub. We have already configured Pycharm and a pipeline on GitHub to deliver code to Puppet Server.

Task:

Override the default parameter responsible for the Zabbix Server network address in the Puppet module Zabbix.

If necessary, in the server console foreman-main We check that the modules were delivered and the parameter was registered correctly.

Now, by default, the address Server=8.8.8.8 will be set for all newly installed Zabbix agents.

Let’s override the value of the Server parameter for one host using Puppet ENC.

Great features, you can easily deliver code to Puppet Server → Puppet Agent and use default parameters in Puppet code, as well as override them in Puppet ENC.

Below we display the implementation of the default and overridden parameter values ​​via Foreman in Puppet ENC.

On the left side of the IDE, the params.pp manifest is open with the default value of the $agent_server parameter (aka server from the Foreman interface).

The bottom of the IDE shows the ssh connection to test-server and the value of the Server parameter from the zabbix_agentd.conf file.

On the right side of the IDE is the connection to the Foreman database and the contents of the lookup_keys and lookup_values ​​tables.

Results:

• For all servers where the Zabbix agent will be installed, the $agent_server parameter is equal to ‘8.8.8.8’

• For test-server.example.com the $agent_server parameter is redefined in the database and is equal to “7.7.7.7”. Priority will be given to the value specified in the database.

Next, let’s look at the Foreman abstractions: Host Groups, Organizations, Locations and the parameters that can be set for them.

Foreman Group hosts, Organization, Location and Parameters

The Foreman system has a flexible logic for grouping hosts and nesting parameters. Thanks to this, you can control the configuration depending on the functional purpose of the server. For example, separate hosts depending on the required software, the location of the servers and their purpose.

It will not be possible to consider it completely in one publication, but as an example, let’s consider the simplest option.

Task:

Creating a group that, when assigned to servers, will install Zabbix Agent with the specified $agent_server parameter, different from the default value.

Steps to solve the problem:

• creating a host group

• overriding the parameter in Puppet ENC for the created host group

• assigning a group to the server test-server.example.com

After applying the Puppet Agent configuration, we see a new parameter in the Zabbix agent config, overridden for a group of hosts test_group.

Now let’s look at the possibility of creating group parameters. These are different parameters from Puppet ENC that can be used in Puppet modules.

We use the test_parameter_zabbix group parameter as the default value of the $agent_server variable in the Puppet manifest.
You must first remove the override from the Puppet ENC tab.

After the Puppet Agent configuration is applied, the report displays the value set using the group parameter.

Similarly, you can manage hosts divided into Organizations And Locations, which allow you to include Groups and individual hosts.

Let’s group the servers into two branches: North and South. These abstractions will have different values ​​for the $agent_server parameter.

Consider the creation of the North Branch organization.

We indicate in the params.pp manifest the created organization parameter $branch_parameter_zabbix, which can be viewed in the parameters table.

After the Puppet Agent configuration is applied, the report displays the value set using the organization setting.

In this chapter we looked at the possibility of cataloging servers and using a hierarchy of parameters.
You can learn more about the hierarchy of parameters at link.

Next, let’s move on to Puppet facts and view them.

View Puppet facts in Foreman

In this part, we will learn how to view Puppet facts in the web interface and in the database.

Using Puppet, information about servers is collected and stored in the form of abstractions called Puppet facts.

There are default facts, for example, the OS name and network address. It is also possible to collect custom facts, such as the presence of mounted network directories, installed software version, etc.

One way to view collected facts is the Monitor → Facts tab of the web interface. It is possible to search for facts by the host attributes, the name of the fact and its meaning, and the presence of the fact.

For example, let’s request all collected facts for the host test-server.

It is also possible to access facts directly from the database.

select distinct hosts.name, fact_names.short_name, fact_values.value
from hosts
join fact_values on hosts.id=fact_values.host_id
join fact_names on fact_values.fact_name_id=fact_names.id
where hosts.name="test-server.example.com";

The screenshot shows the execution of a query in the Foreman database and obtaining facts test-server.

Using this information, you can visualize data, for example using Grafana, conduct analysis and find patterns.

Let’s say we have a task:

Get data about NFS mount points on a host for the entire server fleet and use it in Puppet manifest logic.

This problem can be solved using custom facts.
An example script for assembling a fact about mount points. This script will be executed on all servers, the output will be received by Puppet Server and will be available in the database and in the web interface.

# nfs_search.rb

awk = "/usr/bin/awk"

if File.exist?(awk)
    Facter.add('nfs_search') do
        setcode do
          Facter::Core::Execution.execute('\'awk\' \'$1 ~ /^([0-9]|\/\/)/ {print $1 ","}\' /proc/mounts')
        end
    end
end

In this chapter, we looked at the possibility of collecting information about servers using Puppet facts. These values ​​can be used in manifest logic and system auditing.

Next, we will consider the use of collected facts in Report Templates.

Report templates in Foreman

Report templates uses the ERB template engine to create reports, which can be generated as files via a web interface or queried using the Foreman API.
Let’s consider using such a report and getting it from the web interface. Reports use can use the facts collected.

Task: Get a list of all hosts, their network address and operating system name.

Let’s take a look at the implementation:

The following template was used to generate the data:

<%#
name: test - status
snippet: false
template_inputs:
- name: hosts
  required: false
  input_type: user
  advanced: false
  value_type: search
  resource_type: Host
model: ReportTemplate
-%>

<%- load_hosts(search: input('hosts')).each_record do |host| -%>
<%-   report_row({
        'server_name': host.name,
        'network_address': host.facts['ipaddress'],
        'OS': host.facts['os::name'],
              }) -%>
<%- end -%>
<%= report_render -%>

Reports can be used in solving various tasks, such as infrastructure audit.

All reports available from the web interface can be requested using the Foreman API, which will be discussed below

API Foreman

In this chapter we will look at using the Foreman API.

Task:
Receive report data via http request.

An example of solving a problem using Postman:

This feature can be useful in implementing automation or integration.

Let’s solve the same problem using a Python class.

#!/usr/bin/python3
import requests
from pprint import pprint
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


class ReportGenerator:
    def __init__(self, api_url, auth):
        self.api_url = api_url
        self.auth = auth

    def generate_report(self):
        json_data = {
            "report_format": "json"
        }

        response = requests.post(self.api_url,
                                 auth=(self.auth['user'], self.auth['pass']),
                                 json=json_data,
                                 verify=False)

        return response.json()


if __name__ == "__main__":
    auth_credentials = {'user': 'your_login', 'pass': 'your_pass'}
    api_url="https://foreman-main.example.com/api/report_templates/148/generate"

    report_generator = ReportGenerator(api_url, auth_credentials)
    api_response = report_generator.generate_report()
    pprint(api_response)

Result of script execution:

The Foreman API is very functional and will help with automation.
The publication describes a simple example of unloading a Report Template.

API features include:

• creating/deleting hosts

• revocation of certificates

• overriding parameters at any level

• assigning Puppet manifests, groups, organizations and more

You can find out about all the API capabilities on the official website Foreman.

Conclusion

Using the Foreman framework and Puppet in an IaC approach is an effective way to manage infrastructure.
The methods used at Foreman guarantee order, repeatability, safety and high dynamism of all parts of the systems. Foreman will complement the infrastructure landscape, significantly reduce the level of technical contradictions, and provide the opportunity for integration and automation.

Thank you for reading this far. This post is an attempt to convey the user experience. We briefly touched on the functionality that is associated with the IaC approach, trying not to overload the text with technical details.

Feel free to use the comments on this article and our contacts, we are really interested in feedback. It is planned to begin a series of publications about Foreman and its components.

Contacts:
Mailing address george.article.feedback@gmail.com

Authors of the publication:
Minashvili Georgy, Volya Yana, Cherdakov Igor.