Vagrant virtual machine for team development

The topic of creating vagrant virtual machines has already been discussed on Habré with comments “why do I need this?”. In my article I want to answer this question. When you have a project and new employees join your team along the way, the problem arises that they do not have identical conditions to run and test the project on their machines. And here the Vagrant virtual machine comes to the rescue. Using the example of the django project created in my etl article, I want to show how each new member of your project will deploy a development environment with all dependencies in four console commands. This will allow you to start the work process as soon as possible.

As an alternative to a virtual machine, you can use docker containers. But there is a caveat that to work with containers, appropriate qualifications are required.

So, let’s imagine that we have a django project that we want to develop as a team.

We will create a virtual machine for further development with preinstalled pyenv for python versioning and poetry to manage dependencies. We will also configure the interpreter in the IDE PyCharmPro And VS Code.

All sources from this article are posted in repository.

Installing the software

On the host (main) operating system, each project participant must have:

Let’s designate the structure of the project

The little django project is in the etl folder. Project configuration folder config, application folder etl_app. Read about other files and folders below.

.           
├── Makefile
├── Vagrantfile
├── etl
│   ├── README.md
│   ├── __init__.py
│   ├── config
│   │   ├── __init__.py
│   │   ├── asgi.py
│   │   ├── celery.py
│   │   ├── settings.py
│   │   ├── settings_local.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── etl_app
│   │   ├── __init__.py
│   │   ├── apps.py
│   │   ├── etl.py
│   │   └── tasks.py
│   └── manage.py
├── poetry.lock
├── pyproject.toml
└── vagrant
    ├── playbook.yml
    ├── postgres.yml
    ├── requirements.yml
    └── templates
        └── settings_local.j2

Getting started with Ansible

Ansible is a configuration management system for Linux. We will configure the created virtual machine through a script (hereinafter referred to as playbooks) in the format YAML with a description of the required states of the controlled system. A script is a description of the state of a system’s resources that it should be in at a given time, including packages installed, services running, files created, and more. Ansible checks that each of the resources in the system is in the expected state and tries to fix the state of the resource if it is not as expected.

To perform tasks, a system of modules is used. Each task is its name, the module used and a list of parameters that characterize the task. The system supports variables, filters for processing variables (support is provided by the library Jinja2), conditional task execution, parallelization, file patterns. The addresses and settings of the target systems are contained in the “inventory” files. Grouping is supported. To implement a set of tasks of the same type, there is a system of roles.

From all of the above, we will use roles to write less code, and a templating engine Jinja2.

We write all playbooks in the vagrant folder.

requirements.txt

In our project, the role will be used python_developer from the roles repository ansible-galaxy. She (role) is a ready-made set of tasks that will install pyenv and poetry for us. We register the role in the requirements.txt dependency file:

- name: staticdev.python-developer
  src: https://github.com/staticdev/ansible-role-python-developer.git

postgres.yml

In this playbook, we will write a set of tasks for creating a database and schemas in it:

---
- name: Add postgreuser permission for createbd
  become: true
  become_user: postgres
  shell: psql -U postgres -c 'ALTER USER {{ postgres_user }} CREATEDB'

- name: Create database
  become: true
  become_user: postgres
  postgresql_db:
    name: "{{ postgres_db }}"
    encoding: UTF-8
    owner: "{{ postgres_user }}"

- name: Create schemas
  become: true
  become_user: postgres
  community.postgresql.postgresql_schema:
    db: "{{ postgres_db }}"
    name:
      - main
      - my_schema
    owner: "{{ postgres_user }}"

Pay attention to lines like {{ some_var }} – these are variables. We will ask them in the next playbook.

playbook.yml

This is where we write our main script. It determines what needs to be installed on our virtual machine, and also uses the playbooks described earlier.

---
- hosts: all
  vars:
    app_settings_dir: config
    postgres_user: sergei
    postgres_password: sergei
    postgres_db: demo
    python_version: 3.10.6
  roles:
    - role: staticdev.python-developer
      vars:
        pyenv_global:                   
          - "{{ python_version }}"
        pyenv_python_versions:
          - "{{ python_version }}"
  tasks:
    - name: add GPG key
      ansible.builtin.apt_key:
        url: https://keyserver.ubuntu.com/pks/lookup?op=get&fingerprint=on&search=0x1655A0AB68576280
        id: 68576280
        state: present
      become: yes

    - name: Install base needed apt-packages
      ansible.builtin.apt:
        pkg:
          - wget
          - unzip
          - redis-server
          - wget
          - python3-psycopg2
          - acl
        state: latest
      become: yes
      become_method: sudo

    - name: Set up Postgres repo
      shell: |
        echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list
        wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
      args:
        warn: no
      become: yes

    - name: Install postgresql
      apt:
        name: postgresql-15
        update_cache: yes
      become: yes

    - name: Start and enable postgres
      ansible.builtin.service:
        name: postgresql
        enabled: yes
        state: started
      become: yes

    - name: Create user
      community.postgresql.postgresql_user:
        name: "{{ postgres_user }}"
        password: "{{ postgres_password }}"
      become: yes
      become_user: postgres

    - name: Create db
      include_tasks: postgres.yml

    - name: Ensure Redis is started
      service: name=redis-server state=started enabled=yes
      become: yes
      become_method: sudo

    - name: Exporting broker url to env
      lineinfile:
        dest: "/etc/environment"
        state: present
        regexp: "^REDIS_URL="
        line: "REDIS_URL=redis://127.0.0.1:6379/0"
      become: yes
      become_method: sudo

    - name: Exporting db url to env
      lineinfile:
        dest: "/etc/environment"
        state: present
        regexp: "^DATABASE_URL="
        line: "DATABASE_URL=postgres://{{ postgres_user }}:{{ postgres_password }}@127.0.0.1:5432/{{ postgres_db }}"
      become: yes
      become_method: sudo

    - name: Make settings file from template
      template:
        src: /vagrant/vagrant/templates/settings_local.j2
        dest: "/vagrant/etl/{{ app_settings_dir }}/settings_local.py"

    - name: set python version for project
      ansible.builtin.command: /home/vagrant/.local/bin/poetry env use /home/vagrant/pyenv/shims/python
      args:
        chdir: /vagrant/

Playbook sections:

• vars – set variables

• roles – run roles (a predefined set of tasks). pyenv_python_versions – a list of python versions that you want to install on the VM, pyenv_global– which of these versions to make global.

• tasks – a list of tasks that form a set of console commands. Only tasks still track the responses of executed commands. If any of the commands return an error, the playbook will stop running. In the example above, I’m installing Redis (lines 29 and 73). You are free to install any set of packages.

Let’s take a closer look at the problem Make settings file from template. This command takes the settings_local.j2 file template from the src path, runs it through the Jinja2 template engine – see DEBUG in the code snippet below. It then copies the completed module to the dest location. The most obvious benefit is the ability to exclude the settings_local.py file of the project’s configuration folder from the git index. As a consequence, settings_local.py is indeed a local (sorry for the tautology) module. This technique allows you to use other variable values ​​for the sale.

Was settings_local.j2:

import os
import dj_database_url

DEBUG = {{ debug|lower|capitalize }}

...

Became settings_local.py:

"""Локальные настройки проекта"""
import os
import dj_database_url

DEBUG = True

...

Preparing to launch the Vagrant virtual machine

To start the virtual machine, we need a Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  config.vm.network "forwarded_port", guest: 8000, host: 8001
  config.vm.network "forwarded_port", guest: 5432, host: 5433
  config.vm.provider "virtualbox" do |v|
    v.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
    v.customize ["modifyvm", :id, "--uartmode1", "file", File::NULL]
    v.memory = 2048
  end

  # ansible_local
  config.vm.provision "ansible_local" do |ansible|
    ansible.verbose = "vv"
    ansible.galaxy_role_file="vagrant/requirements.yml"
    ansible.galaxy_roles_path="/home/vagrant/.ansible/roles"
    ansible.galaxy_command = 'ansible-galaxy install --ignore-errors --force --role-file=%{role_file}'
    ansible.playbook = "vagrant/playbook.yml"
  end
end

Line number explanations:

• 5 – operating system image

• 7- forwarding the 8000th port of the VM, which is usually used to run the Django development server, to the 8001st port of the host machine. Both ports can be changed to your liking.

• 8 – port 5432 of the Postgres DBMS forwarded to port 5433 of the host machine. Port 5433 can be changed at your discretion.

• 12 – the amount of RAM in MB.

• 16 – we declare a block of commands to run ansible.

• 17 – level of detail in the console, regulated by the number of characters v.

• 18 – path to dependency files.

• 19 – path to store asnible role files. More about this in the documentation. In this case, they will be stored in the user’s folder and not appear in the project.

• 20 – console command to install ansible dependencies. I customized it with--ignore_errorsso that there is no error when reinstalling dependencies.

• 21 – the path to the main playbook, which starts all the “magic”.

Starting the virtual machine

Creation and each subsequent launch is done using the console command vagrant up. The complete algorithm looks like this:

1. In the project directory, run the console command vagrant up.

2. Immediately (without waiting for step 1) launch the Oracle VM virtual box window. Make the created virtual machine active (no additional windows need to be opened). Until the VM is fully launched, keep this window active.

The first launch can take 15-20 minutes because the VM is being created and packages are being installed. At this time, make sure that the computer does not go into “sleep”. You may need a VPN for the first run.

Other commands for managing the VM:

• vagrant halt – stop VM

• vagrant ssh – go to VM

• vagrant destroy – “disassemble” VM

Using make

To work with a virtual machine, the commands from the previous section are enough for us. But in addition, we can take out the most commonly used ones in make. The installation process on Ubuntu is described Hereon Windows Here.

Let’s create a Makefile:

# инициализировать виртуальную машину
up:
	vagrant up

# установить зависимости проекта
install:
	vagrant ssh -c "cd /vagrant/ && poetry install"

# сделать дата-миграции бд
migrate:
	vagrant ssh -c "cd /vagrant/etl && poetry run python manage.py makemigrations && poetry run python manage.py migrate"

# запустить сервер разработки
runserver:
	vagrant ssh -c "cd /vagrant/etl && poetry run python manage.py runserver 0.0.0.0:8000"

Pay attention to the last command. Thanks to port forwarding in Vagrantfile, the development server will be available on the host machine at http://localhost:8001.

Setting up the IDE

To fully integrate the interpreter into the IDE shell, the latter must be able to work through an ssh connection. Connection options:

• host 127.0.0.1

• port 2222. If you start more than one VM at the same time, then each subsequent one will have a different port. Be careful. You will see the port in the report of the VM initialization command.

• username vagrant

• key pair (connection goes by key, the path is relative to the directory where the Vagrantfile is located): .vagrant/machines/default/virtualbox/private_key

PyCharmPro

Unfortunately, integration with the interpreter in the Community version is excluded, because there is no ssh connection functionality. In this case, you can go to the virtual machine in the terminal and perform all code launches from the virtual machine console.

Connecting to a remote database in Community is also not possible. But pgAdmin (more than) fully replace the functionality of PyCharm Pro in this part.

Setting up an interpreter

The instruction is available at link on the IDE developer site. In addition to the ssh connection parameters, we need the path to the interpreter inside the VM. It can be obtained with a compound command in the virtual machine consolecd /vagrant/ && poetry run which python

Database connection

We also connect to the database via ssh. The instruction is available at link on the IDE developer site. Get connection information from the beginning of the current section. Login / password / database name we have specified in playbook.yml in the vars section.

VS Code

Integration with the interpreter is performed in two steps:

1) Fix the connection setting to in the VM in the ssh client configuration of your host machine. For this run the command vagrant ssh-config on the host machine in the project folder. The output will be something like this:

Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile <some path>/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

Add these lines to your ssh client config file. This is usually ~/.ssh/config. default replace with something you understand and related to the project.

2) Set up an ssh connection interpreter in VS Code. Make sure you have Microsoft’s Remote SSH and Python plugins installed first.

Connecting to VM:

In the virtual machine console with a compound commandcd /vagrant/ && poetry run which python get the path to the interpreter. For example /home/vagrant/.cache/pypoetry/virtualenvs/etl-base-vs8V2ZPt-py3.10/bin/python

Specify this path in the Python plugin settings:

Detailed instructions are available at link on the IDE developer site.

Summing up

We have created a virtual machine with no load in the form of a GUI. They showed how you can work with it using programs installed on the host machine. For “full happiness” you can install a ssh client with a graphical interface to manage files in a familiar environment, as an option – MobaXterm.

At the beginning of the article, I promised that each participant will be able to deploy your project on their machine in four console commands. Here they are:

git clone <repo>  # клонируем репозиторий
make up  # поднимаем ВМ
make install  # устанавливаем зависимости проекта
make migrate  # создаем требуемые таблицы базы данных

Thank you for your attention, I look forward to your comments and will be happy to answer all questions.