how to start your project from scratch with Pyenv and Poetry
This is an adapted translation of the article. Modern Python part 1: start a project with pyenv & poetry Faozi Braza, Data Engineer. The narration is conducted on behalf of the original author.
The translation was done for the platform of courses on programming Hexlet
Almost all novice developers are faced with the fact that understanding the syntax of a programming language and good engineering practices is not enough to start programming. Before you write your first working program, you need to create the right development environment — one that implements good coding practices, improves productivity, and facilitates project interaction and communication.
The packaging and tooling process for Python is often described as cumbersome and complex. As a result, several open source projects have emerged in recent years with the goal of making it easier to manage Python packages within your work projects. Here, we’ll look at how to use two of them: Pyenv – to manage and install various versions of Python, and Poetry – for managing packages and virtual environments.
This article is the first in a series that introduces modern Python best practices. In this series of articles, all examples are based on the implementation of a simple project – it is a Python function that summarizes the data present in pandas DataFrame… The function prints out the number of rows and columns and the frequency of each data type present in the pandas DataFrame.
Training
Installing pyenv
For installation pyenv you need some dependencies specific to your operating system. They are necessary because pyenv installs Python by building from source. To learn about the required dependencies for your OS, read this documentation… After installing the dependencies, you can install pyenv… For this it is better to use pyenv-installerwhich automates the process.
curl https://pyenv.run | bash
After that, you can already install any version of Python on your system. You can find out all available versions of Python using a special command:
pyenv install --list
In our case, we will install the classic CPython versions 3.7.10, 3.8.7, 3.9.2:
pyenv install 3.7.10
Downloading Python-3.7.10.tar.xz...
> https://www.python.org/ftp/python/3.7.10/Python-3.7.10.tar.xz
Installing Python-3.7.10...
Installed Python-3.7.10 to /home/aabur/.pyenv/versions/3.7.10
After installing the versions, you can see them by running the command:
pyenv versions
system
3.7.10
3.8.7
3.9.2
It’s clear that pyenv identified recently installed Python versions, and the default version installed on your system. The symbol in front of system means that the global version currently in use is the system version. pyenv allows you to manage Python versions at different levels: globally and locally. Let’s say we are going to install version 3.7.10 as a global version.
pyenv global 3.7.10
Let’s take another look at our versions:
pyenv versions
system
* 3.7.10 (set by /home//.pyenv/version)
3.8.7
3.9.2
You see that pyenv established 3.7.10 as our global Python version. This will not change operations that require the use of the system version. The path that you can read between the parentheses matches the path that points to the required Python version. pyenv intercepts Python commands using executables injected into your PATH, determines which version of Python you need to use, and passes the commands to the version of Python you want. Feel free to read the full documentationto better understand the functionality and capabilities pyenv…
Don’t be confused by the semantics. Changing the global version will not affect your system version. The system version corresponds to the version your OS uses to perform certain tasks or run background processes, which depend on that particular Python version. Do not change the system version to another, otherwise you may face a number of problems with your operating system! This version is usually updated along with your OS. The global version is just the version that pyenv will use to execute Python commands / programs globally.
Poetry installation
poetry allows you to efficiently manage dependencies and packages in Python. It fulfills the same role as setup.py or pipenvbut has more flexibility and functionality. You can specify the libraries your project depends on in the file pyproject.toml… Thereafter poetry will install or update them at your request. In addition, this tool allows you to isolate your working project in an isolated environment. Finally, you can use poetry to directly publish your package to Pypi…
The final preparatory step is to install poetry by running the command:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
Project creation
We’ll look at how to create a project and isolate it inside a Python environment using pyenv and poetry…
Installing Python version using pyenv
First, let’s create a directory called my_awesome_project and move to it:
mkdir my_awesome_project && cd $_
Once inside the project folder, install the local version of Python that we’ll be using (we’ll be using Python 3.8.7). This will indicate poetry use a local Python version defined by pyenv:
pyenv local 3.8.7
This will create a file .python-version inside our project. This file will be read pyenv and will force it to install a specific local Python version. As a result, every directory or file created after that will depend on the local version of Python, not the global one.
Building a project with poetry
Poetry provides a robust CLI for building, configuring, and updating your Python project and its dependencies. Use the following command to create a Python project:
poetry new <project_name>
This command generates a default project. The content of the new project is as follows:
.
└── summarize_dataframe
├── README.rst
├── pyproject.toml
├── summarize_dataframe
│ └── init.py
└── tests
├── init.py
└── test_summarize_dataframe.py
Pay attention to the file pyproject.toml… This is where we define all of our project’s metadata, dependencies, scripts, and more.
[tool.poetry]
name = "summarize_dataframe"
version = "0.1.0"
description = ""
authors = [" "]
[tool.poetry.dependencies]
python = "^3.8"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
We see several sections in our standard file pyproject.toml…
[tool.poetry]: This section contains metadata about our package. You can put the package name, short description, author details, version of your project, and so on here. All details are optional here, but will be necessary if you decide to publish the package to Pypi…[tool.poetry.dependencies]: This section contains all the required dependencies for our package. You can specify specific version numbers of these packages (packageX = "1.0.0") or use conventions. It also sets the version of Python that we want to use in the project. In our casepython = "^3.8"defines the minimum version required for our application to work. Here is Python 3.8, this is based on our local version defined withpyenv…[tool.poetry.dev-dependencies]: This section contains all the dependencies needed to develop the project. However, these dependencies are not required to run the application and will not be loaded when the package is built.[build-system]: Do not touch this section if you have not updated the versionpoetry…
You can see the full list of available sections for the file pyproject.toml here…
Installing and activating the virtual environment
There are two approaches here: either you know in advance all the dependencies you need and you can directly modify the file pyproject.toml as appropriate, or you decide to add them later as needed. In our example, we are going to gradually add dependencies as we write the code. Therefore, you only need to initialize the project and create a virtual environment. To do this, you need to run the command in the directory of your project:
poetry install
Creating virtualenv summarize-dataframe in /Users/aabur/Documents/GitHub/AABur/modern_python/summarize_dataframe/.venv
Updating dependencies
Resolving dependencies... (7.3s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (2.4.7)
• Installing attrs (21.2.0)
• Installing more-itertools (8.11.0)
• Installing packaging (21.2)
• Installing pluggy (0.13.1)
• Installing py (1.11.0)
• Installing wcwidth (0.2.5)
• Installing pytest (5.4.3)
Installing the current project: summarize_dataframe (0.1.0)
First, a virtual environment is created. In our case, in the project folder in the directory .venv…
You can create a virtual environment in any other directories. To do this, you need to edit the configuration poetry… For more details follow this documentation…
Then, poetry will read the file pyproject.toml and will install all dependencies specified in this file. If no dependency versions are defined, poetry will download the latest version of the packages. At the end of the operation, a file is created poetry.lock… It contains all packages and their exact versions. Remember that if the file poetry.lock already exists, the version numbers defined in it take precedence over those defined in the file pyproject.toml… Finally, you must save the file poetry.lock in the project repository so that all contributors working on the project use the same version of the dependencies.
Now, let’s activate the environment we just created with the following command:
poetry shell
Spawning shell within /Users/aabur/Documents/GitHub/AABur/modern_python/summarize_dataframe/.venv
The command creates a child process that inherits from the parent Shell, but does not change its environment. It isolates and protects any changes you make to the project environment.
Creating a git repository
At the last stage, we will create a git repository, add the files README.md and .gitignore and push everything to our remote repository.
git init
echo ".*n!.gitignore" > .gitignore
echo "# Summarize dataframe" > README.md
git add .
git commit -m "build: first commit. Environment built"
git remote add origin git@github.com:AABur/summarize_dataframe.git
git branch -M main
git push -u origin main
Conclusion
Here we have covered how to install and manage different versions of Python on our machine using pyenv… We have demonstrated how to use pyenv local to install a specific version of Python in your project and then create a virtual environment with poetry… Usage poetry greatly simplifies the process of creating a project, offering simple and varied tools for customizing it. In addition, it includes the minimum build system requirements as defined in PEP 518…
Brief Memo
pyenv
pyenv install --list
pyenv global
pyenv local
poetry
poetry new
poetry install
poetry shell
