Creating a virtual environment for your Python project

In this article we will see how to use virtual environments to manage Python projects. We will see that it is possible to have a different Python version for each project, and also how packages and dependencies are controlled.

If you prefer to watch a video, click the player below (the video is in Portuguese). The full article follows the video.

Why use virtual environments?

#

Like other programming languages, Python has its own way of downloading, storing, and managing packages and modules. Of particular interest to us is understanding how and where these packages are stored.

Most Unix-derived systems, such as macOS and the most common Linux distributions, already come with some Python version preinstalled, since some system tools depend on the language. I am a Linux user and currently have Linux Mint 20 installed, so the examples here come from that distribution, but they can easily be replicated on others and on a Mac. Windows has no system tools that depend on Python, so it usually does not come with any version preinstalled, but installing one is very easy, as a simple internet search will show.

As described above, Linux distributions have tools that depend on Python. Even though version 3 of the language was released in 2008, many projects still use some micro version of Python 2, often for backward-compatibility reasons. The last micro version of Python 2 was 2.7.18, released in April 2020, and its support ended in early 2020. There are several interesting discussions about this long support period for such an old version; I recommend starting here if you are interested in the subject. And if you are not quite sure how version numbers are represented in Python, read here.

It is easy to tell whether both versions are present on your Linux system. In the terminal, type which python and then which python3. The paths returned are the locations of the Python 2 and Python 3 interpreters, respectively.

$ which python3
/usr/bin/python3

In distributions based on Ubuntu 20.04, such as Linux Mint 20, which I use, Python 2 was officially removed from the repositories, but it can still be accessed with the python2 command (and not just python, as was customary in earlier distributions).

By default, the shell that opens when you type python in the terminal in most distributions is Python 2. To open a Python 3 shell, just use the python3 command.

Most system packages are stored in folders inside a system folder. To identify that folder, we can open a Python shell in the terminal and enter the following commands:

>>> import sys
>>> sys.prefix

The result will usually be the system folder /usr. That folder has the following structure, with folders for executables, libraries, and source code.

# tree /usr -L
/usr
├── bin
├── games
├── include
├── lib
├── libexec
├── local
├── sbin
├── share
└── src

You can read the documentation of sys.prefix here.

Third-party packages, usually installed with the pip manager, will be installed in one of the folders given by the commands:

>>> import site
>>> site.getsitepackages()
['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']

Documentation of the site package.

Notice that these are system folders: all projects will use these same folders. Let us consider the following scenario to see why this is a problem.

Suppose you have two projects: Project A and Project B. Both depend on the Pandas library, but Project A is older and was built on version 0.25.3, while Project B is brand new and uses version 1.0.5, which is not fully backward compatible with 0.25.3. So Project A cannot be upgraded without rewriting some pieces of code, which is not feasible if the project is to be delivered on time. As the system stands, a single version of Pandas will be installed for the whole system, in a pandas folder inside the folder returned by the commands above, making it impossible to work on both projects at the same time.

This is where virtual environments come into play.

What is a virtual environment?

#

In essence, virtual environments let you create an isolated environment for each Python project. That way, each project can have its own dependencies, regardless of the dependencies of other projects.

In our example, we just need to create one environment for Project A, where Pandas 0.25.3 will be installed, and another for Project B, where version 1.0.5 will be installed.

There is no limit to how many environments you can create, other than the physical space on your disk, since they are just directories. Let us look at the main tools available for creating and managing virtual environments.

venv

#

The first tool we will see for creating virtual environments is venv. It is part of the Python 3 standard library, so there is nothing to install. In this article we will focus only on Python 3, given that support for Python 2 has been discontinued.

First, we need a folder for each project:

mkdir projects && mkdir projects/project_{A,B}

Structure of a venv environment

#

Now, inside each folder, we will create a virtual environment which, at first, we will think of as a simple folder inside the project. It is common to name this folder env or .env (remember that a leading dot marks a hidden folder). You can use any name, but we will keep to the convention here and use .env. So, inside each project folder, we run the command

python3 -m venv .env

Notice that the command makes it explicit that the interpreter of these environments will be the Python 3 interpreter. Let us look at the structure of the .env directories created in each project:

$ tree .env -L 3
.env
├── bin
    ├── activate
    ├── easy_install
    ├── pip
    ├── python -> python3
    ├── python3 -> /usr/bin/python3
├── include
├── lib
    ├── python3.8
        ├── site-packages
├── lib64 -> lib
├── pyvenv.cfg
└── share
    └── python-wheels

Among these folders, we can highlight:

• bin: files that interact with the virtual environment;
• include: a copy of the Python version along with the site-packages folder where each dependency will be installed.

There are also copies and symbolic links (symlinks) for some tools and the interpreter executables. All of this ensures that Python commands and code run within the context of the current environment, guaranteeing isolation. We will see more details soon.

Also in the bin folder there are activate scripts which, as the name suggests, are used to actually activate the environment so that the shell uses the environment’s Python executable and site-packages folder, not the system’s.

Activating a venv environment

#

To activate the environment, inside the project folder, run

$ source .env/bin/activate
(.env) $

Notice that the prompt now shows the environment name, .env, as a prefix, indicating that it is active.

Understanding how venv works

#

Now, when we run which python, the path that appears refers to the virtual environment and no longer to the system (compare with the output at the beginning of the article). So, in the folder of our Project A:

$ which python
/home/chico/projects/project_A/.env/bin/python

Let us now deactivate the virtual environment. All it takes is the following command

$ deactivate

With the environment deactivated, let us look at the $PATH environment variable. As a reminder, $PATH tells the shell which directories to search for executables in response to user commands. Let us activate the environment again to see how $PATH changed:

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

(.env) $ echo $PATH
/home/chico/projects/project_A/.env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Notice that, with the environment activated, the environment’s bin directory is the first path in $PATH. In practice, this means it will be the first directory the system searches for executables, so the shell will use the environment’s Python instance and not the one available to the whole system.

Solving our scenario

#

In our scenario, we have two projects that depend on the Pandas package, but on different versions of it. In the Project A folder, after activating the environment, we need to run the following command to install the specific version of the package in question:

(.env) $ pip install pandas==0.25.3

The command will install the Pandas package in the requested version and the necessary dependencies, such as the NumPy package. After the installation, we can check the packages in the environment with the command

(.env) $ pip list
Package           Version
----------------- ----------
numpy             1.19.1
pandas            0.25.3
pip               20.0.2
pkg-resources     0.0.0
python-dateutil   2.8.1
pytz              2020.1
setuptools        44.0.0
six               1.15.0

Let us now deactivate the Project A environment and go to the Project B directory, activating the environment of this latter project. Since this project has just started, we want to begin with the latest Pandas version, so there is no need to specify the version: the latest one will be installed, which at the time of writing is 1.1.0.

(.env) $ pip install pandas

Running pip list again, we see that the latest version was indeed installed. So the two projects are isolated and have the versions they need.

(.env) $ pip list
Package           Version
----------------- ----------
numpy             1.19.1
pandas            1.1.0
pip               20.0.2
pkg-resources     0.0.0
python-dateutil   2.8.1
pytz              2020.1
setuptools        44.0.0
six               1.15.0

Sharing a venv environment

#

In an even more realistic scenario, the projects would have many more dependencies, and installing them one by one, especially when a project is already structured as in the case of Project A, is unproductive. So, in projects that use venv, it is quite common to have a requirements.txt file. To generate the file from an already structured environment:

(.env) $ pip freeze > requirements.txt

In our simplified case of Project A, this is the structure of the file:

(.env) $ cat requirements.txt
numpy==1.19.1
pandas==0.25.3
python-dateutil==2.8.1
pytz==2020.1
six==1.15.0

It is a very simple structure, showing the project’s dependencies and their versions.

To create an environment from the file, you create a folder for the project with an environment as per the instructions above, copy the requirements.txt file to the folder, activate the environment, and then run the command:

(.env) $ pip install -r requirements.txt

That way, all collaborators on the project can have the same development environment, quickly and easily.

Making the environment show up in Jupyter Notebook

#

If the project uses Jupyter Notebook, for the environment’s interpreter and packages to be loaded you need to install and run, inside the environment:

(.env) $ pip install ipykernel
(.env) $ ipython kernel install --user --name=project_name

Removing the environment

#

Since an environment created with venv is built around the virtual environment folder, to delete the environment you just deactivate it and remove the folder:

(.env) $ deactivate
$ rm -r .env

conda

#

Many people start with Python through distributions such as Anaconda, which aim to simplify the management and development of Python projects, especially those aimed at the scientific field. Those who start this way end up coming into contact with the conda manager, which, by the way, works for any language, not just Python. conda can also manage virtual environments. These last sentences already highlight important differences between conda and pip, which many beginners wrongly consider equivalent, but there are others that go beyond the scope of this article, so I strongly recommend this article on the Anaconda project website. It is also worth noting that the approach and structure of conda virtual environments differ significantly from those of venv, so I advise reading the documentation of conda, which presents these differences in detail and describes the contexts where one approach or the other is more suitable.

We have already covered installing Anaconda here on the site. Check it out at this link. Since the focus of this article is creating and managing virtual environments, let us get straight to the point.

By default, environments created by conda are installed in the envs folder inside the conda directory. This way, the commands below can be run in any directory, since they do not affect it. You can change this behavior to have the environment inside a folder in your project, similar to venv, but there are some considerations about this change that go beyond the scope of this article, so I recommend reading the documentation on the subject.

Understanding the base environment

#

When the Anaconda installation finishes, a virtual environment containing the Python interpreter and the installation’s packages, named base, is created automatically. The default installation settings make the environment activate automatically, so when you open the terminal it should show the environment in the prompt prefix. We can check which Python interpreter will be used:

(base) $ which python
/opt/anaconda/bin/python

As expected, the path shown is the Anaconda installation directory.

If you want to list all the packages installed by Anaconda:

(base) $ conda list

The result should be a long list of packages.

Creating and activating a conda environment

#

Considering our two-project scenario, let us create an environment for Project A. To create an environment:

(base) $ conda create --name env_project_A

An environment created this way will be completely empty, with no packages and no interpreter. Once you have an environment created, to activate it all you need is:

$ conda activate env_project_A

With the environment activated, we can verify that it is indeed empty:

(env_project_A) $ which python
(env_project_A) $ conda list

Neither command returns anything.

Installing a Python interpreter in a conda environment

#

To install a Python interpreter in the environment you created:

(env_project_A) $ conda install python

Installing this way will install the latest Python version available in the conda repositories. If you want a specific version, just specify it:

(env_project_A) $ conda install python=3.7

With an interpreter installed, we can check its path:

(env_project_A) $ which python
/home/chico/.conda/envs/env_project_A/bin/python

Installing dependencies in a conda environment

#

To install packages in the environment, just pass the environment name and the package name. If only the name is passed, the latest version of the package available in the conda repository will be installed. If you want a specific version, just pass the version number. Take a look:

(env_project_A) $ conda install -n env_project_A scipy
(env_project_A) $ conda install -n env_project_A pandas=0.25.3

Creating a conda environment with interpreter and dependencies

#

Everything we have done so far could be done in a single command when creating the environment. That is, you can create an environment and already pass it the Python version to be used and the packages you want:

$ conda create --name env_project_A python=3.7 scipy pandas=0.25.3

Use the conda list command to check the installed packages and their respective versions.

Here we already see a big advantage of conda: the ability to install environments with different Python versions. Another advantage is that conda automatically checks for dependency conflicts among the packages to be installed, which is very useful in large projects with many dependencies, especially when updating them.

Checking existing conda environments

#

To check which environments managed by conda exist:

(base) $ conda info --envs
# conda environments
#
env_project_A          /home/chico/.conda/envs/env_project_A
env_project_B          /home/chico/.conda/envs/env_project_B
base                 *  /opt/anaconda

The environment marked with * is the one currently active.

Sharing a conda environment

#

After setting up the environment with the packages you are interested in, it is worth exporting these settings so that others involved in the project can have an identical development environment, or so that you can replicate the environment at some point in the future. conda has a logic similar to venv and its requirements.txt file, except that in the case of conda the format is YAML and the file structure is a bit different to conform to the way conda works. To generate this file the environment must be active and, usually, the file is named environment.yml, although it can have any name:

(env_project_A) $ conda env export > environment.yml

Let us look at the structure of this file:

name: env_project_A
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - blas=1.0=mkl
  - ca-certificates=2020.6.24=0
  - certifi=2020.6.20=py37_0
  - intel-openmp=2020.1=217
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - mkl=2020.1=217
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.1.0=py37h23d657b_0
  - mkl_random=1.1.1=py37h0573a6f_0
  - ncurses=6.2=he6710b0_1
  - numpy=1.19.1=py37hbc911f0_0
  - numpy-base=1.19.1=py37hfa32c7d_0
  - openssl=1.1.1g=h7b6447c_0
  - pandas=0.25.3=py37he6710b0_0
  - pip=20.2.1=py37_0
  - python=3.7.7=hcff3b4d_5
  - python-dateutil=2.8.1=py_0
  - pytz=2020.1=py_0
  - readline=8.0=h7b6447c_0
  - setuptools=49.2.1=py37_0
  - six=1.15.0=py_0
  - sqlite=3.32.3=h62c20be_0
  - tk=8.6.10=hbc83047_0
  - wheel=0.34.2=py37_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
prefix: /home/chico/.conda/envs/env_project_A

It is a somewhat more complex structure than the requirements.txt we saw earlier with venv. It shows the environment name, the channel to download packages from, the Python interpreter version for the environment, and the path the file was exported from.

To create an environment from a previously created environment file:

$ conda env create -f environment.yml

Deactivating and removing a conda environment

#

To deactivate the environment in use:

(env_project_A) $ conda deactivate

To completely delete an environment and the packages installed in it:

conda remove --name your_env_name --all

pyenv

#

Even though we have seen that the operating system already comes with Python, the available version is often not the latest. And it is never a good idea to directly upgrade the operating system’s version, because some applications may break, especially if they depend on a language feature that was removed in a later update with no backward-compatibility option. We have already seen that conda lets you install different Python versions per environment, but it is not the only option. Now we will talk about pyenv.

The project repository is very instructive, even explaining some things already covered in this article, such as how $PATH works.

Installing pyenv

#

It is advisable to check the repository for the installation instructions, because the project needs some dependencies to work. I will list below the ones required at the time of writing, but I recommend checking the repository to make sure nothing has changed:

$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev python-openssl git

$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

At the end of the previous command, instructions appear on the screen to be added to the configuration file of bash (or of any other shell you use, such as ZSH). So, at the end of your .bashrc file, or equivalent, add what appears on the screen, which should be something similar to the following:

export PATH="~/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Restart your shell for the changes to take effect:

$ exec $SHELL

Checking whether it was installed:

$ pyenv

If all goes well, the usage instructions and the main available commands will appear on the screen.

Using pyenv

#

To check all the interpreter versions available for installation:

$ pyenv install -l

Notice that there are many versions, including some still under development (with the -dev suffix).

To install a specific version:

$ pyenv install 3.8.5

You can check the versions available on the system:

$ pyenv versions
* system
3.8.5

Notice that it highlights which version is the system’s original (system). The version marked with an asterisk is the one currently active.

To avoid the risk of modifying the system version, you can set the default Python of your choice:

$ pyenv global 3.8.5

You may need to restart the terminal to see the effect of the command above. Confirming which Python is active:

$ which python
/home/chico/.pyenv/shims/python

$ python -V
Python 3.8.5

It is also worth checking that pip is using the same path shown by the which python command:

$ pip --version

That way, we make sure the installed packages are really tied to the Python we want and not to the system Python.

Now you can combine this pyenv feature, which gives you several Python versions if you need them, with venv to have environments with those versions.

pipenv

#

Last but not least, we have pipenv. As you can see in the project repository, it is a very ambitious tool that aims to:

• unify the behavior of pip and venv;
• fix the problems that can occur in projects with requirements.txt;
• increase security by using hashes (cryptographic signatures);
• detail the project’s dependencies clearly;
• separate development dependencies from runtime dependencies.

Installing pipenv

#

Installation is quite simple:

$ pip install pipenv

To confirm that the installation was done correctly, just use the pipenv command. A list of the main commands will be shown.

Keeping the pipenv environment folder in the project folder

#

Like conda, pipenv by default sets up all environments in a system folder. Some people may prefer each environment to stay in the folder of its project. To do that, add the following line to ~/.bashrc (or the equivalent for your shell):

export PIPENV_VENV_IN_PROJECT=1

Restart the shell or run source ~/.bashrc for the change to take effect. Throughout the rest of this article, I will assume this was done and, therefore, that the environment folder is inside the project folder.

Creating a virtual environment with pipenv

#

To use it, enter the project folder. Now, the following command will already create a virtual environment for the project:

$ pipenv install

An environment will be created with the Python interpreter active at the time of creation. We can verify that the environment was created by looking at the new items in the project folder:

$ ls -l -a
Pipfile
Pipfile.lock
.venv

Structure of a Pipfile

#

There is a .venv folder, whose structure is similar to the one we described when talking about venv, and two Pipfile files. We can think of these files as similar to the requirements.txt or environment.yml files mentioned in other sections. However, the way pipenv manages dependencies is more robust.

First let us investigate the Pipfile file:

$ cat Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version="3.8"

We see that it has the repository source, PyPI; the parameter that says to use SSL; and the Python version. Notice that there are sections, still blank because this is a new project, that separate the development packages from the project’s actual packages.

Installing a dependency with pipenv

#

To install a dependency, for which we will use Pandas in our Project A as an example, we run:

pipenv install pandas==0.25.3

If no specific version were used, the latest version would be installed.

Let us look at the Pipfile again:

$ cat Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
pandas = "==0.25.3"

[requires]
python_version="3.8"

We see that it lists the Pandas package with the requested version. If no version had been passed, a “*” would be shown next to the package name.

We know that Pandas has its own dependencies and that these were certainly installed too. To see how these dependencies relate to each other and to Pandas:

$ pipenv graph
pandas==0.25.3
  - numpy [required: >=1.13.3, installed: 1.19.1]
  - python-dateutil [required: >=2.6.1, installed: 2.8.1]
    - six [required: >=1.5, installed: 1.15.0]
  - pytz [required: >=2017.2, installed: 2020.1]

Notice that we now see all the Pandas dependencies and which version ranges are compatible. This enables the so-called deterministic build, where you know exactly all the versions used in the project. This management is very important when you develop locally and plan to deploy to production exactly what was developed locally. For this fine-grained control, we have the Pipfile.lock file, a long file describing all direct and indirect dependencies, with their cryptographic signatures, sources, and versions.

Installing development dependencies

#

Over the course of a project, we will have some dependencies that are useful only during development and do not need to go to production. Examples are testing libraries such as pytest, or libraries we use to keep the code style in line with the language standards, such as flake8. To install a dependency for development only:

$ pipenv install --dev pytest
$ pipenv install --dev flake8

Such dependencies are in the [dev-packages] section of the Pipfile.

$ cat Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"
flake8 = "*"

[packages]
pandas = "==0.25.3"

[requires]
python_version="3.8"

Sharing a pipenv environment

#

To start a project from a Pipfile, as when, for example, we clone a project from GitHub, we run:

$ pipenv install -d

Notice that the -d flag was passed, indicating that development dependencies will also be installed. If that is not necessary, just drop the flag.

When we run the previous command, the whole dependency graph will be rebuilt. If you want a deterministic build with the exact versions present in the Pipfile.lock file, we use the sync command, which has the -d flag to also install development dependencies:

$ pipenv sync -d

Activating a pipenv environment

#

pipenv has its own shell that can be activated:

$ pipenv shell
(project A) $

Now we can use the installed libraries

(project A) $ pytest .

To deactivate, just use the exit command.

If you want to run a command without having to enter and leave the shell, we can use run:

$ pipenv run pytest .

Conclusion

#

In this article we saw how Python dependencies are stored and managed by different tools, as well as how we can have different Python versions for each project.

I tried to include what I consider the minimum necessary to understand virtual environments, their importance, and the basics of using them. The subject is very broad and there are still more tools and approaches that I did not mention here, such as Poetry, because I chose to write about the ones I am most familiar with from having used them in projects. Part of your progress is getting to know the tools, their limitations, and the contexts where they are most suitable. It is also very important to consult the documentation of each tool.