11. Using packages#
When you are programming, usually you are solving a problem - the programming is simply a means to an end. If that’s the case, then packaging is a means to a means to an end.
Packaging is of my favorite subjects, because it’s absolutely critical as soon as you:
Work on more than one thing
Share your work with anyone (even if not as a package)
Work in more than one place
Upgrade or change anything on your computer
Unfortunately, packing has a lot of historical cruft, bad practices that have easy solutions today but are still propagated.
We will split our focus into two situations, then pull both ideas together.
11.1. Installing a package#
You will see two very common recommendations:
pip install <package> # Use only in virtual environment!
pip install --user <package> # Almost never use
Don’t use them unless you know exactly what you are doing! The first one will try to install globally, and if you don’t have permission, will install to your user site packages (as of a recent pip update). In global site packages, you can get conflicting versions of libraries, you can’t tell what you’ve installed for what, it’s a mess. And user site packages are worse, because all installs of Python on your computer share it, so you might override and break things you didn’t intend to.
The solution depends on what you are doing:
11.1.1. Safe libraries#
There are likely a few libraries (possibly one) that you just have to install globally. Go ahead, but be careful (and always use your system package manager instead if you can, like brew on macOS or the Windows ones - Linux package managers tend to be too old to use for Python libraries).
Ideas for safe libraries: the other libraries you see listed in this lesson! It’s likely better than bootstrapping them. In fact, you can get away with just one:
11.1.2. pipx: pip for executables!#
If you are installing an “application”, that is, it has a script end-point and you don’t expect to import it, do not use pip; use pipx. It will isolate it in a virtual environment, but hide all that for you, and then you’ll just have an application you can use with no global/user side effects!
pip install pipx # Easier to install like this
pipx install black
black myfile.py
Now you have “black”, but nothing has changed in your global site packages! You cannot import black or any of it’s dependencies! There are no conflicting requirements (more common in pip 20.3+, which now will refuse to install two packages that have incompatible requirements).
11.1.2.1. Directly running applications#
Pipx also has a very powerful feature: you can install and run an application in a temporary environment!
For example, this works just as well as the second two lines above:
pipx run black myfile.py
The first time you do this, pipx create a venv and puts black in it, then runs it. If you run it again, it will reuse the cached environment if it hasn’t been cleaned up yet, so it’s fast.
Another example:
pipx run build
This is great for CI! Pipx is installed by default in GitHub Actions (GHA); you do not need
actions/setup-pythonto run it.
If the command and the package have different names, then you may have to write this with a --spec, though pipx has a way to customize this, and it will try to guess if there’s only one command in the package. You can also pin exactly, specify extras, etc:
pipx run --spec cibuildwheel==2.14.1 cibuildwheel --platform linux
11.1.3. Environment tools#
There are other tools we are about to talk about, like virtualenv, poetry, pipenv, nox, tox, etc. that you could also install with pip (or better yet, with pipx), and are not too likely to interfere or break down if you use pip. But keep it to a minimum or use pipx.
11.1.4. Nox and Tox#
You can also use a task runner tool like nox or tox. These create and manage virtual environment for each task (called sessions in nox). This is a very simple way to avoid making and entering an environment, and is great for less common tasks, like scripts and docs.
11.1.5. Python launcher#
The Python launcher for Unix (a Rust port of the one bundled with Python on Windows by a Python core developer) supports virtual environments in a .venv folder. So if you make a virtual environment with python -m venv .venv or virtualenv .venv, then you can just run py <stuff> instead of python <stuff> and it uses the virtual environment for you. This feature has not been back-ported to the Windows version yet.
11.2. Environments#
There are several environment systems available for Python, and they generally come in two categories. The Python Packaging Authority supports PyPI (Python Package Index), and all the systems except one build on this (usually by pip somewhere). The lone exception is Conda, which has a completely separate set of packages (often but not always with matching names).
11.2.1. Environment specification#
All systems have an environment specification, something like this:
requests
rich >=9.8
This is technically a valid requirements.txt file. If you wanted to use it, you would do:
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
Use deactivate to “leave” the virtual environment.
These two tools (venv to isolate a virtual environment) and the requirements file let you set up non-interacting places to work for each project, and you can set up again anywhere.
11.2.2. Locking an environment#
But now you want to share your environment with someone else. But let’s say rich updated and now something doesn’t work. You have a working environment (until you update), but your friend does not, theirs installed broken (this just happened to me with IPython and jedi, by the way). How do you recover a working version without going back to your computer? With a lock file! This would look something like this:
requests ==2.25.1
rich ==9.8.0
typing-extensions ==3.7.4
...
This file lists all installed packages with exact versions, so now you can restore your environment if you need to. However, managing these by hand is not ideal and easy to forget. If you like this, pdm, poetry, and pipenv. You can look into it off-line, but we are moving on. We’ll encounter this idea again.
11.2.3. Dev environments or Extras#
Some environment tools have the idea of a “dev” environment, or optional components to the environment that you can ask for. Look for them wherever fine environments are made.
When you install a package via pip or any of the (non-locked) methods, you can also ask for “extras”, though you have to know about them beforehand. For example, pip install rich[jupyter] will add some extra requirements for interacting with notebooks. These add requirements only, you can’t change the package with an extra.
11.2.4. Conda environments#
If you use Conda, the environment file is called environment.yaml. The one we are using can be seen here:
import rich
from rich.syntax import Syntax
from pathlib import Path
filepath = Path("../environment.yml")
rich.print(Syntax(filepath.read_text(), "yaml", theme="default"))
name: level-up-your-python channels: - conda-forge dependencies: - black - ipykernel >=6.0 - ipython >=8.0 - ipywidgets - jupyter-book - jupyterlab >=3 - line_profiler - matplotlib >=3.5 - mypy >=0.9r760 - nb_conda_kernels - numba >=0.55 - numpy >=1.20 - pandas >=1.3 - pip - pybind11 >=2.10 - pytest >7 - pytest-mock - python ==3.10.* - rich >=11
You can specify pip dependencies, too:
- pip:
- i_couldnt_think_of_a_library_missing_from_conda