Task runners

Task runners#

A task runner, like make (fully general), rake (Ruby general), invoke (Python general), tox (Python packages), or nox (Python semi-general), is a tool that lets you specify a set of tasks via a common interface. These can be a crutch, allowing poor packaging practices to be employed behind a custom script, and they can hide what is actually happening.

However, there is a tool that is really useful: nox. Nox has two strong points that help with the above concerns. First, it is very explicit, and even prints what it is doing as it operates. Unlike the older tox, it does not have any implicit assumptions built-in. Second, it has very elegant built-in support for both virtual and Conda environments. This can greatly reduce new contributor friction with your codebase.

A daily developer is not expected to use nox for simple tasks, like running tests or linting. You should not rely on nox to make a task that should be made simple and standard (like building a package) complicated. You are not expected to use nox for linting on CI, or often even for testing on CI, even if those tasks are provided for users. Nox is a few seconds slower than running directly in a custom environment - but for new users, and rarely run tasks, it is much faster than explaining how to get setup or manually messing with virtual environments. It is also highly reproducible, creating and destroying the temporary environment each time.

You should use nox to make it easy and simple for new contributors to run things. You should use nox to make specialized developer tasks easy. You should use nox to avoid making single-use virtual environments for docs and other rarely run tasks.

Nox doesn’t handle binary builds very well, so for compiled projects, it might be best left to just specialized tasks.

Nox#

Installing#

Installing nox should be handled like any other Python application. You should either use a good package manager, like brew on macOS, or you should use pipx; either permanently (pipx install nox) or by running pipx run nox instead of nox.

Introduction#

Nox is a tool for running tasks, called “sessions”, inside temporary virtual environments. It is configured through Python and is designed to resemble pytest. The file it looks for is called noxfile.py by default. This is an example of a simple nox file:

import nox


@nox.session
def tests(session: nox.Session) -> None:
    """
    Run the unit and regular tests.
    """
    session.install(".[test]")
    session.run("pytest", *session.posargs)

This will create a session called tests. The function receives the “session” argument, which gives you access to the virtual environment it creates. You can use .install() to install inside the environment, and .run() to run inside the environment. We are also using session.posargs to allow extra arguments to be passed through to pytest. There are more useful methods as well.

You can run this using:

$ nox -s tests

You can see all defined sessions (along with the docstrings) using:

$ nox -l

It is a good idea to list the sessions you want by default by setting nox.options.sessions near the top of your file:

nox.options.sessions = ["lint", "tests"]

This will keep you from running extra things like docs by default.

Parametrizing#

You can parametrize sessions by any item, for example Python version.

# Shortcut to parametrize Python
@nox.session(python=["3.9", "3.10", "3.11", "3.12", "3.13"])
def my_session(session: nox.Session) -> None: ...


# General parametrization
@nox.session
@nox.parametrize("letter", ["a", "b"], ids=["a", "b"])
def my_session(session: nox.Session, letter: str) -> None: ...

The optional ids= parameter can give the parametrization nice names, like in pytest.

If a user does not have a particular version of Python installed, it will be skipped. You can use a Docker container to run in an environment where all Python’s (3.6+) are available:

$ docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_28_x86_64:latest pipx run nox

Useful sessions#

Things like bumping the versions can be made sessions - since nox handles the environment for you, you can use any Python dependencies you like, and not have to worry about installing anything. Here are some commonly useful sessions that will likely look similar across different projects:

Lint#

Developers should be using pre-commit directly, but this helps new users.

@nox.session
def lint(session: nox.Session) -> None:
    """
    Run the linter.
    """
    session.install("pre-commit")
    session.run(
        "pre-commit", "run", "--show-diff-on-failure", "--all-files", *session.posargs
    )

Tests#

import nox


@nox.session
def tests(session: nox.Session) -> None:
    """
    Run the unit and regular tests.
    """
    session.install(".[test]")
    session.run("pytest", *session.posargs)

Docs#

@nox.session
def docs(session: nox.Session) -> None:
    """
    Build the docs. Pass "serve" to serve.
    """

    session.install(".[docs]")
    session.chdir("docs")
    session.run("sphinx-build", "-M", "html", ".", "_build")

    if session.posargs:
        if "serve" in session.posargs:
            print("Launching docs at http://localhost:8000/ - use Ctrl-C to quit")
            session.run("python", "-m", "http.server", "8000", "-d", "_build/html")
        else:
            print("Unsupported argument to docs")

This supports setting up a quick server as well, run like this:

$ nox -s docs -- serve

Build (pure Python)#

For pure Python packages, this could be useful:

from pathlib import Path
import shutil

DIR = Path(__file__).parent.resolve()


@nox.session
def build(session: nox.Session) -> None:
    """
    Build an SDist and wheel.
    """

    build_p = DIR.joinpath("build")
    if build_p.exists():
        shutil.rmtree(build_p)

    session.install("build")
    session.run("python", "-m", "build")

Examples#

A standard powered by nox package in Pure Python in Scikit-HEP is Hist.

A package that happens to use PDM (like Poetry but better) is Scikit-HEP UHI, which is powered by nox. Nox can setup a conda environment with ROOT (slow, but only nox and conda are required). There also is a version bump session, and does some custom logic too.

The complex testing procedure powering Scikit-HEP Cookie is powered by nox. It allows the complex CI jobs that generate projects and lint/test/build them to be run locally with no other setup.

PyPA’s cibuildwheel also is powered by nox, running pip-tools’ compile on every Python version to pin dependencies, as well as providing a standard interface to update Python and project listing update scripts. The docs job there runs mkdocs instead of Sphinx. Other PyPA projects using nox include pip, pipx, manylinux, packaging, and packaging.python.org.

Contents