12. Creating packages#

Now, let’s change gears and look at creating our own packages. If you want to make your code accessible to someone else to use via pip install, you need to make it a package. In fact, as you’ll see at the end of this section, even if you just want to develop an application, it’s much better to be working in a package. I won’t show you the internals of setting up a setuptools package, but we’ll just go over how you work with it and how it is distributed.

To install a local package, use:

pip install .

This will copy the files into site-packages. If you want to actively develop a module, use this instead (setuptools only, command varies on other tools):

pip install -e .

This uses symlink(s) so that you can edit the local files and immediately see the changes (after restarting Python, as usual).

If you want to produce an SDist for distributing the source, use

pip install build
python -m build --sdist

If you want to produce a wheel for distributing, use

python -m build --wheel

You’ll see old tutorials directly call python setup.py ...; if you can possibly avoid doing that, please do! The setup.py file is still a good idea for setuptools, but it’s not even required there (and doesn’t exist for any other packaging software). (It’s also quite valid to use pipx to install build, but remember the command is pyproject-build if you do that).

12.1. Distributions#

12.1.1. Wheel: fast and simple#

A wheel is just a normal zipped file with the extension .whl. It contains folders that get copied to specific locations, and a metadata folder.

It does not contain setup.py/setup.cfg/pyproject.toml.

Why use wheels?

  • Secure installs - arbitrary code does not run

  • Fast installs - files are just copied inplace

  • Reliable - does not depend on pretty much anything being on user’s machine, including setuptools!

  • Faster first imports - pip makes .pyc files when it installs

  • Can be tagged for Python version, OS, and/or architecture (supports binaries!).

See https://pythonwheels.com

12.1.2. SDist: Source distribution#

This is a .tar.gz file holding the files needed to make a wheel. It is often a subset of the files in the GitHub repo, though sometimes it contains generated files, like version.py or maybe Cython/SWIG generated source files. If there is no matching wheel (only for projects with binary components, in general), then pip gets the SDist and builds/installs manually.

12.2. PDM/Hatch/Flit/Poetry: A breath of fresh air#

See https://learn.scientific-python.org/development/guides/packaging-simple for a complete setup!

Let’s look at an all-in-one solution: PDM. It is a bit younger than Poetry, the current leader of all-in-one solutions, but it follows standards much better. There are some caveats:

  • Should be pure Python (no compiled extensions in your code)

  • Should be PyPI based (no Conda integration AFAIK)

I’m strongly against some of the decisions in Poetry and to a much lesser extent, PDM, along with many of the other PyPA members and Python core developers. These decisions were mostly made for “application” uses, so they are only problematic when making a library. You can avoid them, just follow the following rules:

  • Never add an upper limit to your Python version. ^3.8 should be changed to >=3.8. Poetry will force you to add an upper limit if a package you include does this, though, so the bad practice percolates.

  • Never add an upper limit to a project you don’t heavily depend on unless you know you really don’t support some version. It’s much more likely that you will support the next version than you won’t, and makes a mess for solving this later, and forces you to constantly “maintain” the upper limit.

12.2.1. Step 1: make a new project#

!pdm new tmp_project

/usr/bin/sh: 1: pdm: not found

%ls tmp_project/

ls: cannot access 'tmp_project/': No such file or directory

%cat tmp_project/pyproject.toml

cat: tmp_project/pyproject.toml: No such file or directory

The following commands I’ll demo in a shell, if I have time.

# Create a virtual environment, start the poetry.lock file
pdm install

# "Enter" the environment (Ctrl-D or exit to exit)
pdm shell

# Run without entering the environment
pdm run ...

# Add a new package (--dev to make it development only)
# Modifies your pyproject.toml
pdm add rich

# Update the environment and lock files
pdm update

# You can use python -m build, or you can use pdm build
# You can publish to PyPI with pdm publish
# And that's package + environment management!

When you publish your package, it makes completely normal wheels, so pip install works exactly as expected.

New developers can start developing right away by getting your repository and running pdm install. They even get the dev dependencies by default! (which was a brilliant choice, IMO). They start with the lock file if it exists, so they always get what you have, and anyone can run pdm update if needed.

With PDM, you can even select any PEP 621 backend, including the excellent Hatchling, and the (too) minimal flit-core! Poetry does not support standards like this, at least yet.

12.3. Hatch / hatchling#

The “Hatch” tool is like PDM/Poetry, but is based on multiple environments. This allows it to be a “true” all-in-one tool by replaing nox/tox. It comes with a fantastic “Hatchling” backend that is currently the nicest PEP 517 builder; this is what I nearly always use.

Hatch doesn’t support locking environments yet (was waiting on an official solution, but that’s been hard to agree on). But Hatching is currently the nicest and most extensable PEPL 517 builder available! I’d highly reommend using it (even with PDM, which is what I usually do).

12.4. Flit: Lightweight, (too) simple#

Flit is great for simple projects that don’t need all the bells and wistles. Ironically, it’s currently more stable that setuptools is or will be till Python 3.12, since setuptools is fighting through the distutils deprecation process. The PyPA is likely to start moving some core packages to using Flit. Short guide for Flit:

  • Consider using the flit command line tool for a streamlined experience (though you don’t need to, and I don’t)

  • Use the PEP 621 (new metadata) format - it’s better and can be used more places (like with PDM!)

  • One design feature/problem is that SDists are exactly tars of your repo - there’s no build step. If you need that, look elsewhere. This includes Git-based versioning, sadly..

  • Ahh, another problem: Standards-based SDist builds do not use Git info - so you have to check the files explicitly.

12.5. Setuptools: Classic, powerful, verbose#

The most powerful (and originally, forced by pip) tool is setuptools. This is a collection of hacks built on top of distutils, which is a collections of hacks to build packages (which was the standard library tool that is now deprecated and may be removed in Python 3.12). There are some awful examples around on using it, so look at https://learn.scientific-python.org/development for a proper example.

The short version:

  • Use declarative setup.cfg for everything you can

    • Use file: to read files

    • Always use find: for packages - include or exclude if you need to

    • Always set python_requires!

  • Logic goes in setup.py; often it’s just from setuptools import setup; setup()

    • Binary extensions go here too

    • You don’t need this file at all much of the time.

  • Always include a pyproject.toml, often it’s just 5 or so lines

  • Check your MANIFEST.in to make sure it’s not missing things going into the SDist