14. pytest: Unit Testing#
# Small local extension
%load_ext save_and_run
I highly recommend taking some time to learn advanced pytest, as anything that makes writing tests easiser enables more and better testing, which is always a plus!
14.1. Tests should be easy#
Always use pytest. The built-in unittest is very verbose; the simpler the writing of tests, the more tests you will write!
%%save_and_run unittest
import unittest
class MyTestCase(unittest.TestCase):
def test_something(self):
x = 1
self.assertEqual(x, 2)
F
======================================================================
FAIL: test_something (tmp.MyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/level-up-your-python/level-up-your-python/notebooks/tmp.py", line 6, in test_something
self.assertEqual(x, 2)
AssertionError: 1 != 2
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
Contrast this to pytest:
%%save_and_run pytest
def test_something():
x = 1
assert x == 2
============================= test session starts ==============================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/runner/work/level-up-your-python/level-up-your-python/notebooks
plugins: mock-3.12.0, anyio-4.1.0
collected 1 item
tmp.py F [100%]
=================================== FAILURES ===================================
________________________________ test_something ________________________________
def test_something():
x = 1
> assert x == 2
E assert 1 == 2
tmp.py:3: AssertionError
=========================== short test summary info ============================
FAILED tmp.py::test_something - assert 1 == 2
============================== 1 failed in 0.06s ===============================
pytest still gives you clear breakdowns, including what the value of x actually is, even though it seems to use the Python assert statement! You don’t need to set up a class (though you can), and you don’t need to remember 50 or so different self.assert* functions! pytest can also run unittest tests, as well as the old nose package’s tests, too.
Approximately equals is normally ugly to check, but pytest makes it easy too:
%%save_and_run pytest
from pytest import approx
def test_approx():
.3333333333333 == approx(1/3)
============================= test session starts ==============================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/runner/work/level-up-your-python/level-up-your-python/notebooks
plugins: mock-3.12.0, anyio-4.1.0
collected 1 item
tmp.py . [100%]
============================== 1 passed in 0.01s ===============================
14.2. Tests should test for failures too#
You should make sure that expected errors are thrown:
%%save_and_run pytest --no-header
import pytest
def test_raises():
with pytest.raises(ZeroDivisionError):
1 / 0
============================= test session starts ==============================
collected 1 item
tmp.py . [100%]
============================== 1 passed in 0.00s ===============================
You can check for warnings as well, with pytest.warns or pytest.deprecated_call.
14.3. Tests should stay easy when scaling out#
pytest uses fixtures to represent complex ideas, like setup/teardown, temporary resources, or parameterization.
A fixture looks like a function argument; pytest recognizes them by name:
%%save_and_run pytest --no-header
def test_printout(capsys):
print("hello")
captured = capsys.readouterr()
assert "hello" in captured.out
============================= test session starts ==============================
collected 1 item
tmp.py . [100%]
============================== 1 passed in 0.00s ===============================
Making a new fixture is not too hard, and can be placed in the test file or in conftest.py:
%%save_and_run pytest --no-header
import pytest
@pytest.fixture(params=[1,2,3], ids=["one", "two", "three"])
def ints(request):
return request.param
def test_on_ints(ints):
assert ints**2 == ints*ints
============================= test session starts ==============================
collected 3 items
tmp.py ... [100%]
============================== 3 passed in 0.01s ===============================
We could have left off ids, but for complex inputs, this lets the tests have beautiful names.
Now you will get three tests, test_on_ints-one, test_on_ints-two, and test_on_ints-three!
Fixtures can be scoped, allowing simple setup/teardown (use yield if you need to run teardown). You can even set autouse=True to use a fixture always in a file or module (via conftest.py). You can have conftest.py’s in nested folders, too!
Here’s an advanced example, which also uses monkeypatch, which is a great way for making things hard to split into units into unit tests. Let’s say you wanted to make a test that “tricked” your code into thinking that it was running on different platforms:
%%save_and_run pytest --no-header
import platform
import pytest
@pytest.fixture(params=['Linux', 'Darwin', 'Windows'], autouse=True)
def platform_system(request, monkeypatch):
monkeypatch.setattr(platform, "system", lambda : request.param)
def test_anything():
assert platform.system() in {"Linux", "Darwin", "Windows"}
============================= test session starts ==============================
collected 3 items
tmp.py ... [100%]
============================== 3 passed in 0.01s ===============================
Now every test in the file this is in (or the directory that this is in if in conftest) will run three times, and each time will identify as a different platform.system()! Leave autouse off, and it becomes opt-in; adding platform_system to the list of arguments will opt in.
14.4. Tests should be organized#
You can use pytest.mark.* to mark tests, so you can easily turn on or off groups of tests, or do something else special with marked tests, like tests marked “slow”, for example. Probably the most useful built-in mark is skipif:
%%save_and_run pytest --no-header
import pytest
@pytest.mark.skipif("sys.version_info < (3, 8)")
def test_only_on_37plus():
x = 3
assert f"{x = }" == "x = 3"
============================= test session starts ==============================
collected 1 item
tmp.py . [100%]
============================== 1 passed in 0.00s ===============================
Now this test will only run on Python 3.8, and will be skipped on earlier versions. You don’t have to use a string for the condition, but if you don’t, add a reason= so there will still be nice printout explaining why the test was skipped.
You can also use xfail for tests that are expected to fail (you can even strictly test them as failing if you want). You can use parametrize to make a single parameterized test instead of sharing them (with fixtures). There’s a filterwarnings mark, too.
Many pytest plugins support new marks too, like pytest-parametrize. You can also use custom marks to enable/disable groups of tests, or to pass data into fixtures.
14.5. Tests should test the installed version, not the local version#
Your tests should run against an installed version of your code. Testing against the local version might work while the installed version does not (due to a missing file, changed paths, etc). This is one of the big reasons to use /src/package instead of /package, as python -m pytest will pick up local directories and pytest does not. Also, there may come a time when someone (possibly you) needs to run your tests off of a wheel or a conda package, or in a build system, and if you are unable to test against an installed version, you won’t be able to run your tests! (It happens more than you might think).
14.5.1. Mock expensive or tricky calls#
If you have to call something that is expensive or hard to call, it is often better to mock it. To isolate parts of your own code for “unit” testing, mocking is useful too. Combined with monkey patching (shown in an earlier example), this is a very powerful tool!
Say we want to write a function that calls matplotlib. We could use pytest-mpl to capture images and compare them in our test, but that’s an integration test, not a unit test; and if something does go wrong, we are stuck comparing pictures, and we don’t know how our usage of matplotlib changed from the test report. Let’s see how we could mock it. We will use the pytest-mock plugin for pytest, which simply adapts the built-in unittest.mock in a more native pytest fashion as fixtures and such.
%%save_and_run pytest --no-header --disable-pytest-warnings
import pytest
from pytest import approx
import matplotlib.pyplot
from types import SimpleNamespace
def my_plot(ax):
ax.plot([1,3,2], label=None, linewidth=1.5)
@pytest.fixture
def mock_matplotlib(mocker):
fig = mocker.Mock(spec=matplotlib.pyplot.Figure)
ax = mocker.Mock(spec=matplotlib.pyplot.Axes)
line2d = mocker.Mock(name="step", spec=matplotlib.lines.Line2D)
ax.plot.return_value = (line2d,)
# Patch the main library if used directly
mpl = mocker.patch("matplotlib.pyplot", autospec=True)
mocker.patch("matplotlib.pyplot.subplots", return_value=(fig, ax))
return SimpleNamespace(fig=fig, ax=ax, mpl=mpl)
def test_my_plot(mock_matplotlib):
ax = mock_matplotlib.ax
my_plot(ax=ax)
assert len(ax.mock_calls) == 1
ax.plot.assert_called_once_with(
approx([1.0, 3.0, 2.0]),
label=None,
linewidth=approx(1.5),
)
============================= test session starts ==============================
collected 1 item
tmp.py . [100%]
============================== 1 passed in 1.48s ===============================
We’ve just mocked the parts we touch in our plot function that we need to test. We use spec= to get the mock to just respond to the same things that the original object would have responded to. We can set return values so that our objects behave like the real thing.
If this changes, we immediately know exactly what changed - and this runs instantly, we aren’t making any images! While this is a little work to set up, it pays off in the long run.
The documentation at pytest-mock is helpful, though most of it just redirects to the standard library unittest.mock.