Level Up Your Python#
A course in intermediate Python for a beginner ready to move up. Binder version and Live WebAssembly version available too!
Henry Schreiner
Introduction#
Expected Knowledge#
You should already know:
Basic Python syntax
Functions
Basic classes (will cover advanced usages mostly)
Basic NumPy (will mention, but not cover)
Git - CRITICAL FOR ANY SOFTWARE WORK!
And we will be using notebooks in JupyterLab.
About the author#
Most important link: https://iscinumpy.dev
PyPA member. Scikit-HEP admin, scikit-build admin, member of IRIS-HEP.
Projects#
pybind11 • cibuildwheel • build • scikit-build • boost-histogram • Hist • UHI • Vector • CLI11 • Plumbum • GooFit • Particle • DecayLanguage • Conda-Forge ROOT • POVM • Jekyll-Indico
Interests:#
Packaging and building • Bindings • Building a HEP analysis toolchain in Python, JITable
My books and workshops#
Modern CMake • CMake Workshop • Computational Physics Class • Python CPU, GPU, Compiled minicourses
Tempering your expectations#
What can you expect?#
I don’t know what you know;
We don’t have time to study any topic in depth.
So, we will move fast, and cover a lot.
You are not expected to able to master everything you see.
Instead, you are expected to:#
Know what is possible, so you know to look for it;
Get pointers on where to look (lots of links!);
Refer back to this material later.
Theory: New features in programming#
Programming is all about organization. This is not always obvious, and has some odd consequences. Let’s look at one: new features remove functionality from the user. And that’s a good thing.
Don’t believe me? Pick one. Let’s go with an old, simple one you should already know: goto vs. loops (for/while). (This is my favorite example, even though Python thankfully came along late enough to not even have goto in the first place, except as an April fools joke or a proof of concept library.)
You have total power with goto! Jump from anywhere, to anywhere! You can recreate all loops (for loops, for each loops, while loops, do while (C) loops) with it, and more (like functions)! So why are loops the newer, better feature?
goto (partially hypothetical in Python)
i = 0
label .start
print(f"Hi {i}")
i + 1
if i <= 10:
goto .start
Compare to for loop:
for i in range(10):
print(f"Hi {i}")
Hi 0
Hi 1
Hi 2
Hi 3
Hi 4
Hi 5
Hi 6
Hi 7
Hi 8
Hi 9
A programmer has to spend time to recognize what is happening in the first example - in the second example, even a fairly new Python programmer will immediately say “that prints 0 to 9”. The second example lets you build more complex programs, because you are working at a ‘higher level’, humans tend to do better which high level concepts (while computers work up from low level).
Also, we now need several features to make up for the loss of goto; the for loop, the while loop, and functions. Each is more restricted, with less functionality, but better readability and composability.
We will see lots of examples of this – in section 2, especially.
Notebooks#
We will be using notebooks today. Notebooks are fantastic for teaching, quick experimentation, for developing, or for driving a final analysis product. They are not for serious programming - that happens in .py files. Once you write something and get it working, move it to a .py file and add a test. Then import it into your notebook!
Python version#
We will be using Python 3.11. I’ll try to point out when something is newer than 3.8. NEP 29 mandates that data science libraries currently support 3.9+ (support dropped 42 months after release), while general Python EOL is 3.8+ (5 year support window). IPython and NumPy have already dropped Python 3.8.
Key upcoming dates:
Python |
Release |
NEP 29 drop |
General EOL |
|---|---|---|---|
~~Python 3.7~~ |
~~Jun 2018~~ |
~~Dec 2021~~ |
~~Jun 2023~~ |
Python 3.8 |
Oct 2019 |
~~Apr 2023~~ |
Oct 2024 |
Python 3.9 |
Oct 2020 |
Apr 2024 |
Oct 2025 |
Python 3.10 |
Oct 2021 |
Apr 2025 |
Oct 2026 |
Python 3.11 |
Oct 2022 |
Apr 2026 |
Oct 2027 |
Python 3.12 |
Oct 2023 |
Apr 2027 |
Oct 2028 |
Python 3.X |
Oct 2011+X |
Apr 2015+X |
Oct 2016+X |
Since Python 3.8, Python releases yearly, so you can expect a new Python release every October, and an EOL every December (April a year before for data science).
Note that SPEC 0 seems to be replacing NEP 29, and it has an even shorter support cycle, 36 months. So releases get dropped in October three years after they debut.
Extra: saving and running a file from Jupyter#
For teaching purposes, we will be running some tools (pytest and mypy) from notebooks; this is not what they were designed to do, so we will use the following small extension to save a cell to a file and then run it with a Python module. I’m using a third-party library, rich, to render this with nice syntax highlighting in the notebook.
import rich
from rich.syntax import Syntax
from pathlib import Path
filepath = Path("save_and_run.py")
rich.print(Syntax(filepath.read_text(), "python", theme="default"))
from IPython.core.magic import Magics, magics_class, cell_magic import sys import subprocess from pathlib import Path @magics_class class AutoMagics(Magics): @cell_magic def save_and_run(self, line, cell, local_ns=None): commands = line.split() filename = "tmp.py" Path(filename).write_text(cell) subprocess.run( [sys.executable, "-m", *commands, filename], # This is just to support colors in the notebook env={"FORCE_COLOR": "1", "MYPY_FORCE_COLOR": "1", "TERM": "xterm-color"}, ) # Why sys.executable here? # # We are running inside a virtual environment, which also has mypy installed. # However, this is a shell command (the starting `!`), so it will not # necessarily run in the same environment. So we'll use the current python # interpreter `sys.executable` and use the `python -m mypy` expression instead # of plain `mypy` to make sure we get our installed MyPy. You don't usually run # mypy from a notebook. def load_ipython_extension(ipython): ipython.register_magics(AutoMagics)