Level Up Your Python#

A course in intermediate Python for a beginner ready to move up. Binder version and Live WebAssembly version available too!

Henry Schreiner


Expected Knowledge#

You should already know:

  • Basic Python syntax

  • Functions

  • Basic classes (will cover advanced usages mostly)

  • Basic NumPy (will mention, but not cover)


And we will be using notebooks in JupyterLab.

About the author#

Henryiii's github stats

Most important link: https://iscinumpy.dev

PyPA member. Scikit-HEP admin, scikit-build admin, member of IRIS-HEP.


pybind11cibuildwheelbuildscikit-buildboost-histogramHistUHIVectorCLI11PlumbumGooFitParticleDecayLanguageConda-Forge ROOTPOVMJekyll-Indico


Packaging and building • Bindings • Building a HEP analysis toolchain in Python, JITable

My books and workshops#

Modern CMakeCMake WorkshopComputational Physics Class • Python CPU, GPU, Compiled minicourses

Tempering your expectations#

What can you expect?#

  • I don’t know what you know;

  • We don’t have time to study any topic in depth.

So, we will move fast, and cover a lot.

You are not expected to able to master everything you see.

Instead, you are expected to:#

  1. Know what is possible, so you know to look for it;

  2. Get pointers on where to look (lots of links!);

  3. Refer back to this material later.

Theory: New features in programming#

Programming is all about organization. This is not always obvious, and has some odd consequences. Let’s look at one: new features remove functionality from the user. And that’s a good thing.

Don’t believe me? Pick one. Let’s go with an old, simple one you should already know: goto vs. loops (for/while). (This is my favorite example, even though Python thankfully came along late enough to not even have goto in the first place, except as an April fools joke or a proof of concept library.)

You have total power with goto! Jump from anywhere, to anywhere! You can recreate all loops (for loops, for each loops, while loops, do while (C) loops) with it, and more (like functions)! So why are loops the newer, better feature?

goto (partially hypothetical in Python)

i = 0
label .start
print(f"Hi {i}")
i + 1
if i <= 10:
    goto .start

Compare to for loop:

for i in range(10):
    print(f"Hi {i}")
Hi 0
Hi 1
Hi 2
Hi 3
Hi 4
Hi 5
Hi 6
Hi 7
Hi 8
Hi 9

A programmer has to spend time to recognize what is happening in the first example - in the second example, even a fairly new Python programmer will immediately say “that prints 0 to 9”. The second example lets you build more complex programs, because you are working at a ‘higher level’, humans tend to do better which high level concepts (while computers work up from low level).

Also, we now need several features to make up for the loss of goto; the for loop, the while loop, and functions. Each is more restricted, with less functionality, but better readability and composability.

GOTO xkcd comic

We will see lots of examples of this – in section 2, especially.


We will be using notebooks today. Notebooks are fantastic for teaching, quick experimentation, for developing, or for driving a final analysis product. They are not for serious programming - that happens in .py files. Once you write something and get it working, move it to a .py file and add a test. Then import it into your notebook!

Python version#

We will be using Python 3.11. I’ll try to point out when something is newer than 3.8. NEP 29 mandates that data science libraries currently support 3.9+ (support dropped 42 months after release), while general Python EOL is 3.8+ (5 year support window). IPython and NumPy have already dropped Python 3.8.

Key upcoming dates:



NEP 29 drop

General EOL

~~Python 3.7~~

~~Jun 2018~~

~~Dec 2021~~

~~Jun 2023~~

Python 3.8

Oct 2019

~~Apr 2023~~

Oct 2024

Python 3.9

Oct 2020

Apr 2024

Oct 2025

Python 3.10

Oct 2021

Apr 2025

Oct 2026

Python 3.11

Oct 2022

Apr 2026

Oct 2027

Python 3.12

Oct 2023

Apr 2027

Oct 2028

Python 3.X

Oct 2011+X

Apr 2015+X

Oct 2016+X

Since Python 3.8, Python releases yearly, so you can expect a new Python release every October, and an EOL every December (April a year before for data science).

Note that SPEC 0 seems to be replacing NEP 29, and it has an even shorter support cycle, 36 months. So releases get dropped in October three years after they debut.

Extra: saving and running a file from Jupyter#

For teaching purposes, we will be running some tools (pytest and mypy) from notebooks; this is not what they were designed to do, so we will use the following small extension to save a cell to a file and then run it with a Python module. I’m using a third-party library, rich, to render this with nice syntax highlighting in the notebook.

import rich
from rich.syntax import Syntax
from pathlib import Path

filepath = Path("save_and_run.py")
rich.print(Syntax(filepath.read_text(), "python", theme="default"))
from IPython.core.magic import Magics, magics_class, cell_magic                                                    
import sys                                                                                                         
import subprocess                                                                                                  
from pathlib import Path                                                                                           
class AutoMagics(Magics):                                                                                          
    def save_and_run(self, line, cell, local_ns=None):                                                             
        commands = line.split()                                                                                    
        filename = "tmp.py"                                                                                        
            [sys.executable, "-m", *commands, filename],                                                           
            # This is just to support colors in the notebook                                                       
            env={"FORCE_COLOR": "1", "MYPY_FORCE_COLOR": "1", "TERM": "xterm-color"},                              
# Why sys.executable here?                                                                                         
# We are running inside a virtual environment, which also has mypy installed.                                      
# However, this is a shell command (the starting `!`), so it will not                                              
# necessarily run in the same environment. So we'll use the current python                                         
# interpreter `sys.executable` and use the `python -m mypy` expression  instead                                    
# of plain `mypy` to make sure we get our installed MyPy. You don't usually run                                    
# mypy from a notebook.                                                                                            
def load_ipython_extension(ipython):