Level Up Your Python

A course in intermediate Python for a beginner ready to move up.

Henry Schreiner

Introduction

Expected Knowledge

You should already know:

  • Basic Python syntax

  • Functions

  • Basic classes (will cover advanced usages mostly)

  • Basic NumPy (will mention, but not cover)

  • Git - CRITICAL FOR ANY SOFTWARE WORK!

And we will be using notebooks in JupyterLab.

About the author

Henryiii's github stats

Most important link: https://iscinumpy.dev

PyPA member. Scikit-HEP admin, scikit-build admin, member of IRIS-HEP.

Interests:

Packaging and building • Bindings • Building a HEP analysis toolchain in Python, JITable

My books and workshops

Modern CMakeCMake WorkshopComputational Physics Class • Python CPU, GPU, Compiled minicourses

Tempering your expectations

What can you expect?

  • I don’t know what you know;

  • We don’t have time to study any topic in depth.

So, we will move fast, and cover a lot.

You are not expected to able to master everything you see.

Instead, you are expected to:

  1. Know what is possible, so you know to look for it;

  2. Get pointers on where to look (lots of links!);

  3. Refer back to this material later.

Theory: New features in programming

Programming is all about organization. This is not always obvious, and has some odd consequences. Let’s look at one: new features remove functionality from the user. And that’s a good thing.

Don’t believe me? Pick one. Let’s go with an old, simple one you should already know: goto vs. loops (for/while). (This is my favorite example, even though Python thankfully came along late enough to not even have goto in the first place, except as an April fools joke or a proof of concept library.)

You have total power with goto! Jump from anywhere, to anywhere! You can recreate all loops (for loops, for each loops, while loops, do while (C) loops) with it, and more (like functions)! So why are loops the newer, better feature?

goto (partially hypothetical in Python)

i = 0
label .start
print(f"Hi {i}")
i + 1
if i <= 10:
    goto .start

Compare to for loop:

for i in range(10):
    print(f"Hi {i}")
Hi 0
Hi 1
Hi 2
Hi 3
Hi 4
Hi 5
Hi 6
Hi 7
Hi 8
Hi 9

A programmer has to spend time to recognize what is happening in the first example - in the second example, even a fairly new Python programmer will immediately say “that prints 0 to 9”. The second example lets you build more complex programs, because you are working at a ‘higher level’, humans tend to do better which high level concepts (while computers work up from low level).

Also, we now need several features to make up for the loss of goto; the for loop, the while loop, and functions. Each is more restricted, with less functionality, but better readability and composability.

GOTO xkcd comic

We will see lots of examples of this – in section 2, especially.

Notebooks

We will be using notebooks today. Notebooks are fantastic for teaching, quick experimentation, for developing, or for driving a final analysis product. They are not for serious programming - that happens in .py files. Once you write something and get it working, move it to a .py file and add a test. Then import it into your notebook!

Python version

We will be using Python 3.10. I’ll try to point out when something is newer than 3.7. NEP 29 mandates that data science libraries currently support 3.8+ (support dropped 42 months after release), while general Python EOL is 3.7+ (5 year support window). IPython and NumPy have already dropped Python 3.7.

Key upcoming dates:

Python

Release

NEP 29 drop

General EOL

~~Python 3.6~~

~~Dec 2016~~

~~Jun 2020~~

~~Dec 2021~~

Python 3.7

Jun 2018

~~Dec 2021~~

Jun 2023

Python 3.8

Oct 2019

Apr 2023

Oct 2024

Python 3.9

Oct 2020

Apr 2024

Oct 2025

Python 3.10

Oct 2021

Apr 2025

Oct 2026

Python 3.X

Oct 2011+X

Apr 2015+X

Oct 2016+X

Since Python 3.8, Python releases yearly, so you can expect a new Python release every October, and an EOL every December (April a year before for data science).

Extra: saving and running a file from Jupyter

For teaching purposes, we will be running some tools (pytest and mypy) from notebooks; this is not what they were designed to do, so we will use the following small extension to save a cell to a file and then run it with a Python module. I’m using a third-party library, rich, to render this with nice syntax highlighting in the notebook.

import rich
from rich.syntax import Syntax
from pathlib import Path

filepath = Path("save_and_run.py")
rich.print(Syntax(filepath.read_text(), "python", theme="default"))
from IPython.core.magic import Magics, magics_class, cell_magic                              
                                                                                             
import sys                                                                                   
import subprocess                                                                            
from pathlib import Path                                                                     
                                                                                             
                                                                                             
@magics_class                                                                                
class AutoMagics(Magics):                                                                    
    @cell_magic                                                                              
    def save_and_run(self, line, cell, local_ns=None):                                       
        commands = line.split()                                                              
        filename = "tmp.py"                                                                  
        Path(filename).write_text(cell)                                                      
                                                                                             
        subprocess.run(                                                                      
            [sys.executable, "-m", *commands, filename],                                     
            # This is just to support colors in the notebook                                 
            env={"FORCE_COLOR": "1", "MYPY_FORCE_COLOR": "1", "TERM": "xterm-color"},        
        )                                                                                    
                                                                                             
                                                                                             
# Why sys.executable here?                                                                   
#                                                                                            
# We are running inside a virtual environment, which also has mypy installed.                
# However, this is a shell command (the starting `!`), so it will not                        
# necessarily run in the same environment. So we'll use the current python                   
# interpreter `sys.executable` and use the `python -m mypy` expression  instead              
# of plain `mypy` to make sure we get our installed MyPy. You don't usually run              
# mypy from a notebook.                                                                      
                                                                                             
                                                                                             
def load_ipython_extension(ipython):                                                         
    ipython.register_magics(AutoMagics)