Everything we’ve been doing has built on itself, and we seemed to be going somewhere; the pinnacle of this direction was actually not context managers, but async/await. All of this feeds into async/await, which was formally introduced as a language component in Python 3.6. However, we did skip a necessary step; we didn’t talk about generators (iterators can actually “send” values in, not just produce them, but there’s no specific construct for doing that, like there is for consuming values in a for loop). The main reason we didn’t try to reach async though is that I’ve never found a great use for it in scientific programming; it is much more intrusive than normal threading, it doesn’t really “live” side-by-side with normal synchronous programming all that well (it’s better now, though), and the libraries for it are a little young. Feel free to investigate on your own, though! I’ve also discussed the mechanisms behind it in detail in my blog a few years ago.
Let’s whet your appetite with a quick example, though:
import asyncio
# This is an "async" function, like a generator
async def slow(t: int) -> int:
print(f"About to sleep for {t} seconds")
await asyncio.sleep(t)
print(f"Slept for {t} seconds")
return t
# Gather runs its arguments in parallel when awaited on
await asyncio.gather(slow(3), slow(1), slow(2))
# Only works if running in an eventloop already, like IPython or with python -m asyncio
# Otherwise, use: asyncio.run(...)About to sleep for 3 seconds
About to sleep for 1 seconds
About to sleep for 2 seconds
Slept for 1 seconds
Slept for 2 seconds
Slept for 3 seconds
[3, 1, 2]Notice there are no locks! We don’t have to worry about printing being overleaved, because it’s not running at the same time. Only the explicit “await” lines “wait” at the same time!
Starting in Python 3.11, TaskGroups make this really easy to use:
async with asyncio.TaskGroup() as tg:
res1 = tg.create_task(slow(3))
res2 = tg.create_task(slow(1))
res3 = tg.create_task(slow(2))
print(res1.result(), res2.result(), res3.result())About to sleep for 3 seconds
About to sleep for 1 seconds
About to sleep for 2 seconds
Slept for 1 seconds
Slept for 2 seconds
Slept for 3 seconds
3 1 2
Besides looking nicer, this also provides grouped errors (also Python 3.11+) if multiple tasks fail.
async def busted(s: str) -> None:
raise RuntimeError(s)
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(busted("First error"))
tg.create_task(busted("Second error"))
except* RuntimeError as error_group:
print(error_group.exceptions)(RuntimeError('First error'), RuntimeError('Second error'))
Why use async?¶
The reason this is “better” than threaded programming might not be obvious, but it’s this: code you write runs in the order you see. Most of the challenges of threaded programming, like mutexes and locks, simply aren’t an issue with async programming, because it’s not running in threads. You can still delegate work to threads, but keep the outer logic in sequence.
When you read async code, the await’s are important:
async def f(): # The async tells you this will be a coroutine (needs to be awaited)
do_a()
do_b() # This will always run right after do_a, nothing will run in between
await do_c() # Execution pauses here, and other coroutines might start and run till their awaits
do_d()Oh, and Jupyter notebooks are async, so it’s a great way to do things like auto-updating plots and output in notebooks.