Disclamer
We are focusing on Python 3 for most of the course, but for reference, I'll note when features became available for this set of slides only. Most things are applicable to both versions. Also, if using Python 2, include the following line:
from __future__ import print_function, division, absolute_import
Python 3 came out 10 years ago. Python 2 is dying in 2020. Packages are dropping support. Please convince everyone you work with to go to Python 3 if they can!
Basics of Python
- The ultimate source: Python docs
- Variables are dynamically typed
my_int = 3 # Python 2 has two integers, normal and long
my_float = 3.0
my_str = "hi" # Strings come in different flavors, like r"raw".
my_complex = 1j
my_bool = True
# None is it's own type. Odd.
my_list = [0, 1, 2, 3] # Can be changed later
my_tuple = (0, "anything") # Can't be changed later
my_dict = {"one": 1, "two": 2} # Order preserved in Python 3.6+
my_set = {1, 2, 3}
- Special values:
special_values = (None, True, False, Ellipsis)
- Last uncaptured value:
_ - Indexing arrays:
[value]- Multi-dimensional arrays supported by language, but not in standard library
- Slicing arrays:
[first:last:step]- Can omit
:stepor either/both values - Does not include final value
- The length of the result is
last - first ...means "all the rest"
- Can omit
Overview of syntax:
- Indented blocks with
: - Classes and functions create scope
- Function parameters can be optional or keyword only
- Iteration built into the language
for x in iterable: ...
listing = [x for x in iterable if True]
listing = list(iterable)
Common stumbling blocks:
- Mutable vs Immutable (only some built-ins immutable)
- Assignment replaces the variable
x = [1, 2]
y = x
x[:] = [3, 4]
What is printed with print(x,y) ?
print(x, y)
x = [5, 6]
print(x, y)
Keywords
Exhaustive and complete list:
False class finally is return
None continue for lambda try
True def from nonlocal while
and del global not with
as elif if or yield
assert else import pass async
break except in raise await
Python 2: add print and exec, remove nonlocal
Python before 3.7: remove async and await
Built in functions
Every function Python 3.7 offers:
| ... | ... | ... | ... | ... |
|---|---|---|---|---|
abs() |
dict() |
help() |
min() |
setattr() |
all() |
dir() |
hex() |
next() |
slice() |
any() |
divmod() |
id() |
object() |
sorted() |
ascii() |
enumerate() |
input() |
oct() |
staticmethod() |
bin() |
eval() |
int() |
open() |
str() |
bool() |
exec() |
isinstance() |
ord() |
sum() |
bytearray() |
filter() |
issubclass() |
pow() |
super() |
bytes() |
float() |
iter() |
print() |
tuple() |
callable() |
format() |
len() |
property() |
type() |
chr() |
frozenset() |
list() |
range() |
vars() |
classmethod() |
getattr() |
locals() |
repr() |
zip() |
compile() |
globals() |
map() |
reversed() |
__import__() |
complex() |
hasattr() |
max() |
round() |
breakpoint() |
delattr() |
hash() |
memoryview() |
set() |
You could even get this list in Python itself!
# All builtins are available in the builtins module
import builtins
# Iterate over the contents of the module
for item in dir(builtins):
# Items are strings; only look at the ones]
# that start with a lower case letter
if item[0] != item[0].upper():
print(item)
x = True
if x:
print("x was true")
else:
print("x was not true")
The for each loop:
a = "abcdefg"
for item in a:
print(item)
You can also use for item in range(N) to count over a range of values, or while CONDITION:. You can use for (and if) inside [], (), or {} to build the data structures mentioned above inplace.
bad_way_to_make_a_set_of_tens = {x for x in range(100) if x % 10 == 0}
bad_way_to_make_a_set_of_tens
With
With lets you take code like this:
f = open(filename)
txt = f.read()
f.close() # Don't forget me!
and write instead:
with open(filename) as f:
txt = f.read()
# File automatically closed here!
With simply runs code at the start and at the end of a block. It also promises the code at the end runs, even if there's an error!
def my_function(value, another=None):
if another is not None:
return another
else:
return value
print(my_function(2), my_function(3, 4), my_function(3, another=4))
def my_iterator():
for i in range(5):
yield i * 2
for j in my_iterator():
print(j)
Decorators are objects that can modify functions or classes:
@change_a_function
def something():
return 1
is the same as
def something():
return 1
something = change_a_function(something)
class MyFunctionLike(object):
def __init__(self, value=2):
self.value = value
def __call__(self, value):
return value * self.value
f2 = MyFunctionLike(2)
f3 = MyFunctionLike(3)
print(f2(2), f3(2))
- Lots of special methods are available
- Can customize almost all behavior
- Lookup for normal members goes
object -> class -> parent class - Lookup for special methods starts with the class
- Member lookup can be intercepted
Creating an object looks like this:
- Capture the local variables in class statement
- Process the variables using metaclass (normally type) (improved in Python 3)
class.__new__->create objectobject.__init__: prepare object
Special decorators:
@property: Make a property function@classmethod: Make a method runnable from the class@staticmethod: Make a method not depend on anything in class
class MyTest(object):
@property
def x(self):
return 3
@staticmethod
def y():
return 2
print(MyTest.y(), MyTest().x)
def simple(x):
"This is help"
return x * 2
print(dir(simple))
help(simple)
import inspect
print(inspect.getsource(simple))
Signature details are available in Python 2, but are much more elegant in Python 3.
inspect.signature(simple)
inspect.getfile(inspect)
def make_an_error():
raise RuntimeError("I'm an error!")
# Try uncommenting this
# make_an_error()
try:
make_an_error()
except RuntimeError as e:
print(e, "But I don't care, continuing anyway")
You can catch errors like ImportError and then fix the problem, etc.
The only exception: segfault (C error) (but can still get some info about it in Python 3)
There are two blocks that also show up in Python: finally and else. Finally will run regardless of if an exception was caught or not, and else will run only if an exception was not caught. You can also use else on a loop, where it only runs if break was not used to exit the loop. (While we are at it, Python provides continue to keep going in the loop as well)
for i in range(10):
print("Breaking out:")
# break
else:
print("Did not break out")
Anatomy of a package
Folders and packages are intrinsically linked in Python.
import one
from two import three
from four.five import six
onecould beone.pyorone/__init__.pythreecould be in a package or another package- All package folders should contain
__init__.py - Relative imports
from .oneare available inside a package
Making an installable package
- Running code requires that it be in
sys.path(made from$PATHand$PYTHONPATHand the local dir) - Installing a package puts the code or a link to it in system or local
site-packages
To be installable:
- Make a setup.py file in root folder, follow examples
- Use
pip install -e .(recommended) orpython setup.py develop(not recommended)
History of Python: 2 vs. 3
- Python started in 1991 by Guido Van Rossum
- Based on the learner's language ABC
- Released at the right time
- Python 2.4 had a huge user base by the time Python 3 was released
So why is Python 3 better and why is Python 2 still common?
TL;DR: Python 3 removes bad features and backward incompatible is slow when you have millions of lines of code.
See the great talk
Changes
- Many of the improvements were backported to 2.6 and 2.7 (from 3.0 and 3.1)
- Some stylistic changes (print is a function, etc)
- Some removals not back-portable (function iterator unpacking)
- Most changes covered by the packages: six or future
Iterator unpacking
- Better support for
*listand**dict, especially in 3.5
first, *rest, last = range(10)Keyword only arguments
- No longer need to fake this with manual keyword dictionary, more natural
Iterators
- Iterators instead of duplicate functions
for i in range(1000000000000):
print("Your computer will die in Python 2!")
May require wrapping some functions with
listYou can now
yield fromanother iterator (3.4)
Comparison fixed
- You can no longer compare everything
"two" < 1is now an error instead of the wrong answer!
Type support
- Still duck-typed (dynamically typed)
- Added function (3.0) and variable (3.6) annotations
- Tools like PyCharm and mypy can do static analysis
Matrix multiplication operator (3.5)
- The
@symbol was added for matrix multiplication
F-strings (3.6)
- Format strings allow inline python in strings
x = 3
print(f"This is {x} and x+1={x+1}")
print("This is {} and x+1={}".format(x, x + 1))
print("This is %i and x+1=%i" % (x, x + 1))
As you could see, Python is simple. To do real work in Python, you need libraries.
The best library for Python: the Python standard library (included).
- Usually pretty good, covers many use cases
- Does not evolve, better options usually available
- Most Python 3 additions available as external libraries for Python 2
Small sampling
sys,os: system toolsre: regular expressionsmath,timeand friends,pathlib(3 or backport),functools,itertools,pdb, ...
See PMOTW3 (or PMOTW2).
There are many more common packages that can be downloaded from pypi with pip:
For scientists: The big three
numpy: Numerical python (matrix and array)scipy: More specialized additions to numpymatplotlib: The classic plotting packagepandas: A replacement forRor Excel, tabular data
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
x = np.arange(100)
plt.plot(x, np.sin(x))
plt.show()
Other useful libraries
sympy: Symbolic mathematics as Python objectsplumbum: Set of tools for command line programming (calling other programs, command line arguments, paths, remote connections, color)numba: Compile python into CPU, OpenMP, or CUDA codepytest: Better testing for your code in pythonsetuptools: The continued development of Python packagingtensorflow: Google's computation engine designed for MLCython: The C++ and Python mix for compiled python-like code
IPython language extensions:
%line_magic
%%cell_magic
help?
!command
- History (input and output cells stored)
- IPython has access to the original text before processed by Python
- IPython looks for special methods to print formatted objects
- Example:
%debugafter an exception drops you intopdb - Example:
%timeitwill run a command multiple times and get the timing info
Jupyter Notebooks
- Powerful interactive environment
- Great for remote servers or websites
- Inline plots, widgets, etc
- Cells in markdown, with code and $m \alpha \tau h$ support
- Converts to PDF, slides, etc.
- Great for prototyping
- Fledgling project: nteract for desktop application
- Future project: Juypter Lab for web environment with multiple components
Useful shortcuts (more in help->keyboard shortcuts)
- Shift-tab: Help
- Shift-tab-tab: More help (etc.)
- Tab: Auto-completion (existing objects only for IPython < 5)
JupyterLab
The newest and shiniest member of the IPython/Jupyter family. It's a home for notebooks and other types of windows, all in a single browser window. The notebook has some nice updates, too, like drag-n-drop cells and collapsible input. Not ready for major sites yet, but good enough for you to run it on your own.
Packaging in Python
pip
The Package Installer for Python, any package should be able to be installed with this.
- Easy to get (ensurepip in latest Python 2.7 or Python 3.4+,
getpip.pybefore that) - Can download binaries now instead of just source
- Interacts with environments nicely
pip install a_package
pip install --user local_package
pip install --upgrade pip
pip install -e .
Anaconda
- A science focused distribution of Python
- Contains Python 2 or/and 3
- Highly optimized Numpy and friends with help from Intel
- Contains the Conda package manager and ~100 packages, with ~400 optional packages
Conda
- Designed to manage binary packages (mostly Python related)
- Built in support for environments
- Can manage Python itself!
conda create -n my_python_27_env python=2.7 anaconda
source activate my_python_27_env
conda install paramiko
pip install plumbum
IDEs for Python
PyCharm
- If you need to develop a package, PyCharm is impressive
- All the features you could ever want, plus many more
- Free community edition is great
Spyder
- An open-source Matlab like IDE
- Used to be part of Python(x,y) for Windows, now on pip and in anaconda
IDLE
- A basic editor that comes with every Python