CPython

CPython#

  • Let’s see how bindings work before going into C++ binding tools

  • This is how CPython itself is implemented (and PyPy supports this too)

C reminder: static means visible in this file only

import sys

if sys.platform.startswith("darwin"):
    %set_env CPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
    %set_env LIBRARY_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib

First, here’s the code we want to wrap:

%%writefile pysimple.c

#include <Python.h>

float square(float x) {
    return x*x;
}

Writing pysimple.c

Okay, now we need to manually build the Python -> C -> Python conversion. All Python objects, regardless of type, are PyObject*. All the necessary conversions are provided using Py* functions.

  • PyArg_ParseTuple(PyObject*, format, *result) will parse a tuple into a C.

  • PyFloat_FromDouble(double) will convert a C item into a PyObject*.

%%writefile -a pysimple.c

static PyObject* square_wrapper(PyObject* self, PyObject* args) {
  float input, result;

  if (!PyArg_ParseTuple(args, "f", &input)) {
      return NULL;
  }
    
  result = square(input);
    
  return PyFloat_FromDouble(result);
}

Appending to pysimple.c

Next, we need a PyMethodDef. This is a structure that holds functions. The structure is “name”, wrapper function, argument type, and doc string. Since C doesn’t know when an array ends, you use a null terminated row at the end to signal that that you are done. The argument type is an item from this list; I’m excited about the new METH_FASTCALL in 3.7+…

Looking back, this could have been a little simpler with METH_O, which takes exactly one argument and therefore you’d avoid the Tuple.

%%writefile -a pysimple.c

static PyMethodDef pysimple_methods[] = {
   { "square", square_wrapper, METH_VARARGS, "Square function" },
   { NULL,     NULL,           0,             NULL }
};

Appending to pysimple.c

Now we need a module structure. I won’t go into it in great detail; PyModuleDef_HEAD_INIT is always there. Then the name of the module, then the module docstring (or NULL), then a size for subinterpreters (-1 to disable), the the methods you defined above.

%%writefile -a pysimple.c

static struct PyModuleDef pysimple_module = {
    PyModuleDef_HEAD_INIT, "pysimple", NULL, -1, pysimple_methods
};

Appending to pysimple.c

Finally we have the last bit; you need to define a symbol with a specific name, PyInit_<name>, where <name> is the name of the module.

%%writefile -a pysimple.c

PyMODINIT_FUNC PyInit_pysimple(void) {
    return PyModule_Create(&pysimple_module);
}

Appending to pysimple.c

To compile our module, we need to use a build system of some sort. This is the correct way to compile it, and cross platform. You can just compile it by hand, which I’ve done here in the past, but it’s very hard to get it right, and very system specific. I’m going to use setuptools for this example; you may need to install it on Python 3.12+.

%%writefile setup.py

from setuptools import setup, Extension

module1 = Extension('pysimple',
                    sources=['pysimple.c']
                   )

setup(name='pysimple', ext_modules=[module1])

Writing setup.py

For simplicity, I’m going to run setup.py directly; never do this in a real package! We’ll show a much better way to do all this soon.

!python3 setup.py build_ext --inplace

running build_ext

building 'pysimple' extension
creating build/temp.linux-x86_64-cpython-312
gcc -pthread -B /home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/include -fPIC -O2 -isystem /home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/include -fPIC -I/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/include/python3.12 -c pysimple.c -o build/temp.linux-x86_64-cpython-312/pysimple.o

creating build/lib.linux-x86_64-cpython-312
gcc -pthread -B /home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/compiler_compat -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib -Wl,-rpath-link,/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib -L/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib -Wl,-rpath-link,/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib -L/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib build/temp.linux-x86_64-cpython-312/pysimple.o -o build/lib.linux-x86_64-cpython-312/pysimple.cpython-312-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-cpython-312/pysimple.cpython-312-x86_64-linux-gnu.so ->

Run:#

import pysimple

pysimple.square(2.0)

4.0

pysimple.square(1)

1.0

pysimple.__file__

'/home/runner/work/se-for-sci/se-for-sci/content/week09/02-cpython/pysimple.cpython-312-x86_64-linux-gnu.so'

Python limited API: PEP 384#

One possible benefit of building your own Python extensions this way is that you can use the Python limited API. This defines a reduced superset of functionality that you must stick to, but in return, you now can use a single compiled extension with multiple versions of Python, from a minimum (3.2 or later) to the current version (and future versions!).

~~Notice that sadly~~, PyBuffer is not part of the limited API. (Added in 3.11!)

Nanobind (a light weight set of the core pybind11 functionality for C++17+ and Python 3.8+) supports the limited ABI starting for Python 3.12.

To use:

  • Define #define Py_LIMITED_API 0x03080000 at the top of your file (above the Python.h include). This number is the minimum python version you want to support (0x03020000, or 3.2, is the earliest). Currently supported Python is 3.9+, just to let you know.

  • Set py_limited_api=True in the setuptools Extension or set a similar feature in scikit-build-core/meson-python

  • Make sure you don’t use any of the (now missing) Py* functions that are not part of the limited API.

Exercise#

Try to convert the above code to use the Python limited API by making the two recommended changes. How did the file name change? What did you have to do to get it to load the new file? (note: you also need to delete the more-specific Python 3.8 file)

Contents