pybind11

pybind11

pybind11#

  • Similar to Boost::Python, but easier to build

  • Pure C++11 (no new language required), no dependencies

  • Builds remain simple and don’t require preprocessing

  • Easy to customize result

  • Great Gitter community

  • Used in GooFit 2.1+ for CUDA too [CHEP talk]

  • Used in boost-histogram, SciPy, PyTorch, Google, and more.

  • Big updates to build system in 2.6, great developers ;)

Downsides:

  • Still verbose (but highly customized interface is likely worth it in most cases)

UNIX setup only! Next section will include Windows compatible builds.

import os
import sys
from pybind11 import get_include

inc = "-I " + get_include()
plat = "-undefined dynamic_lookup" if "darwin" in sys.platform else "-fPIC"
pyinc = !python3-config --cflags

print(f"{inc = }\n")
print(f"{plat = }\n")
print(f"{pyinc.s = }\n")

inc = '-I /home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/lib/python3.12/site-packages/pybind11/include'

plat = '-fPIC'

pyinc.s = '-I/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/include/python3.12 -I/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/include/python3.12  -fno-strict-overflow -Wsign-compare -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -ffunction-sections -pipe -isystem /home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/python-split_1728056517259/work=/usr/local/src/conda/python-3.12.7 -fdebug-prefix-map=/home/runner/work/se-for-sci/se-for-sci/.pixi/envs/default=/usr/local/src/conda-prefix -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto -DNDEBUG -O3 -Wall'

Let’s look at our example from before:

%%writefile pysimple.cpp

#include <pybind11/pybind11.h>

namespace py = pybind11;

float square(float x) {
    return x*x;
}

PYBIND11_MODULE(pysimple, m) {
    m.def("square", &square);
}

Writing pysimple.cpp

!c++ -std=c++11 pysimple.cpp -shared {inc} {pyinc.s} -o pysimple.so {plat}

import pysimple

pysimple.square(3)

9.0

Pybind11 supports C++11 lambdas naturally, and has an overload system:

%%writefile pytempl.cpp

#include <pybind11/pybind11.h>

namespace py = pybind11;

template<class T>
T square(T x) {
    return x*x;
}

PYBIND11_MODULE(pytempl, m) {
    m.def("square", [](int value){return square(value);});
    m.def("square", [](double value){return square(value);});
}

Writing pytempl.cpp

!c++ -std=c++11 pytempl.cpp -shared {inc} {pyinc.s} -o pytempl.so {plat}

import pytempl

pytempl.square(3)

9

pytempl.square(3.0)

9.0

Of course, in C++ you usually want to get access to classes:

%%writefile SimpleClass.hpp
#pragma once

class Simple {
    int x;
    
public:

    Simple(int x): x(x) {}
    
    int get() const {
        return x;
    }
    
};

Writing SimpleClass.hpp

%%writefile pybindclass.cpp

#include <pybind11/pybind11.h>
#include "SimpleClass.hpp"

namespace py = pybind11;

PYBIND11_MODULE(pybindclass, m) {
    py::class_<Simple>(m, "Simple")
        .def(py::init<int>())
        .def("get", &Simple::get)
    ;
}

Writing pybindclass.cpp

!c++ -std=c++11 pybindclass.cpp -shared {inc} {pyinc.s} -o pybindclass.so {plat}

import pybindclass

x = pybindclass.Simple(4)
x.get()

4

More binding situations#

Let’s take a look at other binding situations.

%%writefile VectorClass.hpp
#pragma once

class Vector2D {
    double x;
    double y;
    
public:

    Vector2D(double x, double y): x(x), y(y) {}
    
    float get_x() const {
        return x;
    }
    
    float get_y() const {
        return y;
    }
    
    void set_x(float val) {
        x = val;
    }
    
    void set_y(float val) {
        y = val;
    }

    
    Vector2D& operator+= (const Vector2D& other) {
        x += other.x;
        y += other.y;
        return *this;
    }
    
    Vector2D operator+ (const Vector2D& other) const {
        return Vector2D(x + other.x, y + other.y);
    }
};

Writing VectorClass.hpp

The binding code for this will show a few more features than before.

%%writefile vectorclass.cpp

#include <pybind11/pybind11.h>
#include <pybind11/operators.h>
#include "VectorClass.hpp"

namespace py = pybind11;
using namespace pybind11::literals;

PYBIND11_MODULE(vectorclass, m) {
    py::class_<Vector2D>(m, "Vector2D")
        .def(py::init<double, double>(), "x"_a, "y"_a)
        .def_property("x", &Vector2D::get_x, &Vector2D::set_x)
        .def_property("y", &Vector2D::get_y, &Vector2D::set_y)
        .def(py::self += py::self)
        .def(py::self + py::self)
        .def("__repr__", [](py::object self){
            return py::str("{0.__class__.__name__}({0.x}, {0.y})").format(self);
        })
    ;
}

Writing vectorclass.cpp

Important / new points:

  • Initializers are really easy, even if overloaded. Just use py::init<...>().

  • You can specify argument names with "..."_a and using namespace pybind11::literals.

  • Properties allow you to specify a get and set function.

  • The pybind11/operators.h header lets you describe operations with py::self

  • You can manually add special functions

  • There doesn’t have to be a method; you can bind a lambda function as well.

  • You can use a class instance OR a py::object, and you can cast between them with py::cast (not shown).

  • Many python classes are provided, like py::str, with common methods, like .format.

  • You can access any python method with .attr (not shown).

Feel free to play with this example. Most importantly, remember to restart the notebook, because Python caches imports (which we’ve seen before).

!c++ -std=c++11 vectorclass.cpp -shared {inc} {pyinc.s} -o vectorclass.so {plat}

import vectorclass

v = vectorclass.Vector2D(1, 2)
print(f"{v.x = }, {v.y = }")

v.x = 1.0, v.y = 2.0

v

Vector2D(1.0, 2.0)

v + v

Vector2D(2.0, 4.0)

vectorclass.Vector2D(x=2, y=4)

Vector2D(2.0, 4.0)

Pybind11 has fantastic documentation, and supports a lot of situations.

  • Embedded mode (runs in reverse, calling Python from C++)

  • Smart pointers support

  • Variant support (C++17 or custom variants like from Boost)

  • NumPy support (without requiring NumPy during the compile!)

  • Eigen support (built-in extension)

  • Several external extensions provided (JSON, Abisel, and more)

Design details:

  • Functions make a call-chain to handle overloads. This means there is a little more overhead in calling a function than some other systems.

  • The focus of the library is making size efficient extensions, if faster == bigger, it probably won’t fly.

Contents