Saturday, August 27, 2016


Lazily Exposing an std::vector to Python as a Numpy Array with pybind11

Suppose one wishes to expose to Python multi-threaded C++ code that generates output in the form of flat integer or floating point value arrays, and these arrays should appear as Numpy arrays in Python. The two common approaches are (replacing "element" with bool, std::uint16_t, int, float, double, etc):

With pybind11, there is a better way: 1) keep any std::vector<> that may be exposed to Python in an std::shared_ptr<std::vector>, 2) expose the concrete std::vector<element> types used with std::shared_ptr<std::vector<>> as the associated "holder" type for each, with an appropriate .def_buffer call, 3) and in response to requests from Python, lazily retrieve (causing instantiation of) the Python object wrapping the vector requested, feed this to Numpy, and cache and return the resulting Numpy array.

This arrangement may sound complicated, but it is, by far, the most natural and flexible of all approaches: without resort to Python reference counting or requirement to acquire the GIL, a vector exposed in this manner is not garbage collected until both the last outstanding Python reference and the last outstanding C++ reference are gone. This is awesome.

Let's break down the rather dense instructions presented above.


Keep any std::vector<> that may be exposed to Python in an std::shared_ptr<std::vector<>>.

There not much to this. struct Foo { std::vector<int> v; }; changes to Foo { std::shared_ptr<std::vector<int>> v; };, and any v. changes to v->


Expose the concrete std::vector<element> types used with std::shared_ptr<std::vector<>> as the associated "holder" type for each, with an appropriate .def_buffer call.
py::class_<std::vector<std::uint64_t>, std::shared_ptr<std::vector<std::uint64_t>>>(m, "_HistogramBuffer")
    .def_buffer([](std::vector<std::uint64_t>& v) {
        return py::buffer_info(
            { v.size() },
            { sizeof(std::uint64_t) });


In response to requests from Python, lazily retrieve (causing instantiation of) the Python object wrapping the vector requested, feed this to Numpy, and cache and return the resulting Numpy array. The lines of code where this is done are in bold; the rest is provided as minimal context, so that you have some chance of figuring out what I'm talking about :)

template<typename T>
struct StatsBase
    static void expose_via_pybind11(py::module& m);

    StatsBase(const StatsBase&) = delete;
    StatsBase& operator = (const StatsBase&) = delete;
    virtual ~StatsBase() = default;

    std::tuple<T, T> extrema;
    std::size_t max_bin;

    std::shared_ptr<std::vector<std::uint64_t>> histogram;
    // A numpy array that is a read-only view of histogram. Lazily created in response to get_histogram_py calls.
    std::shared_ptr<py::object> histogram_py;

    py::object& get_histogram_py();

template<typename T>
void StatsBase<T>::expose_via_pybind11(py::module& m)
    std::string s = std::string("_StatsBase_") + component_type_names[std::type_index(typeid(T))];
    py::class_<StatsBase<T>, std::shared_ptr<StatsBase<T>>>(m, s.c_str())
        .def_readonly("extrema", &StatsBase<T>::extrema)
        .def_readonly("max_bin", &StatsBase<T>::max_bin)
        .def_readonly("histogram_buff", &StatsBase<T>::histogram)
        .def_property_readonly("histogram", [](StatsBase<T>& v){return v.get_histogram_py();});

template<typename T>
  : extrema(0, 0),
    histogram(new std::vector<std::uint64_t>(bin_count<T>(), 0)),

template<typename T>
py::object& StatsBase<T>::get_histogram_py()
        py::object buffer_obj = py::cast(histogram);
        histogram_py.reset(new py::object(PyArray_FromAny(buffer_obj.ptr(), nullptr, 1, 1, 0, nullptr), false), &safe_py_deleter);
    return *histogram_py;

StatsBase<T>::get_histogram_py() is a bit complex; let's break it down:

If the StatsBase<T> instance in question does not already have a non-null histogram_py pointer...

py::object buffer_obj = py::cast(histogram);
Get a Python object wrapping our std::shared_ptr<std::vector<std::uint64_t>> instance. This wrapper will be as we specified to pybind11 and will therefore have a buffer protocol interface understood by Numpy.

histogram_py.reset(new py::object(PyArray_FromAny(buffer_obj.ptr(), nullptr, 1, 1, 0, nullptr), false), &safe_py_deleter);
Use the PyArray_FromAny call to make a Numpy array that is a view of our vector and keep the resulting PyObject* in a pybind11 PyObject* wrapper that will decrement its refcount appropriately when destroyed. Store this in an std::shared_ptr with a GIL-safe deleter in order to avoid crashing in the case where a C++ background thread is the last thing with a reference to a StatsBase instance that has been accessed from a no-longer-extant Python reference.

return *histogram_py;
Return a C++ reference to the py::object representing the Numpy array.

This example is from real world code (it may be necessary to look in the new_ndimage_statistics branch, but I expect to merge this into master within the next few days). Apologies for not making a minimal example. If you'd like one or have any questions, please ask!


Post a Comment

Subscribe to Post Comments [Atom]

<< Home


July 2009   August 2009   September 2009   October 2009   November 2009   December 2009   January 2010   September 2010   December 2010   January 2011   February 2011   April 2011   June 2011   August 2011   February 2012   June 2012   July 2012   August 2012   October 2012   November 2012   January 2014   April 2014   June 2014   August 2014   September 2014   October 2014   January 2015   March 2015   April 2015   June 2015   November 2015   December 2015   January 2016   June 2016   August 2016   January 2017   March 2017   April 2018  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]