On 19.10.2015 06:24, Wenzel Jakob wrote:

I would be open to it but have my doubts about the feasibility of a merge. Consider the difference in code size alone: Boost.Python (without dependencies like MPL etc.) uses 26K lines of code, compared to about 2K for pybind11 (3K with all extensions). Apart from that, the libraries take very different internal design decisions, which would likely break existing software that ventures beyond the basic .def() syntax.

Hi Wenzel, Indeed, I would be very interested in a detailed comparison. Modernizing Boost.Python by using C++11 features (for example) and stripping out obsolete compiler support is one way to move forward. That in itself will help a lot, and may even allow Boost.Python to strip off dependencies to other Boost libraries. I'd be curious to learn about those design decisions that you are alluding to that lead to incompatibilities. Such a document may ultimately also be important for potential users when they consider the alternatives. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Hi, it would take a long time to discuss all differences, but I can give some examples. There are basically three ways of interfacing with Python objects in pybind11. 1. using wrapper classes like pybind11::object (analogous to boost::python::object) 2. by creating bindings that map a C++ type to Python — this is done using pybind11::class_ (analogous to boost::python::class_) 3. by declaring a partial template overload that does transparent conversions between different types. Boost.Python’s approach for communicating type information (item 2. in the above list) between modules entails linking against a shared library with a few containers storing the relevant data. In comparison, pybind11 installs a __pybind11__ capsule object in the global scope for this purpose, which avoids the library dependency. Any extra binding library that is loaded just registers its types there. In terms of the underlying implementation, 1. and 2. are pretty basic, and 3. is where a lot of the interesting things happen. This is basically a big list of partial template overloads of a class named type_caster which try to match various common types recursively. I’ll show just one example of how C++11 can considerably simplify implementation details here. For instance, consider the converter which enables transparent conversions between std::tuple<…> and Python’s ‘tuple’ class. Among other things, pybind11 uses this to convert function arguments to Python objects. The top-level signature matches an arbitrary tuple (that could even be nested, or other kinds of type concoctions … :)) I’ll expand the snippet literal programming-style, adding code to the <…> part. template <typename... Tuple> class type_caster<std::tuple<Tuple...>> { typedef std::tuple<Tuple...> type; enum { size = sizeof...(Tuple) }; <…> }; The first thing we’ll do is to declare sub-converters to deal with the individual tuple element types. The decay template simplifies the base type as much as possible by stripping type modifiers like pointers, references, const, etc. (those are handled separately) <…> += std::tuple<type_caster<typename decay<Tuple>::type>...> value; The following function takes a tuple from Python and converts it into the corresponding C++ object, returning false if the conversion wasn’t possible. It expects a special type index_sequence<0,1,2,3,…., N-1> as an argument, where N is the length of the tuple. This is a pretty common workaround to enable something resembling a loop over variadic template arguments rather than writing a messy recursive function. <…> += protected: template <size_t ... Indices> bool load(PyObject *src, index_sequence<Indices...>) { if (!PyTuple_Check(src)) return false; if (PyTuple_Size(src) != size) return false; std::array<bool, size> results {{ (PyTuple_GET_ITEM(src, Indices) != nullptr ? std::get<Indices>(value).load(PyTuple_GET_ITEM(src, Indices)) : false)... }}; for (bool r : results) if (!r) return false; return true; } The following function function calls the above protected function with the needed index_sequence <…> += public: bool load(PyObject *src) { return load(src, typename make_index_sequence<sizeof...(Tuple)>::type()); } which is constructed using a much shorter recursive implementation that runs at compile time: template<size_t ...> struct index_sequence { }; template<size_t N, size_t ...S> struct make_index_sequence : make_index_sequence <N - 1, N - 1, S...> { }; template<size_t ...S> struct make_index_sequence <0, S...> { typedef index_sequence<S...> type; }; Here is another very short example that I like: this converts a Python function into a std::function<> using a stateful lambda closure that invokes the function object’s call() function. With this partial template overload, we can easily call functions that take std::function<>s as argument using Python functions. Something similar is also possible for the reverse direction. template <typename Return, typename... Args> struct type_caster<std::function<Return(Args...)>> { typedef std::function<Return(Args...)> type; public: bool load(PyObject *src_) { if (!PyFunction_Check(src_)) return false; object src(src_, true); value = [src](Args... args) -> Return { object retval(handle(src).call(std::move(args)...)); return retval.template cast<Return>(); }; return true; } <…> protected: type value; }. The codebase contains many other examples. For instance, the optional auto-vectorization support over NumPy array arguments is something that would have been very painful to do with C++03. Best, Wenzel

On Oct 19, 2015, at 2:15 PM, Jim Bosch <talljimbo@gmail.com> wrote:

At first glance, this looks great. I've been tinkering with something like this for a while now, but it's never amounted to anything more than a C++11 learning project, and this looks quite solid in comparison.

It may be a long time before I get a chance to evaluate pybind11 for use in my own projects, but as a prelude to that I was wondering if you could say anything about support for custom converters and cross-module type conversion, which I didn't see mentioned in the docs (though I just skimmed them). Are you using the same sort of global registry Boost.Python used? If so, I'm curious how that works with a header-only library.

Jim

On Sun, Oct 18, 2015 at 9:56 PM, Wenzel Jakob <wenzel@inf.ethz.ch <mailto:wenzel@inf.ethz.ch>> wrote: Hello all,

after being a long-time Boost.Python user, I’ve been working on an alternative that makes more effective use of recent C++11-capable compilers. The overall syntax and ideology are very similar to Boost.Python, but the implementation only requires a few header files with a a vastly smaller amount of code thanks to C++11 lambda functions, tuples and variadic templates. There is also dedicated support for Python’s buffer protocol and NumPy arrays, which is useful for scientific computing applications.

So far it’s only used by a few projects, but I think it could be useful to this audience.

Code: https://github.com/wjakob/pybind11 <https://github.com/wjakob/pybind11> Documentation: http://pybind11.readthedocs.org/en/latest/ <http://pybind11.readthedocs.org/en/latest/>

Best, Wenzel

_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org <mailto:Cplusplus-sig@python.org> https://mail.python.org/mailman/listinfo/cplusplus-sig <https://mail.python.org/mailman/listinfo/cplusplus-sig>

_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig

Hi, I became curious about this myself and ran a simple benchmark for automatically generated binding code of increasing size. The compilation times for Boost.Python and pybind11 turn out to be fairly similar. However, there is a significant difference in terms of the size of the compilation result, which is almost twice as large for Boost.Python. See the details here: http://pybind11.readthedocs.org/en/latest/benchmark.html <http://pybind11.readthedocs.org/en/latest/benchmark.html> Best, Wenzel

On Oct 19, 2015, at 9:58 PM, Francesco Biscani <bluescarni@gmail.com> wrote:

This looks really neat!

Do you have any measure on the memory/cpu performance wrt Boost.Python when compiling large bindings? I would expect that variadic templates and all the other C++11 goodies are more efficient than the preprocessor magic used in Boost Python, from this point of view.

Cheers,

Francesco.

On 18 October 2015 at 14:56, Wenzel Jakob <wenzel@inf.ethz.ch <mailto:wenzel@inf.ethz.ch>> wrote: Hello all,

after being a long-time Boost.Python user, I’ve been working on an alternative that makes more effective use of recent C++11-capable compilers. The overall syntax and ideology are very similar to Boost.Python, but the implementation only requires a few header files with a a vastly smaller amount of code thanks to C++11 lambda functions, tuples and variadic templates. There is also dedicated support for Python’s buffer protocol and NumPy arrays, which is useful for scientific computing applications.

So far it’s only used by a few projects, but I think it could be useful to this audience.

Code: https://github.com/wjakob/pybind11 <https://github.com/wjakob/pybind11> Documentation: http://pybind11.readthedocs.org/en/latest/ <http://pybind11.readthedocs.org/en/latest/>

Best, Wenzel

_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org <mailto:Cplusplus-sig@python.org> https://mail.python.org/mailman/listinfo/cplusplus-sig <https://mail.python.org/mailman/listinfo/cplusplus-sig>

_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig

I use that by default for compiling Python bindings. It should not make any difference for just a single file (including this testcase), but I found that it yields consistently smaller shared libraries when dealing with lots of compilation units. Wenzel

On Oct 20, 2015, at 1:33 PM, Neal Becker <ndbecker2@gmail.com> wrote:

I noticed you used -flto when building the shared libs. Do you find this makes a difference?

_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig