Tips for exposing classes with own memory management model
Dear list, Apologies if this has been asked before, but I'm struggling to find anything strictly related.. Background ---------- This library I'm trying to wrap uses its own memory management model, where almost every class derives from an object with loads of memory management-related member functions; it also has a couple of friend classes related to counting and locking. I don't intend to expose any of these memory-related functions or friend classes to Python, but I was thinking that performance could be quite badly affected if both Python and C++ code are performing separate memory management implementations. Optimal memory usage -------------------- I would suppose that memory usage on class instances would probably contain unnecessary bloat too, as I think each exposed class instantiation would allocate memory for a normal PyObject as well as unexposed C++ member functions. Right thing to do ----------------- I initially hoped to use a 'return_internal_reference' CallPolicy on the class_<..> init calls, but I doubt that is The Right Thing To Do. Would it be a better design to define a PyTypeObject for this C++ base class and its friends? If I did, could I still use functions in boost::python? I don't think PyTypeObject's are supposed to be derived, so I don't have a clue what extra I'd have to do to make it work with Boost::Python. How should one proceed with this? Links to archived emails or documentation would be great.. If I can conjure up something good enough for Boost, I'd be happy to contribute, if possible. Thanks for your time, and kind regards, Alex
On Apr 18, 2013 9:12 AM, "Alex Leach" <beamesleach@gmail.com> wrote:
Dear list,
Apologies if this has been asked before, but I'm struggling to find
anything strictly related..
Background ----------
This library I'm trying to wrap uses its own memory management model,
where almost every class derives from an object with loads of memory management-related member functions; it also has a couple of friend classes related to counting and locking. I don't intend to expose any of these memory-related functions or friend classes to Python, but I was thinking that performance could be quite badly affected if both Python and C++ code are performing separate memory management implementations.
Optimal memory usage --------------------
I would suppose that memory usage on class instances would probably
contain unnecessary bloat too, as I think each exposed class instantiation would allocate memory for a normal PyObject as well as unexposed C++ member functions.
Right thing to do -----------------
I initially hoped to use a 'return_internal_reference' CallPolicy on the
class_<..> init calls, but I doubt that is The Right Thing To Do.
Would it be a better design to define a PyTypeObject for this C++ base
class and its friends? If I did, could I still use functions in boost::python? I don't think PyTypeObject's are supposed to be derived, so I don't have a clue what extra I'd have to do to make it work with Boost::Python.
How should one proceed with this? Links to archived emails or
documentation would be great.. If I can conjure up something good enough for Boost, I'd be happy to contribute, if possible.
If you go with writing your own PyTypeObject, you will indeed have a lot more control, but it will greatly limit how much Boost.Python you can use (no class_, for instance, at least), and you'll need to dive deep into the Boost.Python implementation to learn how and when you can use it. I'd only consider recommending this approach if you wanted to wrap one or two simple classes this way and then use regular Boost.Python for everything else. I think the best solution would probably be to use shared_ptr with a custom deleter, as that gives you control over how your objects are allocated while giving Boost.Python an object it knows how to handle extremely well. One key ingredient of this is that instead of wrapping C++ constructors, you'll want to wrap factory functions that return shared_ptrs. You can even wrap such functions as Python constructors using make_constructor. All that said, my first recommendation would be to try to wrap it (or at least a subset of it) without trying to get the optimal memory performance first, and only fix it if it actually is a performance problem. You might be surprised at where the time ends up going. Jim
Thank you for the quick response! On Thu, 18 Apr 2013 15:24:09 +0100, Jim Bosch <talljimbo@gmail.com> wrote:
If you go with writing your own PyTypeObject, you will indeed have a lot more control, but it will greatly limit >how much Boost.Python you can use (no class_, for instance, at least), and you'll need to dive deep into the >Boost.Python implementation to learn how and when you can use it. I'd only consider recommending this approach if >you wanted to wrap one or two simple classes this way and then use regular Boost.Python for everything else.
No class_<..> type would be a problem, as I've already exposed a lot of its derived classes this way.. There's got to be about 100 in total. :-\
I think the best solution would probably be to use shared_ptr with a custom deleter, as that gives you control over >how your objects are allocated while giving Boost.Python an object it knows how to handle extremely well. One key >ingredient of this is that instead of wrapping C++ constructors, you'll want to wrap factory functions that return
shared_ptrs. You can even wrap such functions as Python constructors using make_constructor.
I've already had to do this once, so I've got some experience with the technique, although can't remember exactly why it was needed. Thank you for a viable option, though!
All that said, my first recommendation would be to try to wrap it (or at least a subset of it) without trying to >get the optimal memory performance first, and only fix it if it actually is a performance problem. You might be >surprised at where the time ends up going.
:) Lol, yes, I can see how attempting to define a new PyTypeObject could become very time-consuming! If I were to go this route though (I probably will), is there a Boost Python registry or something where I can register these new types? I just thought to look into list.hpp, as I figure the boost::python::list would probably make use of the PyListType, which could be thought of as a similar PyTypeObject specialisation to one I might like to create. ...... Now I've done a little more digging, I think I have an idea of how to do it, which I'll detail now.. Any tips, pointers or advice in Boost::Python'ising it would be appreciated, as I have next to no experience with the Python-C API beyond Boost. Resources --------- Worth mentioning that the best (simplest) resources I've found to go by, are: [1] - For the C-Python side: http://docs.python.org/2/extending/newtypes.html [2] - For Boost-Python side: http://www.boost.org/doc/libs/1_53_0/boost/python/list.hpp Steps to expose:- ----------------- a. - Create mytypeobject.h, a raw C-Python extension type, as explained in [1]. a.i. - Create unit-test, by deriving such a type in Python, making a few instances, deleting some and leaving others to the garbage collector. b. Create mytypeobject.hpp, where a type similar to boost::python::list is declared. Register it, with for example:- // From list.hpp:- // namespace converter { template <> struct object_manager_traits<list> : pytype_object_manager_traits<&PyList_Type,list> { }; } What else? ---------- I very much doubt it will be this simple, as any exposed class_<> would probably still attach a typical PyObject, thereby using PyTypeObject's memory management functions, rather than that of any type object specialisation. Example ------- For example, let's suppose PyListObject has some functionality I need in an exposed class, and I need the garbage collector to use PyListType instead of PyTypeObject, when managing its memory. Is there a way to attach a PyListObject to a class_<> instance, instead of a PyObject? Perhaps an alternative to the class_<> template could be designed to use any arbitrary type, instead of the default PyTypeObject. Something which could be used like this would be cool:- boost::python::type_object_<PyListType, PyList, [boost::noncopyable]> ("alt_list", "PyListObject wrapped in Boost.", init<>()) .def("append", PyList_Append) /// ... ; How much farther do you think I would need to dig into Boost internals to implement such functionality? Worth it? Cheers, Alex
Jim
On 04/18/2013 11:44 AM, Alex Leach wrote:
Thank you for the quick response!
On Thu, 18 Apr 2013 15:24:09 +0100, Jim Bosch <talljimbo@gmail.com> wrote:
If you go with writing your own PyTypeObject, you will indeed have a lot more control, but it will greatly limit how much Boost.Python you can use (no class_, for instance, at least), and you'll need to dive deep into the Boost.Python implementation to learn how and when you can use it. I'd only consider recommending this approach if you wanted to wrap one or two simple classes this way and then use regular Boost.Python for everything else.
No class_<..> type would be a problem, as I've already exposed a lot of its derived classes this way.. There's got to be about 100 in total. :-\
I think the best solution would probably be to use shared_ptr with a custom deleter, as that gives you control over how your objects are allocated while giving Boost.Python an object it knows how to handle extremely well. One key ingredient of this is that instead of wrapping C++ constructors, you'll want to wrap factory functions that return shared_ptrs. You can even wrap such functions as Python constructors using make_constructor.
I've already had to do this once, so I've got some experience with the technique, although can't remember exactly why it was needed. Thank you for a viable option, though!
All that said, my first recommendation would be to try to wrap it (or at least a subset of it) without trying to get the optimal memory performance first, and only fix it if it actually is a performance problem. You might be surprised at where the time ends up going.
:) Lol, yes, I can see how attempting to define a new PyTypeObject could become very time-consuming!
If I were to go this route though (I probably will), is there a Boost Python registry or something where I can register these new types? I just thought to look into list.hpp, as I figure the boost::python::list would probably make use of the PyListType, which could be thought of as a similar PyTypeObject specialisation to one I might like to create.
You can't really register the types themselves. All you can do is register custom converters for them, i.e. You'll need to read the code and comments in the "converter" subdirectories of the Boost.Python source to really learn how to do that, though I think there are some how-tos scattered about the web. That would be enough to allow you to wrap functions and member functions that take these objects using boost::python::make_function and the like, and you could then add those to your type object using C API calls (PyObject_SetAttr) or their Boost.Python equivalents. Even so, you're starting down what seems like a really painful road.
......
Now I've done a little more digging, I think I have an idea of how to do it, which I'll detail now.. Any tips, pointers or advice in Boost::Python'ising it would be appreciated, as I have next to no experience with the Python-C API beyond Boost.
Resources ---------
Worth mentioning that the best (simplest) resources I've found to go by, are:
[1] - For the C-Python side: http://docs.python.org/2/extending/newtypes.html [2] - For Boost-Python side: http://www.boost.org/doc/libs/1_53_0/boost/python/list.hpp
Steps to expose:- -----------------
a. - Create mytypeobject.h, a raw C-Python extension type, as explained in [1].
a.i. - Create unit-test, by deriving such a type in Python, making a few instances, deleting some and leaving others to the garbage collector. b. Create mytypeobject.hpp, where a type similar to boost::python::list is declared. Register it, with for example:-
// From list.hpp:- // namespace converter { template <> struct object_manager_traits<list> : pytype_object_manager_traits<&PyList_Type,list> { }; }
Doing this sort of thing will allow you to get a Python object that's an instance of your PyTypeObject. It might be a useful bit of utility code, but it's really not directly what you want, I think, which is to be able to convert between Python instances of your class and C++ instances.
What else? ----------
I very much doubt it will be this simple, as any exposed class_<> would probably still attach a typical PyObject, thereby using PyTypeObject's memory management functions, rather than that of any type object specialisation.
Example -------
For example, let's suppose PyListObject has some functionality I need in an exposed class, and I need the garbage collector to use PyListType instead of PyTypeObject, when managing its memory.
Is there a way to attach a PyListObject to a class_<> instance, instead of a PyObject? Perhaps an alternative to the class_<> template could be designed to use any arbitrary type, instead of the default PyTypeObject. Something which could be used like this would be cool:-
boost::python::type_object_<PyListType, PyList, [boost::noncopyable]> ("alt_list", "PyListObject wrapped in Boost.", init<>()) .def("append", PyList_Append) /// ... ;
How much farther do you think I would need to dig into Boost internals to implement such functionality? Worth it?
It's pretty much definitely not worth it, IMO; you'd have to essentially duplicate and rewrite major parts of Boost.Python to support putting a custom PyTypeObject in a class_. The class_ infrastructure relies very heavily not just on its own PyTypeObject hiearchy, but also on a custom metaclass. In fact, now that I think about it, you'll probably need to do some of that even if you don't try to use class_ or something like it. I was originally thinking that maybe you could get away with essentially wrapping your own classes just using the Python C API directly (i.e. following the approach in the "extending and embedding" tutorial in the official Python docs), but then use Boost.Python to wrap all of your functions and handle type conversion. But even that seems like it's pretty difficult. So I guess the summary is that I think you may be making a mistake by taking this approach, but I'm sure you'll learn something either way. You've been warned ;-) Jim
Hi, Thanks again for the fast response! On Thu, 18 Apr 2013 17:14:05 +0100, Jim Bosch <jbosch@astro.princeton.edu> wrote:
You can't really register the types themselves. All you can do is register custom converters for them, i.e. You'll need to read the code and comments in the "converter" subdirectories of the Boost.Python source to really learn how to do that, though I think there are some how-tos scattered about the web.
Registered converters do sound another good option; I have seen some nice how-tos describing their usage. Thanks again for another decent suggestion. So far, I've found the class_<> template so easy to use that I've hardly needed to touch converters, nor any form of registry.
That would be enough to allow you to wrap functions and member functions that take these objects using boost::python::make_function and the like, and you could then add those to your type object using C API calls (PyObject_SetAttr) or their Boost.Python equivalents. Even so, you're starting down what seems like a really painful road.
Yes, I think you're right; it does sound a very long and painful road! But when I use a generator to iterate through containers holding millions of instances - which I totally intend to do - I'd like the base classes of these instances to be as lightweight as possible. I don't know much (anything) about the implementations, but I've read that Cython's memoryview's are amazingly efficient at extracting lightweight instances from C/C++ containers. I think numpy arrays might be similar in this regard. From what I've been seen today, I imagine there are specialised PyTypeObject's somewhere in their midsts.
Doing this sort of thing will allow you to get a Python object that's an instance of your PyTypeObject. It might be a useful bit of utility code, but it's really not directly what you want, I think, which is to be able to convert between Python instances of your class and C++ instances.
Yes, I was hoping this memory-managed base class could be used very similarly to both Python's and Boost's 'object'. Registered converters seem like a must in this regard.
It's pretty much definitely not worth it, IMO; you'd have to essentially duplicate and rewrite major parts of Boost.Python to support putting a custom PyTypeObject in a class_. The class_ infrastructure relies very heavily not just on its own PyTypeObject hiearchy, but also on a custom metaclass.
I wouldn't want to repeat anything - I'm familiar with the principles D.R.Y. and KISS... Inheriting functionality from class_<> and overriding specific members would be preferable, assuming its constructors doesn't explicitly use PyTypeObject. Is that impossible, given the current class_ template inheritance chain?
In fact, now that I think about it, you'll probably need to do some of that even if you don't try to use class_ or something like it. I was originally thinking that maybe you could get away with essentially wrapping your own classes just using the Python C API directly (i.e. following the approach in the "extending and embedding" tutorial in the official Python docs), but then use Boost.Python to wrap all of your functions and handle type conversion. But even that seems like it's pretty difficult.
Yes, that does sound tough! I've struggled to understand the C-Python docs before, and can't waste too much time atm repeating the process..
So I guess the summary is that I think you may be making a mistake by taking this approach, but I'm sure you'll learn something either way. You've been warned ;-)
Thanks for the warnings, and please excuse my naiivety. It was late last night when my real problem made me think of defining a new PyTypeObject, but I should definitely steer clear of any potential pitfalls and time-thieves atm. I should just get it working, first off... Actual problem -------------- My actual problem? (Sorry, it's hard to explain without an inheritance diagram... So I just made one - see the png attached.) Yesterday, I witnessed first hand the diamond of death multiple inheritance problem. I had extended the Base class to the memory-managed class - lets call these BaseObj and ManagedObj, respectively - with an additional 'Print(void)' method, to be exposed in Python as PyBaseObj.__repr__. ADerived (and about 100 other classes) all inherit from ManagedObj, and I thought it could be possible to call 'this->Print()', in e.g. PyADerived's methods. As the libraries I'm using don't use virtual inheritance, I soon learnt this would be impossible, but I figured I could add 'bases<PyBaseObj, PyManagedObj>' to the 'class_<PyDerivedObj, ...>' template constructor, and call it from C++ with: 'bp::object(aderived_obj).attr("__repr__")()'. So I've done that, which seems to work fine, but there do seem to be problems in the registry when calling certain derived class methods. I've got the old error:- Boost.Python.ArgumentError: Python argument types in PyBDerived.GetA(PyBDerived) did not match C++ signature: GetA(PyBDerived{lvalue}) The return type (which isn't mentioned in the above error) is a reference to another class - PyADerived - which also indirectly derives from ManagedObj. I can't figure out exactly why this breaks - I've played a lot with constness, to no avail - but I think the fix might be to expose ManagedObj directly, and not a class derived from it, which is what I've been doing fairly repeatedly up until now... BaseObj, however, would have to be derived, as it has pure-virtual member functions. It would probably have been much more sensible to write the Print method as a free function, and attach it to ManagedObj with def(...), rather than exposing as a derived class member function. I don't actually need any functionality in BaseObj that I can't call from a derived class, so I think I should just skip exposing that and attach the 'Print' method directly to the exposed ManagedObj... Apologies for the rambling and the lengthening emails! Thanks again for taking the time to write out your thoughts. Kind regards, Alex -- Using Opera's mail client: http://www.opera.com/mail/
Hi again, Whilst hoping for a reply, I thought I'd add some further insights I've learnt about the current PyTypeObject scheme. On Thu, 18 Apr 2013 17:14:05 +0100, Jim Bosch <jbosch@astro.princeton.edu> wrote:
I was originally thinking that maybe you could get away with essentially wrapping your own classes just using the Python C API directly (i.e. following the approach in the "extending and embedding" tutorial in the official Python docs), but then use Boost.Python to wrap all of your functions and handle type conversion. But even that seems like it's pretty difficult.
Well, this is basically what I did... I started playing around with noddy_NoddyType[1], to see what I could learn about Boost.Python's current way of things. I don't want to break my code just yet, nor fall any more of a victim to the early optimisation problem, but here we go anyway.. Exposing a simple type ---------------------- Wrapping noddy_NoddyObject is about as simple as it gets: ///////////////////////////////////////////////////////////////// // // File: noddy.hpp #include "noddy.h" // (code from [1]) struct NoddyClass : noddy_NoddyObject { NoddyClass(void) : noddy_NoddyObject() {} ~NoddyClass() {} }; // Register converter for classes derived from NoddyClass. // i.e. Tell Boost::Python to use noddy_NoddyType members. // // N.B. For this to compile, noddy_NoddyType must be forward // declared like PyListType. // // i.e. // PyAPI_DATA(PyTypeObject) noddy_NoddyType; // // Then, don't(!) declare noddy_NoddyType as static. namespace boost { namespace python { namespace converter { template <> struct object_manager_traits<NoddyClass> : pytype_object_manager_traits<&noddy_NoddyType, NoddyClass> { }; } } } ///////////////////////////////////////////////////////////////// // // File: noddy.cpp #include "noddy.hpp" #include <boost/python/module.hpp> #include <boost/python/class.hpp> BOOST_PYTHON_MODULE(noddy) { boost::python::class_<NoddyClass, boost::noncopyable> ("Noddy", "Empty Python object using custom PyTypeObject", boost::python::init<>()[ ...CallPolicies... ]) // ... ; } ----------------- Analysis -------- Now, I'm not exactly experienced when it comes to disassembling and the like, but an easy observation to make, from the Python side, is to use sys.getsizeof() :-
import sys, noddy nod = noddy.Noddy() print sys.getsizeof(nod) 80
Okay, 80 bytes apparently. And a plain old Python object?
obj = object() print sys.getsizeof(obj) 16
If I use only the C-Python API, compiling the code exactly as listed in [1], then:-
print sys.getsizeof(nod) 16
Relatively, that's quite a vast size difference (x5), so I thought I'd look for an explanation. I found it in a couple of places: the python wiki [2], and instance.hpp[3]. Sequence objects ---------------- As mentioned on the wiki, amongst other things, a wrapped object is the size of "the extra size required to allow variable-length data in the instance". This is fixed in instance.hpp, where Boost uses the 'PyObject_VAR_HEAD' macro at the top of the instance template. This is instead of the comparatively much smaller 'PyObject_HEAD' macro used in simple objects. A fairer test then, might be to compare a Python list() to a Boost.Python-wrapped NoddyObject:-
print sys.getsizeof(list()) 72
The relative size difference is now much smaller (x10/9), but I can't explain the last lost 8 bytes. There are apparently "zero or more bytes of padding required to ensure that the instanceholder is properly aligned." I don't really understand why this is necessary; isn't the compiler supposed to decide how to align objects? Well, apparently numpy arrays do the same thing[4]. NumPy ----- So, what about numpy arrays?
print sys.getsizeof(numpy.array([])) 80
Oh, that's impressive! It's identical. With that in mind, I would believe that the object is as good as it gets for sequence types. But what about simple numpy datatypes?
print sys.getsizeof( numpy.float64() ) 24
Ah, now there's a noticeable difference (x10/3). But, I think Boost objects could be made identical in this respect, with some time and dedication... ------------------ It doesn't appear that anyone has ever had an issue with this design, but it seems to me that there is large room for improvement in memory efficiency, when it comes to simple data-types managing only one thing at a time. Notes ----- * I can't see any effect from the object_manager_traits<> call. Doesn't seem to matter if I use it or not, but that's probably because I haven't done anything special with noddy_NoddyType. * I tried some different CallPolicies on the class_<> init method: the default; return_internal_reference; and copy_const_reference. None of these affect object size. * A couple of open source extension modules where I've seen PyTypeObject's used: Numpy API docs [4] and PythonQt source code [5]. ------------------ Now I'm guessing that there are no plans to improve the memory usage of bp::object's. From what Jim has said and from what I've seen in 'class.hpp', 'object_base.hpp' and other header files, it would probably require some dramatic modifications. But could it be as simple as adding a template to instance.hpp? Probably not.. It's not currently in my capacity to rewrite large portions of boost python; I'm far too new to Boost and C++ in general to begin to attempt this really, and I've got a thesis to write atm, too... Back to previous use case ------------------------- From what I've seen in my extension library (and using the classes from my last email), every ManagedObj could be designated as a simple datatype; there are (far fewer) dedicated containers for managing multiple instances of them; these would require using the current instance system, but most of my exposed classes would benefit from a smaller Python instance. If anyone has any thoughts or ideas on how to squeeze extra performance from simple PyTypeObject's, please do share! Kind regards, Alex [1] - http://docs.python.org/2/extending/newtypes.html [2] - http://wiki.python.org/moin/boost.python/InternalDataStructures [3] - http://www.boost.org/doc/libs/1_53_0/boost/python/object/instance.hpp [4] - http://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html [5] - http://pythonqt.sourceforge.net/PythonQtClassWrapper_8h_source.html
participants (3)
-
Alex Leach -
Jim Bosch -
Jim Bosch