Mailman 3 June 2007 - Cplusplus-sig

using intrusive_ptr objects with boost python
by Jeff Webb July 9, 2007

July 9, 2007

Let me first say that I am extremely impressed with boost python's elegant and powerful design. I would not have thought that it was possible to integrate C++ and python code in such a nice manner. Thanks for all the hard work and thoughtful design. I am writing a C++ library from scratch, and I want to develop boost python bindings for it simultaneously. My C++ library will instantiate many small objects that hold references to other objects. I had come up a consistent scheme for managing the objects using C++, but when I started trying to wrap the library using boost, I found that this scheme didn't mesh well with python's reference counting form of memory management. After quite a bit of reading, experimentation, and contemplation, it appears to me that using reference-counting smart pointers in C++ would be the most consistent and seamless way to provide a rich python binding for this type of library. Do others agree with this assertion? I wrote a simple test library using shared_ptr objects and found that wrapping it was very natural. Things 'just worked' as I would have hoped. The source code for this library is attached to this email as 's_ptr_lib.cpp'. A sample ipython session is shown below. In [1]: from s_ptr_lib import * In [2]: c1 = C('c1'); c2 = C('c2') C(c1) C(c2) In [3]: c1.set_ptr(c2) In [4]: ptr = c1.get_ptr() In [5]: ptr is c2 Out[5]: True In [6]: del(c2) In [7]: del(ptr) In [8]: del(c1) ~C(c1) ~C(c2) Very nice! Unfortunately, I think the overhead of using a shared_ptr will be too high, since my library will instantiate many small objects. I then discovered the intrusive_ptr, and I think it would be a better choice. I then converted my library to use intrusive_ptrs instead of shared_ptrs. The source code for this library is attached as 'i_ptr_lib.cpp'. When I tried to use this library from python, I found that the intrusive_ptr was not as well supported as the shared_ptr. A sample ipython session is shown below: In [1]: from i_ptr_lib import * In [2]: c1 = C('c1'); c2 = C('c2') C(c1) C(c2) In [3]: c1.set_ptr(c2) ------------------------------- Boost.Python.ArgumentError Traceback (most recent call last) ArgumentError: Python argument types in C.set_ptr(C, C) did not match C++ signature: set_ptr(C {lvalue}, boost::intrusive_ptr<C>) It seems that I will need to supply all the necessary type conversions for the intrusive_ptr. I was hoping to not have to dig that deep into boost python's template magic at this point... Is there any good documentation on how to do this? I found the following post that seemed promising: http://mail.python.org/pipermail/c++-sig/2007-February/011961.html I downloaded the 'register_intrusive_ptr_from_python.hpp' and 'intrusive_ptr_from_python.hpp' file attachments from the above link and then modified my i_ptr_lib.cpp program by adding '#include register_intrusive_ptr_from_python.hpp' near the top, and this line after the end of the class_<C> declaration: boostPatch::register_intrusive_ptr_from_python_and_casts( (C *)0, class_<C>::metadata::bases() ); I then tested the library again: In [1]: from i2_ptr_lib import * In [2]: c1 = C('c1'); c2 = C('c2') C(c1) C(c2) In [3]: c1.set_ptr(c2) In [4]: del(c2) ~C(c2) In [5]: c1.get_ptr() ------------------------ exceptions.TypeError Traceback (most recent call last) TypeError: No to_python (by-value) converter found for C++ type: boost::intrusive_ptr<C> The set_ptr method no longer generates an exception, but c2's lifetime is not tied to c1 like it is for the shared_ptr version (the destructor gets called when c2 gets deleted). Another type conversion problem pops up in 'In[5]'. Am I on the right path here? Is it possible to make the intrusive_ptr work as seamlessly as the shared_ptr, or is there some fundamental problem with what I hope to achieve? Is there some technical reason why this is not already implemented, or is it just that there hasn't been a need up to this point? If I should continue down this path, can someone give me a skeleton of the code I need to write (and how to get it 'registered') to provide all the functionality that exists for the shared_ptr class? Thanks again, Jeff Webb s_ptr_lib.cpp -------------------------------------------------------------- #include <boost/python.hpp> #include <boost/shared_ptr.hpp> #include <string> #include <iostream> using namespace std; using namespace boost; using namespace boost::python; class C; typedef shared_ptr<C> CPtr; class C { public: C(const string &s) : name(s) { cout << "C(" << name << ")" << endl; } ~C() { cout << "~C(" << name << ")" << endl; } void set_ptr(CPtr ptr) { this->ptr = ptr; } void clear_ptr() { ptr.reset(); } CPtr get_ptr() { return ptr; } private: string name; CPtr ptr; }; BOOST_PYTHON_MODULE(s_ptr_lib) { class_<C>("C", init<const string&>()) .def("set_ptr", &C::set_ptr) .def("clear_ptr", &C::set_ptr) .def("get_ptr", &C::get_ptr) ; } i_ptr_lib.cpp -------------------------------------------------------------- #include <boost/python.hpp> #include <boost/intrusive_ptr.hpp> #include <string> #include <iostream> using namespace std; using namespace boost; using namespace boost::python; class C; typedef intrusive_ptr<C> CPtr; namespace boost { void intrusive_ptr_add_ref(C * ptr); void intrusive_ptr_release(C * ptr); } class C { public: C(const string &s) : name(s) { cout << "C(" << name << ")" << endl; } ~C() { cout << "~C(" << name << ")" << endl; } void set_ptr(CPtr ptr) { this->ptr = ptr; } void clear_ptr() { intrusive_ptr_release((C*) ptr.get()); } CPtr get_ptr() { return ptr; } private: friend void boost::intrusive_ptr_add_ref(C * ptr); friend void boost::intrusive_ptr_release(C * ptr); string name; CPtr ptr; int ref_count; }; namespace boost { inline void intrusive_ptr_add_ref(C * ptr) { ++(ptr->ref_count); } void intrusive_ptr_release(C * ptr) { if (--(ptr->ref_count) == 0) delete ptr; } } BOOST_PYTHON_MODULE(i_ptr_lib) { class_<C>("C", init<const string&>()) .def("set_ptr", &C::set_ptr) .def("clear_ptr", &C::set_ptr) .def("get_ptr", &C::get_ptr) ; }

3 5

docstring in enum
by Piotr Jaroszyński July 9, 2007

July 9, 2007

Hello, Is it somehow possible to add docstring to enum_? I tried with PyObject_SetAttrString, but it doesn't work. Don't you think that docstrings would be useful in enums as well? -- Best Regards, Piotr Jaroszyński

2 3

Re: [C++-sig] Boost.Python Memory Leaks?
by Ralf W. Grosse-Kunstleve July 2, 2007

July 2, 2007

This has come up many times before: Boost.Python doesn't support PyFinalize(). There is no way to cleanup the global type conversion registry. It wasn't on the agenda of the people funding Boost.Python version 2 development (i.e. embedding wasn't on the list of things considered important). The leaks are harmless. Just don't use PyFinalize() and ignore the memory occupied by the few globals Boost.Python instantiates. My way of finding "real" leaks is very simple: wrap your code in an endless loop and monitor the process with the top command (Linux or OSX). Let it run for a couple of hours and check again. I've done this many times (to check my own new C++ code). There are no "real" leaks in Python or Boost.Python. Ralf ____________________________________________________________________________________ Shape Yahoo! in your own image. Join our Network Research Panel today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7

5 8

type traits
by Piotr Jaroszyński June 30, 2007

June 30, 2007

Hello, I thought (and saw) that boost.python is using type traits to figure out the real type of polymorphic pointers/references and hence I am really surprised that with the following code it doesn't happen: #include <boost/python.hpp> namespace bp = boost::python; struct Base { }; struct Derived : Base { }; const boost::shared_ptr<Base> get_derived() { static boost::shared_ptr<Base> foo(boost::shared_ptr<Derived>(new Derived())); return foo; } BOOST_PYTHON_MODULE(type_magic) { bp::class_<Base>("Base", bp::no_init); bp::register_ptr_to_python<boost::shared_ptr<Base> >(); bp::class_<Derived, bp::bases<Base> >("Derived", bp::no_init); bp::def("get_derived", &get_derived); } in python get_derived gives me: <type_magic.Base object at 0x2adb9e8ba848> For the "saw" part, I have working code, which does exactly what I expected (getting a Derived object in python) and I can't really find a difference between it and this basic testcase, but I am probably just missing something. Any clues what that could be? -- Best Regards, Piotr Jaroszyński

2 2

Pickling Enums
by Tom Brooks June 30, 2007

June 30, 2007

Hello, I've searched the web, and this question has come up a couple of times, but I haven't been able to find an answer. My problem is that I get an error when trying to pickle an enum that's been exposed with Boost::Python. enum BlendType { NoBlend, Transparent, Additive }; BOOST_PYTHON_MODULE(Renderer) { enum_<BlendType>("BlendType") .value("NoBlend",NoBlend) .value("Transparent",Transparent) .value("Additive",Additive) .export_values() ; } The error is very long, but here's the end part of it: ... save(element) File "C:\Python24\lib\pickle.py", line 293, in save f(self, obj) # Call unbound method with explicit self File "C:\Python24\lib\pickle.py", line 760, in save_global raise PicklingError( pickle . PicklingError : Can't pickle <class 'Renderer.BlendType'>: it's not found as __main__.Renderer.BlendType If it matters, this pickling happens when I pickle another class. I have no idea where to start, since you can't def_pickle or even .def for an enum_. How do I go about doing this? Thanks in advanced

1 0

[pygccxml] caching fails due to pickling recursion limit
by Kirill Lapshin June 30, 2007

June 30, 2007

Hi Roman, everybody, First of all my apologies if it is a wrong list for pygccxml discussion. We are using py++/pygccxml on moderately sized project, and recently py++ started to fail silently on us. It runs gccxml and then silently quits. It turned out the offending piece of code was pickling cache. Something fails in cPickle and terminates the whole process. Most likely it is a stack overflow, because if cPickle replaced with pickle, then we get recursion limit runtime error. One can raise recursion limit and pickle will save cache just fine, but really slowly (not surprising at all). At the moment we've worked around this problem by disabling cache, however it would be nice to fix it properly. After reading a bit about pickling, it looks like (c)pickle don't work very well with recursive structures, which declarations obviously are. I just want to list some ideas I have on how to tackle this problem, and I would love to get some comments on their feasibility, or better yet alternative approaches. 1. Simple one, just set all parents to None before pickling and rebuild parents upon loading. Should be relatively simple, I'll try it first thing. Hopefully that would kill fair amount of recursion, but not all of it (e.g. class A refers to class B and B to A). 2. Add unique id to each declaration, have a global store that saves declarations by id, and define for each declaration __getstate__, __setstate__ that would serialize ids, rather then declaration instances. Then we just pickle the global store. In reality somewhat more complex approach needed to tackle recursive references. References: Reversible flatting out: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302535 Some details: we are running: Python 2.4.3 on Windows XP Py++/pygccxml 0.9 gccxml -- somewhat old CVS snapshot Kirill

2 3

Re: [C++-sig] Boost.Python Memory Leaks?
by Ralf W. Grosse-Kunstleve June 29, 2007

June 29, 2007

What you see is the best we could do given the time and money we had. To give people like you a chance to improve on what we did, we released everything as open source. We hope this encourages others to contribute back. I have a couple of years of my life in Boost.Python (and Dave a lot more, of course). It works perfectly for my purposes. IMHO embedding is bad design from the outset. :-) Ralf ____________________________________________________________________________________ Finding fabulous fares is fun. Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains. http://farechase.yahoo.com/promo-generic-14795097

1 0

Seeking advice on wrapping library with boost::python
by Bobby R. Ward June 29, 2007

June 29, 2007

I'm looking to create a python module wrapping a C library and C is most certainly not my specialty. The library has structs in the the form struct sub_sub_item { // stuff }; struct sub_item { //stuff int num_sub_sub_items; sub_sub_item *sub_sub_items; }; struct item { //stuff int num_sub_items; sub_item *sub_items; }; The module would have to allow adding and removing from the arrays and wrapping the "stuff". Performance is not critical but it is fairly important. What would be the best way to go about creating this? Would I be better off creating the module some other way?

1 0

Using py++ GCCXML error (type definition ?)
by Coockie_jr June 28, 2007

June 28, 2007

Hi! I've trying to use py++ for my project, it seems like a great tool but I feel a bit lost and I need your help ! I've installed gccxml,pygccxml, boost, python and py++ in their latest version but I'm having problems when it comes to use py++ ( gui ) on my project files! I'm currently testing it on very simple declaration files (no templates .., only simple functions) but I'm having gccxml errors during the code generation. "Error occured while running GCC-XML: C:/projet/include/Address.h:11: error: syntax error before `&' token" And here is the Address.h's 11th line : Address(Inet& ip,unsigned short port); The problem obliviously seems to be the Inet type (which is defined in an other .h file in the same directory as Address.h) that GCCXML doesn't recognize. But I don't know how to solve this problem. If I comment this line py++ works great. Being completely noob in boost.pyhton and py++ and afraid i'm missing an important step. :( Is this the case ? Thank you and sorry for my simple English ... -- View this message in context: http://www.nabble.com/Using-py%2B%2B-GCCXML-error-%28type-definition--%29-t… Sent from the Python - c++-sig mailing list archive at Nabble.com.

2 9

Boost.Python Memory Leaks?
by Lawrence Spector June 27, 2007

June 27, 2007

I was testing out some of my Boost.Python code using the Boost.Test framework. During the running, I got memory leaks reported. I ran a simple test to verify, which produced the following output: Running 1 test case... *** No errors detected Detected memory leaks! Dumping objects -> {168} normal block at 0x00366488, 12 bytes long. Data: < > 10 A5 03 10 A0 A5 03 10 00 00 00 00 {166} normal block at 0x003663F8, 12 bytes long. Data: < > C0 A2 03 10 80 A3 03 10 00 00 00 00 {164} normal block at 0x00366368, 12 bytes long. Data: < > D0 84 03 10 00 00 00 00 00 00 00 00 {163} normal block at 0x00366330, 8 bytes long. Data: < > D0 84 03 10 00 00 00 00 {161} normal block at 0x003662A0, 12 bytes long. Data: < > B0 A1 03 10 F0 A1 03 10 00 00 00 00 {159} normal block at 0x00366210, 12 bytes long. Data: < > B0 A0 03 10 F0 A0 03 10 00 00 00 00 {157} normal block at 0x00366180, 12 bytes long. Data: <p > 70 9E 03 10 00 9F 03 10 00 00 00 00 {155} normal block at 0x003660F0, 12 bytes long. Data: <` > 60 9D 03 10 A0 9D 03 10 00 00 00 00 {153} normal block at 0x00366060, 12 bytes long. Data: <P > 50 9C 03 10 90 9C 03 10 00 00 00 00 {151} normal block at 0x00365FD0, 12 bytes long. Data: <@ @ > 40 9A 03 10 40 9B 03 10 00 00 00 00 {149} normal block at 0x00365F40, 12 bytes long. Data: < > C0 98 03 10 00 99 03 10 00 00 00 00 {147} normal block at 0x00365EB0, 12 bytes long. Data: < > B0 96 03 10 80 97 03 10 00 00 00 00 {145} normal block at 0x00365E20, 12 bytes long. Data: < > A0 95 03 10 E0 95 03 10 00 00 00 00 {143} normal block at 0x00365D90, 12 bytes long. Data: < > 90 94 03 10 D0 94 03 10 00 00 00 00 {141} normal block at 0x00365D00, 12 bytes long. Data: < > 80 93 03 10 C0 93 03 10 00 00 00 00 {139} normal block at 0x00365C70, 12 bytes long. Data: <p > 70 92 03 10 B0 92 03 10 00 00 00 00 {137} normal block at 0x00365BE0, 12 bytes long. Data: <` > 60 91 03 10 A0 91 03 10 00 00 00 00 {135} normal block at 0x00365B50, 12 bytes long. Data: <P > 50 90 03 10 90 90 03 10 00 00 00 00 {133} normal block at 0x00365AC0, 12 bytes long. Data: < > B0 8E 03 10 80 8F 03 10 00 00 00 00 {131} normal block at 0x00365A30, 12 bytes long. Data: < > 10 8D 03 10 E0 8D 03 10 00 00 00 00 {129} normal block at 0x003659A0, 12 bytes long. Data: < > 90 8B 03 10 20 8C 03 10 00 00 00 00 Object dump complete. The code that produced this is as follows: BOOST_AUTO_TEST_CASE(testPythonDebugIntegration) { { Py_Initialize(); { boost::python::object m_mainModule((boost::python::handle<>( boost::python::borrowed(PyImport_AddModule("__main__"))))); boost::python::object m_mainNamespace(m_mainModule.attr("__dict__")); } Py_Finalize(); } } If I change the code to either of the following: 1) BOOST_AUTO_TEST_CASE(testPythonDebugIntegration) { { Py_Initialize(); { boost::python::object m_mainModule( ( boost::python::borrowed(PyImport_AddModule("__main__")))); } Py_Finalize(); } } OR 2) BOOST_AUTO_TEST_CASE(testPythonDebugIntegration) { { Py_Initialize(); { } Py_Finalize(); } } Then I get no memory leaks reported. Anyone have any clue why this is happening? The code I used was basically the suggested code from the Python tutorial from the Boost.Python page. Thanks in advance, Lawrence

1 0