Hello, I am considering trying cppyy on a large project, and I have a question. I need to be able to add a reference to Python objects into the c++ classes. In essence: #Python code: x="My python object" inst = cpp_class() inst.add_data(x) inst.do_stuff()#This does not modify x in any way z=inst.get_data(x) So, cpp_class acts as a container for python objects, and it does not need to know what kind of object is inside. I was able to make it work with pypy through basic C API, using SWIG and typemaps by mapping void* into PyObject*, but that is somewhat hackish. I am now thinking how would the same logic look with cppyy extension. PS: building cppyy is pain in the ass... You guys seriously need to think about build logs =)
Hi Alex,
So, cpp_class acts as a container for python objects, and it does not need to know what kind of object is inside. I was able to make it work with pypy through basic C API, using SWIG and typemaps by mapping void* into PyObject*, but that is somewhat hackish. I am now thinking how would the same logic look with cppyy extension.
this should be supported directly (PyROOT does), but where in CPython handing a PyObject* is trivial, in PyPy one has to be created. My best guess is that the route would be cpyext to get such a PyObject*, then hand that over. I don't think it can be hacked around (a void* argument will peel the contained C++ object, which will fail, since it's a real Python object). Identity preservation (on the return) should be easy, and the C++ side should keep the refcount up for the proper duration. Let me figure that one out. Is as good a reason as any to have some fun and do some coding again for the first time in weeks. :P
PS: building cppyy is pain in the ass... You guys seriously need to think about build logs =)
Sorry (but this is why at CERN we have a global file system (afs) and simply install there for all users :) ) ... So, is there anything in the instructions that was missing and needs to be added? Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
keskiviikko 30 toukokuu 2012 18:52:25 wlavrijsen@lbl.gov kirjoitti:
Hi Alex,
So, cpp_class acts as a container for python objects, and it does not need to know what kind of object is inside. I was able to make it work with pypy through basic C API, using SWIG and typemaps by mapping void* into PyObject*, but that is somewhat hackish. I am now thinking how would the same logic look with cppyy extension.
this should be supported directly (PyROOT does), but where in CPython handing a PyObject* is trivial, in PyPy one has to be created. My best guess is that the route would be cpyext to get such a PyObject*, then hand that over.
I don't think it can be hacked around (a void* argument will peel the contained C++ object, which will fail, since it's a real Python object).
Identity preservation (on the return) should be easy, and the C++ side should keep the refcount up for the proper duration.
Let me figure that one out. Is as good a reason as any to have some fun and do some coding again for the first time in weeks. :P
Thanks! I have accumulated some understanding about the process, and the issue can be worked around by having a dictionary that would map ints into python objects and vice versa before those are sent to c++, so when c++ functions feth them back they can be retreived. However, this would incur certain overhead, which may be rather annoying. Anyway, the whole point was to test if it will be any faster then cpyext, so unless this feature is properly resolved the test does not make any massive amount of sense (mostly due to this extra overhead). I'll probably run it anyway though=) Thanks for looking into it, its good to hear that somebody cares!
PS: building cppyy is pain in the ass... You guys seriously need to think about build logs =)
Sorry (but this is why at CERN we have a global file system (afs) and simply install there for all users :) ) ... So, is there anything in the instructions that was missing and needs to be added?
With respect to build, I installed root from gentoo repository, and the problem there is that headers are in /usr/include/root and the actual libs are in /usr/lib/root, but the pypy build expects libs to be in /usr/include/root/lib if you specify SYSROOT=/usr/include/root. So instead I ended up having to symlink those together to replicate whatever pypy build was expecting. This is indeed a hax, but probably if you install root from source package it is not needed. Anyway, it should be benefitial to add a header and library check in the beginning of the build to be certain that it does not fail after 20 mins of compiling. But for now symlink works just fine. With respect to genreflex command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes on standard headers. Also, any headers that are not absolutely necessary to define the interface to Python should be removed if possible, that does make life a lot easier, even with SWIG, because it makes wrappers smaller. Best Regards, Alex PS: If you want I could give you benchmark results of cppyy vs cpyext+SWIG when we get it running, probably it will be of some interest. I could give you the entire code, but it would not be much use since it requires a ton of other libraries.
Hi Alex,
Thanks! I have accumulated some understanding about the process, and the issue can be worked around by having a dictionary that would map ints into python objects and vice versa before those are sent to c++, so when c++ functions feth them back they can be retreived.
good you found a workaround, as using cpyext from cppyy is turning out to be a bigger can of worms that I expected. :} Will take a tad bit longer: I have some working code, but only post-translation. For some reason, the resulting objects look nothing as expected in the test and as passed, they go into the CPython interpreter (this is a test, after all), which then bombs. Post-translation, it's looking fine, though, and working as expected. Performance ... is absolutely deplorable: slower than PyROOT on CPython! My guess is that in the micro-benchmark, the JIT gets no chance of making a trace out of the loop iteration, since the loop iteration requires the PyObject in every call (the PyObject that I'm passing is the iterator variable itself). I'll have to look into more detail .... However, if I do not pass the loop iterator, but pass a constant, it is very, very zippy (though far from what can be done by unwrapping the constant). My guess here is that the JIT can move the constant out of the loop iteration, and this thus spares all those cpyext calls. Iow., it isn't actually fast, but the JIT has its way with the loop. Again, I need to look into that. So, I can't check in any code that doesn't pass a test, and since you'd have to rebuild anyway, let's hold off with checking this code in (unless you are happy with a patch?). I'll get to it soon enough.
With respect to build, I installed root from gentoo repository, and the problem there is that headers are in /usr/include/root and the actual libs are in /usr/lib/root, but the pypy build expects libs to be in /usr/include/root/lib if you specify SYSROOT=/usr/include/root.
Oops. Yes, that is something that can be solved easily enough.
With respect to genreflex command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes on standard headers.
That shouldn't be ... on my box I have gcc 4.6.2 and that does not pose any problems. What version of gccxml do you have? Mine's at 0.9.0.
Also, any headers that are not absolutely necessary to define the interface to Python should be removed if possible, that does make life a lot easier, even with SWIG, because it makes wrappers smaller.
Have you looked into providing a selection.xml to genreflex to select the needed classes? Also, another common option is to collect all desired headers in a single header file (with simple #include's) and feed that to genreflex, together with a selection.xml that simply uses "*" patterns.
PS: If you want I could give you benchmark results of cppyy vs cpyext+SWIG when we get it running, probably it will be of some interest.
My benchmarks are pure C++. :) But yes, I'm interested in any result, although for now, the binary benchmark "it works yes/no" is already interesting. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
good you found a workaround, as using cpyext from cppyy is turning out to be a bigger can of worms that I expected. :} Will take a tad bit longer: I have some working code, but only post-translation. For some reason, the resulting objects look nothing as expected in the test and as passed, they go into the CPython interpreter (this is a test, after all), which then bombs. Cpyext is not needed with cppyy as far as I understand, is it? Post-translation, it's looking fine, though, and working as expected.
Performance ... is absolutely deplorable: slower than PyROOT on CPython! Performance was originally the point, right? Or am I missing something (lol)?
My guess is that in the micro-benchmark, the JIT gets no chance of making a trace out of the loop iteration, since the loop iteration requires the PyObject in every call (the PyObject that I'm passing is the iterator variable itself). I'll have to look into more detail .... I am giving you my benchmark code, right now it uses dictionary to map pyobject id into objects when they leave c++ domain. Obviously, you add about O(2*log(N)) overhead to the code, but that should not be too much anyway. If that can be done in rpython, it could probably be made to work massively faster. Like a module that does object caching for the storage of objects that would be sent to c++ domain. Then in c++ one can do as simple as have an long int index that can be used to resolve back to byobject when the job is done. It also does classical SWIG typemap to do the same job. to use swig use SConstruct build. Or just use the run.sh script to get all the stuff immediately.
With respect to genreflex command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes on standard headers.
That shouldn't be ... on my box I have gcc 4.6.2 and that does not pose any problems. What version of gccxml do you have? Mine's at 0.9.0.
Also, any headers that are not absolutely necessary to define the interface to Python should be removed if possible, that does make life a lot easier, even with SWIG, because it makes wrappers smaller.
Have you looked into providing a selection.xml to genreflex to select the needed classes? Also, another common option is to collect all desired headers in a single header file (with simple #include's) and feed that to genreflex, together with a selection.xml that simply uses "*" patterns. The point is that if the header is included, and it confuses the parser, then
mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have older gcc in a different slot? the patterns have no effect. So it is better not to have extra headers like Qt library=) Alex.
Please, mention in the manual, for the mortals, that the reflection library has to be linked with -lReflex, so that they do not have to figure that out on their own=) BR, Alex. sunnuntai 03 kesäkuu 2012 02:33:15 Alex Pyattaev kirjoitti:
good you found a workaround, as using cpyext from cppyy is turning out to be a bigger can of worms that I expected. :} Will take a tad bit longer: I have some working code, but only post-translation. For some reason, the resulting objects look nothing as expected in the test and as passed, they go into the CPython interpreter (this is a test, after all), which then bombs. Cpyext is not needed with cppyy as far as I understand, is it? Post-translation, it's looking fine, though, and working as expected.
Performance ... is absolutely deplorable: slower than PyROOT on CPython! Performance was originally the point, right? Or am I missing something (lol)?
My guess is that in the micro-benchmark, the JIT gets no chance of making a trace out of the loop iteration, since the loop iteration requires the PyObject in every call (the PyObject that I'm passing is the iterator variable itself). I'll have to look into more detail .... I am giving you my benchmark code, right now it uses dictionary to map pyobject id into objects when they leave c++ domain. Obviously, you add about O(2*log(N)) overhead to the code, but that should not be too much anyway. If that can be done in rpython, it could probably be made to work massively faster. Like a module that does object caching for the storage of objects that would be sent to c++ domain. Then in c++ one can do as simple as have an long int index that can be used to resolve back to byobject when the job is done. It also does classical SWIG typemap to do the same job. to use swig use SConstruct build. Or just use the run.sh script to get all the stuff immediately.
With respect to genreflex command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes on standard headers.
That shouldn't be ... on my box I have gcc 4.6.2 and that does not pose any problems. What version of gccxml do you have? Mine's at 0.9.0.
Also, any headers that are not absolutely necessary to define the interface to Python should be removed if possible, that does make life a lot easier, even with SWIG, because it makes wrappers smaller.
Have you looked into providing a selection.xml to genreflex to select the needed classes? Also, another common option is to collect all desired headers in a single header file (with simple #include's) and feed that to genreflex, together with a selection.xml that simply uses "*" patterns. The point is that if the header is included, and it confuses the parser, then
mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have older gcc in a different slot? the patterns have no effect. So it is better not to have extra headers like Qt library=) Alex.
Hi Alex,
Performance ... is absolutely deplorable: slower than PyROOT on CPython! Performance was originally the point, right? Or am I missing something (lol)?
well, remember that there are two problems here: 1) since there are no PyObject's as such in pypy-c (they have different internal structure), one has the create them, to have a PyObject* pointer to them; and keep track of them to mix the reference counting scheme with the garbage collection scheme (this is all done in/by cpyext). And 2) now that the object is created, pinning down a variable in the .py code, the JIT can't unbox it. The example I used, which was like so: for i in range(N): a.add_data(i) and is rather pathological, of course. But then, all micro-benches are. Conversely, doing: l = [] for i in range(N): l.append(i) a.add_data(l) is very speedy, presumably (I don't know, guessing here as it may also be that there is another optimization for the list that I can't prevent even with the append() calls) because now the cpyext code is in a loop in and of itself, so e.g. lookup of the type of the objects in the container (all the same here) can become constant.
I am giving you my benchmark code
Which clearly is also pathological. :) I can't manage to run the SWIG test from pypy, though? I tried by adding cpyext, but no dice? Anyway, you'll be happy to know that: CPPYY Created container No more items real 0m2.236s user 0m2.156s sys 0m0.071s Cpython Created container No more items real 0m2.758s user 0m2.574s sys 0m0.174s (I verified with your debug=1 that the objects appear.) But that improvement is (I'm sure) only b/c all the other stuff you do around it with manipulating the dict on the python side. That does get JIT'ed just fine. Btw., I had to jack up N by 10x, otherwise all numbers were just noise. :}
right now it uses dictionary to map pyobject id into objects when they leave c++ domain. Obviously, you add about O(2*log(N)) overhead to the code, but that should not be too much anyway. If that can be done in rpython, it could probably be made to work massively faster. Like a module that does object caching for the storage of objects that would be sent to c++ domain. Then in c++ one can do as simple as have an long int index that can be used to resolve back to byobject when the job is done.
From a quick read of the cpyext code, that is rather similar to what is there.
But as per above, the problem isn't with the extra overhead, the problem is with the JIT being prevented to do any work (and with pypy-c being slower than CPython in pure interpreted mode).
mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have older gcc in a different slot?
I do, but those should not be accessible from my pypy work environment. I did rebuild it. Not sure whether that matters. I'll update the docs showing how a different set of headers from another compiler can be selected.
The point is that if the header is included, and it confuses the parser, then the patterns have no effect. So it is better not to have extra headers like Qt library=)
Qt is a story in and of itself, of course. :) Actually, what folks here do, is to link any code from external C++ through python to Qt (C++) by making use of the fact that both ends can handle opaque pointers, represented as long values. (The combination of addressof and bind_object in cppyy.) Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Okay, with respect to the SWIG + pypy, works for me, and about 4x slower then cppyy or cpython ( which show same results more or less). time cppyy ./cppyy_timings.py real 0m0.265s user 0m0.225s sys 0m0.040s time pypy ./pypy_timings.py real 0m2.424s user 0m2.338s sys 0m0.086s time python2 ./pypy_timings.py real 0m0.414s user 0m0.390s sys 0m0.024s So apparently pypy is dramatically bad with SWIG. cppyy seems to work nice. I have, however, encountered an interesting problem. Say I have a dictionary built. I'd like to know which symbols/classes are actually available (if any), but I can not do that. The implication is that in case your library does not have any symbols in it after SUCCESSFUL build, there is no way to figure out why and how that happened. Please advise. Also I can bench the entire thing on a real, quite large app that makes intensive use of pypy and also runs on python2 if needed, but for that we would need to figure why no classes are visible in the dictionary=) BR, Alex sunnuntai 03 kesäkuu 2012 16:44:19 wlavrijsen@lbl.gov kirjoitti:
Hi Alex,
Performance ... is absolutely deplorable: slower than PyROOT on CPython!
Performance was originally the point, right? Or am I missing something (lol)? well, remember that there are two problems here: 1) since there are no PyObject's as such in pypy-c (they have different internal structure), one has the create them, to have a PyObject* pointer to them; and keep track of them to mix the reference counting scheme with the garbage collection scheme (this is all done in/by cpyext). And 2) now that the object is created, pinning down a variable in the .py code, the JIT can't unbox it.
The example I used, which was like so:
for i in range(N): a.add_data(i)
and is rather pathological, of course. But then, all micro-benches are.
Conversely, doing:
l = [] for i in range(N): l.append(i) a.add_data(l)
is very speedy, presumably (I don't know, guessing here as it may also be that there is another optimization for the list that I can't prevent even with the append() calls) because now the cpyext code is in a loop in and of itself, so e.g. lookup of the type of the objects in the container (all the same here) can become constant.
I am giving you my benchmark code
Which clearly is also pathological. :) I can't manage to run the SWIG test from pypy, though? I tried by adding cpyext, but no dice?
Anyway, you'll be happy to know that:
CPPYY Created container No more items
real 0m2.236s user 0m2.156s sys 0m0.071s
Cpython Created container No more items
real 0m2.758s user 0m2.574s sys 0m0.174s
(I verified with your debug=1 that the objects appear.)
But that improvement is (I'm sure) only b/c all the other stuff you do around it with manipulating the dict on the python side. That does get JIT'ed just fine.
Btw., I had to jack up N by 10x, otherwise all numbers were just noise. :}
right now it uses dictionary to map pyobject id into objects when they leave c++ domain. Obviously, you add about O(2*log(N)) overhead to the code, but that should not be too much anyway. If that can be done in rpython, it could probably be made to work massively faster. Like a module that does object caching for the storage of objects that would be sent to c++ domain. Then in c++ one can do as simple as have an long int index that can be used to resolve back to byobject when the job is done.
From a quick read of the cpyext code, that is rather similar to what is there.
But as per above, the problem isn't with the extra overhead, the problem is with the JIT being prevented to do any work (and with pypy-c being slower than CPython in pure interpreted mode).
mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have older gcc in a different slot?
I do, but those should not be accessible from my pypy work environment. I did rebuild it. Not sure whether that matters. I'll update the docs showing how a different set of headers from another compiler can be selected.
The point is that if the header is included, and it confuses the parser, then the patterns have no effect. So it is better not to have extra headers like Qt library=)
Qt is a story in and of itself, of course. :) Actually, what folks here do, is to link any code from external C++ through python to Qt (C++) by making use of the fact that both ends can handle opaque pointers, represented as long values. (The combination of addressof and bind_object in cppyy.)
Best regards, Wim
Hi Alex,
Okay, with respect to the SWIG + pypy, works for me, and about 4x slower then cppyy or cpython ( which show same results more or less).
I'll try the SWIG portion later (as well as all the other points that you raise in this thread; I think the feedback is great). First, I now have the PyObject* tests (whew!) and that has allowed me to push the code. Also, I found one reason why my micro-benchmark was so slow: the objects were all kept alive (ref counting problem), meaning the number of objects to keep track of by cpyext kept growing, thus the lookups became slower and slower. The tests caught that. :) With that solved, micro-bench speeds are okay-ish. Then, rewriting your example to use PyObject* directly, there is one problem: the C++ side now needs an additional Py_IncRef (which is a function call, not a macro, in PyPy, adding extra overhead) since it takes ownership. Keeping the python objects alive on the python side does not work as well, since e.g. ints are apparently not pinned down, like I thought earlier, so you may get a fresh one with the same value on the C++ side. Note that if you use the typedef ref_t, a selection file is needed to tell Reflex about that typedef, or no binding will be generated for it. Alternative, you can use PyObject* directly (which is also a typedef, but I hard-coded that name in cppyy, together with the actual _object, for convenience). Okay, so speeds ... It's not too bad. As discussed before, your method and the cpyext one are mostly the same, so similar should be expected, and it is: CPPYY Created container No more items real 0m2.013s user 0m1.931s sys 0m0.073s Cpython Created container No more items real 0m2.722s user 0m2.556s sys 0m0.157s I then tried with the fast path enabled, but clearly all the time is being spent in cpyext, so I although it does show consistent better timings, the difference is only 0.02 secs. Maybe that could be faster, but cpyext is an impressive piece of work, quite a bit above my PyPy-skills. :} Overall, I'm happy enough with the result (I'm going to need handling of PyObject*'s later for my own nefarious purposes). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Tue, Jun 5, 2012 at 1:53 AM, <wlavrijsen@lbl.gov> wrote:
Overall, I'm happy enough with the result (I'm going to need handling of PyObject*'s later for my own nefarious purposes).
Maybe we can motivate you to look for and fix inefficiencies in cpyext :-) As far as I know it's still fully designed for "it works and we don't care about speed". See for example https://bugs.pypy.org/issue1121 which tries to push optimizations in cpyext but with no-one really interested from the core group. But IMHO the main slow part that should be optimized (probably by being rewritten from scratch) is the correspondance between the proxy PyObject structures and the real PyPy objects, which is exactly the part that interests you too. It should be possible at least to cope with only one dict lookup instead of at least two to do the roundtrip "pypy obj -> cpyext obj -> pypy obj". A bientôt, Armin.
Well, as far as I can see from where I am, the only real good that can come from interfacing with PyObject is the capability to write container classes in C++. On the other hand, it seems that it is not very often needed. For the case when objects just need to be LINKED from the C++ code as references, pure python/Rpython abstraction layer that would keep a backup copy of the object (which can be discarded from the python side) seems like a better idea. The main benefit is that there will be no refcounting problem and no slate pointers - if C++ returns some key that is not available then exception can be raised instead of segfault as with PyObject*. Also, no extra headers will be needed on C++ side, and the datatype can be an integer. Why not? As far as I understand, modifying python objects in C++ with pypy is massively stupid. By the way - the only reason i use C++ in my project is because I have a massive amount of legacy code that uses several C libraries and Qt, as well as openGL, so porting that bunch to python can be quite a hassle. The only problem is that Python side has to send certain requests to C++, those should be cached, processed and reported back up when done. Obviously, request object itself could travel along with the request data for code clarity, but there can just as well be an int. The only problem with ints is the lifetime of the references in the mapping structure, which would have to be managed explicitly in such case. If you are interested, I could send you some thoughts on that issue specifically, as I was planning to do this kind of thing at some point. BR, Alex. tiistai 05 kesäkuu 2012 09:54:25 Armin Rigo kirjoitti:
Hi Wim,
On Tue, Jun 5, 2012 at 1:53 AM, <wlavrijsen@lbl.gov> wrote:
Overall, I'm happy enough with the result (I'm going to need handling of PyObject*'s later for my own nefarious purposes).
Maybe we can motivate you to look for and fix inefficiencies in cpyext :-) As far as I know it's still fully designed for "it works and we don't care about speed". See for example https://bugs.pypy.org/issue1121 which tries to push optimizations in cpyext but with no-one really interested from the core group.
But IMHO the main slow part that should be optimized (probably by being rewritten from scratch) is the correspondance between the proxy PyObject structures and the real PyPy objects, which is exactly the part that interests you too. It should be possible at least to cope with only one dict lookup instead of at least two to do the roundtrip "pypy obj -> cpyext obj -> pypy obj".
A bientôt,
Armin.
Hi Alex,
By the way - the only reason i use C++ in my project is because I have a massive amount of legacy code that uses several C libraries and Qt, as well as openGL, so porting that bunch to python can be quite a hassle. The only problem is that Python side has to send certain requests to C++, those should be cached, processed and reported back up when done.
somehow I have a feeling that here's a general use case in there that should be supported in a more direct way, but I can't quite put my finger on it. One thing I'm missing, for example, is: what is doing the unpacking of the python objects (requests) into something that the C++ side understands? Another option may be that if the PyObject*'s can be reused (e.g. if there is a fixed set of kinds of requests), the overhead is much, much less, as the JIT can then consider them constants. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Armin,
But IMHO the main slow part that should be optimized (probably by being rewritten from scratch) is the correspondance between the proxy PyObject structures and the real PyPy objects, which is exactly the part that interests you too. It should be possible at least to cope with only one dict lookup instead of at least two to do the roundtrip "pypy obj -> cpyext obj -> pypy obj".
I'll need to understand first in more detail how cpyext works, but it is good motivation to know that a dict lookup can be removed. In cppyy, I have a similar, but reversed, problem: keeping track of all C++ objects to guarantee object identity. Since the bound object carries a pointer to the C++ object, a round trip has only one dict lookup, the other "lookup" being a data member access. If I understand correctly what you describe above, then perhaps giving the "PyPyObject by pointer" as payload to the PyObject could help, too. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Wed, Jun 6, 2012 at 9:10 AM, <wlavrijsen@lbl.gov> wrote:
If I understand correctly what you describe above, then perhaps giving the "PyPyObject by pointer" as payload to the PyObject could help, too.
Yes: so far, most PyObjects are small structures containing just the refcount and the cpyext-ified type. Note that for your use case you don't need the latter, which would simplify things a lot. If the goal is "only" to pass around PyObjects as black boxes, as fast as reasonably possible, then the easiest route would be to not use cpyext at all, but instead try to come up with your own mapping logic. This looks quite easier for experimentation purposes. And if you end up with a good and fast solution we can always try later to reuse it in cpyext. Trying to think about the API to use: there is pypy.rlib.rgc.malloc_nonmovable(TP), where TP is a GcStruct. It allocates a normal GC object but tries to make sure that it will never move. Raw pointers to that can be passed around to C++, but won't keep the GcStruct alive by themselves, of couse. I'm not sure if this is the best suited API; maybe we can discuss and make a better one too. For example: 1) a pair of functions to register and unregister a GC pointer that lives outside the GC 2) or maybe a way to register and unregister "external" GcStructs: you'd still allocate and free raw bytes for the PyObject wrapper, but making sure it has a GcStruct shape (i.e. some standard GC headers). You would then register and unregister it to be considered by the GC. This looks easy to implement: it is the same as we consider all the prebuilt GcStructs that exist at translation time. In other words, they are a source of further GC pointers, but which live themselves outside the scope managed by the GC. The difference with 1) is that the GC can still write flags in the header of these objects; it helps a lot generational GCs. A bientôt, Armin.
If the goal is "only" to pass around PyObjects as black boxes, as fast as reasonably possible, then the easiest route would be to not use cpyext at all, but instead try to come up with your own mapping logic. This looks quite easier for experimentation purposes. And if you end up with a good and fast solution we can always try later to reuse it in cpyext. This is precisely the goal for me. The requests in my system are basically a C++ classes (structs, essentially), that are created and submitted to C++ core as pointers, they also hold a pointer to the PyObject that submitted the request. When the request is served, the pyobject can be notified about that without having to go through any sort of lookups.
1) a pair of functions to register and unregister a GC pointer that lives outside the GC That might actually be a good idea. In that case a certain object can tell GC that it wants to have a valid external pointer, and when the object is destroyed, a __del__ function can remove that registration with GC. If anyone uses that pointer afterwards it is probably his/her own problem.
I think also that SWIG's idea with c++ side "owning" python objects is rather lame. Basically, if an object is owned by c++ and not cleaned properly all kinds of strange things can happen... BR, Alex.
I think I've figured it with the missing classes. When supplied with multiple headers, genreflex apparently only saves the last one of them...Obviously, that can be worked around, but still seems like a bug or just me not finding some manual. BR, Alex. sunnuntai 03 kesäkuu 2012 16:44:19 wlavrijsen@lbl.gov kirjoitti:
Hi Alex,
Performance ... is absolutely deplorable: slower than PyROOT on CPython!
Performance was originally the point, right? Or am I missing something (lol)? well, remember that there are two problems here: 1) since there are no PyObject's as such in pypy-c (they have different internal structure), one has the create them, to have a PyObject* pointer to them; and keep track of them to mix the reference counting scheme with the garbage collection scheme (this is all done in/by cpyext). And 2) now that the object is created, pinning down a variable in the .py code, the JIT can't unbox it.
The example I used, which was like so:
for i in range(N): a.add_data(i)
and is rather pathological, of course. But then, all micro-benches are.
Conversely, doing:
l = [] for i in range(N): l.append(i) a.add_data(l)
is very speedy, presumably (I don't know, guessing here as it may also be that there is another optimization for the list that I can't prevent even with the append() calls) because now the cpyext code is in a loop in and of itself, so e.g. lookup of the type of the objects in the container (all the same here) can become constant.
I am giving you my benchmark code
Which clearly is also pathological. :) I can't manage to run the SWIG test from pypy, though? I tried by adding cpyext, but no dice?
Anyway, you'll be happy to know that:
CPPYY Created container No more items
real 0m2.236s user 0m2.156s sys 0m0.071s
Cpython Created container No more items
real 0m2.758s user 0m2.574s sys 0m0.174s
(I verified with your debug=1 that the objects appear.)
But that improvement is (I'm sure) only b/c all the other stuff you do around it with manipulating the dict on the python side. That does get JIT'ed just fine.
Btw., I had to jack up N by 10x, otherwise all numbers were just noise. :}
right now it uses dictionary to map pyobject id into objects when they leave c++ domain. Obviously, you add about O(2*log(N)) overhead to the code, but that should not be too much anyway. If that can be done in rpython, it could probably be made to work massively faster. Like a module that does object caching for the storage of objects that would be sent to c++ domain. Then in c++ one can do as simple as have an long int index that can be used to resolve back to byobject when the job is done.
From a quick read of the cpyext code, that is rather similar to what is there.
But as per above, the problem isn't with the extra overhead, the problem is with the JIT being prevented to do any work (and with pypy-c being slower than CPython in pure interpreted mode).
mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have older gcc in a different slot?
I do, but those should not be accessible from my pypy work environment. I did rebuild it. Not sure whether that matters. I'll update the docs showing how a different set of headers from another compiler can be selected.
The point is that if the header is included, and it confuses the parser, then the patterns have no effect. So it is better not to have extra headers like Qt library=)
Qt is a story in and of itself, of course. :) Actually, what folks here do, is to link any code from external C++ through python to Qt (C++) by making use of the fact that both ends can handle opaque pointers, represented as long values. (The combination of addressof and bind_object in cppyy.)
Best regards, Wim
Hi Alex,
With respect to build, I installed root from gentoo repository, and the problem there is that headers are in /usr/include/root and the actual libs are in /usr/lib/root, but the pypy build expects libs to be in /usr/include/root/lib if you specify SYSROOT=/usr/include/root.
now picking up include and libdirs via root-config, but then it does require that tool to be available through PATH (it should be under $ROOTSYS/bin). If not (e.g. standalone Reflex), it will still fall back to $ROOTSYS/include etc. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Great, that actually works. torstai 07 kesäkuu 2012 15:34:33 wlavrijsen@lbl.gov kirjoitti:
Hi Alex,
With respect to build, I installed root from gentoo repository, and the problem there is that headers are in /usr/include/root and the actual libs are in /usr/lib/root, but the pypy build expects libs to be in /usr/include/root/lib if you specify SYSROOT=/usr/include/root.
now picking up include and libdirs via root-config, but then it does require that tool to be available through PATH (it should be under $ROOTSYS/bin). If not (e.g. standalone Reflex), it will still fall back to $ROOTSYS/include etc.
Best regards, Wim
participants (3)
-
Alex Pyattaev -
Armin Rigo -
wlavrijsen＠lbl.gov