Hi, as detailed in a couple of blog posts in the past (*), for some time now we have been working on making C++ bindings through the Reflex package available on PyPy, in the form of the "cppyy" module. Software is never done, and that is true also in this case, but it has reached a level of maturity at which it can be said to be usable. Initial docs are now up on: http://doc.pypy.org/en/latest/cppyy.html with instructions on how to setup the reflex-support branch and how to test it out. The documentation will be updated with more advanced and detailed topics in the next couple of weeks or so. There's a sizable (non-PyPy) set of unit tests that still need to be worked through, thus development will steadily continue at its current pace. Still, if you find that cppyy almost works for you except for a particular feature, feel free to ask for it to be prioritized. The biggest obstacle for most people (that are not in the field of HEP) will be the current set of dependencies. Although the dependency set for cppyy is really only Reflex, which could be distributed separately, for the CPython equivalent code, the dependency set is a large portion of the ROOT class library. The medium-term plan then is to use Cling, which is based on CLang from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For this, there will be a PyCling on CPython, and all as stand-alone projects to remove the large dependency set. More interesting though, having Cling on the back of the bindings allows far more advanced features, such as dynamic setup of callbacks and cross language inheritance to put Python code into C++ frameworks. This has been prototyped successfully in Clings predecessor (CINT), but would be very hard to do in Reflex. Of course, an important reason for pushing this code out to the community somewhat early, is that it allows anyone so interested to start hacking on it and help shape it! Best regards, Wim (*) http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.ht... http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries... -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
On Mon, Apr 23, 2012 at 7:08 AM, <wlavrijsen@lbl.gov> wrote:
Hi,
as detailed in a couple of blog posts in the past (*), for some time now we have been working on making C++ bindings through the Reflex package available on PyPy, in the form of the "cppyy" module. Software is never done, and that is true also in this case, but it has reached a level of maturity at which it can be said to be usable. Initial docs are now up on:
http://doc.pypy.org/en/latest/**cppyy.html<http://doc.pypy.org/en/latest/cppyy.html>
with instructions on how to setup the reflex-support branch and how to test it out. The documentation will be updated with more advanced and detailed topics in the next couple of weeks or so.
There's a sizable (non-PyPy) set of unit tests that still need to be worked through, thus development will steadily continue at its current pace. Still, if you find that cppyy almost works for you except for a particular feature, feel free to ask for it to be prioritized.
The biggest obstacle for most people (that are not in the field of HEP) will be the current set of dependencies. Although the dependency set for cppyy is really only Reflex, which could be distributed separately, for the CPython equivalent code, the dependency set is a large portion of the ROOT class library. The medium-term plan then is to use Cling, which is based on CLang from llvm (http://root.cern.ch/drupal/**content/cling<http://root.cern.ch/drupal/content/cling>) as the back-end. For this, there will be a PyCling on CPython, and all as stand-alone projects to remove the large dependency set.
More interesting though, having Cling on the back of the bindings allows far more advanced features, such as dynamic setup of callbacks and cross language inheritance to put Python code into C++ frameworks. This has been prototyped successfully in Clings predecessor (CINT), but would be very hard to do in Reflex.
Of course, an important reason for pushing this code out to the community somewhat early, is that it allows anyone so interested to start hacking on it and help shape it!
Best regards, Wim
(*) http://morepypy.blogspot.com/**2011/08/wrapping-c-libraries-** with-reflection.html<http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.ht...> http://morepypy.blogspot.com/**2010/07/cern-sprint-report-** wrapping-c-libraries.html<http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries...> -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net ______________________________**_________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
Quick question - if it's mature, why not merge it to default? I presume it should be turned off, since there is a sizeable dependency, but still having it in default can be good. Cheers, fijal
Hi Maciej,
Quick question - if it's mature, why not merge it to default? I presume it should be turned off, since there is a sizeable dependency, but still having it in default can be good.
the dependency is the main issue: for it to build, it requires headers and libs from Reflex. Completely off (i.e. the module not picked up) could work: there are only a few changes that fall outside of the cppyy module proper (mainly access to the raw internals of arrays and a rule for .cxx in the generated Makefile). However, I'm not sure what the advantage would be, as the branch is regularly kept up to date with merges from default? Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
On Mon, Apr 23, 2012 at 8:42 AM, <wlavrijsen@lbl.gov> wrote:
Hi Maciej,
Quick question - if it's mature, why not merge it to default? I presume it
should be turned off, since there is a sizeable dependency, but still having it in default can be good.
the dependency is the main issue: for it to build, it requires headers and libs from Reflex. Completely off (i.e. the module not picked up) could work: there are only a few changes that fall outside of the cppyy module proper (mainly access to the raw internals of arrays and a rule for .cxx in the generated Makefile). However, I'm not sure what the advantage would be, as the branch is regularly kept up to date with merges from default?
Best regards, Wim
-- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
It'll at the very least run tests nightly so we'll make sure we don't break it (we can install reflex on tannit). It's easier to maintain and I prefer done features on trunk
An important question from userland - would it be benefitial to switch from SWIG to the new interface? When can that be done? Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though. Still, the point is that a new C++ interface is a cool thing to have, especially if it is native to pypy. Cant wait to test it out on a huge c++ extension PS: I would have gladly tested an alpha-beta quality version of the lib (i have some unittests for SWIG bindings, so they should work with new lib also i suppose), but I can not build pypy from sources, not enuf RAM =) Maybe someone could send me a build with new feature for testing?
2012/4/23 Alexander Pyattaev <alexander.pyattaev@tut.fi>
Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though.
Do you have a sample code for this issue? -- Amaury Forgeot d'Arc
Well, I have never been able to isolate it, and the project where it is triggered is rather close-sourced. long story short - make up a SWIG object inside a function, pass it as a pointer to the swig-wrapped function, then call gc.collect(). After that swig calls object's destructor and if the C code is using it, youre screwd. I am not sure if the bug persists on a small scale, however, it only shows during long simulation runs where thousands of objects like that are circulated. If it ever happens on a smaller case I'll let you know. Alex maanantai 23 huhtikuu 2012 10:52:04 Amaury Forgeot d'Arc kirjoitti: 2012/4/23 Alexander Pyattaev <alexander.pyattaev@tut.fi> Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though. Do you have a sample code for this issue? -- Amaury Forgeot d'Arc
Hi Alexander,
An important question from userland - would it be benefitial to switch from SWIG to the new interface? When can that be done?
there are no many major differences between SWIG and this on the Python side (nothing that can't be captured in a compatibility layer). I'm thinking of such things like cvar that is needed in SWIG, but not in cppyy, and the use of templates which need to be named in the .i's in SWIG but are have a more "instantiable" syntax in cppyy (for when the Cling backend becomes available). Also, Reflex can parse more code than can SWIG, so you'd have to be careful in which direction you use things.
Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though.
The GC and cppyy work fine for those objects clearly owned by PyPy. Of course, if the C++ side takes ownership, that would need to be patched up "by hand" or with a rule (if the C++ API is consistent).
PS: I would have gladly tested an alpha-beta quality version of the lib (i have some unittests for SWIG bindings, so they should work with new lib also i suppose), but I can not build pypy from sources, not enuf RAM =) Maybe someone could send me a build with new feature for testing?
What distro (in particular, 32b or 64b and which version of libssl)? I've never distributed a binary version of PyPy, but could give it a try. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Distro is latest gentoo, amd64 with sse2 and sse3 (core i5). libssl is in 2 versions: dev-libs/openssl-0.9.8u:0.9.8 dev-libs/openssl-1.0.0h:0 so pick one that suits you better. Anyway, default binary pypy works fine. Currently i have only one typemap in my *.i file for swig, and it allows PyObject* pointers to go through to c++ side. Otherwise no special features are used, so migration should be pretty painless. And of course if there are bugs they would most likely arise, especially if they are in memory management, because the c++ code is executed millions of times during the program run. BR, Alex PS: sorry for using 2 emails - one is work other home, screwd company policy. maanantai 23 huhtikuu 2012 07:52:49 wlavrijsen@lbl.gov kirjoitti:
Hi Alexander,
An important question from userland - would it be benefitial to switch from SWIG to the new interface? When can that be done?
there are no many major differences between SWIG and this on the Python side (nothing that can't be captured in a compatibility layer). I'm thinking of such things like cvar that is needed in SWIG, but not in cppyy, and the use of templates which need to be named in the .i's in SWIG but are have a more "instantiable" syntax in cppyy (for when the Cling backend becomes available). Also, Reflex can parse more code than can SWIG, so you'd have to be careful in which direction you use things.
Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though.
The GC and cppyy work fine for those objects clearly owned by PyPy. Of course, if the C++ side takes ownership, that would need to be patched up "by hand" or with a rule (if the C++ API is consistent).
PS: I would have gladly tested an alpha-beta quality version of the lib (i have some unittests for SWIG bindings, so they should work with new lib also i suppose), but I can not build pypy from sources, not enuf RAM =) Maybe someone could send me a build with new feature for testing?
What distro (in particular, 32b or 64b and which version of libssl)? I've never distributed a binary version of PyPy, but could give it a try.
Best regards, Wim
Hi Alex,
Distro is latest gentoo, amd64 with sse2 and sse3 (core i5).
there's an older version of (Py)ROOT distributed with gentoo: http://packages.gentoo.org/package/sci-physics/root Do you want to use that package, or pick up a more recent version (doesn't matter for Reflex, I think, as it has been mostly stable)? The pypy binary below is against 5.32, as I had it available.
libssl is in 2 versions: dev-libs/openssl-0.9.8u:0.9.8 dev-libs/openssl-1.0.0h:0 so pick one that suits you better.
Okay; the machine that I had a binary ready for and is probably closest (I'm not sure whether I can find an amd64 box, but Intel binary should do), has 0.9.8e, so that should work. First attempt, build against ROOT 5.32 (latest stable version): http://cern.ch/wlav/pypy-reflex-support-042312.tar.bz2 with md5sum: c6ae683605658fa43e0089a99c82c49b pypy-reflex-support-042312.tar.bz2
Anyway, default binary pypy works fine.
Different machines, different people, etc. :) This will be a little trial and error, I'm afraid, since I haven't distributed PyPy in binary before (for CERN, I simply installed binaries on the global file system, which makes it available to most institutes, by building it on a typical worker node). Btw., if there'll be a number of iterations necessary, then we can take this off the list.
Currently i have only one typemap in my *.i file for swig, and it allows PyObject* pointers to go through to c++ side.
Passing PyObject* is in PyROOT, not yet in cppyy, as I wasn't sure about its representation (I'll probably have to pass it through cpyext first to build an actual PyObject* with the guaranteed expected layout). It's on the TODO list, though.
so migration should be pretty painless.
Also depends on the specific C++ features used of course. Actually, what I hope to gain from this exercise is more that cppyy gives proper and clear diagnostics if features are missing, rather than just crash. Furthermore, any answers to questions you may have can be turned into docs, so that's useful as well.
And of course if there are bugs they would most likely arise, especially if they are in memory management, because the c++ code is executed millions of times during the program run.
What we do in the unit tests, is to call the gc explicitly and then see whether cleanup was done as expected (by having an instance counter on the C++ side). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
(sorry, that was also supposed to go to the ML) Awesome work, keep it up! But: I showed the reflex library you're using to friends and they both suggested clang to parse C++ headers instead. What do you think about that? Cheers, Leonard On Mon, Apr 23, 2012 at 7:08 AM, <wlavrijsen@lbl.gov> wrote:
Hi,
as detailed in a couple of blog posts in the past (*), for some time now we have been working on making C++ bindings through the Reflex package available on PyPy, in the form of the "cppyy" module. Software is never done, and that is true also in this case, but it has reached a level of maturity at which it can be said to be usable. Initial docs are now up on:
http://doc.pypy.org/en/latest/cppyy.html
with instructions on how to setup the reflex-support branch and how to test it out. The documentation will be updated with more advanced and detailed topics in the next couple of weeks or so.
There's a sizable (non-PyPy) set of unit tests that still need to be worked through, thus development will steadily continue at its current pace. Still, if you find that cppyy almost works for you except for a particular feature, feel free to ask for it to be prioritized.
The biggest obstacle for most people (that are not in the field of HEP) will be the current set of dependencies. Although the dependency set for cppyy is really only Reflex, which could be distributed separately, for the CPython equivalent code, the dependency set is a large portion of the ROOT class library. The medium-term plan then is to use Cling, which is based on CLang from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For this, there will be a PyCling on CPython, and all as stand-alone projects to remove the large dependency set.
More interesting though, having Cling on the back of the bindings allows far more advanced features, such as dynamic setup of callbacks and cross language inheritance to put Python code into C++ frameworks. This has been prototyped successfully in Clings predecessor (CINT), but would be very hard to do in Reflex.
Of course, an important reason for pushing this code out to the community somewhat early, is that it allows anyone so interested to start hacking on it and help shape it!
Best regards, Wim
(*) http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.ht...
http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries... -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Addendum: apparently, clang even provides python bindings. http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/ On Mon, Apr 23, 2012 at 11:17 AM, Leonard Ritter <paniq@paniq.org> wrote:
(sorry, that was also supposed to go to the ML)
Awesome work, keep it up!
But: I showed the reflex library you're using to friends and they both suggested clang to parse C++ headers instead. What do you think about that?
Cheers, Leonard
On Mon, Apr 23, 2012 at 7:08 AM, <wlavrijsen@lbl.gov> wrote:
Hi,
as detailed in a couple of blog posts in the past (*), for some time now we have been working on making C++ bindings through the Reflex package available on PyPy, in the form of the "cppyy" module. Software is never done, and that is true also in this case, but it has reached a level of maturity at which it can be said to be usable. Initial docs are now up on:
http://doc.pypy.org/en/latest/cppyy.html
with instructions on how to setup the reflex-support branch and how to test it out. The documentation will be updated with more advanced and detailed topics in the next couple of weeks or so.
There's a sizable (non-PyPy) set of unit tests that still need to be worked through, thus development will steadily continue at its current pace. Still, if you find that cppyy almost works for you except for a particular feature, feel free to ask for it to be prioritized.
The biggest obstacle for most people (that are not in the field of HEP) will be the current set of dependencies. Although the dependency set for cppyy is really only Reflex, which could be distributed separately, for the CPython equivalent code, the dependency set is a large portion of the ROOT class library. The medium-term plan then is to use Cling, which is based on CLang from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For this, there will be a PyCling on CPython, and all as stand-alone projects to remove the large dependency set.
More interesting though, having Cling on the back of the bindings allows far more advanced features, such as dynamic setup of callbacks and cross language inheritance to put Python code into C++ frameworks. This has been prototyped successfully in Clings predecessor (CINT), but would be very hard to do in Reflex.
Of course, an important reason for pushing this code out to the community somewhat early, is that it allows anyone so interested to start hacking on it and help shape it!
Best regards, Wim
(*) http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.ht...
http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries... -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Leonard,
Addendum: apparently, clang even provides python bindings. http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/
yes, but AFAIK it's C only and in any case, those are bindings to CLang, rather than bindings to the code parsed from CLang. You'll run into the same problems in their use from PyPy just as any other extension library, and you'd still have to build up the bindings to user code from those bindings to CLang (which is where the major work resides).
But: I showed the reflex library you're using to friends and they both suggested clang to parse C++ headers instead. What do you think about that?
As said in the mail and the docs: the medium term is to use Cling, which is based on CLang. The difference is the C++ interactivity, which is a better match for Python to allow build-up of callbacks and cross-language inheritance. We started out with Reflex b/c that was a known quantity. Also, for large C++ libraries across many different projects (as we have to deal with), there is nothing in CLang that allows you to cleanly pre-package reflection info like is done with shared libraries in Reflex. For Cling, a method is being developed based on pre-compiled headers. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
participants (6)
-
Alex Pyattaev -
Alexander Pyattaev -
Amaury Forgeot d'Arc -
Leonard Ritter -
Maciej Fijalkowski -
wlavrijsen＠lbl.gov