Hi, any objections to renaming cppyy into _cppyy? I want to be able to do a straight 'pip install cppyy' and then use it w/o further gymnastics (this works today for CPython), but then I can't have 'cppyy' be a built-in module. (You can pip install PyPy-cppyy-backend, but then you'd still have to deal with LD_LIBRARY_PATH and certain cppyy features are PyPy version dependent even as they need not be as they are pure Python.) The pip-installed cppyy will still use the built-in _cppyy for the PyPy specific parts (low-level manipulations etc.). I'm also moving the cppyy documentation out of the pypy documentation and placing it on its own (http://cppyy.readthedocs.io/), given that the CPython side of things now works, too. Yes, no, conditional? Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Maciej, yes, I know. :) I used and use CFFI for inspiration. Now done. The package structure has become: _cppyy for PyPy / \ cppyy (shared) cppyy-backend (shared) \ / CPyCppyy for CPython For pip-purposes, the cppyy wheel is not shared (but it's cheap to install). The backend bdist_wheel can be re-used for PyPy, CPython2, and CPython3. $ virtualenv -p .../pypy/pypy/goal/pypy-c pypy-dev $ MAKE_NPROCS=32 pip install cppyy ... $ python Python 2.7.13 (0b40d2587588, Jul 19 2017, 00:43:13) [PyPy 5.9.0-alpha0 with GCC 5.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. using my private settings ... And now for something completely different: ``Therefore, specific information, I was in an ideal context, I had to realize the faith''
The documentation and (shared) app-level tests live in cppyy. I have some more work to do there. For PyPy-5.7 and -5.8, this can be done:
and things work from there (albeit not niceties such as cppdef() above, but all the genreflex etc. stuff works). Omer, Anto, thanks for pushing. :) I'm actually pretty pleased with this setup: it all works nicer than I expected. Life being what it is, the last time I looked into distutils was apparently Oct 9, 2007 (I just checked my archives). Things have improved quite a bit since then ... Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Sounds like a good idea. Since this is similar to what is done with cffi, I have a few questions: How do you make sure that the pure Python part remains compatible with the backend? Can we include the pure Python part in PyPy, in the way as it's done for cffi? On 2017-07-18 23:33, wlavrijsen@lbl.gov wrote:
Manuel,
How do you make sure that the pure Python part remains compatible with the backend?
I'm thinking of selecting the versions explicitly in the front cppyy package: it already has different dependencies selected based on interpreter: PyPy or CPython. Also, the backend "API" is small and stable, so I expect a large number of versions to work well, including older ones, moving forward. If you have a better idea, then I'm all ears! I'm new to this packaging ...
Can we include the pure Python part in PyPy, in the way as it's done for cffi?
Yes, albeit that it is still necessary to install cppyy_backend (contains a patched version of Clang/LLVM), so I don't see much net gain. And yes, it is technically possible to write a bindings generator that only depends on LLVM during offline bindings generation, not at run-time. But then you'd just have SWIG (albeit with a standards-compliant parser). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi, [replying to self]
it hit me yesterday that instead of generating something offline for cppyy to read, you could also use cppyy_backend directly to generate a pure python module for use with cffi as it knows about sizes, offsets, name mangling, class relations etc., etc., etc. (The point is that cppyy_backend exposes a minimalistic C-API, which itself can be bound with cffi, as _cppyy in fact does.) (There are still wrappers needed for inlined functions, overloaded operator new, virtual inheritance offsets, etc., but those could be stuffed in a .cxx on the side of the cffi-based Python module. Can't solve automatic template instantiation this way, though: cppyy will always be superior.) So, I wrote a quick proof of concept (https://bitbucket.org/wlav/quaff/src): $ cat Simple.h class Simple { public: Simple(); Simple(int); virtual ~Simple(); public: int get_data(); void set_data(int); private: int m_data; }; $ cat Simple.cxx #include "Simple.h" Simple::Simple() : m_data(42) { } Simple::Simple(int i) : m_data(i) { } Simple::~Simple() { } int Simple::get_data() { return m_data; } void Simple::set_data(int i) { m_data = i; } $ g++ -shared -o libsimple.so -fPIC Simple.cxx $ python
What quaff generated looks like this: $ cat simple.py from cffi import FFI _ffi = FFI() _ffi.cdef(""" void _ZN6SimpleC1Ev(void*); void _ZN6SimpleC1Ei(void*, int); int _ZN6Simple8get_dataEv(void*); void _ZN6Simple8set_dataEi(void*, int); """) _dll = _ffi.dlopen('libsimple.so') class Simple(object): def __init__(self, *args): self._cpp_this = _ffi.new('char[]', 16) if len(args) == 0: _dll._ZN6SimpleC1Ev(self._cpp_this, *args) if len(args) == 1: _dll._ZN6SimpleC1Ei(self._cpp_this, *args) def get_data(self, *args): return _dll._ZN6Simple8get_dataEv(self._cpp_this, *args) def set_data(self, *args): return _dll._ZN6Simple8set_dataEi(self._cpp_this, *args) which is nicely standalone for distribution (yes, the result is platform specific, but no more so than a binary .so and way better than a Python extension module). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On 21 July 2017 at 03:17, <wlavrijsen@lbl.gov> wrote:
Ouch, I suppose :-) Explicit C++ mangling. The alternative would be to use ffi.set_source() instead of ffi.dlopen(), and use a C++ compiler again to produce the cffi binding. This reduces cppyy to a generator emitting a cffi build script. But as you mentioned there are additional issues, e.g. with instantiation of templates. A bientôt, Armin.
Armin,
sure, but the mangled name saves a redirection and has less code to generate, was my point. Besides, who looks at generated code. :) Have you seen what comes out of the PyPy translator ...
But as you mentioned there are additional issues, e.g. with instantiation of templates.
And all other dynamic behaviors, incl. deriving Python classes from C++ ones for use in C++-land. (Is currently disabled, as I never implemented that for the PyPy side, only CPython (it uses the Python C-API).) Or general patch-ups (e.g. byte* <-> char*). Or auto-downcasting (favorite). Or class unique-ness. Or adding user-defined classes interactively. As said, in the end it'd be just SWIG but with a better parser (and no Python versioning issues, or new language to learn). I'd like to see the performance of a more complex case, though. E.g. when doing class matches using Python reflection such as isinstance. I've always had a hard time getting these things fast in RPython, and PyPy may have an easier time with proper Python code (another reason to reject C++ wrappers and live with mangled names). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On 21 July 2017 at 09:46, <wlavrijsen@lbl.gov> wrote:
Comparing ABI mode (ffi.dlopen) and API mode (ffi.set_source) for performance, people often fail to realize that the API mode is generally much faster because calls don't go through libffi. That is always the case on CPython. On PyPy, the JIT can compile all calls in API mode, but only "simple enough" calls in ABI mode, making performance more predictable. For the cases where the call is "simple enough", you might think that the API mode adds the marginal cost of a wrapper, but that is mostly wrong too: the JIT replaces a call to a JMP instruction with a call to the jmp target. A bientôt, Armin.
Armin,
people often fail to realize
count me among those. :) Yes, that is surprising.
For the cases where the call is "simple enough", you might think that the API mode adds the marginal cost of a wrapper
To be sure, if the code is really simple, it's typically inlined on the C++ end and then a wrapper is needed anyway (shouldn't, but linkers tends to be too smart by half as we found out). Alright, let me try some micro-benches. If it can outperform CPython/cppyy then I'm interested in pursuing this track. (With cppyy, wrappers are already generated anyway and compiled by the LLVM JIT, but those are fully generic ones, leaving lots of (un)packing and some copying on each call.) Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On 18 July 2017 at 23:33, <wlavrijsen@lbl.gov> wrote:
I want to be able to do a straight 'pip install cppyy' and then use it w/o further gymnastics (this works today for CPython)
Great, I'm +1 on that as well. If the newly-named '_cppyy' module is more minimal than '_cffi_backend' and less likely to change, then yes, I think it's a good plan to include only '_cppyy' with PyPy and distribute the rest with 'pip install'. A bientôt, Armin
Armin,
If the newly-named '_cppyy' module is more minimal than '_cffi_backend'
'more minimal' is a hard to define term, but a 'wc -l' shows that _cffi_backend has 8498 lines of python and _cppyy has 4265. Of course, the latter uses the former, saving lots of code for it. :) Further, _cppyy includes 470 lines in pythonify.py which could move out, as it is not RPython, but that would make it harder to write tests, which would have to be low-level (as in test_cppyy.py) instead of high level (the rest). I wrote some small benchmarks for the PyHPC '16 paper. If I find the time to flesh that out further, I can better judge what can move out of RPython (_cppyy) and into Python (cppyy), without loss of performance. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Maciej, yes, I know. :) I used and use CFFI for inspiration. Now done. The package structure has become: _cppyy for PyPy / \ cppyy (shared) cppyy-backend (shared) \ / CPyCppyy for CPython For pip-purposes, the cppyy wheel is not shared (but it's cheap to install). The backend bdist_wheel can be re-used for PyPy, CPython2, and CPython3. $ virtualenv -p .../pypy/pypy/goal/pypy-c pypy-dev $ MAKE_NPROCS=32 pip install cppyy ... $ python Python 2.7.13 (0b40d2587588, Jul 19 2017, 00:43:13) [PyPy 5.9.0-alpha0 with GCC 5.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. using my private settings ... And now for something completely different: ``Therefore, specific information, I was in an ideal context, I had to realize the faith''
The documentation and (shared) app-level tests live in cppyy. I have some more work to do there. For PyPy-5.7 and -5.8, this can be done:
and things work from there (albeit not niceties such as cppdef() above, but all the genreflex etc. stuff works). Omer, Anto, thanks for pushing. :) I'm actually pretty pleased with this setup: it all works nicer than I expected. Life being what it is, the last time I looked into distutils was apparently Oct 9, 2007 (I just checked my archives). Things have improved quite a bit since then ... Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Sounds like a good idea. Since this is similar to what is done with cffi, I have a few questions: How do you make sure that the pure Python part remains compatible with the backend? Can we include the pure Python part in PyPy, in the way as it's done for cffi? On 2017-07-18 23:33, wlavrijsen@lbl.gov wrote:
Manuel,
How do you make sure that the pure Python part remains compatible with the backend?
I'm thinking of selecting the versions explicitly in the front cppyy package: it already has different dependencies selected based on interpreter: PyPy or CPython. Also, the backend "API" is small and stable, so I expect a large number of versions to work well, including older ones, moving forward. If you have a better idea, then I'm all ears! I'm new to this packaging ...
Can we include the pure Python part in PyPy, in the way as it's done for cffi?
Yes, albeit that it is still necessary to install cppyy_backend (contains a patched version of Clang/LLVM), so I don't see much net gain. And yes, it is technically possible to write a bindings generator that only depends on LLVM during offline bindings generation, not at run-time. But then you'd just have SWIG (albeit with a standards-compliant parser). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi, [replying to self]
it hit me yesterday that instead of generating something offline for cppyy to read, you could also use cppyy_backend directly to generate a pure python module for use with cffi as it knows about sizes, offsets, name mangling, class relations etc., etc., etc. (The point is that cppyy_backend exposes a minimalistic C-API, which itself can be bound with cffi, as _cppyy in fact does.) (There are still wrappers needed for inlined functions, overloaded operator new, virtual inheritance offsets, etc., but those could be stuffed in a .cxx on the side of the cffi-based Python module. Can't solve automatic template instantiation this way, though: cppyy will always be superior.) So, I wrote a quick proof of concept (https://bitbucket.org/wlav/quaff/src): $ cat Simple.h class Simple { public: Simple(); Simple(int); virtual ~Simple(); public: int get_data(); void set_data(int); private: int m_data; }; $ cat Simple.cxx #include "Simple.h" Simple::Simple() : m_data(42) { } Simple::Simple(int i) : m_data(i) { } Simple::~Simple() { } int Simple::get_data() { return m_data; } void Simple::set_data(int i) { m_data = i; } $ g++ -shared -o libsimple.so -fPIC Simple.cxx $ python
What quaff generated looks like this: $ cat simple.py from cffi import FFI _ffi = FFI() _ffi.cdef(""" void _ZN6SimpleC1Ev(void*); void _ZN6SimpleC1Ei(void*, int); int _ZN6Simple8get_dataEv(void*); void _ZN6Simple8set_dataEi(void*, int); """) _dll = _ffi.dlopen('libsimple.so') class Simple(object): def __init__(self, *args): self._cpp_this = _ffi.new('char[]', 16) if len(args) == 0: _dll._ZN6SimpleC1Ev(self._cpp_this, *args) if len(args) == 1: _dll._ZN6SimpleC1Ei(self._cpp_this, *args) def get_data(self, *args): return _dll._ZN6Simple8get_dataEv(self._cpp_this, *args) def set_data(self, *args): return _dll._ZN6Simple8set_dataEi(self._cpp_this, *args) which is nicely standalone for distribution (yes, the result is platform specific, but no more so than a binary .so and way better than a Python extension module). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On 21 July 2017 at 03:17, <wlavrijsen@lbl.gov> wrote:
Ouch, I suppose :-) Explicit C++ mangling. The alternative would be to use ffi.set_source() instead of ffi.dlopen(), and use a C++ compiler again to produce the cffi binding. This reduces cppyy to a generator emitting a cffi build script. But as you mentioned there are additional issues, e.g. with instantiation of templates. A bientôt, Armin.
Armin,
sure, but the mangled name saves a redirection and has less code to generate, was my point. Besides, who looks at generated code. :) Have you seen what comes out of the PyPy translator ...
But as you mentioned there are additional issues, e.g. with instantiation of templates.
And all other dynamic behaviors, incl. deriving Python classes from C++ ones for use in C++-land. (Is currently disabled, as I never implemented that for the PyPy side, only CPython (it uses the Python C-API).) Or general patch-ups (e.g. byte* <-> char*). Or auto-downcasting (favorite). Or class unique-ness. Or adding user-defined classes interactively. As said, in the end it'd be just SWIG but with a better parser (and no Python versioning issues, or new language to learn). I'd like to see the performance of a more complex case, though. E.g. when doing class matches using Python reflection such as isinstance. I've always had a hard time getting these things fast in RPython, and PyPy may have an easier time with proper Python code (another reason to reject C++ wrappers and live with mangled names). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On 21 July 2017 at 09:46, <wlavrijsen@lbl.gov> wrote:
Comparing ABI mode (ffi.dlopen) and API mode (ffi.set_source) for performance, people often fail to realize that the API mode is generally much faster because calls don't go through libffi. That is always the case on CPython. On PyPy, the JIT can compile all calls in API mode, but only "simple enough" calls in ABI mode, making performance more predictable. For the cases where the call is "simple enough", you might think that the API mode adds the marginal cost of a wrapper, but that is mostly wrong too: the JIT replaces a call to a JMP instruction with a call to the jmp target. A bientôt, Armin.
Armin,
people often fail to realize
count me among those. :) Yes, that is surprising.
For the cases where the call is "simple enough", you might think that the API mode adds the marginal cost of a wrapper
To be sure, if the code is really simple, it's typically inlined on the C++ end and then a wrapper is needed anyway (shouldn't, but linkers tends to be too smart by half as we found out). Alright, let me try some micro-benches. If it can outperform CPython/cppyy then I'm interested in pursuing this track. (With cppyy, wrappers are already generated anyway and compiled by the LLVM JIT, but those are fully generic ones, leaving lots of (un)packing and some copying on each call.) Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On 18 July 2017 at 23:33, <wlavrijsen@lbl.gov> wrote:
I want to be able to do a straight 'pip install cppyy' and then use it w/o further gymnastics (this works today for CPython)
Great, I'm +1 on that as well. If the newly-named '_cppyy' module is more minimal than '_cffi_backend' and less likely to change, then yes, I think it's a good plan to include only '_cppyy' with PyPy and distribute the rest with 'pip install'. A bientôt, Armin
Armin,
If the newly-named '_cppyy' module is more minimal than '_cffi_backend'
'more minimal' is a hard to define term, but a 'wc -l' shows that _cffi_backend has 8498 lines of python and _cppyy has 4265. Of course, the latter uses the former, saving lots of code for it. :) Further, _cppyy includes 470 lines in pythonify.py which could move out, as it is not RPython, but that would make it harder to write tests, which would have to be low-level (as in test_cppyy.py) instead of high level (the rest). I wrote some small benchmarks for the PyHPC '16 paper. If I find the time to flesh that out further, I can better judge what can move out of RPython (_cppyy) and into Python (cppyy), without loss of performance. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
participants (6)
-
Antonio Cuni -
Armin Rigo -
Maciej Fijalkowski -
Manuel Jacob -
Omer Katz -
wlavrijsen＠lbl.gov