[pypy-dev] your thoughts on low level optimizations

[wlavrijsen at lbl.gov]

Hi,

On Thu, 1 Sep 2011, Armin Rigo wrote:
> Reflex is another solution that is likely to work very nicely if you
> can rewrite your C module as a C++ module and use the Reflex-provided
> Python API extracted from the C++ module.

for most practical purposes, the rewriting of C -> C++ for wrapping purposes
with Reflex would be a simple matter of:

  $ cat mycppheader.h
  extern "C" {
  #include "mycheader.h"
  }

But using Reflex for C is overkill, given that no reflection information is
absolutely needed. What can also be done, is to generate the reflection info
as part of the build process, and use it to generate annotations for ctypes.
Then put those in a pickle file and ship that.

On Thu, 1 Sep 2011, Gertjan van Zwieten wrote:
> By the way, you mention ctypes *or* libffi as if they are two distinct
> options, but I believe ctypes was built on top of libffi.

Yes, but what I meant in the same sentence was the pair of Python+ctypes and
the pair RPython+libffi. Both are efficient as Armin already explained because
once the JIT is warmed up, no wrapping/unwrapping is needed anymore.

> Lastly, you mention SWIG of equivalent (Boost?) as alternative options.

I mentioned those on the CPython side as reasons why I've never chosen to
make Reflex-based (or CINT-based, rather) bindings available as a standalone
application. They take the same amount of work if reflection information is
not generated yet (in our applications, the reflection info is already there
for the I/O, so the end-user does not need to deal with that as they would
if the choice had fallen on SWIG).

I think a part of the discussion that is missing, is who the target is of
the various tools and who ends up using the product: if I'm an end-user,
installing binary Python extension modules from the package manager that
comes with my OS, then cpyext is probably my best friend. But if I'm a
developer of an extension module, like you are, I would not rely on it, and
instead provide a solution that works best on both, and that could run on
all Pythons from using ctypes to writing custom code.

> This 2008 sprint discussion [2] loosely suggests that there will be no
> direct PyPy-ish implementation of these tools, and instead argues for
> reflex, leading to this week's post.

There's a 2010 post in between, when work was started:

   http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries.html

Work is progressing as time allows and there are some nice results, but it's
not production quality yet. Getting there, though, as the list of available
features shows. However, everytime I throw it at a large class library (large
meaning thousands of classes), there's always something new to tackle so far.

> If reflex offers a one-to-one translation of C++ classes to Python then that
> certainly sounds useful, but unless it is something that I could not equally
> achieve by manual ctypes annotations

That depends on your C++ classes. E.g. for calculations of offsets between a
derived class and virtual base classes, some form of reflection information is
absolutely needed.

Best regards,
            Wim

> PS @Wim, that's interesting. People tend to be a bit confused when I tell
> them I went from earthquake research to printer ink. Now I can explain that
> printer ink is just one step away from high energy particle physics.

Ah. :) Actually, Oce was a detour. A fun one where I learned a lot to be sure,
but I did start out in HEP and astrophysics originally.
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net