[pypy-dev] cpyext: Detecting pypy and other issues

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/01/2011 07:35 AM, Armin Rigo wrote:
> A first warning: even if you manage to compile your C extension with
> PyPy, the interfacing is much slower than with CPython.  Keep that in
> mind, given that according to what you write you care enough about
> performance to try to squeeze a few percents.

First you make it work right then you make it work fast :-)  Even if it is
10% slower than CPython the rest of pypy will be more than 10% faster so
things will still be better overall.  Worst case some combination of my
code, my tests and pypy will be improved by trying to do the port.

>> I need the api PyRun_StringFlags as it is used to execute a Python string
>> and attach the results to the module.
> 
> If it's not already provided, then you are welcome to contribute it to
> PyPy in pypy/module/cpyext/.  It is known that we don't have 100%
> coverage even of Python.h (not to mention other headers like
> compile.h, which will probably never be supported).

It would be helpful if the cpyext site said what to do in these situations.
 For example add missing functions to a wiki page or create a ticket or send
to mailing list etc.  ie what actions should developers like me take when
discovering things like this that ensures the smoothest running of the
project.  (Obviously just fixing it is good but we aren't all in a position
to do that at the moment.)  PyErr_Display is another missing function which
is widely used, but never documented in the Python docs for some reason.

The pypy site also doesn't say where to communicate for cpyext - I assumed
this mailing list was the right place.

>> Do I have to add an incref for everything that has a static pointer to it?
> 
> It depends on details of exactly what you do, but it is likely.  The
> exact rule is, like CPython, that any object is freed as soon as its
> reference counter goes down to zero; but unlike CPython, this does
> *not* include places like a reference stored in the module's
> dictionary.  

Note my problem is not with contents of the module, but with the module itself.

  static PyObject *themodule=Py_InitModule3(...);

Later in one of the methods called back from Python I call
PyObject_AttrString(themodule, "foo") which is crashing.  If I incref
themodule in init then this problem doesn't occur.

The behaviour is as though Py_InitModule3 is setting the starting refcount
to zero, or with the module's refcount becoming zero inside pypy which then
begs the question of how my callback could have been called.

> In other words, if it seems to work on CPython but not on
> PyPy, then it is likely that there is actually a (potentially very
> rare) bug in the CPython version too.

For my tests I build a full debug version of CPython will all memory
optimisations disabled (quite a feat since it squirrels away stuff all over
the place) and then run everything under valgrind.  Combined with 100%
coverage in almost all source files I'm reasonably confident that things are
coded well, although there may be some luck :-)

> For example, the object you
> have in the static variable could be meant to be kept alive by being
> also stored in the module's dictionary.

I did just double check my code and every item that I do PyModule_AddObject
with is also incref first.  That was some diligent coding however many years
ago when I wrote it :-)

> You get easier-to-read tracebacks if you run it without the JIT: "pypy
> --jit threshold=-1".  Of course in order to get the details of, say,
> the local variables, then you need a custom pypy built with debugging
> symbols.

Unfortunately disabling the JIT wasn't much help.  My goal is to produce the
best bug report possible and use the least amount of the pypy team's time.
(It is a lot easier to fix and understand things starting from a good bug
report.)  What would be most helpful is if the cpyext page gave some
instructions to follow in particular making an appropriate pypy and using
valgrind.  (Usually making valgrind work well requires extra options above
and beyond regular debugging builds.)

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk2+OVEACgkQmOOfHg372QTkpQCfc+DL/GFDZCpyMsNEsx5tRhvY
fIUAn0z1EQ+NxZLQ8PwaouXXSRy4JCTo
=vu+1
-----END PGP SIGNATURE-----