[pypy-dev] cpyext performance

[Amaury Forgeot d'Arc]

2012/7/3 Stefan Behnel <stefan_ml at behnel.de>

> > Don't forget that cpyext reference counts are quite different from
> CPython:
> > PySequence_GetItem() needs to *create* a PyObject structure, and the
> > returned object has a refcount of 1.
> > Then Py_DECREF() will really *deallocate* the PyObject structure...
>
> Sure. And using PySequence_GetItem() is still some 10x faster in PyPy than
> taking a borrowed reference using PyList_GET_ITEM() and then calling
> Py_INCREF() on it, which is what Cython does in CPython because it's way
> faster there. The overhead of borrowed references is seriously huge in
> PyPy.
>
> BTW, are PyObject structures currently cached in a free-list somewhere?
> That would be really helpful for the iteration performance.
>

No optimization of any kind have been done in cpyext (it's difficult enough
to get it right...)
A freelist would be a nice thing, but there would still be the cost of
attaching the PyObject to the pypy w_object.

Maybe we should use a weak dictionary to cache the PyObject structure.
This already exists for objects defined and created from C...


> > This is quite more expensive than the simple refcount increment/decrement
> > done by CPython.
> >
> >> OverflowError: array too large
> >
> > Looks like a ctypes bug to me. Which OS, Python, etc. are you using?
>
> Ah - totally, sure. I accidentally ran the system Py2.5 on 64bit Linux.
> Running it with Py2.7 fixes this specific problem, thanks for the hint!
> Although it now names the extension module "nbody.so" instead of
> "nbody.pypy-19.so". Comprend qui peut ...
>
> After figuring out that I was supposed to enable cpyext manually and
> running strace to see what extension module name it is actually looking
> for, I failed to make it load the module it just built regardless of how I
> named it, so I tried building it within the same run as follows:
>
>   pypy/bin/py.py --withmod-cpyext -c 'import setup; import nbody; \
>                                       nbody.test_nbody(1)'  build_ext -i


Ah, but this won't work!
py.py runs on top of CPython, so the PyString_AsString symbol is already
defined by your CPython interpreter!

There is a workaround though: compile your extension module with
   python2.7 pypy/module/cpyext/presetup.py setup.py build_ext -i

presetup.py will patch distutils, and create a module "nbody.pypy-19i.so"
(note the i) which works on top of an *interpreted* pypy.
Among the hacks, all symbols are renamed: #define PyString_AsString
PyPyString_AsString.

Then this should work:
   pypy/bin/py.py --withmod-cpyext -c "import nbody"
*very* slowly of course, but I was able to debug pygames this way!

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120703/d2bdd608/attachment.html>