[pypy-dev] cpyext performance
Amaury Forgeot d'Arc
amauryfa at gmail.com
Tue Jul 3 18:26:33 CEST 2012
2012/7/3 Stefan Behnel <stefan_ml at behnel.de>
> > Don't forget that cpyext reference counts are quite different from
> > PySequence_GetItem() needs to *create* a PyObject structure, and the
> > returned object has a refcount of 1.
> > Then Py_DECREF() will really *deallocate* the PyObject structure...
> Sure. And using PySequence_GetItem() is still some 10x faster in PyPy than
> taking a borrowed reference using PyList_GET_ITEM() and then calling
> Py_INCREF() on it, which is what Cython does in CPython because it's way
> faster there. The overhead of borrowed references is seriously huge in
> BTW, are PyObject structures currently cached in a free-list somewhere?
> That would be really helpful for the iteration performance.
No optimization of any kind have been done in cpyext (it's difficult enough
to get it right...)
A freelist would be a nice thing, but there would still be the cost of
attaching the PyObject to the pypy w_object.
Maybe we should use a weak dictionary to cache the PyObject structure.
This already exists for objects defined and created from C...
> > This is quite more expensive than the simple refcount increment/decrement
> > done by CPython.
> >> OverflowError: array too large
> > Looks like a ctypes bug to me. Which OS, Python, etc. are you using?
> Ah - totally, sure. I accidentally ran the system Py2.5 on 64bit Linux.
> Running it with Py2.7 fixes this specific problem, thanks for the hint!
> Although it now names the extension module "nbody.so" instead of
> "nbody.pypy-19.so". Comprend qui peut ...
> After figuring out that I was supposed to enable cpyext manually and
> running strace to see what extension module name it is actually looking
> for, I failed to make it load the module it just built regardless of how I
> named it, so I tried building it within the same run as follows:
> pypy/bin/py.py --withmod-cpyext -c 'import setup; import nbody; \
> nbody.test_nbody(1)' build_ext -i
Ah, but this won't work!
py.py runs on top of CPython, so the PyString_AsString symbol is already
defined by your CPython interpreter!
There is a workaround though: compile your extension module with
python2.7 pypy/module/cpyext/presetup.py setup.py build_ext -i
presetup.py will patch distutils, and create a module "nbody.pypy-19i.so"
(note the i) which works on top of an *interpreted* pypy.
Among the hacks, all symbols are renamed: #define PyString_AsString
Then this should work:
pypy/bin/py.py --withmod-cpyext -c "import nbody"
*very* slowly of course, but I was able to debug pygames this way!
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the pypy-dev