[pypy-dev] work on libpypy.so

[Roberto De Ioris]

Il giorno 25/lug/2011, alle ore 14:31, Armin Rigo ha scritto:

> 
> Indeed, there is nothing like that so far.  We need to think out the
> kind of C-level API which makes sense for libpypy.so to expose.  There
> are two options: either we tweak a little bit cpyext to expose the
> CPython C API, or we design some higher-performance PyPy-only C API...
> 
> For the latter case the first issue is how to store a reference to a
> PyPy object in C, given that it moves around --- maybe with an
> indirection, which would be a bit similar to what cpyext does but can
> be done with much better performance if we are free to design the API.
> 
> Feel free to come to our IRC channel to discuss things (#pypy on
> irc.freenode.net).
> 
> 

Sorry for the big late/silence but i spent last days working/hacking on cpyext.

I have managed to have a working uWSGI+pypy inverting the initial concept
(not very useful, but this is a base for benchmark) :

1) i build the whole uWSGI stack core (so, without cpython plugin) as a shared library
2) i created a uwsgi_pypy c extension:

http://projects.unbit.it/hg/uwsgi_pypy/file/7bbae6efc126/uwsgi_pypy.c

This extension export a bunch of functions, with the only useful one being uwsgi_pypy.init()
This function calls the original main() of the uWSGI server, so effectively pypy lose the process control.

3) i have developed a very raw/minimal plugin for uWSGI that inspect a symbol in the process space.

http://projects.unbit.it/uwsgi/browser/plugins/symcall/symcall_plugin.c

This symbol is defined in the uwsgi_pypy.c module. So, this is the way uWSGI give control back to pypy at each request.

This is an example WSGI-handler implementation (incomplete, it misses post handling, but can already execute werkzeug test app)

http://projects.unbit.it/hg/uwsgi_pypy/file/7bbae6efc126/uwsgi-pypy.py

Well, running it and benchmarking showed a 10% improvement in response time, so i tried with a more complex app 
(fibonacci's sequence and some prime number generator) and …damn it destroyed cpython :) it served the double of the connections in half of the time….

So, cpyext looks really in good shape, so (probably) the best way would be tweek it to have a libpypy.so.

i will work on the current setup a bit more to have a full WSGI environment, then i will start investigating the original idea (but this time
base on cpyext)

Thanks for the amazing work

--
Roberto De Ioris
http://unbit.it
JID: roberto at jabber.unbit.it