[pypy-dev] Benchmarking PyPy performance on real-world Django app

[Andy]

15 times more memory? That's a lot.

Interestingly Quora reported that their PyPy processes were only 50% larger than CPython ones:

http://www.quora.com/Quora-Infrastructure/Did-Quoras-switch-to-PyPy-result-in-increased-memory-consumption


"our PyPy worker processes themselves take approximately 50% more memory than our equivalent CPython worker processes, although we did not do a large amount of tuning of the GC. Regardless, this wasn't the main cause of our memory blowup.
"In our development, we found that certain functions were not worth being ported from their C libraries to pure Python, things like crypto, lxml, PyML, and a couple other random libraries. Our solution for those functions was to run a parallel CPython process that would do nothing but take arguments via an execnetchannel, and output return values via the same execnet channel.

"The overhead for some of these Python processes, especially for the ones that required a lot of state (for example, PyML) is comparable to the amount of memory taken by the master PyPy process, effectively causing a 2-3x blowup in memory just to maintain the CPython processes; this is our main memory sink for our PyPy branch."

----
I wonder what accounts for this large difference in PyPy memory consumption (50% more vs. 1,400% more). What type of "large amount of tuning of the GC" did Quora do?




________________________________
From: Igor Katson <igor.katson at gmail.com>
To: Maciej Fijalkowski <fijall at gmail.com>
Cc: pypy-dev at python.org
Sent: Friday, October 7, 2011 2:07 PM
Subject: Re: [pypy-dev] Benchmarking PyPy performance on real-world Django app


Tried gunicorn, nothing special, the speed is roughly the same. 
Unfortunately, I noticed that a single instance takes way to much memory 
to bring that to production, where I pay for the actually used memory. 4 
uWSGI workers eat 14 megabytes each, and pypy's memory usage increases, 
and after a couple thousands or requests a single worker took 250mb, 
more than 15 times more.
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
http://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20111007/1055e329/attachment-0001.html>