<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>15 times more memory? That's a lot.</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span>Interestingly Quora reported that their PyPy processes were only 50% larger than CPython ones:</span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span><meta http-equiv="content-type" content="text/html; charset=utf-8"><a
href="http://www.quora.com/Quora-Infrastructure/Did-Quoras-switch-to-PyPy-result-in-increased-memory-consumption">http://www.quora.com/Quora-Infrastructure/Did-Quoras-switch-to-PyPy-result-in-increased-memory-consumption</a><br></span></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">"<span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, Arial, default; font-size: 13px; line-height: 17px; ">our PyPy worker processes themselves take approximately 50% more memory than our equivalent CPython worker processes, although we did not do a large amount of tuning of the GC. Regardless, this wasn't the main cause of our memory blowup.</span></div><meta http-equiv="content-type" content="text/html; charset=utf-8"><span class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, default; font-size: 13px; line-height: 17px; "><br style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">"In our development, we found that certain functions were not worth being ported from their C libraries to pure Python, things like <div class="codeblock inline_codeblock" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(244, 244, 244); display: inline; font-size: 1.1em; white-space: pre-wrap; background-position: initial initial; background-repeat: initial initial; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;
padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; display: inline; font-size: 1.1em; white-space: pre-wrap; "><span class="" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">crypto</span></pre></div>, <div class="codeblock inline_codeblock" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(244, 244, 244); display: inline; font-size: 1.1em; white-space: pre-wrap; background-position: initial initial; background-repeat: initial initial; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px;
padding-bottom: 0px; padding-left: 0px; display: inline; font-size: 1.1em; white-space: pre-wrap; "><span class="" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">lxml</span></pre></div>, <div class="codeblock inline_codeblock" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(244, 244, 244); display: inline; font-size: 1.1em; white-space: pre-wrap; background-position: initial initial; background-repeat: initial initial; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; display:
inline; font-size: 1.1em; white-space: pre-wrap; "><span class="" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">PyML</span></pre></div>, and a couple other random libraries. Our solution for those functions was to run a parallel CPython process that would do nothing but take arguments via an <div class="codeblock inline_codeblock" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(244, 244, 244); display: inline; font-size: 1.1em; white-space: pre-wrap; background-position: initial initial; background-repeat: initial initial; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px;
margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; display: inline; font-size: 1.1em; white-space: pre-wrap; "><span class="" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">execnet</span></pre></div>channel, and output return values via the same <div class="codeblock inline_codeblock" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(244, 244, 244); display: inline; font-size: 1.1em; white-space: pre-wrap; background-position: initial initial; background-repeat: initial initial; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px;
margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; display: inline; font-size: 1.1em; white-space: pre-wrap; "><span class="" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">execnet</span></pre></div> channel.<br style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; "><br style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">"The overhead for some of these Python processes, especially for the ones that required a lot of state (for example, <div class="codeblock inline_codeblock" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px;
padding-right: 0px; padding-bottom: 0px; padding-left: 0px; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(244, 244, 244); display: inline; font-size: 1.1em; white-space: pre-wrap; background-position: initial initial; background-repeat: initial initial; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; display: inline; font-size: 1.1em; white-space: pre-wrap; "><span class="" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">PyML</span></pre></div>) is comparable to the amount of memory taken by the master PyPy process, effectively causing a 2-3x blowup in memory just to maintain the CPython processes; this is our main memory sink for our PyPy
branch."</span><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">----</div><div><font class="Apple-style-span" size="3" style="font-family: 'times new roman', 'new york', times, serif; ">I wonder what accounts for this large difference in PyPy memory consumption (50% more vs. 1,400% more). What type of "</font><span class="Apple-style-span" style="color: rgb(51, 51, 51); line-height: 17px; "><font class="Apple-style-span">large amount of tuning of the GC" did Quora do?</font></span><br></div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="font-size: 12pt; font-family: 'times new roman', 'new
york', times, serif; "><div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "><font size="2" face="Arial"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Igor Katson <igor.katson@gmail.com><br><b><span style="font-weight: bold;">To:</span></b> Maciej Fijalkowski <fijall@gmail.com><br><b><span style="font-weight: bold;">Cc:</span></b> pypy-dev@python.org<br><b><span style="font-weight: bold;">Sent:</span></b> Friday, October 7, 2011 2:07 PM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [pypy-dev] Benchmarking PyPy performance on real-world Django app<br></font><br><br>Tried gunicorn, nothing special, the speed is roughly the same. <br>Unfortunately, I noticed that a single instance takes way to much memory <br>to bring that to production, where I pay for the actually used memory. 4 <br>uWSGI workers eat 14 megabytes each, and pypy's memory usage increases, <br>and after a
couple thousands or requests a single worker took 250mb, <br>more than 15 times more.<br>_______________________________________________<br>pypy-dev mailing list<br><a ymailto="mailto:pypy-dev@python.org" href="mailto:pypy-dev@python.org">pypy-dev@python.org</a><br>http://mail.python.org/mailman/listinfo/pypy-dev<br><br><br></div></div></div></body></html>