Tried deleting the pymongo._cmessage module? It’s a cpyext one, so seems like a good place to start. I think it works without it.<div><br></div><div>A general bisection memory leak finding technique that works in 11 runs or under... Run the code under coverage, so you have the lines that are actually called. Now comment out half the code so that it still runs. Is there a memory leak still? Look at the uncommented lines. No leak? Look at the comment lines. Keep doing it until you spot the problem, or you only have 1 line left.<div><br></div><div>A common resource issue with pypy is how it does cleanup differently. Some code relies on destructors and reference counting to clean things up. Maybe check things like the number of open sockets and files?<br><div><div><br></div><div>objgraph is reeeeeealy good IMHO (haven’t tried it with pypy, not sure it works)</div><div><a href="https://mg.pov.lt/objgraph/">https://mg.pov.lt/objgraph/</a><br><br><br><br>On Thursday, March 28, 2019, Robert Whitcher &lt;<a href="mailto:robert.whitcher@rubrik.com">robert.whitcher@rubrik.com</a>&gt; wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Thanks Antonio...<br><br>Relative to cpyext... I am not sure.  We are not directly, but who knows what is being used by incorporated modules (pymongo, etc..)<div>Trying to create a stripped down test case...</div><div>I filed bug here with some updates: </div><div>    <a href="https://bitbucket.org/pypy/pypy/issues/2982/continual-increase-in-memory-utilization" target="_blank">https://bitbucket.org/pypy/<wbr>pypy/issues/2982/continual-<wbr>increase-in-memory-utilization</a></div><div><br></div><div>Reproduces in pypy, pypy without JIT but *does not* reproduce in CPython so there is some indications there</div></div></div><div><br></div>Once I get the stripped down test case that is not tied in to our entire codebase -- I will publish it in the bug if I can and try other PyPy versions as well.<div><br></div><div>Rob</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 28, 2019 at 10:56 AM Antonio Cuni &lt;<a href="mailto:anto.cuni@gmail.com" target="_blank">anto.cuni@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-size:small">Hi Robert,</div><div style="font-size:small">are you using any package which relies on cpyext? I.e., modules written in C and/or with Cython (cffi is fine).</div><div style="font-size:small">IIRC, at the moment PyPy doesn&#39;t detect GC cycles which involve cpyext objects.</div><div style="font-size:small">So if you have a cycle which does e.g. </div><div style="font-size:small">    Py_foo -&gt; C_bar -&gt; Py_foo</div><div style="font-size:small">(where Py_foo is a pure-python object and C_bar a cpyext object) they will never be collected unless you break the cycle manually.</div><div style="font-size:small"><br></div><div style="font-size:small">Other than that: have you tried running it with PyPy 7.0 and/or 7.1?</div><div style="font-size:small"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 28, 2019 at 8:35 AM Robert Whitcher &lt;<a href="mailto:robert.whitcher@rubrik.com" target="_blank">robert.whitcher@rubrik.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">So I have a process that use PyPy and pymongo in a loop.<div>It does basically the same thing every loop, which query a table in via pymongo and do a few non-save calculations and then wait and loop again </div><div><br></div><div>The RSS of the process continually increased (the PYPY_GC_MAX is set pretty high).</div><div>So I hooked in the GC stats output per: <a href="http://doc.pypy.org/en/latest/gc_info.html" target="_blank">http://doc.pypy.org/en/<wbr>latest/gc_info.html</a></div><div>I also assure that gc.collect() was called at least every 3 minutes.</div><div><br></div><div>What I see is that... The memory while high is fair constant for a long time:<br></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><font face="monospace, monospace">2019-03-27 00:04:10.033-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 144244736</font></div></div></div></div><div><div><div><div><font face="monospace, monospace">...</font></div></div></div></div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:01:46.841-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 144420864</font></div></div></div></div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:02:36.943-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 144269312</font></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div><br></div><div><div>Then it decides (an the exact per-loop behavior is the same each time) to chew up much more memory:<br><br></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:04:17.184-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 145469440</font></div></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:05:07.305-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 158175232</font></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:05:57.401-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 173191168</font></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:06:47.490-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 196943872</font></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:07:37.575-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 205406208</font></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:08:27.659-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 254562304</font></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:09:17.770-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 256020480</font></div></div></div></div></div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:10:07.866-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 289779712</font></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>That&#39;s 140 MB .... Where is all that memory going...</div><div>What&#39;s more is that the PyPy GC stats do not show anything different:</div><div><br></div><div>Here are the GC stats from GC-Complete when we were at <b>144MB</b>:</div><div><br></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><font face="monospace, monospace">2019-03-26 23:55:49.127-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 140632064</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">2019-03-26 23:55:49.133-0600 [-] main_thread(29621)log (async_worker_process.py:308): DBG0: Total memory consumed:</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            GC used:            56.8MB (peak: 69.6MB)</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">               in arenas:            39.3MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">               rawmalloced:          14.5MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">               nursery:              3.0MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            raw assembler used: 521.6kB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            -----------------------------</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            Total:              57.4MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace"><br></font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            Total memory allocated:</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            GC allocated:            63.0MB (peak: 71.2MB)</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">               in arenas:            43.9MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">               rawmalloced:          22.7MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">               nursery:              3.0MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            raw assembler allocated: 1.0MB</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            -----------------------------</font></div></div></div></div></div></div><div><div><div><div><div><div><font face="monospace, monospace">            Total:                   64.0MB</font></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>Here are the GC stats from GC-Complete when we are at <b>285MB</b>:</div><div><br></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div><font face="monospace, monospace">2019-03-27 01:42:41.751-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 285147136</font></div></div></div></div></div></div></div><div><font face="monospace, monospace">2019-03-27 01:42:41.751-0600 [-] main_thread(29621)log (async_worker_process.py:308): DBG0: Total memory consumed:<br></font></div><div><div><div><div><div><div><div><font face="monospace, monospace">            GC used:            57.5MB (peak: 69.6MB)</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">               in arenas:            39.9MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">               rawmalloced:          14.6MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">               nursery:              3.0MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            raw assembler used: 1.5MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            -----------------------------</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            Total:              58.9MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace"><br></font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            Total memory allocated:</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            GC allocated:            63.1MB (peak: 71.2MB)</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">               in arenas:            43.9MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">               rawmalloced:          22.7MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">               nursery:              3.0MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            raw assembler allocated: 2.0MB</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            -----------------------------</font></div></div></div></div></div></div></div><div><div><div><div><div><div><div><font face="monospace, monospace">            Total:                   65.1MB</font></div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>How is this possible?</div><div><br></div><div>I am measuring RSS with:<br><br></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">def get_rss_mem_usage():</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">    &#39;&#39;&#39;</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">    Get the RSS memory usage in bytes</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">    @return: memory size in bytes; -1 if error occurs</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">    &#39;&#39;&#39;</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">    try:</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">        process = psutil.Process(os.getpid())</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">        return process.get_memory_info().rss</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">    except:</font></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div><div><div><div><div><font face="monospace, monospace">        return -1</font></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><font face="monospace, monospace"><div><font face="monospace, monospace"><br></font></div>And cross referencing with &quot;ps -orss -p &lt;pid&gt;&quot; and the RSS values reported are correct....</font><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">I cannot figure out where to go from here with this as it appears that PyPy is leaking this memory somehow...</font></div><div>And I have no idea howto proceed from here...</div><div>I end up having memory problems and getting Memory Warnings for a process that just loops and queries via pymongo<br>Pymongo version is 3.7.1</div><div><br></div><div>This is:</div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div>Python 2.7.13 (<wbr>ab0b9caf307db6592905a80b8faffd<wbr>69b39005b8, Apr 30 2018, 08:21:35)</div></div></div><div><div><div>[PyPy 6.0.0 with GCC 7.2.0]</div></div></div></blockquote><div dir="ltr"><div><br></div><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div dir="ltr"><div style="width:510px"><div> </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
______________________________<wbr>_________________<br>
pypy-dev mailing list<br>
<a href="mailto:pypy-dev@python.org" target="_blank">pypy-dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/pypy-dev" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/pypy-dev</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><div dir="ltr"><div style="width:510px"><div> </div></div></div></div></div>
</blockquote></div></div></div></div>