[pypy-dev] GC error

[Rich Drewes]

On 01/16/2012 02:02 PM, Romain Guillebert wrote:
> Hi
>
> PyPy may use more memory than cpython because of the JIT, can you try
> without the JIT (by passing --jit off to the interpreter) ?

When I run with --jit off it gives a MemoryError at about the same point 
in the run, but with no exception trace:

----
drewes at ladastra:/home/drewes/Desktop/forister2$ pypy --jit off 
grouper2.py --pass2 p5parsed-TSU6.txt
reloading reads . . .
loading reads, on record 1000000
...
loading reads, on record 22000000
loading reads, on record 23000000
MemoryError
----

On another run with --jit off it failed like this:

----
...
loading reads, on record 25000000
loading reads, on record 26000000
Traceback (most recent call last):
   File "app_main.py", line 51, in run_toplevel
   File "grouper2.py", line 993, in <module>
     reads, idquals, l3s, quals=method2pass2(argv[1], 
matchlist=matchlist, maxrec=maxrec, fastq=True)
   File "grouper2.py", line 217, in method2pass2
     reads.append(l2.rstrip())
MemoryError
----

So, the exact point of the error varies.  Is there any part of pypy that 
*requires* physical memory and cannot use swap?  That seems unlikely to 
me, but it is the only guess I can come up with.

At the point of failure the program running with pypy is consuming only 
about 3.7GB resident memory and 3.7GB swap (shown in 'top') and there is 
plenty of swap free.  There is 8GB system RAM but there are other things 
running on the machine too.

When I run my program with cpython, it consumes all available physical 
memory and then the swap space (as shown in 'top') grows up to about 
17GB.  The program runs to completion.  When I run my program with pypy 
on default settings with JIT enabled, it throws that exception shown in 
my last email before it uses much swap space at all (using only about 
4GB swap with >30GB swap still free).

So when pypy throws the memory exception it is using considerably *less* 
memory than the cpython version uses at its peak.  The pypy version 
fails when it has loaded less than half the data and is using less than 
a quarter of the swap space that the cpython program uses without a problem.

[BTW, I didn't find the pypy-issues list when I sent my initial query, 
and this list may not be the right place for this, but thank you for 
responding.]

Rich