[pypy-dev] GC error
Rich Drewes
drewes at interstice.com
Tue Jan 17 00:26:50 CET 2012
On 01/16/2012 02:02 PM, Romain Guillebert wrote:
> Hi
>
> PyPy may use more memory than cpython because of the JIT, can you try
> without the JIT (by passing --jit off to the interpreter) ?
When I run with --jit off it gives a MemoryError at about the same point
in the run, but with no exception trace:
----
drewes at ladastra:/home/drewes/Desktop/forister2$ pypy --jit off
grouper2.py --pass2 p5parsed-TSU6.txt
reloading reads . . .
loading reads, on record 1000000
...
loading reads, on record 22000000
loading reads, on record 23000000
MemoryError
----
On another run with --jit off it failed like this:
----
...
loading reads, on record 25000000
loading reads, on record 26000000
Traceback (most recent call last):
File "app_main.py", line 51, in run_toplevel
File "grouper2.py", line 993, in <module>
reads, idquals, l3s, quals=method2pass2(argv[1],
matchlist=matchlist, maxrec=maxrec, fastq=True)
File "grouper2.py", line 217, in method2pass2
reads.append(l2.rstrip())
MemoryError
----
So, the exact point of the error varies. Is there any part of pypy that
*requires* physical memory and cannot use swap? That seems unlikely to
me, but it is the only guess I can come up with.
At the point of failure the program running with pypy is consuming only
about 3.7GB resident memory and 3.7GB swap (shown in 'top') and there is
plenty of swap free. There is 8GB system RAM but there are other things
running on the machine too.
When I run my program with cpython, it consumes all available physical
memory and then the swap space (as shown in 'top') grows up to about
17GB. The program runs to completion. When I run my program with pypy
on default settings with JIT enabled, it throws that exception shown in
my last email before it uses much swap space at all (using only about
4GB swap with >30GB swap still free).
So when pypy throws the memory exception it is using considerably *less*
memory than the cpython version uses at its peak. The pypy version
fails when it has loaded less than half the data and is using less than
a quarter of the swap space that the cpython program uses without a problem.
[BTW, I didn't find the pypy-issues list when I sent my initial query,
and this list may not be the right place for this, but thank you for
responding.]
Rich
More information about the pypy-dev
mailing list