[pypy-dev] GC error

[Rich Drewes]

On 01/16/2012 09:59 PM, Timo Paulssen wrote:
> Hello,
>
> PyPy has a hard limit on its heap size. It can be specified with this 
> environment variable:
>
>  PYPY_GC_MAX            The max heap size.  If coming near this limit, it
>                         will first collect more often, then raise an
>                         RPython MemoryError, and if that is not enough,
>                         crash the program with a fatal error.  Try values
>                         like '1.6GB'.
Thanks for the suggestion.  I tried this, and it did not seem to change 
the point of failure though the message looks a bit different:

----
drewes at ladastra:/home/drewes$ set | grep PYPY
PYPY_GC_MAX=6GB
drewes at ladastra:/home/drewes$ pypy grouper2.py --pass2 p5parsed-TSU6.txt
reloading reads . . .
loading reads, on record 1000000 ...
... loading reads, on record 14000000
Traceback (most recent call last):
   File "app_main.py", line 51, in run_toplevel
   File "grouper2.py", line 982, in <module>
Using too much memory, aborting
Aborted
----

According to 'top', the program is not actually using anywhere near 6GB 
when the failure occurs.  It is only using about 2GB when it fails.
> Check out the rest of the evariables for the GC in
> pypy/rpython/memory/gc/minimark.py
I will do that.
>
> In general, the pypy GC isn't optimised to know what pages are in 
> memory or in swap, so each major collection will cause every single 
> page to be touched. If a large part of your pypy process will not fit 
> into the swap, you will probably experience very significant slowdowns.
That is good information, thanks.

Rich