[pypy-dev] GC error

[Maciej Fijalkowski]

On Wed, Jan 18, 2012 at 12:30 AM, Rich Drewes <drewes at interstice.com> wrote:
> On 01/16/2012 09:59 PM, Timo Paulssen wrote:
>>
>> Hello,
>>
>> PyPy has a hard limit on its heap size. It can be specified with this
>> environment variable:
>>
>>  PYPY_GC_MAX            The max heap size.  If coming near this limit, it
>>                        will first collect more often, then raise an
>>                        RPython MemoryError, and if that is not enough,
>>                        crash the program with a fatal error.  Try values
>>                        like '1.6GB'.
>
> Thanks for the suggestion.  I tried this, and it did not seem to change the
> point of failure though the message looks a bit different:
>
> ----
> drewes at ladastra:/home/drewes$ set | grep PYPY
> PYPY_GC_MAX=6GB
> drewes at ladastra:/home/drewes$ pypy grouper2.py --pass2 p5parsed-TSU6.txt
>
> reloading reads . . .
> loading reads, on record 1000000 ...
> ... loading reads, on record 14000000
>
> Traceback (most recent call last):
>  File "app_main.py", line 51, in run_toplevel
>  File "grouper2.py", line 982, in <module>
> Using too much memory, aborting
> Aborted
> ----
>
> According to 'top', the program is not actually using anywhere near 6GB when
> the failure occurs.  It is only using about 2GB when it fails.

That means you're running a 32bit program in a 64bit environment

>>
>> Check out the rest of the evariables for the GC in
>> pypy/rpython/memory/gc/minimark.py
>
> I will do that.
>
>>
>> In general, the pypy GC isn't optimised to know what pages are in memory
>> or in swap, so each major collection will cause every single page to be
>> touched. If a large part of your pypy process will not fit into the swap,
>> you will probably experience very significant slowdowns.
>
> That is good information, thanks.
>
> Rich
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev