[pypy-dev] Problem with large allocation in test

[Maciej Fijalkowski]

On Tue, Apr 19, 2011 at 12:59 AM,  <pypy at pocketnix.org> wrote:
> On Mon, Apr 18, 2011 at 06:25:44PM -0400, Joe wrote:
>> I was trying to run the test file:
>> pypy/jit/backend/x86/test/test_rx86_64_auto_encoding.py
>>
>> and was getting the following traceback:
>> http://paste.pocoo.org/show/374129/
>>
>> If you look at the comment on line 17, it's trying to allocate much
>> more memory than I have.  I think it's a total of 21GB, while I only
>> have 4GB.  I'm using 64bit OpenSuSE 11.4 for my operating system. I
>> had the kernel setting overcommit_memory set to 0 (which may be part
>> of the problem).
>>
>> Anyway, after I went into ll2ctypes.py and set far_regions to True, I
>> was able to successfully run the original test.  I don't think setting
>> far_regions to True is the correct solution to the problem, but
>> fiddling with kernel settings on my system is not ideal either.  What
>> would be a better overall solution?
>>
>> If any clarification is needed let me know,
>> Joe
>
>
> your vm.overcommit_ratio should be set to "50" or 50% by default, as
> per
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/vm/overcommit-accounting;hb=HEAD
> this means that any allocations of 6GB will automatically be rejected
> as "not sane" and you should receive the ENOMEM error indicating the
> kernel cannot satisfy the supplied range
>
> there are a couple of ways to fix this
>
> * don't allocate so much ram (did anyone test this before on a 64bit
> host on linux)

The reason for this is that we want to test far jumps (exceeding 4G or
2^32 in address space). How can you do it otherwise?

>
> * change the vm overcommit policy to 1 (allow everything, don't perform
> sanity checks)
>
> * change the overcommit ratio to something that will satisfy the
> allocation (20GB/4GB ~= 5x, so a value of 600% or 600 should do it)
>
> * Make the mapping a rmmap.MAP_PRIVATE and rmmap.PROT_READ only,
> depending on what you are testing this may not be useful as indicated
> in the linked kernel documentation
>
> YYMV, first option is safe, 2nd option you may want to double check
> the documentation and the values you are passing, the 3rd option is
> also safe in that it will allow badly behaved apps to run, not prevent
> apps from running
>
>
> to change either of these values use the following:
>
> * to adjust the allocation policy:
>  sysctl vm.overcommit_memory=<val>
>
> * to adjust the ratio:
>  sysctl vm.overcommit_ratio=<val>
>
> to print the current values (and save them for restoring them after
> you have done tweaking them:
>  sysctl vm.overcommit_memory
> or
>  sysctl vm.overcommit_ratio
>
>
> Hope this is whats causing the issue
> Da_Blitz
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>