[pypy-dev] Problem with large allocation in test

[pypy at pocketnix.org]

On Mon, Apr 18, 2011 at 06:25:44PM -0400, Joe wrote:
> I was trying to run the test file:
> pypy/jit/backend/x86/test/test_rx86_64_auto_encoding.py
> 
> and was getting the following traceback:
> http://paste.pocoo.org/show/374129/
> 
> If you look at the comment on line 17, it's trying to allocate much
> more memory than I have.  I think it's a total of 21GB, while I only
> have 4GB.  I'm using 64bit OpenSuSE 11.4 for my operating system. I
> had the kernel setting overcommit_memory set to 0 (which may be part
> of the problem).
> 
> Anyway, after I went into ll2ctypes.py and set far_regions to True, I
> was able to successfully run the original test.  I don't think setting
> far_regions to True is the correct solution to the problem, but
> fiddling with kernel settings on my system is not ideal either.  What
> would be a better overall solution?
> 
> If any clarification is needed let me know,
> Joe


your vm.overcommit_ratio should be set to "50" or 50% by default, as 
per 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/vm/overcommit-accounting;hb=HEAD 
this means that any allocations of 6GB will automatically be rejected 
as "not sane" and you should receive the ENOMEM error indicating the 
kernel cannot satisfy the supplied range

there are a couple of ways to fix this 

* don't allocate so much ram (did anyone test this before on a 64bit 
host on linux)

* change the vm overcommit policy to 1 (allow everything, don't perform 
sanity checks)

* change the overcommit ratio to something that will satisfy the 
allocation (20GB/4GB ~= 5x, so a value of 600% or 600 should do it)

* Make the mapping a rmmap.MAP_PRIVATE and rmmap.PROT_READ only, 
depending on what you are testing this may not be useful as indicated 
in the linked kernel documentation

YYMV, first option is safe, 2nd option you may want to double check 
the documentation and the values you are passing, the 3rd option is 
also safe in that it will allow badly behaved apps to run, not prevent 
apps from running


to change either of these values use the following:

* to adjust the allocation policy:
  sysctl vm.overcommit_memory=<val>

* to adjust the ratio:
  sysctl vm.overcommit_ratio=<val>

to print the current values (and save them for restoring them after 
you have done tweaking them:
  sysctl vm.overcommit_memory
or
  sysctl vm.overcommit_ratio


Hope this is whats causing the issue
Da_Blitz