[Python-Dev] FreeBSD 7 amd64 and large memory tests

[James Y Knight]

On Sep 17, 2008, at 10:53 AM, Andrew MacIntyre wrote:

> Martin v. Löwis wrote:
>>> I haven't yet tried posting a query to a FreeBSD list, as it could  
>>> simply
>>> be a bug on amd64, but I was wondering whether there was anything  
>>> (other
>>> than deactivating tests and documenting use of ulimit -v on this
>>> platform) that could be done to work around this behaviour.
>> I think it should be possible to debug this (for people with access  
>> to
>> such a system, and appropriate skills).
>> I find it hard to believe that a large malloc will simply crash the
>> process, rather than returning NULL. More likely, there is a NULL
>> returned somewhere, and Python (or libc) fails to check for it.
>
> A simple C program doing a repetitive malloc(), much as pymalloc would
> with continuous demand, does indeed not see any NULL from malloc()  
> when
> swap is exhausted but the process gets KILLed (the allocated memory  
> does
> have to be written to to force the situation...)
>
> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
> best to deal with the problem in the context of the test suite pending
> resolution by FreeBSD.

Linux does the same thing, unless the user has explicitly configured  
that behavior off. Search the web for linux overcommit. It's  
controlled by the vm.overcommit_memory sysctl. Although linux's  
default is some heuristic that might make Python's test case work  
right in most cases, depending on malloc returning NULL is not  
something you can actually depend on.

James