[Python-Dev] FreeBSD 7 amd64 and large memory tests

[Alex Martelli]

Unbelievable as this may seem, this crazy over-committing malloc
behavior is by now "a classic" -- I first fought against it in 1990,
when IBM released AIX 3 for its then-new RS/6000 line of workstations;
in a later minor release they did provide a way to optionally switch
this off, but, like on Linux, it's a system-wide switch, NOT
per-process:-(.

I concur with http://www.win.tue.nl/~aeb/linux/lk/lk-9.html (the best
explanation I know of the subject, and recommended reading) which, on
this subject, says "Linux on the other hand is seriously broken."
(just like AIX 3 was).  Sad to learn that BSD is now also broken in
the same way:-(.


Alex
On Wed, Sep 17, 2008 at 8:00 AM, James Y Knight <foom at fuhm.net> wrote:
>
> On Sep 17, 2008, at 10:53 AM, Andrew MacIntyre wrote:
>
>> Martin v. Löwis wrote:
>>>>
>>>> I haven't yet tried posting a query to a FreeBSD list, as it could
>>>> simply
>>>> be a bug on amd64, but I was wondering whether there was anything (other
>>>> than deactivating tests and documenting use of ulimit -v on this
>>>> platform) that could be done to work around this behaviour.
>>>
>>> I think it should be possible to debug this (for people with access to
>>> such a system, and appropriate skills).
>>> I find it hard to believe that a large malloc will simply crash the
>>> process, rather than returning NULL. More likely, there is a NULL
>>> returned somewhere, and Python (or libc) fails to check for it.
>>
>> A simple C program doing a repetitive malloc(), much as pymalloc would
>> with continuous demand, does indeed not see any NULL from malloc() when
>> swap is exhausted but the process gets KILLed (the allocated memory does
>> have to be written to to force the situation...)
>>
>> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
>> best to deal with the problem in the context of the test suite pending
>> resolution by FreeBSD.
>
> Linux does the same thing, unless the user has explicitly configured that
> behavior off. Search the web for linux overcommit. It's controlled by the
> vm.overcommit_memory sysctl. Although linux's default is some heuristic that
> might make Python's test case work right in most cases, depending on malloc
> returning NULL is not something you can actually depend on.
>
> James
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>