Unbelievable as this may seem, this crazy over-committing malloc
behavior is by now "a classic" -- I first fought against it in 1990,
when IBM released AIX 3 for its then-new RS/6000 line of workstations;
in a later minor release they did provide a way to optionally switch
this off, but, like on Linux, it's a system-wide switch, NOT
per-process:-(.
I concur with http://www.win.tue.nl/~aeb/linux/lk/lk-9.html (the best
explanation I know of the subject, and recommended reading) which, on
this subject, says "Linux on the other hand is seriously broken."
(just like AIX 3 was). Sad to learn that BSD is now also broken in
the same way:-(.
Alex
On Wed, Sep 17, 2008 at 8:00 AM, James Y Knight
On Sep 17, 2008, at 10:53 AM, Andrew MacIntyre wrote:
Martin v. Löwis wrote:
I haven't yet tried posting a query to a FreeBSD list, as it could simply be a bug on amd64, but I was wondering whether there was anything (other than deactivating tests and documenting use of ulimit -v on this platform) that could be done to work around this behaviour.
I think it should be possible to debug this (for people with access to such a system, and appropriate skills). I find it hard to believe that a large malloc will simply crash the process, rather than returning NULL. More likely, there is a NULL returned somewhere, and Python (or libc) fails to check for it.
A simple C program doing a repetitive malloc(), much as pymalloc would with continuous demand, does indeed not see any NULL from malloc() when swap is exhausted but the process gets KILLed (the allocated memory does have to be written to to force the situation...)
I'll take this up with FreeBSD folk, but I'm open to ideas as to how best to deal with the problem in the context of the test suite pending resolution by FreeBSD.
Linux does the same thing, unless the user has explicitly configured that behavior off. Search the web for linux overcommit. It's controlled by the vm.overcommit_memory sysctl. Although linux's default is some heuristic that might make Python's test case work right in most cases, depending on malloc returning NULL is not something you can actually depend on.
James _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com