test_descr.py crashes under the debug build now. I've got too many things piled on my stack to look into it right now, so it would be good if someone could try to feel debug pain on Linux too.
Configuring Python with --with-pydebug, I cannot get test_descr to crash on Linux, using the current CVS. Regards, Martin
[Martin v. Loewis]
Configuring Python with --with-pydebug, I cannot get test_descr to crash on Linux, using the current CVS.
I believe that <wink>. It's not dying with a memory fault, it's dying because code in MS's debug libraries is proactively searching for memory corruption, finding it, then deliberately causing a crash as its way of spelling "hey! your program is insane". I expect you'd need something like Insure++ or efence to have a good chance of catching it on Linux. Sooner or later it will manifest as insane behavior in the release build -- this is "wild store" stuff, like bad pointer arithmetic, or out-of-bounds array indexing. Resulting problems usually occur long after the bad store, and can be symptom-free by luck (until you're doing a demo <0.9 wink>).
Tim Peters writes:
spelling "hey! your program is insane". I expect you'd need something like Insure++ or efence to have a good chance of catching it on Linux. Sooner or
I've started an Electric Fence run on a Mandrake 7.2 box, running the regression tests. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Given that Guido's already done the efence run, I'll ignore the weird result I'm getting. ;-) On a laptop running Mandrake 7.2, Python configured --with-pydebug, and lots of free memory, I get this: beowolf(.../python/debug); LD_PRELOAD=/usr/lib/libefence.so.0.0 ./python ../Lib/test/regrtest.py -l Electric Fence 2.2.0 Copyright (C) 1987-1999 Bruce Perens <bruce@perens.com> Adding parser accelerators ... Done. test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ ElectricFence Exiting: mmap() failed: Cannot allocate memory [255] beowolf(.../python/debug); free total used free shared buffers cached Mem: 523808 204256 319552 0 29112 116824 -/+ buffers/cache: 58320 465488 Swap: 1050800 0 1050800 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Fred L. Drake, Jr. wrote:
Given that Guido's already done the efence run, I'll ignore the weird result I'm getting. ;-) On a laptop running Mandrake 7.2, Python configured --with-pydebug, and lots of free memory, I get this:
beowolf(.../python/debug); LD_PRELOAD=/usr/lib/libefence.so.0.0 ./python ../Lib/test/regrtest.py -l
...
ElectricFence Exiting: mmap() failed: Cannot allocate memory
Using electric fence requires huge amounts of (virtual) memory. Every allocation requires two pages of memory. I doubt running the entire test suite would ever work. I'm having trouble running just test_descr alone with all the tests commented out! I think I'm running into MAX_MAP_COUNT (which defaults to 64k). I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a dynamically configurable max_map_count. We'll see how that goes. Someone should really write a portable debug version of malloc and free for Python that works like the Microsoft implementation. It can't be too hard to lay down poison before and after the hunk of memory on allocation and then check it on deallocation. Neil
[Neil Schemenauer]
Using electric fence requires huge amounts of (virtual) memory. Every allocation requires two pages of memory.
Yikes! [NealS]
I doubt running the entire test suite would ever work. I'm having trouble running just test_descr alone with all the tests commented out!
FWIW, I can provoke the bug by running test_descr.inherits() alone: C:\Code\python\PCbuild>python_d Adding parser accelerators ... Done. Python 2.2a4+ (#24, Oct 5 2001, 18:16:47) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
from test import test_descr [21862 refs] test_descr.inherits() Testing inheritance from basic types...
And then it blows up.
I think I'm running into MAX_MAP_COUNT (which defaults to 64k).
I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a dynamically configurable max_map_count. We'll see how that goes.
You may as well rewrite the kernel while you're at it <wink>.
Someone should really write a portable debug version of malloc and free for Python that works like the Microsoft implementation. It can't be too hard to lay down poison before and after the hunk of memory on allocation and then check it on deallocation.
It's extremely useful for establishing the presence of off-by-one stores, but this is the first time I've seen this particular check trigger in Python. Another trick it plays is to fill free()'ed memory with a different unlikely byte pattern, then delay recycling the memory. When things blow up this "dead landfill" value is amazingly often present (indicating that someone is still trying to read from free'd memory). The third trick is that newly malloc'ed memory is filled with a third "clean landfill" byte, which helps catch use of uninitialized memory. All the special byte values are picked so that they're unlikely to yield valid addresses, or integers. In bang for the buck, these tricks are huge wins. if-only-they-pointed-to-the-true-causes-too-ly y'rs - tim
Using electric fence requires huge amounts of (virtual) memory. Every allocation requires two pages of memory. I doubt running the entire test suite would ever work. I'm having trouble running just test_descr alone with all the tests commented out! I think I'm running into MAX_MAP_COUNT (which defaults to 64k).
The trick is to compile it to a .pyc file without efence. The compiler is vastly malloc-happy. Once you have the .pyc, test_descr runs to completion without errors using efence. --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (5)
-
Fred L. Drake, Jr. -
Guido van Rossum -
Martin v. Loewis -
Neil Schemenauer -
Tim Peters