[Python-Dev] RE: Painful death in debug build

[Tim Peters]

[Neil Schemenauer]
> Using electric fence requires huge amounts of (virtual) memory.  Every
> allocation requires two pages of memory.

Yikes!
[NealS]
> I doubt running the entire test suite would ever work.  I'm having
> trouble running just test_descr alone with all the tests commented out!

FWIW, I can provoke the bug by running test_descr.inherits() alone:

C:\Code\python\PCbuild>python_d
Adding parser accelerators ...
Done.
Python 2.2a4+ (#24, Oct  5 2001, 18:16:47) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_descr
[21862 refs]
>>> test_descr.inherits()
Testing inheritance from basic types...

And then it blows up.

> I think I'm running into MAX_MAP_COUNT (which defaults to 64k).
>
> I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a
> dynamically configurable max_map_count.  We'll see how that goes.

You may as well rewrite the kernel while you're at it <wink>.

> Someone should really write a portable debug version of malloc and free
> for Python that works like the Microsoft implementation.  It can't be
> too hard to lay down poison before and after the hunk of memory on
> allocation and then check it on deallocation.

It's extremely useful for establishing the presence of off-by-one stores,
but this is the first time I've seen this particular check trigger in
Python.  Another trick it plays is to fill free()'ed memory with a different
unlikely byte pattern, then delay recycling the memory.  When things blow up
this "dead landfill" value is amazingly often present (indicating that
someone is still trying to read from free'd memory).  The third trick is
that newly malloc'ed memory is filled with a third "clean landfill" byte,
which helps catch use of uninitialized memory.  All the special byte values
are picked so that they're unlikely to yield valid addresses, or integers.
In bang for the buck, these tricks are huge wins.

if-only-they-pointed-to-the-true-causes-too-ly y'rs  - tim