Mailman 3 March 2002 - Python-Dev

Debug entry points for PyMalloc
by Tim Peters March 23, 2002

March 23, 2002

The thing I've dreaded most about switching to pymalloc is losing the invaluable memory-corruption clues supplied by the Microsoft debug-build malloc. On more than one occasion, they've found wild stores, out-of-bounds reads, reads of uninitialized memory, and reads of free()ed memory in Python. It does this by spraying special bytes all over malloc'ed memory at various times, then checking the bytes for sanity at free() and realloc() times. This kind of stuff is going to be pure hell under pymalloc, because there's no padding at all between chunks pymalloc passes out, and pymalloc stores valid addresses at the start of free()ed memory. So a wild store probably can't be detected as any sort of memory corruption, it will simply overwrite part of some other end-user object -- or corrupt pymalloc's internal pointers linking free()ed memory (and pymalloc simply goes nuts then). Several months ago Martin and I took turns thinking about a memory overwrite problem in the Unicode stuff that showed up under pymalloc, and it was indeed pure hell to track it down. Following is a sketch for teaching pymalloc how to do something similar to the MS scheme. A twist over the MS scheme is adding a "serial number" to the pad bytes, incremented by one for each malloc/realloc. At a crude level, this gives a sense of age to the eyeball; for reproducible memory nightmares, it gives an exact way to set a data, or counting, breakpoint (on the next run) to capture the instant at which a doomed-to-go-bad memory block first gets passed out. I hope that addresses the worst problem the MS scheme still leaves untouched: you can catch memory corruption pretty will with it, but all you know then is that "the byte at this address is bad", and you have no idea what the memory's original purpose in life was. Sketch of Debug Mode for PyMalloc + Three new entry points in obmalloc.c (note: stop #include'ing this; hiding code in include files sucks, and naming an include file .c compounds the confusion): DL_IMPORT(void *) _PyMalloc_DebugMalloc(size_t nbytes); DL_IMPORT(void *) _PyMalloc_DebugRealloc(void *p, size_t nbytes); DL_IMPORT(void) _PyMalloc_DebugFree(void *p); + When WITH_PYMALLOC and PYMALLOC_DEBUG are #define'd, these are mapped to in the obvious way from _PyMalloc_{MALLOC, REALLOC, FREE}: #ifdef WITH_PYMALLOC DL_IMPORT(void *) _PyMalloc_Malloc(size_t nbytes); DL_IMPORT(void *) _PyMalloc_Realloc(void *p, size_t nbytes); DL_IMPORT(void) _PyMalloc_Free(void *p); DL_IMPORT(void *) _PyMalloc_DebugMalloc(size_t nbytes); DL_IMPORT(void *) _PyMalloc_DebugRealloc(void *p, size_t nbytes); DL_IMPORT(void) _PyMalloc_DebugFree(void *p); #ifdef PYMALLOC_DEBUG #define _PyMalloc_MALLOC _PyMalloc_DebugMalloc #define _PyMalloc_REALLOC _PyMalloc_DebugRealloc #define _PyMalloc_FREE _PyMalloc_DebugFree #else /* WITH_PYMALLOC && !PYMALLOC_DEBUG */ #define _PyMalloc_MALLOC _PyMalloc_Malloc #define _PyMalloc_REALLOC _PyMalloc_Realloc #define _PyMalloc_FREE _PyMalloc_Free #endif /* PYMALLOC_DEBUG */ #else /* !WITH_PYMALLOC */ #define _PyMalloc_MALLOC PyMem_MALLOC #define _PyMalloc_REALLOC PyMem_REALLOC #define _PyMalloc_FREE PyMem_FREE #endif /* WITH_PYMALLOC */ + A debug build implies PYMALLOC_DEBUG, but PYMALLOC_DEBUG can be forced in a release build. + No changes to the guts of _PyMalloc_{Malloc, Realloc, Free}. Keep them as lean and as clear of #ifdef obscurity as they are now. + Define three special bit patterns. In hex, they all end with B (for deBug <wink>), and begin with a vaguely mnemonic letter. Strings of these are unlikely to be legit memory addresses, ints, 7-bit ASCII, or floats: #define PYMALLOC_CLEANBYTE 0xCB /* uninitialized memory */ #define PYMALLOC_DEADBYTE 0xDB /* free()ed memory */ #define PYMALLOC_FORBIDDENBYTE 0xFB /* unusable memory */ The debug malloc/free/realloc use these as follows. Note that this stuff is done regardless of whether PyMalloc handles the request directly or passes it on to the platform malloc (in fact, the debug entry points won't know and won't care). + The Debug malloc asks for 16 extra bytes and fills them with useful stuff: p[0:4] Number of bytes originally asked for. 4-byte unsigned integer, big-endian (easier to read in a memory dump). p[4:8] Copies of PYMALLOC_FORBIDDENBYTE. Used to catch under- writes and reads. p[8:8+n] The requested memory, filled with copies of PYMALLOC_CLEANBYTE. Used to catch reference to uninitialized memory. &p[8] is returned. Note that this is 8-byte aligned if PyMalloc handled the request itself. p[8+n:8+n+4] Copies of PYMALLOC_FORBIDDENBYTE. Used to catch over- writes and reads. p[8+n+4:8+n+8] A serial number, from a PyMalloc file static, incremented by 1 on each call to _PyMalloc_DebugMalloc and _PyMalloc_DebugRealloc. 4-byte unsigned integer, big-endian. If "bad memory" is detected later, the serial number gives an excellent way to set a breakpoint on the next run, to capture the instant at which this block was passed out. + The Debug free first uses the address to find the number of bytes originally asked for, then checks the 8 bytes on each end for sanity (in particular, that the PYMALLOC_FORBIDDENBYTEs are still intact). XXX Make this checking a distinct entry point. XXX In case an error is found, print informative stuff, but then what? XXX Die or keep going? Fatal error is probably best. Then fills the original N bytes with PYMALLOC_DEADBYTE. This is to catch references to free()ed memory. The forbidden bytes are left intact. Then calls _PyMalloc_Free. + The Debug realloc first calls _PyMalloc_DebugMalloc with the new request size. Then copies over the original bytes. The calls _PyMalloc_DebugFree on the original bytes. XXX This could, and probably should, be optimized to avoid copying XXX every time.

10 18

PEP 282: "FATAL" a misnomer
by David Goodger March 23, 2002

March 23, 2002

One thing I've found uncomfortable in the PEP (& log4j spec) was the "FATAL" name for the highest logging level. I think it's a misnomer and should be renamed. "FATAL" implies death, which in Python implies a raised and uncaught exception, traceback, and exit. If the logging module enforces such an outcome from a FATAL-level log entry, that would make sense, but I don't think that's the case (nor should it be, at least not invariably). In the logging class I wrote [#]_, I used the term "SEVERE". Paul Svensson mentions "CRITICAL" in his system. I think either one of these expresses the intrinsic meaning better (a bad error, worse than "ERROR", ignored at peril), without the misleading connotations of "FATAL". So, how about renaming "FATAL" to "SEVERE" or "CRITICAL"? .. [#] The "Reporter" class in http://docstring.sf.net/dps/utils.py; with copious documentation in docstrings. Also, this class *does* support raising exceptions via an 'errorlevel' exception threshold. I've mentioned it before; please check it out! -- David Goodger goodger(a)users.sourceforge.net Open-source projects: - Python Docstring Processing System: http://docstring.sourceforge.net - reStructuredText: http://structuredtext.sourceforge.net - The Go Tools Project: http://gotools.sourceforge.net

1 0

-C in Setup files
by Andrew Kuchling March 23, 2002

March 23, 2002

Pete Shinners reported that Distutils doesn't treat the -C option in a Setup file the same as the makesetup script. He's right, but I don't think the -C option is useful in makesetup at all! The relevant bit of code in makesetup is: case $arg in -framework) libs="$libs $arg"; skip=libs; ;; -[IDUCfF]*) cpps="$cpps $arg";; -Xlinker) libs="$libs $arg"; skip=libs;; '-Cfoo' in a Setup file therefore causes -Cfoo to be added to the list of compiler flags, but at least in GCC, that doesn't mean anything; -C indicates that the preprocessor shouldn't strip comments, and it doesn't take an argument. (Anyone know what -C does in other compilers?) Because of this, I think -C in makesetup has been useless since revision 1.1 of makesetup (which is 1994 -- Python 1.0.0), so I doubt anyone has ever used it. Distutils should do something useful with it, and IMHO that would be to make '-C cpparg' add 'cpparg' to the compiler's arguments. Thoughts? Should makesetup be fixed to match? --amk (www.amk.ca) The more I know me, the less I like me. -- The Doctor, in "Time and the Rani"

2 3

asyncore 2.1.1/2.2 incompatibility
by Andrew Kuchling March 23, 2002

March 23, 2002

[Crossposted to python-dev and the Medusa mailing list, and CC'ed to the bug reporter; watch your followups!] Bug #528295 (http://www.python.org/sf/528295) reports a difference in asyncore between 2.1.1 and 2.2. The test program attached to that bug report installs a signal handler so 'kill -HUP <pid>' causes the program to re-read its configuration file. This works in 2.2.1 (asyncore rev. 1.10), but doesn't in 2.2 (rev. 1.30). Revision 1.25, made to fix http://www.python.org/sf/453099, is what breaks it; as of that revision, asyncore's poll() functions catch the EINTR error and quietly go around asyncore's main loop. Before that change, the EINTR exception would propagate upward, so the test program could re-read its configuration file and then call asyncore.loop() again. So, as I see it, it's no longer possible to usefully respond to signals with asyncore in 2.2. Is that acceptable? Or is there some clever way to make 'kill -HUP' work again? It might be a good idea to add a function to asyncore that can be called to signal that the asyncore loop should be exited; signal handlers could then call this function to force an exit. That means asyncore users will have to rewrite their code for 2.2, though. If there's agreement that this is OK, should I try to get it into 2.2.1? --amk (www.amk.ca) It is sometimes a mistake to climb, it is always a mistake never even to make the attempt. -- Dream, in SANDMAN: "Fear of Falling"

5 6

SF bug reminders
by Jeremy Hylton March 23, 2002

March 23, 2002

I dusted off my SF bug reminder script and ran it tonight. Unfortunately, I forgot to update the text of the message it sends. I hope the committers will forgive the silly text about getting things done before 2.2. Jeremy

1 0

Shedding load
by Guido van Rossum March 23, 2002

March 23, 2002

I'm swamped in email. I find I can barely get through a day's email in a day. So I've decided to cut some high-volume sources of mail: I no longer read the SF bugs and patches lists, and I see the checkins list as a digest. If you see a SF patch or bug that deserves my attention, simply assign it to me; SF then sends me an email alerting of its existence, and I believe also subscribes me to any future changes of that specific patch or bug. I'm hoping this will make me a little more productive! --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

RE: [ python-Bugs-533234 ] tm_isdst > 1 Passed to strftime.
by Tim Peters March 23, 2002

March 23, 2002

A reminder that nobody should check in anything without running the test suite on their box first. At least two checkins today from two different people have broken the tests even on their boxes <wink>. The test_strftime mess is explicated in http://python.org/sf/533234

1 0

test_strftime broken?
by Neil Schemenauer March 23, 2002

March 23, 2002

I get: strftime test for Fri Mar 22 18:59:49 2002 Strftime test, platform: linux2, Python version: 2.3a0 Conflict for %b (abbreviated month name): Expected Jan, but got Mar Conflict for %B (full month name): Expected January, but got March Supports nonstandard '%c' format (near-asctime() format) Supports nonstandard '%x' format (%m/%d/%y %H:%M:%S) Supports nonstandard '%Z' format (time zone name) Supports nonstandard '%D' format (mm/dd/yy) Supports nonstandard '%e' format (day of month as number, blank padded ( 0-31)) Conflict for nonstandard '%h' format (abbreviated month name): Expected Jan, but got Mar It goes on and on for many lines. Neil

2 1

Re: [Python-Dev] 2.2.1c1 platform reports.
by Michael Hudson March 22, 2002

March 22, 2002

Hye-Shik Chang <perky(a)fallin.lv> writes: > On Fri, Mar 22, 2002 at 11:44:52AM +0000, Michael Hudson wrote: > > > > OS: FreeBSD > > > > test_math.test_exceptions blows up with "sqrt(-1) does not raise > > ValueError". It seems FreeBSD's libm doesn't set errno in this case, > > and that MacOSX has inherited this behaviour (see below) > > > > OS: OpenBSD > > > > test_signal hangs on the sf compile farm. We've had reports of this > > before. I can't be bothered to dig into it today. > > > > OS: NetBSD > > > > no reports. > > > > All *BSDs are using msun of Sun as math library. > NetBSD and OpenBSD may be same with FreeBSD. I don't think they are, surprisingly. [...] > I'll submit PR to both of FreeBSD and NetBSD. I'm not sure it's a problem. I believe it depends on whether you believe in C89 or C99, for instance. Cheers, M. -- I'd certainly be shocked to discover a consensus. ;-) -- Aahz Maruch, comp.lang.python

5 5

Buidling on insane platforms
by Tim Peters March 22, 2002

March 22, 2002

Following up on a threat I made earlier, I've fiddled my (Windows) build so that: + obmalloc.c is now compiled directly. It's no longer #include'd by object.c. + All the _PyMalloc_XXX entry points are physically inside obmalloc.c now (not some in object.c, and others in obmalloc.c). This was easy to get working under the Windows build because I knew how to do it <wink>. In staring at setup.py, though, I find I have no idea how to get it to work on your platform (let alone on the Mac). I find it hard to believe it could just work by magic. Do Unixish boxes still use Makefile.pre.in? If I add Objects/obmnalloc.o \ to the OBJECT_OBJS definition there, and remove this now-pointless line: Objects/object.o: $(srcdir)/Objects/object.c $(srcdir)/Objects/obmalloc.c is that enough? Goodness knows nobody hesitates to break the Windows build <wink>, but I'd just as soon not break yours if I can help it.

2 1