[Python-Dev] segfault (double free?) when '''-string crosses line

[Tim Peters]

[Guido]
> On Linux, In HEAD 2.5, but only with the non-debug version, I get a
> segfault when I do this:
>
> >>> '''
> ... '''
>
> It seems to occur for any triple-quoted string crossing a line
> boundary. A bit of the stack trace:
>
> #0  0x40030087 in pthread_mutex_lock () from /lib/i686/libpthread.so.0
> #1  0x4207ad18 in free () from /lib/i686/libc.so.6
> #2  0x08057990 in tok_nextc (tok=0x81c71d8) at ../Parser/tokenizer.c:809
> #3  0x0805872d in tok_get (tok=0x81c71d8, p_start=0xbffff338, p_end=0xbffff33c)
>     at ../Parser/tokenizer.c:1411
> #4  0x08059042 in PyTokenizer_Get (tok=0x81c71d8, p_start=0xbffff338,
>     p_end=0xbffff33c) at ../Parser/tokenizer.c:1514
> #5  0x080568a7 in parsetok (tok=0x81c71d8, g=0x814a000, start=256,
>     err_ret=0xbffff3a0, flags=0) at ../Parser/parsetok.c:135
>
>
> Does this ring a bell? Is there already an SF bug open perhaps?
>
> On OSX, I get an interesting error:
>
> python2.5(12998) malloc: ***  Deallocation of a pointer not malloced:
> 0x36b460; This could be a double free(), or free() called with the
> middle of an allocated block; Try setting environment variable
> MallocHelp to see tools to help debug

It rings a bell here only in that the front end had lots of
allocate-versus-free mismatches between the PyObject_ and PyMem_
raw-memory APIs, and this kind of failure smells a lot like that.

For example, the ../Parser/tokenizer.c:809 in the traceback is

				PyMem_FREE(new);

and _one_ way to set `new` is from the earlier well-hidden

			else if (tok_stdin_decode(tok, &new) != 0)

where tok_stdin_decode() can do

	PyMem_FREE(*inp);
	*inp = converted;

where `inp` is its local name for `new`, and `converted` comes from

	converted = new_string(PyString_AS_STRING(utf8),
			       PyString_GET_SIZE(utf8));

and new_string() starts with

	char* result = (char *)PyObject_MALLOC(len + 1);

So that's a mismatch, although I don't know whether it's the one
that's triggering.

When I repaired all the mismatches that caused tests to crash on my
box, I changed affected front-end string mucking to use PyObject_
uniformly (strings are usual small, the small-object allocator is
usually faster than the platform malloc, and half (exactly half :-) of
the crash-causing mismatched pairs were using PyObject_ anyway). 
Someone who understands their way through the sub-maze above is
encouraged to do the same for it.

BTW, your "but only with the non-debug version" is more evidence:  in
a debug build, PyMem_ and PyObject_ calls are all redirected to
Python's obmalloc, to take advantage of its debug-build padding
gimmicks.  It's only in a release build that PyMem_ resolves directly
to the platform malloc/realloc/free.