[Python-Dev] pymalloc killer

[Guido van Rossum]

> There may be hope for this, and another strong reason for pursuing it.
[good stuff snipped]

> A nasty subtlety: if PyMem_NukeIt is called when the GIL isn't held,
> it's going to take a lot of thought to see whether this can be made
> safe when called simultaneously by multiple threads; e.g., the
> finger can change "mid-stream" then, and, worse, some thread that
> *does* hold the GIL may legitimately cause a new arena to be
> allocated midstream, mutating the address vector during the search.
> This may be intractable enough to kill the idea of mapping PyMem_XXX
> to this.

I know of two places that calls PyMem_Malloc outside the GIL:
PyOS_StdioReadline in Parser/myreadline.c and call_readline() in
Modules/readline.c.  The memory thus allocated is returned by
PyOS_Readline() (which calls either function through a hook pointer),
and both its callers (the tokenizer and raw_input()) free the result
using PyMem_DEL or PyMem_FREE (these two seem to be used
synonymically).  The tokenizer code may also apply PyMem_RESIZE to it
(it incorporated the input line in its buffer structure).

This would have to be fixed by changing the allocations to use
malloc() (as they did up to 1.5.2 :-) and by changing the consumers to
use free() and realloc().  (An alternative would be to let
PyOS_Readline() copy the memory to PyMem_Malloc'ed memory.)

This is part of a "hook" API that allows 3rd parties to provide their
own alternative to PyOS_Readline.  This was put in at the request of
some folks at LLNL who were providing their own GUI that fed into
Python and who had some problems with sending it to stdin.  I don't
think anybody else has used this.  There is not a single mention of
PyOS_Readline in the entire set of Python documentation.

Given the alternatives:

1. introduce new APIs PyMalloc_{New,Del}Object and tell all extension
   writers that they have to changes their extensions once again to
   use these brand new APIs if they want to benefit from pymalloc; or

2. fix the issues with PyOS_Readline, make PyMem_{DEL,FREE,Del,Free}
   synonyms for Tim's clever PyMem_NukeIt, and continue to promote
   using PyObject_{New,Del} for use by extension writers;

I'm all for #2.

--Guido van Rossum (home page: http://www.python.org/~guido/)