[Python-Dev] segfault (double free?) when '''-string crosses line

[Tim Peters]

[Neal Norwitz]
> http://python.org/sf/1467512 fixes the problem for me on linux.  It
> converts all the PyMem_* APIs to PyObject_* APIs.  Assigned to Guido
> until he changes that. :-)

Thanks!  I didn't take that route, instead not changing anything I
didn't take the time to understand first.  For example, it's a pure
loss to change tok_new() to use PyObject_ (as the patch does), since a
struct tok_state is far too big for obmalloc to handle (e.g., the
`indstack` member alone is too big -- that's an array of 100 ints, so
consumes at least 400 bytes by itself).  Using PyObject_ here just
wastes some time figuring out that tok_state is too big for it, and
passes the calls on to the system malloc/free.

I thought about switching the readline thingies at the time, but
quickly hit a wall:  I have no idea where the source for

#ifdef __VMS
extern char* vms__StdioReadline(FILE *sys_stdin, FILE *sys_stdout,
char *prompt);
#endif

may be (it's apparently not checked in, and Google couldn't find it
either).  Does anyone still work on the VMS port?  If not, screw it
;-)

> There are several more places in the core that should probably use
> PyObject_* memory APIs since the alloced memory is small.  412 uses of
> PyMem_* in */*.c.  Most of those are in modules where it is probably
> appropriate.  But PyFutureFeatures could really use PyObject_* given
> it's only 8 bytes. (Python/future.c and Python/compile.c).

If we allocate one of those per PyAST_Compile, I'm not sure the
difference would be measurable ;-)

> Modules/_bsddb.c has a scary line:  #define PyObject_Del PyMem_DEL

That one's fine -- it's protected by an appropriate #ifdef:

#if PYTHON_API_VERSION <= 1007
    /* 1.5 compatibility */
#define PyObject_New PyObject_NEW
#define PyObject_Del PyMem_DEL
#endif

The name "PyObject_Del" didn't exist in Pythons that old, so
"something like that" is necessary to write C that works with all
Pythons ever released.