[Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
Gregory P. Smith
greg at krypto.org
Fri Dec 21 02:47:40 CET 2012
On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson <trent at snakebite.org> wrote:
> This seems odd to me so I wanted to see what others think. The unit
> test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings
> will eventually hit subprocess.Popen._communicate.
> The `mswindows` implementation of this method relies on threads to
> buffer stdin/stdout. That'll eventually result in PyOs_StdioReadline
> being called without the GIL being held. PyOs_StdioReadline calls
> PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC.
Those threads are implemented in Python so how would the GIL ever not be
> On a debug build, these macros are redirected to their _PyMem_Debug*
> counterparts. The call hierarchy for _PyMem_DebugMalloc looks like
> void *
> _PyMem_DebugMalloc(size_t nbytes)
> return _PyObject_DebugMallocApi(_PYMALLOC_MEM_ID, nbytes);
> /* generic debug memory api, with an "id" to
> identify the API in use */
> void *
> _PyObject_DebugMallocApi(char id, size_t nbytes)
> uchar *p; /* base address of malloc'ed block */
> uchar *tail; /* p + 2*SST + nbytes ==
> pointer to tail pad bytes */
> size_t total; /* nbytes + 4*SST */
> total = nbytes + 4*SST;
> if (total < nbytes)
> /* overflow: can't represent total as a size_t */
> return NULL;
> p = (uchar *)PyObject_Malloc(total);
> if (p == NULL)
> return NULL;
> Both bumpserialno() and PyObject_Malloc affect global state. The
> also has a bunch of LOCK() and UNLOCK() statements, but these end up
> * Python's threads are serialized,
> * so object malloc locking is disabled.
> #define SIMPLELOCK_DECL(lock) /* simple lock declaration */
> #define SIMPLELOCK_INIT(lock) /* allocate (if needed) and ... */
> #define SIMPLELOCK_FINI(lock) /* free/destroy an existing */
> #define SIMPLELOCK_LOCK(lock) /* acquire released lock */
> #define SIMPLELOCK_UNLOCK(lock) /* release acquired lock */
> * This malloc lock
> #define LOCK() SIMPLELOCK_LOCK(_malloc_lock)
> #define UNLOCK() SIMPLELOCK_UNLOCK(_malloc_lock)
> #define LOCK_INIT() SIMPLELOCK_INIT(_malloc_lock)
> #define LOCK_FINI() SIMPLELOCK_FINI(_malloc_lock)
> The PyObject_Malloc() one concerns me the most, as it affects huge
> amounts of global state. Also, I just noticed PyOs_StdioReadline()
> can call PyErr_SetString, which will result in a bunch of other
> calls that should only be made whilst the GIL is held.
> So, like I said, this seems like a bit of a head scratcher. Legit
> issue or am I missing something?
> Python-Dev mailing list
> Python-Dev at python.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev