[Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...

[Trent Nelson]

On Thu, Dec 20, 2012 at 05:47:40PM -0800, Gregory P. Smith wrote:
>    On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson <trent at snakebite.org>
>    wrote:
> 
>          This seems odd to me so I wanted to see what others think.  The unit
>          test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings
>          will eventually hit subprocess.Popen._communicate.
> 
>          The `mswindows` implementation of this method relies on threads to
>          buffer stdin/stdout.  That'll eventually result in
>      PyOs_StdioReadline
>          being called without the GIL being held.  PyOs_StdioReadline calls
>          PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC.
> 
>    Those threads are implemented in Python so how would the GIL ever not be
>    held?
>    -gps

    PyOS_Readline drops the GIL prior to calling PyOS_StdioReadline:

        Py_BEGIN_ALLOW_THREADS
--------^^^^^^^^^^^^^^^^^^^^^^
    #ifdef WITH_THREAD
        PyThread_acquire_lock(_PyOS_ReadlineLock, 1);
    #endif

        /* This is needed to handle the unlikely case that the
         * interpreter is in interactive mode *and* stdin/out are not
         * a tty.  This can happen, for example if python is run like
         * this: python -i < test1.py
         */
        if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout)))
            rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt);
-----------------^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        else
            rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout,
                                                 prompt);
        Py_END_ALLOW_THREADS


        Trent.