[Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...

[Kristján Valur Jónsson]

I ran into this the other day.  I had put in hooks in the PyMem_MALLOC to track memory per tasklet, and it crashed
in those cases because it was being called without the GIL.  My local patch was simply to _not_ release the GIL.
Clearly, calling PyMem_MALLOC without the GIL is an API violation.

K

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-
> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Trent Nelson
> Sent: 21. desember 2012 03:13
> To: Gregory P. Smith
> Cc: Python-Dev
> Subject: Re: [Python-Dev] Possible GIL/threading issue involving subprocess
> and PyMem_MALLOC...
> 
> On Thu, Dec 20, 2012 at 05:47:40PM -0800, Gregory P. Smith wrote:
> >    On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson <trent at snakebite.org>
> >    wrote:
> >
> >          This seems odd to me so I wanted to see what others think.  The unit
> >          test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings
> >          will eventually hit subprocess.Popen._communicate.
> >
> >          The `mswindows` implementation of this method relies on threads to
> >          buffer stdin/stdout.  That'll eventually result in
> >      PyOs_StdioReadline
> >          being called without the GIL being held.  PyOs_StdioReadline calls
> >          PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC.
> >
> >    Those threads are implemented in Python so how would the GIL ever not
> be
> >    held?
> >    -gps
> 
>     PyOS_Readline drops the GIL prior to calling PyOS_StdioReadline:
> 
>         Py_BEGIN_ALLOW_THREADS
> --------^^^^^^^^^^^^^^^^^^^^^^
>     #ifdef WITH_THREAD
>         PyThread_acquire_lock(_PyOS_ReadlineLock, 1);
>     #endif
> 
>         /* This is needed to handle the unlikely case that the
>          * interpreter is in interactive mode *and* stdin/out are not
>          * a tty.  This can happen, for example if python is run like
>          * this: python -i < test1.py
>          */
>         if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout)))
>             rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt); ----------------
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>         else
>             rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout,
>                                                  prompt);
>         Py_END_ALLOW_THREADS
> 
> 
>         Trent.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com