[Python-Dev] pthreads, fork, import, and execvp

[Nick Coghlan]

Guido van Rossum wrote:
> Yeah, I think imports inside functions are overused.

Rotem took the time to explain the problem to me more fully in private email, 
and it appears that using import statements inside functions is currently 
fundamentally unsafe in a posix environment where both multiple threads and 
fork() are used. If any thread other than the one invoking fork() holds the 
import lock at the time of the fork then import statements in the spawned 
process will deadlock.

So I believe fixing this properly would indeed require assigning a new lock 
object in one of the two places Rotem suggested.

Cheers,
Nick.

> 
> On 5/9/06, Rotem Yaari <vmalloc at gmail.com> wrote:
>> Hello everyone!
>>
>> We have been encountering several deadlocks in a threaded Python
>> application which calls  subprocess.Popen (i.e. fork()) in some of its
>> threads.
>>
>> This has occurred on Python 2.4.1 on a 2.4.27 Linux kernel.
>>
>>  Preliminary analysis of the hang shows that the child process blocks
>> upon entering the  execvp function, in which the import_lock is acquired
>> due to the following line:
>>
>> def _ execvpe(file,  args,  env=None):
>>     from  errno import ENOENT, ENOTDIR
>>     ...
>>
>> It is known that when forking from a  pthreaded application, acquisition
>> attempts on locks which were already locked by other threads while
>> fork() was called will deadlock.
>>
>> Due to these oddities we were wondering if it would be better to extract
>> the above import line from the  execvpe call, to prevent lock
>> acquisition attempts in such cases.
>>
>> Another workaround could be re-assigning a new lock to import_lock
>> (such a thing is done with the global interpreter lock) at PyOS_AfterFork or
>> pthread_atfork.
>>
>> We'd appreciate any opinions you might have on the subject.
>>
>> Thanks in advance,
>>
>> Yair and  Rotem
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
> 
> 


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org