[pypy-dev] Dead loop occurs when using python-daemon and multiprocessing together in PyPy 4.0.1

hubo hubo at jiedaibao.com
Mon Dec 28 01:00:35 EST 2015


I think the multi-threaded cases are not really problems. The race condition only happens when another thread closed the file descriptor, and the only knowing case of closing file descriptors without knowing what they do is during fork (daemon) process. It always produces unpredictable results if you fork with multiple threads without planning anyway. For GIL, I think it is acceptable for PyPy to be different with CPython in very rare cases. Users should understand they have different implement details. It is only important that common use cases like python-daemon works as expected.

2015-12-28 

hubo 



发件人:Armin Rigo <arigo at tunes.org>
发送时间:2015-12-27 07:42
主题:Re: [pypy-dev] Dead loop occurs when using python-daemon and multiprocessing together in PyPy 4.0.1
收件人:"Vincent Legoll"<vincent.legoll at gmail.com>
抄送:"pypy-dev"<pypy-dev at python.org>

Hi Vincent, 

On Sat, Dec 26, 2015 at 1:45 PM, Vincent Legoll 
<vincent.legoll at gmail.com> wrote: 
> OK, I understand we want to have behavior as identical to cpython as possible. 
> 
> So does that mean implementing the same : "fstat() & check we're still 
> on the same inode & device as before" ? 
> Do we also need to do the "could have been opened in another thread" dance ? 
> And what about the CLO_EXEC thing, this does seem sensible to do too, 

This is what I started to do, but I stopped because there are 
additional issues on PyPy: you need to make sure that the GIL is not 
released at places where CPython does not release it either. 
Otherwise, you're opening the code to "race conditions" again. 

The goal would be to be *at least* as safe as CPython.  There are a 
lot of corner cases that are not discussed in the source code of 
CPython.  I'm pretty sure that some of them are possible (but rare). 
As an extreme example, if one thread does os.urandom() and another 
thread does os.close(4) in parallel, where 4 happens to be the file 
descriptor returned by open() in urandom.c, then it is possible that 
the open()'s result is closed after open() but before urandom.c 
reacquires the GIL.  Then urandom.c gets a closed file descriptor.  If 
additionally the other thread (or a 3rd one) opens a different file at 
file descriptor 4, then urandom.c will think it successfully opened 
/dev/urandom but actually the file descriptor is for some unrelated 
file.  However, this is probably an issue that cannot be dealt with at 
all on Posix even in C.  That would mean that it is always wrong to 
close() unknown file descriptors from one thread when other threads 
might be running...  This would not hurt being clarified. 


A bientôt, 

Armin. 
_______________________________________________ 
pypy-dev mailing list 
pypy-dev at python.org 
https://mail.python.org/mailman/listinfo/pypy-dev 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20151228/b0eab3be/attachment.html>


More information about the pypy-dev mailing list