[pypy-dev] Dead loop occurs when using python-daemon and multiprocessing together in PyPy 4.0.1

[Maciej Fijalkowski]

That's very interesting, can you produce a standalone example that does not
use multiprocessing? That would make it much easier to fix the bug (e.g.
os.fork followed by os.urandom failing)

On Wed, Dec 23, 2015 at 1:54 PM, hubo <hubo at jiedaibao.com> wrote:

> Thanks for the response. Should I put it directly in the bug tracker?
>
> FYI, I've located the reason to be the incompatibility with python-daemon
> (or rather the standard unix-daemon behavior) and PyPy *posix.urandom*
> implementation.
>
> It seems that in PyPy 4.0.1, when module *random* loaded, a file
> descriptor is created on /dev/urandom. I think PyPy implementation use the
> shared descriptor to read from /dev/urandom. Sadly when python-daemon fork
> the process and turns it into an unix daemon, it closes all the currently
> open file descriptors. After that all os.urandom calls failed with OSError.
> I think maybe the other functions of Random class is also using the file
> descriptor in C code and just never detects if the return value is 0, and
> causes the dead loop.
>
> I think the problem will be solved if the implementation re-open the
> handle when it is closed somehow.
>
> multiprocessing is using random internally. Also there are lots of other
> modules using random, like email etc. The dead loop occurs when you use any
> of the libraries in a daemon.
>
>
>
> 2015-12-23
> ------------------------------
> hubo
> ------------------------------
>
> *发件人：*Maciej Fijalkowski <fijall at gmail.com>
> *发送时间：*2015-12-23 19:35
> *主题：*Re: [pypy-dev] Dead loop occurs when using python-daemon and
> multiprocessing together in PyPy 4.0.1
> *收件人：*"hubo"<hubo at jiedaibao.com>
> *抄送：*"pypy-dev"<pypy-dev at python.org>
>
> Hi hubo
>
> Can you put it as a bug report? Those things get easily lost on the
> mailing list (and sadly I won't look at it right now, multiprocessing
> scares me)
>
> On Wed, Dec 23, 2015 at 12:03 PM, hubo <hubo at jiedaibao.com> wrote:
>
>> Hello devs,
>>
>> A (possible) dead loop is found when I use python-daemon and
>> multiprocessing together in PyPy 4.0.1, which does not appear in Python(2.6
>> or 2.7). Also it does not appear in earlier PyPy versions (2.0.2)
>>
>> *Reproduce*:
>>
>> First install python-daemon:
>> pypy_pip install python-daemon
>>
>> Use the following test script (also available in attachment):
>>
>> #!/usr/bin/pypy
>> import daemon
>> import multiprocessing
>> def test():
>>     q = multiprocessing.Queue(64)
>> if __name__ == '__main__':
>>     with daemon.DaemonContext():
>>         test()
>>
>> When executing the script with pypy:
>> pypy test.py
>>
>> The background service does not exit, and is consuming 100% CPU:
>> ps aux | grep pypy
>> root      7769 99.1  0.5 235332 46812 ?        R    17:52   2:09 pypy
>> test.py
>> root      7775  0.0  0.0 103252   804 pts/1    S+   17:54   0:00 grep pypy
>>
>>
>>
>> Executing the script with python:
>> python2.7 test.py
>> And the background service normally exits.
>>
>> *Environment:*
>> I'm using CentOS 6.5, with portable PyPy distribution for linux (
>> https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-4.0.1-linux_x86_64-portable.tar.bz2
>> )
>> I run the script on system built-in python (python 2.6.6), a compiled
>> CPython (2.7.11), and pypy from epel-release(pypy 2.0.2, python 2.7.2), and
>> the problem does not appear. Though the compiled CPython is 2.7.11 and PyPy
>> 4.0.4 is python 2.7.10, I think that does not matter much.
>>
>> Please contact if you have any questions or ideas.
>>
>>
>> 2015-12-23
>> ------------------------------
>> hubo
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20151223/a6348f21/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Clip(12-23-17-55-26)(2).png
Type: image/png
Size: 2516 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20151223/a6348f21/attachment.png>