[pypy-dev] Dead loop occurs when using python-daemon and multiprocessing together in PyPy 4.0.1

Carl Friedrich Bolz cfbolz at gmx.de
Wed Dec 23 08:57:35 EST 2015


Hi Maciek, 

Of course it's a difference between pypy and cpython. if you close all fds, cpython's random module still works, but PyPy's doesn't. 

Cheers, 
Carl Friedrich

On December 23, 2015 2:54:23 PM GMT+01:00, Maciej Fijalkowski <fijall at gmail.com> wrote:
>well, ok, but that does not sound like a pypy bug then - "close all
>existing fds and complain that some of them are closed" is a bit not
>good -
>maybe it's a bug in python-daemon and the PEP?
>
>On Wed, Dec 23, 2015 at 3:36 PM, hubo <hubo at jiedaibao.com> wrote:
>
>> No, the python-daemon module is critical in this problem, because it
>is
>> the python-daemon module who closed the fd to /dev/urandom. When
>process
>> swith to daemon, it forks itself, and then close all open fds
>(including
>> stdin, stdout and stderr), so it also closes the fd for /dev/urandom
>which
>> is used by PyPy library. It is the standard behavior defined by
>> https://www.python.org/dev/peps/pep-3143/#daemoncontext-objects and
>also
>> the standard behavior for unix daemons. And unfortunately there is
>not a
>> way to prevent the fd to be closed without knowing exactly what
>number it
>> is on.
>>
>> Without python-daemon (or similar libraries), it is only possible to
>> reproduce the problem by closing the fd (usually 4) forcely, but it
>does
>> not make much sense.
>>
>>
>> 2015-12-23
>> ------------------------------
>> hubo
>> ------------------------------
>>
>> *发件人:*Maciej Fijalkowski <fijall at gmail.com>
>> *发送时间:*2015-12-23 21:22
>> *主题:*Re: Re: Re: [pypy-dev] Dead loop occurs when using python-daemon
>and
>> multiprocessing together in PyPy 4.0.1
>> *收件人:*"hubo"<hubo at jiedaibao.com>
>> *抄送:*"pypy-dev"<pypy-dev at python.org>
>>
>> can you reproduce the OSError problem without having the daemon
>module
>> involved either?
>>
>> On Wed, Dec 23, 2015 at 3:14 PM, hubo <hubo at jiedaibao.com> wrote:
>>
>>> I can only reproduce the *OSError* problem. Maybe the CPU 100% is
>not
>>> really a dead lock, but rather some kind of automatic crash report?
>>> Although it is quite easy to crash the program with os.urandom, it
>>> only stops responding when the crash happens in system libraries
>like
>>> multiprocessing or email.
>>>
>>> The posix.urandom problem is quite easy to reproduce:
>>>
>>> #!/usr/bin/pypy
>>> import os
>>> os.urandom(16)
>>> def test():
>>>     print repr(os.urandom(16))
>>> import daemon
>>> import sys
>>> if __name__ == '__main__':
>>>     with daemon.DaemonContext(initgroups=False,
>>> stderr=sys.stderr,stdout=sys.stdout):
>>>         test()
>>>
>>> (stderr and stdout is kept open to show console messages in the
>daemon.
>>> initgroups=False is a workaround on python-daemon not working in
>Python2.6)
>>>
>>> Or, with module random:
>>>
>>> #!/usr/bin/pypy
>>> import random
>>> def test():
>>>     random.Random()
>>> import daemon
>>> import sys
>>> if __name__ == '__main__':
>>>     with daemon.DaemonContext(initgroups=False,
>>> stderr=sys.stderr,stdout=sys.stdout):
>>>         test()
>>> And when run scripts with pypy:
>>>
>>> pypy test3.py
>>>
>>> it crashes with OSError:
>>> Traceback (most recent call last):
>>>   File "test2.py", line 13, in <module>
>>>     test()
>>>   File "test2.py", line 6, in test
>>>     random.Random()
>>>   File
>"/opt/pypy-4.0.1-linux_x86_64-portable/lib-python/2.7/random.py",
>>> line 95, in __init__
>>>     self.seed(x)
>>>   File
>"/opt/pypy-4.0.1-linux_x86_64-portable/lib-python/2.7/random.py",
>>> line 111, in seed
>>>     a = long(_hexlify(_urandom(2500)), 16)
>>> OSError: [Errno 9] Bad file descriptor
>>>
>>> It is still not clear why it causes dead loop (or long-time no
>>> responding) in multiprocessing (should have thrown an ImportError)
>and the
>>> exact condition for the file descriptor of /dev/urandom appears
>(just call
>>> os.urandom and import random does not reproduce the result), but I
>believe
>>> it is definitely linked to the problem.
>>>
>>> 2015-12-23
>>> ------------------------------
>>> hubo
>>> ------------------------------
>>>
>>> *发件人:*Maciej Fijalkowski <fijall at gmail.com>
>>> *发送时间:*2015-12-23 20:07
>>> *主题:*Re: Re: [pypy-dev] Dead loop occurs when using python-daemon
>and
>>> multiprocessing together in PyPy 4.0.1
>>> *收件人:*"hubo"<hubo at jiedaibao.com>
>>> *抄送:*"pypy-dev"<pypy-dev at python.org>
>>>
>>> That's very interesting, can you produce a standalone example that
>does
>>> not use multiprocessing? That would make it much easier to fix the
>bug
>>> (e.g. os.fork followed by os.urandom failing)
>>>
>>> On Wed, Dec 23, 2015 at 1:54 PM, hubo <hubo at jiedaibao.com> wrote:
>>>
>>>> Thanks for the response. Should I put it directly in the bug
>tracker?
>>>>
>>>> FYI, I've located the reason to be the incompatibility with
>>>> python-daemon (or rather the standard unix-daemon behavior) and
>PyPy
>>>> *posix.urandom* implementation.
>>>>
>>>> It seems that in PyPy 4.0.1, when module *random* loaded, a file
>>>> descriptor is created on /dev/urandom. I think PyPy implementation
>use the
>>>> shared descriptor to read from /dev/urandom. Sadly when
>python-daemon fork
>>>> the process and turns it into an unix daemon, it closes all the
>currently
>>>> open file descriptors. After that all os.urandom calls failed with
>OSError.
>>>> I think maybe the other functions of Random class is also using the
>file
>>>> descriptor in C code and just never detects if the return value is
>0, and
>>>> causes the dead loop.
>>>>
>>>> I think the problem will be solved if the implementation re-open
>the
>>>> handle when it is closed somehow.
>>>>
>>>> multiprocessing is using random internally. Also there are lots of
>other
>>>> modules using random, like email etc. The dead loop occurs when you
>use any
>>>> of the libraries in a daemon.
>>>>
>>>>
>>>>
>>>> 2015-12-23
>>>> ------------------------------
>>>> hubo
>>>> ------------------------------
>>>>
>>>> *发件人:*Maciej Fijalkowski <fijall at gmail.com>
>>>> *发送时间:*2015-12-23 19:35
>>>> *主题:*Re: [pypy-dev] Dead loop occurs when using python-daemon and
>>>> multiprocessing together in PyPy 4.0.1
>>>> *收件人:*"hubo"<hubo at jiedaibao.com>
>>>> *抄送:*"pypy-dev"<pypy-dev at python.org>
>>>>
>>>> Hi hubo
>>>>
>>>> Can you put it as a bug report? Those things get easily lost on the
>>>> mailing list (and sadly I won't look at it right now,
>multiprocessing
>>>> scares me)
>>>>
>>>> On Wed, Dec 23, 2015 at 12:03 PM, hubo <hubo at jiedaibao.com> wrote:
>>>>
>>>>> Hello devs,
>>>>>
>>>>> A (possible) dead loop is found when I use python-daemon and
>>>>> multiprocessing together in PyPy 4.0.1, which does not appear in
>Python(2.6
>>>>> or 2.7). Also it does not appear in earlier PyPy versions (2.0.2)
>>>>>
>>>>> *Reproduce*:
>>>>>
>>>>> First install python-daemon:
>>>>> pypy_pip install python-daemon
>>>>>
>>>>> Use the following test script (also available in attachment):
>>>>>
>>>>> #!/usr/bin/pypy
>>>>> import daemon
>>>>> import multiprocessing
>>>>> def test():
>>>>>     q = multiprocessing.Queue(64)
>>>>> if __name__ == '__main__':
>>>>>     with daemon.DaemonContext():
>>>>>         test()
>>>>>
>>>>> When executing the script with pypy:
>>>>> pypy test.py
>>>>>
>>>>> The background service does not exit, and is consuming 100% CPU:
>>>>> ps aux | grep pypy
>>>>> root      7769 99.1  0.5 235332 46812 ?        R    17:52   2:09
>pypy
>>>>> test.py
>>>>> root      7775  0.0  0.0 103252   804 pts/1    S+   17:54   0:00
>grep
>>>>> pypy
>>>>>
>>>>>
>>>>>
>>>>> Executing the script with python:
>>>>> python2.7 test.py
>>>>> And the background service normally exits.
>>>>>
>>>>> *Environment:*
>>>>> I'm using CentOS 6.5, with portable PyPy distribution for linux (
>>>>>
>https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-4.0.1-linux_x86_64-portable.tar.bz2
>>>>> )
>>>>> I run the script on system built-in python (python 2.6.6), a
>compiled
>>>>> CPython (2.7.11), and pypy from epel-release(pypy 2.0.2, python
>2.7.2), and
>>>>> the problem does not appear. Though the compiled CPython is 2.7.11
>and PyPy
>>>>> 4.0.4 is python 2.7.10, I think that does not matter much.
>>>>>
>>>>> Please contact if you have any questions or ideas.
>>>>>
>>>>>
>>>>> 2015-12-23
>>>>> ------------------------------
>>>>> hubo
>>>>>
>>>>> _______________________________________________
>>>>> pypy-dev mailing list
>>>>> pypy-dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>>>
>>>>>
>>>>
>>>
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20151223/809e9586/attachment-0001.html>


More information about the pypy-dev mailing list