Mailman 3 January 2016 - pypy-dev

Re: [pypy-dev] Dead loop occurs when using python-daemon and multiprocessing together in PyPy 4.0.1
by Maciej Fijalkowski 05 Jan '16

05 Jan '16

well, ok, but that does not sound like a pypy bug then - "close all existing fds and complain that some of them are closed" is a bit not good - maybe it's a bug in python-daemon and the PEP? On Wed, Dec 23, 2015 at 3:36 PM, hubo <hubo(a)jiedaibao.com> wrote: > No, the python-daemon module is critical in this problem, because it is > the python-daemon module who closed the fd to /dev/urandom. When process > swith to daemon, it forks itself, and then close all open fds (including > stdin, stdout and stderr), so it also closes the fd for /dev/urandom which > is used by PyPy library. It is the standard behavior defined by > https://www.python.org/dev/peps/pep-3143/#daemoncontext-objects and also > the standard behavior for unix daemons. And unfortunately there is not a > way to prevent the fd to be closed without knowing exactly what number it > is on. > > Without python-daemon (or similar libraries), it is only possible to > reproduce the problem by closing the fd (usually 4) forcely, but it does > not make much sense. > > > 2015-12-23 > ------------------------------ > hubo > ------------------------------ > > *发件人：*Maciej Fijalkowski <fijall(a)gmail.com> > *发送时间：*2015-12-23 21:22 > *主题：*Re: Re: Re: [pypy-dev] Dead loop occurs when using python-daemon and > multiprocessing together in PyPy 4.0.1 > *收件人：*"hubo"<hubo(a)jiedaibao.com> > *抄送：*"pypy-dev"<pypy-dev(a)python.org> > > can you reproduce the OSError problem without having the daemon module > involved either? > > On Wed, Dec 23, 2015 at 3:14 PM, hubo <hubo(a)jiedaibao.com> wrote: > >> I can only reproduce the *OSError* problem. Maybe the CPU 100% is not >> really a dead lock, but rather some kind of automatic crash report? >> Although it is quite easy to crash the program with os.urandom, it >> only stops responding when the crash happens in system libraries like >> multiprocessing or email. >> >> The posix.urandom problem is quite easy to reproduce: >> >> #!/usr/bin/pypy >> import os >> os.urandom(16) >> def test(): >> print repr(os.urandom(16)) >> import daemon >> import sys >> if __name__ == '__main__': >> with daemon.DaemonContext(initgroups=False, >> stderr=sys.stderr,stdout=sys.stdout): >> test() >> >> (stderr and stdout is kept open to show console messages in the daemon. >> initgroups=False is a workaround on python-daemon not working in Python2.6) >> >> Or, with module random: >> >> #!/usr/bin/pypy >> import random >> def test(): >> random.Random() >> import daemon >> import sys >> if __name__ == '__main__': >> with daemon.DaemonContext(initgroups=False, >> stderr=sys.stderr,stdout=sys.stdout): >> test() >> And when run scripts with pypy: >> >> pypy test3.py >> >> it crashes with OSError: >> Traceback (most recent call last): >> File "test2.py", line 13, in <module> >> test() >> File "test2.py", line 6, in test >> random.Random() >> File "/opt/pypy-4.0.1-linux_x86_64-portable/lib-python/2.7/random.py", >> line 95, in __init__ >> self.seed(x) >> File "/opt/pypy-4.0.1-linux_x86_64-portable/lib-python/2.7/random.py", >> line 111, in seed >> a = long(_hexlify(_urandom(2500)), 16) >> OSError: [Errno 9] Bad file descriptor >> >> It is still not clear why it causes dead loop (or long-time no >> responding) in multiprocessing (should have thrown an ImportError) and the >> exact condition for the file descriptor of /dev/urandom appears (just call >> os.urandom and import random does not reproduce the result), but I believe >> it is definitely linked to the problem. >> >> 2015-12-23 >> ------------------------------ >> hubo >> ------------------------------ >> >> *发件人：*Maciej Fijalkowski <fijall(a)gmail.com> >> *发送时间：*2015-12-23 20:07 >> *主题：*Re: Re: [pypy-dev] Dead loop occurs when using python-daemon and >> multiprocessing together in PyPy 4.0.1 >> *收件人：*"hubo"<hubo(a)jiedaibao.com> >> *抄送：*"pypy-dev"<pypy-dev(a)python.org> >> >> That's very interesting, can you produce a standalone example that does >> not use multiprocessing? That would make it much easier to fix the bug >> (e.g. os.fork followed by os.urandom failing) >> >> On Wed, Dec 23, 2015 at 1:54 PM, hubo <hubo(a)jiedaibao.com> wrote: >> >>> Thanks for the response. Should I put it directly in the bug tracker? >>> >>> FYI, I've located the reason to be the incompatibility with >>> python-daemon (or rather the standard unix-daemon behavior) and PyPy >>> *posix.urandom* implementation. >>> >>> It seems that in PyPy 4.0.1, when module *random* loaded, a file >>> descriptor is created on /dev/urandom. I think PyPy implementation use the >>> shared descriptor to read from /dev/urandom. Sadly when python-daemon fork >>> the process and turns it into an unix daemon, it closes all the currently >>> open file descriptors. After that all os.urandom calls failed with OSError. >>> I think maybe the other functions of Random class is also using the file >>> descriptor in C code and just never detects if the return value is 0, and >>> causes the dead loop. >>> >>> I think the problem will be solved if the implementation re-open the >>> handle when it is closed somehow. >>> >>> multiprocessing is using random internally. Also there are lots of other >>> modules using random, like email etc. The dead loop occurs when you use any >>> of the libraries in a daemon. >>> >>> >>> >>> 2015-12-23 >>> ------------------------------ >>> hubo >>> ------------------------------ >>> >>> *发件人：*Maciej Fijalkowski <fijall(a)gmail.com> >>> *发送时间：*2015-12-23 19:35 >>> *主题：*Re: [pypy-dev] Dead loop occurs when using python-daemon and >>> multiprocessing together in PyPy 4.0.1 >>> *收件人：*"hubo"<hubo(a)jiedaibao.com> >>> *抄送：*"pypy-dev"<pypy-dev(a)python.org> >>> >>> Hi hubo >>> >>> Can you put it as a bug report? Those things get easily lost on the >>> mailing list (and sadly I won't look at it right now, multiprocessing >>> scares me) >>> >>> On Wed, Dec 23, 2015 at 12:03 PM, hubo <hubo(a)jiedaibao.com> wrote: >>> >>>> Hello devs, >>>> >>>> A (possible) dead loop is found when I use python-daemon and >>>> multiprocessing together in PyPy 4.0.1, which does not appear in Python(2.6 >>>> or 2.7). Also it does not appear in earlier PyPy versions (2.0.2) >>>> >>>> *Reproduce*: >>>> >>>> First install python-daemon: >>>> pypy_pip install python-daemon >>>> >>>> Use the following test script (also available in attachment): >>>> >>>> #!/usr/bin/pypy >>>> import daemon >>>> import multiprocessing >>>> def test(): >>>> q = multiprocessing.Queue(64) >>>> if __name__ == '__main__': >>>> with daemon.DaemonContext(): >>>> test() >>>> >>>> When executing the script with pypy: >>>> pypy test.py >>>> >>>> The background service does not exit, and is consuming 100% CPU: >>>> ps aux | grep pypy >>>> root 7769 99.1 0.5 235332 46812 ? R 17:52 2:09 pypy >>>> test.py >>>> root 7775 0.0 0.0 103252 804 pts/1 S+ 17:54 0:00 grep >>>> pypy >>>> >>>> >>>> >>>> Executing the script with python: >>>> python2.7 test.py >>>> And the background service normally exits. >>>> >>>> *Environment:* >>>> I'm using CentOS 6.5, with portable PyPy distribution for linux ( >>>> https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-4.0.1-linux_x86_… >>>> ) >>>> I run the script on system built-in python (python 2.6.6), a compiled >>>> CPython (2.7.11), and pypy from epel-release(pypy 2.0.2, python 2.7.2), and >>>> the problem does not appear. Though the compiled CPython is 2.7.11 and PyPy >>>> 4.0.4 is python 2.7.10, I think that does not matter much. >>>> >>>> Please contact if you have any questions or ideas. >>>> >>>> >>>> 2015-12-23 >>>> ------------------------------ >>>> hubo >>>> >>>> _______________________________________________ >>>> pypy-dev mailing list >>>> pypy-dev(a)python.org >>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>> >>>> >>> >> >

5 15

Py3.3 Import Test Question
by marky1991 . 05 Jan '16

05 Jan '16

(Sending an email instead of just asking in irc so I don't keep spamming the question) In the 3.3 branch, there is this failing test: http://buildbot.pypy.org/summary/longrepr?testname=AppTestMagic.%28%29.test… I made this change to make it not fail: https://bitbucket.org/marky1991/pypy/commits/afb2b3cd9535399043722f3d822762… The call chain is thus (mostly pseudocode here): (test code) reload(module) lib-python/3/imp.py, in reload loader.load_module(module.name) lib-python/3/importlib/_bootstrap.py, in load_module init_builtin(module.name) pypy/module/imp/interp_imp.py, in init_builtin space.getbuiltinmodule(module.name) pypy/interpreter/baseobjspace.py sys.modules.get(module.name) The last method in that chain, space.getbuiltinmodule, has a parameter force_init, which defaults to False. If you pass it as true, we avoid the (rpython equivalent to) sys.modules.get(module_name) code, getting the module object and successfully calling init() on it. (The failing test needs reload() to invoke module.init()) In the commit pasted above, I changed init_builtin to call space.getbuiltinmodule with force_init=True. This fixes reload, but I would think that this would be wrong for nonreload scenarios. (I've checked the py3k branch, which has this test (which passes locally), and it passes force_init=True if the finder's modtype is C_BUILTIN) However, I don't see a good way to conditionalize the passing of force_init. Does anyone have any suggestions as to how to handle this issue? (Given what I've found in py3k's code, is my change actually a valid fix?) If anyone has time to reply, feel free to just ping me in irc if I'm on. Thanks

2 2

Leysin Winter Sprint 2016
by Armin Rigo 04 Jan '16

04 Jan '16

Hi all, ===================================================================== PyPy Leysin Winter Sprint (20-27th February 2016) ===================================================================== The next PyPy sprint will be in Leysin, Switzerland, for the eleventh time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. ------------------------------ Goals and topics of the sprint ------------------------------ The details depend on who is here and ready to work. The list of topics is mostly the same as last year (did PyPy became a mature project with only long-term goals?): * cpyext (CPython C API emulation layer): various speed and completeness topics * cleaning up the optimization step in the JIT, change the register allocation done by the JIT's backend, or more improvements to the warm-up time * finish vmprof - a statistical profiler for CPython and PyPy * Py3k (Python 3.x support), NumPyPy (the numpy module) * STM (Software Transaction Memory), notably: try to come up with benchmarks, and measure them carefully in order to test and improve the conflict reporting tools, and more generally to figure out how practical it is in large projects to avoid conflicts * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. ----------- Exact times ----------- I have booked the week from Saturday 20 to Saturday 27. It is fine to leave either the 27 or the 28, or even stay a few more days on either side. The plan is to work full days between the 21 and the 27. You are of course allowed to show up for a part of that time only, too. ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connection with wireless installed. You can also arrange your own lodging elsewhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue). Please *confirm* that you are coming so that we can adjust the reservations as appropriate. The options of rooms are a bit more limited than on previous years because the place for bed-and-breakfast is shrinking: what is guaranteed is only one double-bed room and a bigger room with 5-6 individual beds (the latter at 50-60 CHF per night, breakfast included). If there are more people that would prefer a single room, please contact me and we'll see what choices you have. There are a choice of hotels, many of them reasonably priced for Switzerland. Please register by Mercurial:: https://bitbucket.org/pypy/extradoc/ https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2… or on this mailing list if you do not yet have check-in rights. You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around, and at least one EU-format power strip. -------Armin Rigo

2 1

numpypy unit tests
by Sergey Matyunin 03 Jan '16

03 Jan '16

Hello, I am curious about the status of unit tests of numpy for pypy. Where can I get actual information about it? I have installed pypy 4.0.1, numpypy for it (tag 4.0.1) and launched numpy tests using numpy.test() Looks like there is a plenty of failed tests: > Ran 4157 tests in 15.592s > FAILED (KNOWNFAIL=5, SKIP=116, errors=648, failures=149) Is it the actual state or may be I did something wrong? -- Sergey

5 8

Leysin Winter sprint?
by Armin Rigo 01 Jan '16

01 Jan '16

Hi all, Due to public pressure :-) I'm trying to organize this winter's sprint, in Leysin, Switzerland. This is a fully public sprint of 7 days, with possible skiing. If you'd like to come, please reply to this e-mail with dates that are ok for you. I will give some priority to core people, but anyone else is welcome to have preferences too. Traditionally, it is around mid-January, but this year as far as I understand there is some push to have it in February or March instead---which is fine too. The following week should be avoided if possible, as it is holidays for the canton's schools: 20-28 Feb 2016. A bientôt, Armin.

7 11