Mailman 3 January 2015 - pypy-dev

cppyy questions
by Omer Katz 22 Jan '15

22 Jan '15

I'm trying to use protobuf with PyPy and I've been quite successful doing so with cppyy. I generated the protobuf in C++ and used reflex to generate the bindings. I've encountered some problems that I don't know how to deal with and the documentation doesn't describe what you can do to resolve them. I discovered that If you don't specify --deep when generating the reflex bindings and you try to pass a string to the C++ side you get a segfault. I'm guessing that's a bug. I can't catch exceptions that are being raised from C++ (it's also undocumented). I ensured that protobuf's FatalException has reflex bindings but the process crashes on an exception. e.SerializeAsString() [libprotobuf FATAL google/protobuf/message_lite.cc:273] CHECK failed: IsInitialized(): Can't serialize message of type "MyProtobufType" because it is missing required fields: header terminate called after throwing an instance of 'google::protobuf::FatalException' what(): CHECK failed: IsInitialized(): Can't serialize message of type "MyProtobufType" because it is missing required fields: header Aborted (core dumped) How can I catch that exception? The documentation is unclear how you can pass a pointer to a Python variable e.g.: str = "" e.SerializeToString(str) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-7-993880892d74> in <module>() ----> 1 e.SerializeToString(str) TypeError: none of the 5 overloaded methods succeeded. Full details: bool google::protobuf::MessageLite::SerializeToString(std::string*) => TypeError: cannot pass str as basic_string<char> bool google::protobuf::MessageLite::SerializeToString(std::string*) => TypeError: cannot pass str as basic_string<char> bool google::protobuf::MessageLite::SerializeToString(std::string*) => TypeError: cannot pass str as basic_string<char> bool google::protobuf::MessageLite::SerializeToString(std::string*) => TypeError: cannot pass str as basic_string<char> bool google::protobuf::MessageLite::SerializeToString(std::string*) => TypeError: cannot pass str as basic_string<char> Best Regards, Omer Katz.

3 14

Numpy Topics
by Steven Jackson 21 Jan '15

21 Jan '15

Hey I'd like to know if the proposed numpy projects list at https://bitbucket.org/pypy/extradoc/src/extradoc/planning/micronumpy.txt is still up to date, and if so what is meant by "a good sort function." If it's just a matter of implementing a known good algorithm, that seems like a good way to start contributing to pypy. The advice on http://doc.pypy.org/en/latest/project-ideas.html suggested posting this question to #pypy on IRC, which I attempted to do through http://webchat.freenode.net/ but I never got a response. It was my first time trying to communicate over IRC, so I'm not sure if I did something incorrectly while trying to join the channel (I saw buildbot messages but no one else speaking) or if the lack of activity was simply due to time difference (I'm on USA east coast time, while I'm aware that much of the pypy-dev community is located in Europe). Any help with either the original question or joining the IRC discussion would be greatly appreciated :) -- Steven Jackson

4 5

OrderedDict.move_to_end()
by Armin Rigo 21 Jan '15

21 Jan '15

Hi all, About the new OrderedDict and how to support `move_to_end(last=False)` in the py3k branch: an implementation of the correct complexity is possible. It would piggy-back on the part of `lookup_function_no` that acts as counter for how many entries at the start are known to be deleted. This number is present to allow for a good implementation of `popitem(last=False)`, so that it doesn't have to scan a larger and larger area of deleted items. So you can use the same number in reverse. As long as this number is greater than zero, you can insert the new item at position "this number minus one". When it is zero, you resize and reindex the dictionary by adding an extra argument to the relevant functions which would force it to artificially reserve n free entries at the start. If n is proportional to "num_live_items", maybe 1/8 or 1/16 of it, it should be enough to give amortized constant time to the operation. A bientôt, Armin.

1 0

Sudden failures during compile-c
by Timothy Baldridge 21 Jan '15

21 Jan '15

Recently my builds on linux with --opt=jit have started failing with the following error: [translation:info] Error: [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/goal/translate.py", line 316, in main [translation:info] drv.proceed(goals) [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/driver.py", line 539, in proceed [translation:info] return self._execute(goals, task_skip = self._maybe_skip()) [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/tool/taskengine.py", line 114, in _execute [translation:info] res = self._do(goal, taskcallable, *args, **kwds) [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/driver.py", line 276, in _do [translation:info] res = func() [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/driver.py", line 505, in task_compile_c [translation:info] cbuilder.compile(**kwds) [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/c/genc.py", line 375, in compile [translation:info] extra_opts) [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/platform/posix.py", line 198, in execute_makefile [translation:info] self._handle_error(returncode, stdout, stderr, path.join('make')) [translation:info] File "/home/travis/build/pixie-lang/externals/pypy/rpython/translator/platform/__init__.py", line 151, in _handle_error [translation:info] raise CompilationError(stdout, stderr) [translation:ERROR] CompilationError: CompilationError(err=""" [translation:ERROR] pixie_vm_threads.c: In function ‘pypy_g_do_yield_thread’: [translation:ERROR] pixie_vm_threads.c:475:21: error: ‘pypy_g_do_yield_thread_reload’ undeclared (first use in this function) [translation:ERROR] pixie_vm_threads.c:475:21: note: each undeclared identifier is reported only once for each function it appears in [translation:ERROR] make[1]: *** [pixie_vm_threads.gcmap] Error 1 [translation:ERROR] """) [translation] start debugger... > /home/travis/build/pixie-lang/externals/pypy/rpython/translator/platform/__init__.py(151)_handle_error() -> raise CompilationError(stdout, stderr) (Pdb+) I find this odd as it works just fine without the JIT, and compiles fine on OS X. The code in question is basically a copy-and-paste from PyPy's code: https://github.com/pixie-lang/pixie/blob/master/pixie/vm/threads.py#L90 Any ideas why this would suddenly have started failing recently? I normally build against a pretty recent version of PyPy master, so did something change in the pypy source? Thanks again for any help, Timothy

2 4

pypy2.5 with stdlib-2.7.9?
by Matti Picus 20 Jan '15

20 Jan '15

I would like to start a release cycle of pypy 2.5.0 which seems to be quite a jump from 2.4 in terms of performance. The major blocker for me is stdlib-2.7.9, especially the improved ssl support. Could we get a show of hands for: - yes I will make an effort to help finish stdlib-2.7.9 - nah, just give up and release with stdlib-2.7.8 It would be nice to get the version out before the sprint Matti If there are other blocker issues and/or branches, now is the time to mention them

3 3

Send desktop notifications to everyone that downloads your software.
by Be like AVG, Zone Alarm and Microsoft. 19 Jan '15

19 Jan '15

Be like AVG, Zone Alarm and Microsoft and send Desktop Notifications to everyone that downloads your software. AVG, Zone Alarm and Microsoft and many other major software sellers now send full colour desktop notifications to everyone that downloads their software. These notifications appear directly onto their desktops informing them about upgrades, new products and server notifications and more. These companies can communicate with everyone that downloads their software. You have probably seen them and wondered how they did that. As a result they have found that their sales have increased and they have a high customer satisfaction rate. You can now do the same! Using our Dymantex desktop messaging system that is even better than the ones used by the major companies you can send full colour interactive desktop notifications so that you too can offer upgrades, tell them about special offers and give technical support to everyone that downloads your software! We offer you a Free, no obligation trial of our Dymantex Desktop Notification system. Please Press Here for more information and we send it to you. Dymantex The Desktop Notification System This is a B2B comminication. If received in error please accept our apologise

1 0

Next sprint in Leysin, Switzerland (20-28 Feb 2015)
by Armin Rigo 15 Jan '15

15 Jan '15

Hi all, Here is the next sprint announcement! ============================================================ PyPy Leysin Winter Sprint (20-28th February 2015) ============================================================ The next PyPy sprint will be in Leysin, Switzerland, for the tenth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. ------------------------------ Goals and topics of the sprint ------------------------------ The details depend on who is here and ready to work. We might touch topics such as: * cleaning up the optimization step in the JIT, change the register allocation done by the JIT's backend, or improvements to the warm-up time * STM (Software Transaction Memory), notably: try to come up with benchmarks, and measure them carefully in order to test and improve the conflict reporting tools, and more generally to figure out how practical it is in large projects to avoid conflicts * vmprof - a statistical profiler for CPython and PyPy work, including making it more user friendly. * Py3k (Python 3.x support), NumPyPy (the numpy module) * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. ----------- Exact times ----------- For a change, and as an attempt to simplify things, I specified the dates as 20-28 Februrary 2015, where 20 and 28 are travel days. We will work full days between the 21 and the 27. You are of course allowed to show up for a part of that time only, too. ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) Please *confirm* that you are coming so that we can adjust the reservations as appropriate. In the past, the rates were around 60 CHF a night all included in 2-person rooms, with breakfast. Now, the rooms available are either single-person (or couple), or rooms for 3 persons. The latter choice is recommended and should be under 60 CHF per person. Please register by Mercurial:: https://bitbucket.org/pypy/extradoc/ https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2… or on the pypy-dev mailing list if you do not yet have check-in rights: http://mail.python.org/mailman/listinfo/pypy-dev You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around, and at least one EU-format power strip. -------------------------------- Armin Rigo

1 0

can_enter_jit - what is it and did something change?
by Yuriy Taraday 12 Jan '15

12 Jan '15

Hello. I'm poking around lang-js [0] looking if I can make it better (just for fun). First issue I've stumbled upon is mysterious "Bad can_enter_jit() placement" error. It looks like when we do a jump in user code [1], we should do "can_enter_jit" with new instruction pointer, not the old one. I've changed it (s/=pc/=new_pc/) and everything seems to work fine. It looks like in PyPy it works the same way [2] My questions are: - Did something change wrt can_enter_jit since that code had been written? I mean besides check that raises that error [3]. - Is it correct to not have any _user_ code run between can_enter_jit and jit_merge_point calls but have some _interpreter_ code there? Error message says that there should be _any_ code between them which seems hardly possible. - Why do we need to place this hint in the beginning of the next iteration of the loop but not at the end of current iteration? It would seem logical to say "you might want to compile from here (start of the loop, one of jit_merge_points) and here (end of the loop, can_enter_jit hing) if you like". - It looks like another piece of old RPython code - tutorial (BF implementation [4]) works fine and gets JIT speedup even without can_enter_jit calls. How does this work? - Would it be more beneficial to analyse user code and provide can_enter_jit hint only for real loops instead of all backward jumps? I tried to google around can_enter_jit but it leads mostly to PyPy commit history so I hope to find answers here. [0] https://bitbucket.org/pypy/lang-js [1] https://bitbucket.org/pypy/lang-js/src/e2275b2/js/jscode.py#cl-240 [2] https://bitbucket.org/pypy/pypy/src/d0f031c/pypy/module/pypyjit/interp_jit.… [3] https://bitbucket.org/pypy/pypy/commits/b538c2f [4] https://bitbucket.org/brownan/pypy-tutorial -- Kind regards, Yuriy.

2 4

Re: [pypy-dev] Ordered dict in PyPy
by Laura Creighton 12 Jan '15

12 Jan '15

Can we talk the CPython developers into raising RunTimeError for concurrent modifications? Laura

5 7

Ordered dict in PyPy
by Armin Rigo 12 Jan '15

12 Jan '15

Hi all, The all-ordered-dicts branch is not quite ready for merging, but getting there. In one word, it makes all dicts ordered by default, by a subtle change of the internal model which also makes them *more* compact. An annoying detail is the OrderedDict subclass. We can simplify it a lot in PyPy, but not completely remove it. It adds a few methods like popitem(last=False) or __reversed__() which are not available on the base class, and it has a different notion of equality (two OrderedDicts are equal only if they store items in the same order). Moreover, it's still useful to have a class OrderedDict in PyPy, if only to say "I really want this dict to be ordered and I want a CPython-compatible way to express that". One annoyance is what to do with iteration over OrderedDicts. The CPython logic doesn't raise RuntimeError for concurrent modifications. In fact it doesn't care at all: it gives more or less what we could reasonably expect in simple cases (like not deleting items, only adding new ones). Of course in less simple cases it gives nonsense or crashes obscurely, like after you deleted the item most recently returned by an iterator. What should we do? We can (1) raise RuntimeError as soon as any change is detected, like dict, and propagate this to offending applications; (2) write some slightly different approximation of "what you'd expect"; or (3) come up with a sane and exact definition of what to expect and implement that; or (4) find some hack that happens to give the same result as CPython in the cases where CPython's result makes sense, but not necessarily in all cases. Right now the branch implements 2, but I think either 1 or 3 (but not 4) would be a better idea. Thoughts? A bientôt, Armin.

1 0