Mailman 3 June 2006 - pypy-dev

0.9 release meeting 11AM (GMT+2)
by hpk＠trillke.net June 17, 2006

June 17, 2006

Hi folks, today at 11AM (GMT+2) we'll meet on #pypy-release for a meeting regarding the upcoming 0.9 release Topics will be: * extension compiler * stackless features * (high level backends / snapshot of work) * logic/constraint works * feature freeze (branching off?) * documentation about all this and whatever we decide to be relevant for it. best, holger

3 2

stackless (the module) exception handling
by Stephan Diehl June 15, 2006

June 15, 2006

this is all very confusing, but I'll do my best. The fun stuff is happening in interp_coroutine.py which looks at the moment like this: -------------------------------------------------------------- def _execute(self, incoming_frame): syncstate.switched(incoming_frame) state = self.costate try: try: self.thunk.call() finally: self.finished() self.thunk = None resume_point("coroutine__bind", self, state) except CoroutineExit: # ignore a shutdown exception pass except Exception, e: # redirect all unhandled exceptions to the parent syncstate.things_to_do = True syncstate.temp_exc = e while self.parent is not None and self.parent.frame is None: # greenlet behavior is fine self.parent = self.parent.parent return state.update(self.parent) ------------------------------------------- I've inserted the 'self.finished()' hook yesterday, to give applevel coroutines (in this case tasklets) the chance to do their last rites. Specifically, I needed that so that the stackless scheduler has a chance to stay in sync. Now, the tasklet.finished method contains a 'schedule'. What happens is that the Exception handling in _execute above is ignored. Bad. As a next test, I've moved the 'self.finished()' call right before the return. This certainly gives the exception handling a chance to do it's thing and the stackless code still does it things. But now I get the following error message from py.py: Traceback (most recent call last): File "/usr/local/bin/py.py", line 207, in ? sys.exit(main_(sys.argv)) File "/usr/local/bin/py.py", line 118, in main_ if not main.run_toplevel(space, doit, verbose=Options.verbose): File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 80, in run_toplevel f() File "/usr/local/bin/py.py", line 105, in doit main.run_file(args[0], space=space) File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 67, in run_file run_string(istring, filename, space) File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 58, in run_string _run_eval_string(source, filename, space, False) File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 47, in _run_eval_string retval = pycode.exec_code(space, w_globals, w_globals) File "/home/stephan/projects/pypy-dist/pypy/interpreter/eval.py", line 26, in exec_code return frame.run() File "/home/stephan/projects/pypy-dist/pypy/interpreter/eval.py", line 162, in resume executioncontext.leave(self) File "/home/stephan/projects/pypy-dist/pypy/interpreter/executioncontext.py", line 38, in leave self.framestack.pop() File "/home/stephan/projects/pypy-dist/pypy/interpreter/miscutils.py", line 34, in pop return self.items.pop() IndexError: pop from empty list ---------------------------------------------------- So, somehow I either have nice stackless scheduler clean up code, or I have proper exception handling. If somebody has a nice idea what to do, I'd be very happy to hear about that. Cheers Stephan

1 0

pypy-c vs cpython tests
by Michael Hudson June 13, 2006

June 13, 2006

I've spent an aggravating little while running CPython's regrtest with a recent pypy-c (default options, so boehm and no fancy stuff). After a bit of trial and error, this was the result: 69 tests failed: GC Warning: Finalization cycle involving d66c120 GC Warning: Finalization cycle involving a8e06c0 GC Warning: Finalization cycle involving d286d08 GC Warning: Finalization cycle involving d180780 GC Warning: Finalization cycle involving d12a108 test___all__ test_anydbm test_calendar test_cfgparser test_class test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_coercion test_commands test_compare test_compiler test_cookielib test_cpickle test_decimal test_descr test_descrtut test_dis test_distutils test_doctest2 test_dumbdbm test_extcall test_file test_filecmp test_fileinput test_frozen test_gc test_getargs test_gettext test_glob test_inspect test_iter test_mailbox test_marshal test_mhlib test_mimetools test_minidom test_multibytecodec test_new test_os test_peepholer test_pep292 test_pickle test_pickletools test_posixpath test_profile test_pyclbr test_random test_repr test_scope test_sort test_strftime test_stringprep test_strptime test_structseq test_tarfile test_tempfile test_time test_traceback test_transformer test_ucn test_unicodedata test_weakref test_whichdb test_xmlrpc test_xpickle test_zipfile 97 tests skipped: test__locale test_aepack test_al test_applesingle test_asynchat test_audioop test_bsddb test_bsddb185 test_bsddb3 test_bz2 test_capi test_cd test_cgi test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_crypt test_csv test_curses test_dbm test_dl test_doctest test_email test_email_codecs test_fcntl test_fork1 test_gdbm test_getargs2 test_gl test_grp test_gzip test_hotshot test_httplib test_imageop test_imaplib test_imgfile test_imp test_ioctl test_largefile test_linuxaudiodev test_locale test_logging test_macfs test_macostools test_mimetypes test_mmap test_nis test_normalization test_openpty test_ossaudiodev test_pep277 test_plistlib test_poll test_popen test_popen2 test_pty test_pwd test_pyexpat test_queue test_regex test_resource test_rgbimg test_robotparser test_sax test_scriptpackages test_select test_signal test_site test_socket test_socket_ssl test_socketserver test_strop test_subprocess test_sunaudiodev test_sundry test_symtable test_tcl test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_timeout test_timing test_unicode_file test_urllib test_urllib2 test_urllib2net test_urllibnet test_winreg test_winsound test_zipimport test_zlib Segmentation fault I had to run with '-x test_datetime test_format test_shutil test_shelve test_str test_string test_unicode test_userstring' in the end. test_datetime, test_shutil and test_format kill the build. the string tests fail in ways that suggest that the boehm gc really didn't expect the usage patterns they use. I don't remember why I skipped test_shelve now :) I count around 250 test_* files in CPython's test suite, so we actually pass around 40% of them, and a lot of the skips and plenty of the fails are missing (parts of) extension modules, so we're not really in *that* bad a shape. But fixing the crashers would be nice. We need to get some automated version of this set up. Cheers, mwh -- ARTHUR: Why should he want to know where his towel is? FORD: Everybody should know where his towel is. ARTHUR: I think your head's come undone. -- The Hitch-Hikers Guide to the Galaxy, Episode 7

1 1

christian is ill
by Stephan Diehl June 12, 2006

June 12, 2006

Hi all, I've just talked with Christian. Unfortunatelly, he's ill and at a hospital. He won't be able to work for a couple of days. I guess that it's a good bet that he won't be available before friday. Cheers Stephan

1 0

Re: [pypy-dev] [pypy-svn] r28615 - in pypy/dist/pypy/module/stackless: . test
by Christian Tismer June 10, 2006

June 10, 2006

arigo(a)codespeak.net wrote: > Author: arigo > Date: Sat Jun 10 15:07:59 2006 > New Revision: 28615 > > Modified: > pypy/dist/pypy/module/stackless/interp_coroutine.py > pypy/dist/pypy/module/stackless/test/test_interp_coroutine.py > Log: > Long test and one-liner fix for coroutines and app-coroutines, Ah, well. This is a typical one. It is always hard to tell "who am I" in the coroutine stuff. By default, it is someone else :-) -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

Core sprint before EuroPython
by Armin Rigo June 9, 2006

June 9, 2006

Hi all, ------------------------- Leysin Summer 2006 sprint ------------------------- As mentionned previously, there will be another sprint in Switzerland just before EuroPython. This one is meant for more "core" people in the broad sense, i.e. anyone that has already some knowledge about PyPy. For newcomers we recommend the post-EuroPython sprint at the conference venue :-) The main sprint focus will be on the JIT, but we can also work on other areas, as usual. This pre-EP sprint will take place from the 28th of June (included), to the 2nd of July (probably excluded, to have a break and travel to Geneva). The location is Leysin, about 1h45 away from Geneva by train. The sprint place is the same as during the previous Leysin sprints: Chalet Ermina, http://www.ermina.ch/ . I am arranging a group booking as usual. List of participants (add yourself there): http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-summer-2006/people… A bientot, Armin

1 0

The sound of a sprint :-)
by Christian Tismer June 8, 2006

June 8, 2006

http://tismerysoft.de/~tismer/movies/sprint-ddorf/CIMG0648.AVI recorded in the backyard of building 25.12 of HHU Duesseldorf. You go through the cafeteria and pass two hallways, then there is an open air space with all this sound Python food.

2 2

PyPy post-EP sprint (6-9th july)
by holger krekel June 7, 2006

June 7, 2006

Geneva/CERN Post-EuroPython PyPy sprint 6-9th July ==================================================== The next PyPy sprint will be held at CERN/Geneva right after the EuroPython conference. We specifically invite newcomers and people interested to get to know and get into PyPy development. We are going to work on a variety of topics and will give introductions to various aspects of PyPy (on the 6th morning). Some foreseen focus will be on the following topics: * The extension module compiler. Using it for implementing extension modules both for PyPy and CPython from the same source code. (for example if you have a Need for Speed :) * work and experiment with our to-be-relased 0.9 stackless features * work on high level backends: .NET, Javascript etc. * optimization of core Python data types, making full use of PyPy's flexible architecture and python-implemented (and then translated) type system. * You may even dare to dive into ongoing work on the JIT compiler * experimenting with novel security systems for Python, enabled by PyPy * and so on :) Registration ---------------------- If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). Special note to Students! ---------------------------- As you might know, there are three students in Google's Summer of Code program who are working on projects related to PyPy. In addition we are planning our own "Summer of PyPy": we can cover the expenses of attending some of our sprints and provide mentorship. In this vein, if you have an interesting idea for a project relating to PyPy you should send us ("pypy-tb(a)codespeak.net") a proposal. We will consider it get back to you on whether we can fund your travel and accomodation. We expect to able to fund between 4 and 6 students in this way. As we still need to figure out budget/money issues, for now we can only promise for now that we can fund this Post-EP2006 sprint for such students. But don't hesitate to come to our irc channel, ask around and produce proposals! .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: people.html

1 0

Duesseldorf Sprint Report Days 1-4
by Michael Hudson June 7, 2006

June 7, 2006

Hi pypy-dev! So we're sitting in the geography^Wcomputer science department of Heinrich-Heine-Universitaet Duesseldorf, the alleged employers of Armin and Michael, and (also alleged) educator of Carl. We've been doing the usual sprinty things of drinking beer, getting up late, going to bed even later (and so still not really getting much sleep) and hacking. An early task was finishing off the mostly complete weakref support. After drawing an extremely confusing diagram: http://codespeak.net/~mwh/blackboard.jpg Carl and Arre realized that there were a few strange cases that our model cannot support (in the case of a object and a weakref-with-callback to the same object both being in the same trash cycle, it is unclear whether the callback will be called). We plan to deal with this by calling anyone who claims to care insane. They then looked at getting CPython's test_weakref to run (something that's been on the "that would be nice" list for a looong time). This uncovered an amusing problem, a result of three facts: 1. the conservative Boehm collector cannot distinguish between a pointer and an integer that happens to have the same bit pattern as the pointer (this is what is meant by "conservative"). 2. the hash of a weakref object is the hash of the referenced object. 3. the default hash of a Python instance is it's id, which is usually the memory address cast to an int. This made weakref.WeakKeyDictionary rather poorly named. This was fixed by changing the id when using Boehm; the pointer is cast to an integer and then bitwise complemented. Accidentally, this means that that this: bool(id(object()) & 1) is now a check for pypy-c linked with the Boehm GC (use this fact with care). After solving the weak-keys-are-strong problem and adding a few calls to gc.collect() to CPython's test_weakref (to give Boehm a hurry up), test_weakref now passes. This leads on to the work that Anders and Holger did during the first day of the sprint: improving PyPy's "compliance score": http://codespeak.net/~hpk/pypy-testresult/ After a couple of months of relative neglect, this measure of how accurately we implement CPython's behaviour had slipped to around 80%. After fixing a few problems with the new interp-level implementation of the complex type and the above weakref work, we are back well over 95%. Michael helped Samuele remember exactly what he was thinking of in a branch that previously had only been the subject of hacking in the small hours of the night before his vacation. The goal of the nocturnal head bashing was to introduce the concept of an "explicit resume point" in an RPython program: a named, arbitrary location in the code and a way of transferring control to that point together with values for all live local variables. For example: def f(x, y, z): r = x + y rstack.resume_point("rp_f", r, z) return r + z defines a resume point called "rp_f" and records that the values of r and z must be supplied when the state is jumped to, like so: state = rstack.resume_state_create(None, "rp_f", 3, 4) result = rstack.resume_state_invoke(state) Something like this is a necessary piece in the puzzle that is implementing tasklet pickling, or more properly tasklet unpickling. This was a nice illustration of the "build one to throw away" concept: we were able to see that what Samuele had implemented as one operation really needed to be implemented as two (resume_state_create and resume_state_invoke in the above examples), and the rest of the coding only required moderate amounts of head bashing. By this morning, they had written a test that manually copied a (interp-level) coroutine, which validates the approach to at least some extent. You can also use explicit resume points to write some extremely confusing programs (as we discovered by mistake when debugging). The other major pieces of the tasklet pickling puzzle are being able to pickle and unpickle the core interpreter types such as frames, generators, tracebacks, ..., placing enough explicit resume points in the source of the Standard Interpreter to be able to resume a pickled tasklet and integration of the various pieces. Christian and Eric (working remotely; he arrives in Duesseldorf today) worked on the pickling and unpickling of interpreter types and have now implemented this for enough types that we now have no real option but to face the scary task of making something useful. For the last couple of days, the attention of Holger, Armin (still recovering from the 'flu, that's why we haven't mentioned him yet), Arre and a bit of Carl have been focussed on the elusive "ext-compiler": a tool designed to take an RPython module in a form suitable for making an "extension" for PyPy and translating it into an efficient extension module for CPython. This would clearly already be useful for purely algorithmic code, but the new-ish rctypes package of PyPy allows it to be used for modules that wrap an external library. While most of the basic technology is present for this, the tool is still far too rough to be unleashed even on unsuspecting Summer of Code students, never mind mere humans. Holger and Arre fleshed out ideas for how to build a more usable interface and wrote some documentation explaining this interface (who said open source projects are rubbish at documentation?). Armin began implementing support for typedefs, i.e. exposing custom types in the module interface, but ran into a small army of confusing levels of abstraction, which shift around as soon as you aren't staring at them. We're sure they're no match for Armin's planet-sized [#]_ brain though :) .. [#] http://mail.python.org/pipermail/python-dev/2003-January/032600.html While some of the above work involved much discussion and occasional voluminous cursing, two or three people have been sitting in the corner muttering about strange things like "ootypes" and "dot-net" and "javaskript", as well as producing a stream of high quality checkins (most of them have the log message "More ootypesystem tests", though). PyPy semi-regular Nik and newcomers and Summer of Code participants Antonio and Maciej worked on the ootypesystem backends, fixing many bugs and implementing new features, to the point that the richards and rpystone benchmarks can be run using the llinterpreter (now increasingly badly named; rinterpreter would prooooobably be better, but would have other unhelpful connotations). They couldn't run these benchmarks without a small modification: removing the print at the end that tells you how fast they ran (running things on the llinterpreter is so horrendously slow that this doesn't defeat the point of the excercise as you might at first think). The problem with printing is that it involves calling an external function, something that was not at all supported in the ootypesystem's world. So they decided to fix that :) And they did. Maciej also worked on the newer, ootype-using version of Javascript backend, and can now write python programs which bounce bub-n-bros characters around a browser window. This was a good way of getting Armin's attention :) Pozdrawiam, mwh & Carl Friedrich -- "Well, the old ones go Mmmmmbbbbzzzzttteeeeeep as they start up and the new ones go whupwhupwhupwhooopwhooooopwhooooooommmmmmmmmm." -- Graham Reed explains subway engines on asr

1 0

Vancouver Python Workshop
by Paul Prescod June 6, 2006

June 6, 2006

We would be very appreciative if one or more of you PyPy experts would come to the Vancouver Python Workshop and inform of us of your progress on this exciting project. For invited talks like this, we offer a substantially reduced discount on our already minimal registration fees. In addition, we already have some really good talks organized on non-PyPy topics, including Python 3000 and IronPython, so the conference should be fun for whoever decides to come. http://www.vanpyz.org/conference Paul Prescod Vancouver Python Workshop

2 1