Mailman 3 December 2005 - pypy-dev

gothenburg sprint report #2
by Michael Hudson 12 Dec '05

12 Dec '05

So, the sprint is over, we are on a ferry but we *still* haven't escaped the internet... Christian and Richard spent the last couple of days thinking hard about the many possible ways of exposing stackless features to application-level and by the end of the sprint had pretty much considered them all. This means they will have no choice but to write some code for the mixed module soon :) Armin and his team of helpers (well, mostly Michael and Samuele to be honest) continued to work on JIT-related things, and continued to manufacture both working code and extremely strange bugs. Eventually the Test Driven Development style was halted for a quick but useful Cafe-cake-and-thinking-hard Driven Development moment (also attended by Carl). By the end of the sprint there was support in the abstract interpreter for "virtual structures" and "virtual arrays" which are the abstract interpreter's way of handling objects that are usually allocated on the memory heap but are sufficiently known ahead of time for actual allocation to be unnecessary. Now that probably didn't make much sense, so here's an example: def f(x): l = [x] return l[0] When CPython executes this code it allocates a list, places x into it, extracts it again and throws the list away. When the abstract interpeter sees the statement "l = [x]" it just records the information that "l is a list containing x" so when it sees "l[0]" it already knows what this is -- just "x". Then as nothing in the function ever needed l as a list, it just evaporates. Anders L and Nik continued on the socket module and managed to write a test that contained a simple server and client that could successfully talk to each other (after fighting some mysterious problems that with processes that refused to die). Carl continued his work on __del__, implementing support for it in the C backend when using the Boehm garbage collector which had the minor disadvantage of slowing pypy-c down by a factor of ten, as every instance of a user-defined class was registered with the GC as having a finalizer. This is apparently not something that the Boehm GC appreciates, so on Sunday Carl and Samuele implemented a different strategy. Now only instances of user-defined classes that actually define __del__ (at class definition time, no sneaky cls.__del__ = ... nonsense) get registered as needing finalization. Carl was also awarded his first "obscure Python bug" medal for making CPython dump core when he tried to test a hacky way to implement weakrefs (now SF bug #1377858). Arre and Eric continued their optimization drive and implemented a 'fastcall' path for both functions and methods which accelerates the common case of calling a Python function or method with the correct, fixed number of arguments. This improved the results of a simple function-call benchmark by a factor of about two. Result! A notable feature of this sprint is that Armin and Christian were implementing things very much like other things they had implemented before, painfully, in C -- namely stackless and psyco -- again, but this time in Python, and much more enjoyably :) Even more than this, they both managed to work their way through designs and ideas that had taken months to sort through the first time in days or even hours. We'd love to attribute all this to the magical qualities of Python but practice probably counts for something too :) So another sprint ends, and a productive one at that. As usual, we all need to sleep for a week, or at least until pypy-sync on Thursday... Cheers, mwh & Carl Friedrich

2 1

Re: [pypy-svn] r21001 - pypy/extradoc/Chris EWT-travel
by Günter Jantzen 11 Dec '05

11 Dec '05

Hallo Christian, besser so! Alles was Du hier eincheckst ist für jeden Abonennten von pypy-svn lesbar der sich durch die diffs hangelt. Abgesehen davon: * Aber Hallo! * Viele Grüße aus Köln und noch eine schöne Adventszeit Günter tismer(a)codespeak.net schrieb: > Author: tismer > Date: Sat Dec 10 14:51:18 2005 > New Revision: 21001 > > Removed: > pypy/extradoc/Chris EWT-travel/ > Log: > sorry > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn >

1 0

Compliance test suite broken
by Jacob Hallén 10 Dec '05

10 Dec '05

Running the compliance test suite with a testresults directory installed results in the following traceback for (what I presume is) all the tests: Traceback (application-level): File "/home/jacob/src/pypy/dist-pypy/pypy/tool/pytest/regrverbose.py", line 3 in <module> from test import test_support ImportError: cannot import name 'test_support' I'd be happy if someone who understands what broke can fix it, since I'd like to tackle some of the compliancy problems. Jacob

2 1

This Week in PyPy 6
by Michael Hudson 09 Dec '05

09 Dec '05

Introduction ============ This is the sixth of what will hopefully be many summaries of what's been going on in the world of PyPy in the last week. I'd still like to remind people that when something worth summarizing happens to recommend if for "This Week in PyPy" as mentioned on: http://codespeak.net/pypy/dist/pypy/doc/weekly/ where you can also find old summaries. This week features the first IRC summary from Pieter Holtzhausen, a feature that will hopefully continue. There were about 150 commits to the pypy section of codespeak's repository in the last week (a relatively small number for a sprint week -- lots of thinking going on here). The Sprint! =========== This is covered in more detail in the `sprint report`_, but seems to be going well. There has been work on the JIT, supporting larger integers and sockets in RPython, making the stackless option more useful, performance, compiler flexibility, documentation and probably even more. .. _`sprint report`: http://codespeak.net/pipermail/pypy-dev/2005q4/002656.html IRC Summary =========== Thanks again to Pieter for this. We need to talk about formatting :) **Friday** http://tismerysoft.de/pypy/irc-logs/pypy/%23pypy.log.20051202:: [00:04] Arigo states it is time to merge the PBC branch. Merging henceforth commences. [15:46] Pedronis and mwh discusses the simplification of the backend selection of the translator. Some translator planning documents checked in later. **Saturday** http://tismerysoft.de/pypy/irc-logs/pypy/%23pypy.log.20051203:: [15:45] Stakkars mentions the idea he posted to pypy-dev, that involves the substitution of CPython modules piecewise with pypy generated modules. Pedronis replies that he has thought of a similar approach to integrate pypy and Jython, but that this effort needs to be balanced with the fact that the pypy JIT currently needs attention. **Sunday** http://tismerysoft.de/pypy/irc-logs/pypy/%23pypy.log.20051204:: [14:03] Stakkars asks about the necessity of 3 stacks in the l3interpreter that Armin has been working on. One for floats, ints and addresses. After remarks about easier CPU support, Arigo replies that there is simply no sane way to write RPython with a single one. [18:26] Gromit asks how ready pypy is for production usage. He is interested in pypy as a smalltalk-like environment, since he deems objects spaces to be reminiscent of smalltalk vm images. [18:31] Stakkars states that he believes the project should postpone advanced technologies, in favour of getting the groundwork to a level where the project really becomes a CPython alternative. **Monday** http://tismerysoft.de/pypy/irc-logs/pypy/%23pypy.log.20051205:: [01:44] Pedronis running counting microbenchmarks, one 4.7 times slower than CPython, the other one 11.3 times. Function calling takes its toll in the latter. **Tuesday, Wednesday**:: [xx:xx] Sprint background radiation. Braintone rings like a bell. Not much to report. **Thursday** http://tismerysoft.de/pypy/irc-logs/pypy/%23pypy.log.20051208:: [17:55] Stakkars guess that RPython may get basic coroutine support, and is excited about that. [18:05] Stakkars votes for having stackless enabled all the time. The advantages: - real garbage collection - iterator implementation without clumsy state machines [20:19] Rhamphoryncus wonders whether dynamic specialization (e.g. psyco) can possibly improve memory layout. [20:46] Sabi is glad that long long is now supported (courtesy of mwh and Johahn). He yanks out his work around. EU-related Talks ================ On Monday Holger spoke at a German EU office workshop in Bonn and two days later he, Alastair and Bea spoke at a more union-wide EU workshop in Brussels. Both talks were very well received and while ostensibly we were telling the EU about our project, we gained much immediately useful information about how the EU actually adminsters projects such as ours. -- If you don't use emacs, you're a pathetic, mewling, masochistic weakling and I can't be bothered to convert you. -- Ron Echeverri

1 0

Gothenburg sprint report
by Carl Friedrich Bolz 09 Dec '05

09 Dec '05

Hi PyPy-dev! Finally we have found some time to write a sprint report on what is day three or day five of the sprint depending on how you count. The reason for the uncertainty is that on Monday us lifers^Wfulltimers gathered for two days of EU-report writing ahead of our review meeting in January. We'll spare you the details, but as most of the technical reports are also part of the documention on our website, it's worth mentioning that two documents: http://codespeak.net/pypy/dist/pypy/doc/low-level-encapsulation.html http://codespeak.net/pypy/dist/pypy/doc/translation-aspects.html received significant updates in these days. The former is a more friendly read, start with that one. Consensus opinion is that two days of proof-reading and generally attempting to write nice prose is even more exhausting than coding. It was thus with some relief that on Wednesday morning we met to plan our programming tasks. The task that most combined enormity and urgency was starting work on the the Just-In-Time compiler. Samuele, Armin, Eric, Carl and Arre all worked on this. Eric and Arre implemented yet another interpreter for a very simple toy language as we needed something that would be simpler to apply a JIT to than the PyPy behemoth and then moved on to looking at pypy-c's performance. Carl, Armin and Samuele began writing a partial specializer for low-level graphs, which is to say a way to rewrite a graph in the knowledge that certain values are constants. This is related to the JIT effort because a JIT will partially specialize the interpreter main-loop for a particular code object. The combination of the two sub-tasks will allow us to experiment with ways of applying these specialization techniques. Anders L and Nik repeatedly bashed their heads on the brick wall of issues surrounding a working socket module, and are making good progress although running a useful program is still a way off. The usual platform dependent horrors are, predictably, horrific. Michael and the sole sprint newcomer this time, Johan, implemented support in the translation toolchain for C's "long long" type which involved taking Johan on a whirlwind tour of the annotator, rtyper and C backend. By Thursday lunchtime they had a translated pypy that supported returning a long long value to app-level from exactly one external function (os.lseek) and declared success. Richard and Christian worked on exposing the raw stackless facilities that have existed for some time in a useful way. At this point they have written a 'tasklet' module that is usable by RPython which is probably best considered as an experiment on ways to write a module that can expose stackless features to application level. Adrien and Ludovic worked on increasing the flexibility of our parser and compiler, in particular aiming to allow modification of the syntax at runtime. Their first success was to allow syntax to be *removed* from PyPy: a valuable tool for making web frameworks harder to write:: Clearly, the only way to cut down on the number of Web frameworks is to make it much harder to write them. If Guido were really going to help resolve this Web frameworks mess, he'd make Python a much uglier and less powerful language. I can't help but feel that that's the wrong way to go ;). (from Titus Brown's Advogato diary of 7 Dec 2005: http://www.advogato.org/person/titus/diary.html?start=134) But they have also now allowed the definition of rules that can modify the syntax tree on the fly. On Friday morning a task group reshuffle gave Michael a unique opportunity to work with Armin on the 'JIT' and Carl and Johan became a new '__del__' taskforce. By the end of the day, '__del__' was supported in the genc backend when using the reference counting garbage collector. This also involved changing details at all levels of the translation process, so by now Johan has seen a very little bit of very large amounts of PyPy... Arre and Eric (with some crucial hints from Richard) had a successful hunt for performance problems: they changed about 5 lines of code affecting the way we use the Boehm GC which resulted in a remarkable 30-40% speed up in pystone after translation. Now if they can change 10 lines to get an 80% improvement we _will_ be impressed :) That's all for now. We'll write a report on the last two days, err, sometime. :) Cheers, Michael & Carl Friedrich

1 0

Comments from an observer
by Ben.Young＠risk.sungard.com 09 Dec '05

09 Dec '05

Dear PyPy'ers, First of all I would like to say that I think PyPy is an amazing project and that you have all done a really great job. Also the comments I have on the project are not aimed at any people in the project, more just at the general direction it appears to be going in. PyPy is on the edge of something great. A maintainable, powerful, flexible, fast interpreter is just what the python community needs. However just when it seems that PyPy can start to have some real significance in the Python world it seems like these benefits are being delayed for more research work which may take a long time. For instance a way of writing a rpython module that could be compiled to a Cpython extension or a PyPy extension would allow people to start using PyPy now, and at the same time make faster, powerful extensions for CPython while maintaining an upgrade path to PyPy. This would bring PyPy to the attention of a lot of people giving more testers/developers. Also, most people on #pypy seem to ask about using pypy to compile their simple python programs to c. Now, this doesn't seem like a great deal of work away (better error messages etc), but they are (politely) told that this is not what rpython is for. Now if rpython is not for this, why did you write PyPy in it? The same arguments could be applied to most programs (python is easier to read/maintain/write). I really can't see why something as useful as rpthon should remain an implementation detail, and again, exposing it would bring great exposure and benefits to the project. I don't want to come across like a moaner (and indeed, that's why I stop writing on #pypy as felt I couldn't be enough of a positive voice), and the only reason I'm writing this is because I think so much of the project and think it has so much potential. The last thing I want to see is for PyPy to become a great implemention with many powerful features, but then find that it had missed its time by not being "results driven" enough. The world doesn't need another powerful research/university language, it needs a great production language and with PyPy I think Python could be that language. Anyway, enough of my ranting. I'm sorry if I've offended anyone or completely missed the point. I'll go back to being a hopefull lurker again! Cheers, Ben

7 11

Thoughts on Stackless support
by Christian Tismer 09 Dec '05

09 Dec '05

Hi pypy-dev, Richard and I are prototyping stuff for Stackless support. While doing this (only testing stuff, still no start on a module), certain things struck us to be considered: The existing support for yielding the current frame is very poweful, being some one-shot continuation. There is only a tiny wrapper necessary which updates this continuation on every shot, and we have the concept of coroutines in RPython. (Greenlets are almost the same) Instead of just going for Greenlets/Tasklets on application level, I think it makes very much sense to have coroutine support built into RPython all the time. We can implement all kinds of stuff with this, like generators and iterators would be very nice to implement without having to think about state. We just need something that yields, and let the rest be done by the builtin magic. But maybe a much better reason to do this is the ability to write real garbage collectors, because stack unwinding gives us total control over all the objects! Having that said, I now really agree that we should end up rewriting the stackless support as a graph transformation. ATM, things are happening at the very end, without knowledge of the rtyper. But if we really use stack unwinding to implement iterators, it would be great to do this in an explicit way, so that we can apply inlining to it. Explicit support of unwinding would mean to insert the extra checking into the code, and to build state tuples which are chained into the heap emulated stack. It would be great to also have a way to avoid all of this, if iterators get inlined. Right now, I have a hard time imaginating how this would work, because we are using global structures for the chain of stack records. Maybe it is no problem at all. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

How pypy & all Python implentations test for 'correctness'? Are there standard unit tests somewhere?
by Christian Seberino 09 Dec '05

09 Dec '05

I'm curious how the various python implementations like JPython, IronPython, Pypy, ??? can VERIFY they are not deviating from 'correctness' (whatever that means). Is there a set of unit tests somewhere? Is that the best way to approach the problem? Chris

2 1

Ooops
by Ben.Young＠risk.sungard.com 08 Dec '05

08 Dec '05

Sorry about sending that last one to the bounces list. Cheers, Ben

1 0

Branch merged
by Armin Rigo 03 Dec '05

03 Dec '05

Hi, The somepbc-refactoring branch has been merged. Thanks to Michael for participation! What is still broken: * pypy/bin/translator.py -- the Translator class is going away, the functionality we need should probably be moved into this bin/translator.py for now. But ideally this should be replaced by an interface based on translator/goal/driver.py. * pdbplus, the pdb extension you get at the end of translate_pypy.py, has probably commands that need fixing. * translate_pypy --backend=llvm crashes apparently when building non-standalone targets; standalone targets appear to work fine (didn't try the whole of PyPy, however). All llvm tests work too, so it's hard to understand exactly what the problem is. * Christian: we did not completely port your r19917 because it needs some adaptations to the new world. The Translator class is being replaced by a much thinner TranslationContext. Most importantly, your changes to translator/c/pyobj.py haven't been merged -- they will also need adaptation to work on graphs instead of functions, as we did for the rest of pyobj.py. It was also difficult to know exactly what was needed because of the absence of tests. However, there is a tag of the trunk before we merged, so that a working pypy with your changes is still in http://codespeak.net/svn/pypy/tag/dist-ext-someobject . That's it ! A bientot, Armin & Samuele (& Michael, sadly not on-line when we merged -- or more probably happily so :-)

4 3