Mailman 3 March 2007 - pypy-dev

codespeak/pypy environment overview
by holger krekel 09 Mar '07

09 Mar '07

Hi folks, if you are interested, it'd be great if some people (hi e.g. Alexander! :) could look into a new technical report, describing PyPy's development infrastructure and codespeak.net, particularly if you think about helping with maintaining codespeak.net in the future. I put the current version (mostly written by Guido Wesdorp, CF and me): http://codespeak.net/~hpk/D02.1-deliverablereport.pdf Feel free to give feedback here or privately, there also is a ".txt" version next to it which you may use for sending typo-fixes or refinements. best & thanks, holger -- merlinux GmbH Steinbergstr. 42 31139 Hildesheim http://merlinux.de tel +49 5121 20800 75 (fax 77)

1 0

Hildesheim Coding Sprint Report
by Michael Hudson 07 Mar '07

07 Mar '07

Hello pypy-dev! This is the report for the second, coding, half of PyPy's **25th** sprint, and the 18th and final sprint of the EU funded period. We are so completely tired that we don't have the energy to write a witty entry, so we'll skip that bit and start with describing the now usual tutorial for those participants who are less familiar with PyPy code base. This time Carl Friedrich was talking to Georg Brandl, an interested CPython developer, and Anders (only the third Anders to be involved in the project!) Sigfridsson, a PhD student at the Interaction Design Centre at the University of Limerick. Anders originally became involved in the project as an observer of the sprint process for his group's research when we sprinted at the University of Limerick, and that was partly why he was here this time, but it seems he found PyPy interesting enough to learn Python in the mean time and participate on the coding less at this sprint (or maybe he thought the view from the sprint trenches would be more revealing!). This sprint has seen many, many small tasks completed. On the first morning, Holger and Armin improved the readline module to the point of being useful -- supporting line editing and history, but not completion -- and hooked it into the interpreter sufficiently that the interactive interpreter and pdb both use it when available. At the same time Richard and Michael were hunting a bug Richard had discovered translating his own code, which was generally referred to as "the rdict bug" but turned out to be a bug in the garbage collector. Carl Friedrich and his band of helpers (mostly Anders, Georg, Alexander) worked on experimental reimplementations of Python lists, one using a theoretically optimal overallocation strategy and another using chunked storage to reduce the cost of list resizing. Sadly both resulted in a measurable slow down. This can be seen as yet more evidence that theory is different from practice... CF's other target for reimplementation at this sprint was strings. With help and moral support from Armin, he reimplemented strings according to the design from the "ropes paper" of Boehm, Atkinson & Plass: http://www.cs.ubc.ca/local/reading/proceedings/spe91-95/spe/vol25/issue12/s… The predictable effect was that "typical Python code", written in the knowledge of how strings are implemented in Python today, takes a small (at most 10%) performance hit, but an arguably more natural and naive style of string handling becomes efficient. And some completely crazy code (like hash('a'*sys.maxint)) becomes very fast too... Continuing the theme of making things slower by object reimplementation, Armin supplied an implementation of general dictionaries as a hash table whose collision resolution is via chained lists instead of open addressing. Next! As opposed to the above, Armin and CF implemented caching of app-level character (i.e. strings of length 1) objects, which was a clear win, improving the pystone benchmark by around 10%. There have been many discussions recently about optimizing the lookup of global variables, and during one that took place here about various corner cases, Armin and Carl Friedrich and Samuele removed from PyPy some of the strange things CPython does to determine what the __builtins__ are for the execution of a given frame -- of course, depending on the value of PyPy's five millionth configuration option. Holger and Antonio came up with yet another optimization idea along these lines, which can be found in doc/discussion/chained_getattr.txt. Going back to the first day, Anto and Samuele worked on analyzing why pypy-cli was being reported as 50+ times slower than CPython on the benchmark page: http://tuatara.cs.uni-duesseldorf.de/benchmark.html To do this they wrote some small benchmarks in RPython and stared at some code, but the main problem seems to be that Mono on PowerPC just isn't that good: running pypy-cli using Microsoft's runtime shows performance just 3-4 times slower than pypy-c. After this, they worked on streamlining PyPy's much complained about external function interface (and also broke translation a few times in the process). The last sprint saw the introduction of a more general registry-based interface for external functions, and Samuele and Anto began by moving the math module over to using this interface. This was harder than what had gone before because these functions depend on header files, so some modifications to the C and LLVM backends were necessary. On the last day, Anto made some small improvements to pypy-cli's performance and Samuele made the taint object space translatable. On the first day Georg and Alexander tried to see how fast a PyPy could get if there was no Global Interpreter Lock (GIL). By disabling the GIL and making the exception state thread-local on the genc-level, they could easily get a binary that at least didn't crash if multiple threads were not modifying internal stuff concurrently. Running 4 Pystone instances (from 4 different modules) on this pypy-c let the process use 381% of cpu time, but the resulting figures were disappointing: running the 4 Pystone instances in parallel was less than 25% faster than running them in series, as opposed to being 300% faster in the best case. Both concluded that the garbage collector used (Boehm) is not very well suited for the heavy-duty memory allocation pattern of PyPy in case of multiple threads. After this, they implemented some of Python 2.5's features in the interpreter, in particular support for __index__ and some extensions to string and dict methods. Anders and Anders worked very productively on fixing some of the bugs in PyPy's issue tracker, implementing the -m command line option in pypy-c, much improved handling of EINTR results from syscalls (which makes most difference when pressing ^C on the command line), allowing buffer objects to be passed to socket.send and preventing modifications to builtin types. Holger and Stephan worked in the direction of moving the currently app-level and extremely slow string interpolation code into RPython by separating out the code that analyzes the format string from the code that access the values to be interpolated. Maciej and Guido worked a little on the javascript backend, both generally tidying and improving compatibility with Internet Explorer. Guido should not be allowed to forget saying "I am happy to work with Internet Explorer" during one of the daily status meetings :-) Stephan and Arre worked on fixing the last remaining bugs in the rewrite of the application-level stackless code that Stephan had been working on for some time. Later Stephan joined Armin and Christian in a discussion about the best API to provide for the new "composable coroutine" concept. They feel that the concept is powerful to encompass threads, greenlets, coroutines, threads, syslets and the best way to barbecue ribs. You can read about the basic idea in the "Composability" section of PyPy's stackless documentation: http://codespeak.net/pypy/dist/pypy/doc/stackless.html and further insight is unlikely to be provided by this diagram: http://python.net/crew/mwh/stackless.jpg The basic conclusion was that this is a very nice and natural model for a lot of things, at least once you've whacked your head into the right shape. A task that occupied various people at various times of the sprint was that of benchmarking, the goal being to determine how much effect the object and other optimizations have. Michael had over the previous month or so some written some scaffolding code to allow various benchmarks to be run and the results recorded. At the sprint he added a benchmark using docutils to process 'translation.txt' from pypy's own documentation and Guido added another using his own 'templess' templating system. Holger worked on getting some code written for bzr that makes nice graphs out of benchmark to parse the benchmark data produced by PyPy's benchmark runs. Maciej worked on the lib/distributed code that demonstrates the PyPy's transparent proxies. After a bit of effort, he was able to write a demo that implemented a remote version of pdb by simply creating a traceback object that proxied all operations to a remote process. Michael and Richard spent a day or so on the LLVM backend, which of late hasn't been so much "maintained" as "held together by increasingly large amounts of sticky tape". After some refactoring of the way the backend handled options, they removed a layer of hacks around the issue of FixedSizedArrays and implemented them properly, and also added support for the direct_* pointer operations produced by rctypes. Michael spent some time using Shark, an OS X profiling application, and found some OS specific flags and tweaks that improved the performance of pypy-c on OS X/PPC by around 20%. As readers of pypy-dev will know by now, there were discussions about how PyPy is going to continue after the end of the EU funding period. However, we don't have to summarize them here because we can just link to Armin's mail: http://codespeak.net/pipermail/pypy-dev/2007q1/003577.html Cordiali Saluti, mwh & Carl Friedrich

1 0

Hildesheim report-writing-sprint report
by Lene Wagner 06 Mar '07

06 Mar '07

Hi pypy-dev! Welcome to a slightly unconventional sprint report (only fair for this slightly unconventional sprint): The first three days of the Trillke-sprint were fully dedicated to EU-reporting, considering that it may be slightly more fun to write them knowing that everybody else is doing the same, and that's what this report is about. The coding sprint report will follow, once the writers will have managed to travel home and recover a bit. Each of the 14 work packages that PyPy promised to do to the EU will conclude with at least one report due in the next month, and the target of the report sprint was to wrap up most of them and even deliver some to the EU already, also to have some spare time for coding works in March. In addition, there were the so-called 'review recommendations' - some first project period feedback by the Commission, requesting that the reports should meet scientific standards, include references and in general be useful for other than filling the EU's archives purposes. So, everybody was busy collecting papers, writing footnotes and consider overall sense-making strategies of the EU-reports, many of the results can be found here: http://codespeak.net/pypy/dist/pypy/doc/index-report.html A big effort went into the D12 report (prototypes, backends) taken care of by Samuele, Arre, Anto, Guido, Armin and overall coordinated by Holger, who also got Niko on the boat for remotely writing a section on the JVM backend. And then there were D02.1, D14.2, D01.2-4, D13.1, D03.1, D09.1 not to speak of the periodic/final EU management related reports and (sigh) the D06.1 report, being worked on by Holger, Anto, Arre, Stephan, Samuele, Lene, Carl Friedrich, Bea, Armin, Niko, Guido, and even the jet-lagged Michael and Richard helped with reviews and consultancy. Carl Friedrich, the very patient report release manager, took care for overall finalization and pdf-ing of reports apart from driving and starting the D06.1 report. Bea, Holger and Lene sat together dicussing ideas for overall activity reporting. On Wednesday, we finally managed to send out a pile of reports to the EU project officer, also presenting a commonly discussed table of planned report deliveries. And then the OLPC laptop came - introduced by Christian bringing it directly from Pycon to the PyPy group - which more or less stopped all report writing in exchange for people gathering around the machine and starting hacking. But luckily it was only Wednesday afternoon when it arrived and the belly- dancers were about to conquer the sprint room anyway :) (Actually, Armin continued working with the OLPC on the breakday and managed to port the Bub-n-Bros client to it, getting one step closer to the PyPy3000 goal of running Bub-n-Bros on every imaginable platform.) Cheers, Lene

1 0

PyPy after EU funding
by Armin Rigo 05 Mar '07

05 Mar '07

Hi all, At the Trillke sprint, we discussed some plans about how to structure the post-EU Open Source project period. This resulted on Sunday in a general session of which you find notes here: extradoc/minute/post-eu-structure.txt We are thinking about setting up a group of individuals caring for conceptual project integrity, releases etc. taking an overall project perspective into account. The main goal would be to keep an overall project consistency: given that PyPy is by now a relatively large project, it seems important that we try to keep such an overall view. The issue that we are trying to avoid here is that of developments going in directions that are completely disruptive for other aspects of the project. The initial candidates for the group were discussed and agreed upon at the sprint. Note that the view on the post-EU period is just evolving: we are still quite busy with working towards 1.0 and wrapping up EU reports towards the final review end May in Bruxelles. Feedback, questions, comments welcome! If there are further discussions or needs, we might target e.g. a #pypy-sync meeting early April to refine and conclude on things. Otherwise we will consider the sprint discussion result as the current basis of operation. A bientot, Armin -- e-mail written in common with input from Holger, Samuele, Bea, Lene, etc.

1 0

pypy ./translate.py command completion
by Maciek Fijalkowski 04 Mar '07

04 Mar '07

I've written small command line completion to ./translate.py from pypy/translator/goal. Note: works only from that directory (not sure how to make in working in general case) http://codespeak.net/svn/user/fijal/completetranslate.py

1 0

pylint 0.17 containing the RPython checker is available
by Sylvain Thénault 01 Mar '07

01 Mar '07

Hi there ! I'm pleased to announce that the new pylint release containing the rpython checker is available on logilab's web site. This first attempt contains the following checks: * unavailable keywords / builtins * multiple inheritance * mixing multiple types * non homogeneous list * global modification * negative slice index * using %r in format string * warn about special method that are not implicitly called By default the rpython checker is deactivated. Activate it using : pylint --rpython-mode -rn ... (-rn is disabling statistics reports) or pylint --enable-checker=rpython ... to get only rpython checks (though in this case you won't be warn about regular errors). Another interesting thing is the rpython dedicated testing framework, testing that checked things are actually not translatable. I've the idea that this may be useful to generate some kind of documentation for features supported by rpython or not, and help spread information when a feature that wasn't supported is introduced in rpython. That's another story though... If you're interested, check pylint/test/test_rpycompilation.py. Please report any false positives you find or things you can think about that would be interesting to check. Regards, -- Sylvain Thénault LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Python et calcul scientifique: http://www.logilab.fr/science

1 0