Mailman 3 November 2005 - pypy-dev

Re: [pypy-svn] r19917 - in pypy/dist/pypy: annotation objspace/flow translator translator/c
by Armin Rigo Nov. 16, 2005

Nov. 16, 2005

Hi Christian, On Wed, Nov 16, 2005 at 06:08:23AM +0100, tismer(a)codespeak.net wrote: > Modified: > pypy/dist/pypy/annotation/bookkeeper.py > pypy/dist/pypy/objspace/flow/objspace.py > pypy/dist/pypy/translator/c/pyobj.py > pypy/dist/pypy/translator/translator.py That's fine so far, but you should know that we have a branch in progress about the annotator, so if you plan to make larger changes to any of the translation parts, please discuss them first :-) Also, the subject of how to write C extension modules in RPython could do with being discussed on pypy-dev first anyway. Currently it mostly kind-of-works already, as a by-product of the way the whole thing was developed, because it allowed continuous testing. But we need to stop and design and think a bit more precisely about what we want, if indeed we want that now. A bientot, Armin.

2 1

This Week in PyPy 1
by Michael Hudson Nov. 15, 2005

Nov. 15, 2005

(this summary is also on the web at http://codespeak.net/pypy/dist/pypy/doc/weekly/summary-2005-11-04.html) Introduction ============ This is the first of what will hopefully be many summaries of what's been going on in the world of PyPy in the last week. First, I'd like to make a request: help me write these things. As is mentioned in the page about This Week in PyPy: http://codespeak.net/pypy/dist/pypy/doc/weekly/ as and when something worth summarizing happens, be it on IRC, on a mailing list or off in the blogosphere, add an entry to this file: http://codespeak.net/pypy/dist/pypy/doc/weekly/log (if you can) or email me about it (if you can't). This week noone at all has done anything like this, which I'll forgive because it's the first week :) Please, please do get into the habit of doing this though, at least if you think writing this summary isn't a complete waste of time. Release of PyPy 0.8.0 ===================== The biggest thing that's happened in the past week was clearly the release of PyPy 0.8.0. You can read the release announcement at: http://codespeak.net/pypy/dist/pypy/doc/release-0.8.0.html This release went fairly smoothly compared to some releases, mainly because we weren't rushing to get some feature or other into the release. Import Analysis =============== In an effort to understand what code is used where in PyPy, Michael Hudson wrote a tool to analyse the import structure of PyPy, culminating in a several megabyte HTML report which you can find at: http://starship.python.net/crew/mwh/importfunhtml/pypy/ For example, this is a list of all the modules that reference pypy.objspace.flow.model.Constant (one of the more referenced names in PyPy): http://starship.python.net/crew/mwh/importfunhtml/pypy/objspace/flow/model/… Of course, this work ended up duplicating some of the things done by tools such as pylint and pyflakes and has the potential to be useful for projects other than PyPy, so I hope to clean it up and maybe make it a pylint plugin soon-ish. A RPythonC tool? ================ A fairly common topic of discussion on #pypy starts with people who want to write RPython code and then use PyPy to translate it to efficient C. This was again the case on Monday evening (look from about 19:30 onwards): http://tismerysoft.de/pypy/irc-logs/pypy/%23pypy.log.20051031 While "officially speaking" supporting such things is not a goal of the PyPy project (RPython is essentially an implementation detail) the frequency of raising of the subject means that there probably is some interest in a "rpythonc" type tool that would compile an RPython program. A fairly serious problem, though, is that when the target of compilation turns out not to be RPython, working out *why* can be arbitrarily difficult, even for an experienced PyPyer. For these reasons, it seems unlikely that such a tool will be written all that soon (at least, I'm not going to do it :). PyPy-sync ========= The main discussion at the weekly pypy-sync meeting was planning for the GËteborg sprint in December: http://codespeak.net/pypy/extradoc/minute/pypy-sync-11-03-2005.txt -- <dash> if python is an orchestra, overloaded operators are "miscellaneous percussion" -- from Twisted.Quotes

3 2

PyPy sprint announcement: Gothenburg 7th - 11th December 2005
by Carl Friedrich Bolz Nov. 15, 2005

Nov. 15, 2005

Gothenburg PyPy Sprint II: 7th - 11th December 2005 ====================================================== (NOTE: internal EU-only sprint starts on the 5th!) The next PyPy sprint is scheduled to be in December 2005 in Gothenborg, Sweden. Its main focus is heading towards phase 2, which means JIT work, alternate threading models and logic programming (but there are also other possible topics). We'll give newcomer-friendly introductions. To learn more about the new PyPy Python-in-Python implementation look here: http://codespeak.net/pypy Goals and topics of the sprint ------------------------------ We have released pypy-0.8.0_, which is officially a "research base" for future work. The goal of the Gothenburg sprint is to start exploring new directions and continue in the directions started at the Paris sprint. The currently scheduled main topics are: - The L3 interpreter, a small fast interpreter for "assembler-level" flow graphs. This is heading towards JIT work. - Stackless: write an app-level interface, which might be either Tasklets, as in "Stackless CPython", or the more limited Greenlets. - Porting C modules from CPython. (_socket is not finished) - Optimization/debugging work in general. In particular our thread support is far from stable at the moment and unaccountably slow. - Experimentation: logic programming in Python. A first step might be to try to add logic variables to PyPy. .. _`pypy-0.8.0`: http://codespeak.net/pypy/dist/pypy/doc/release-0.8.0.html Location & Accomodation ------------------------ The sprint will be held in the apartment of Laura Creighton and Jacob Halen which is in Gotabergsgatan 22. The location is central in Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, where many lines call. .. _tram: http://www.vasttrafik.se Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. (You can have a 10% discount there; ask in the pypy-sprint mailing list for details. We also have some possibilites of free accomodation.) .. _`SGS Veckobostader`: http://www.sgsveckobostader.com Exact times ----------- The public PyPy sprint is held Wednesday 7th - Sunday 11th December 2005. There is a sprint for people involved with the EU part of the project on the two days before the "official" sprint. Hours will be from 10:00 until people have had enough. It's a good idea to arrive a day before the sprint starts and leave a day later. In the middle of the sprint there usually is a break day and it's usually ok to take half-days off if you feel like it. Network, Food, currency ------------------------ Sweden is not part of the Euro zone. One SEK (krona in singular, kronor in plural) is roughly 1/10th of a Euro (9.15 SEK to 1 Euro). The venue is central in Gothenburg. There is a large selection of places to get food around, from edible-and-cheap to outstanding. You normally need a wireless network card to access the network, but we can provide a wireless/ethernet bridge. Sweden uses the same kind of plugs as Germany. 230V AC. Registration etc.pp. -------------------- Please subscribe to the `PyPy sprint mailing list`_, introduce yourself and post a note that you want to come. Feel free to ask any questions there! There also is a separate `Gothenburg people`_ page tracking who is already thought to come. If you have commit rights on codespeak then you can modify yourself a checkout of http://codespeak.net/svn/pypy/extradoc/sprintinfo/gothenburg-2005/people.txt .. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`Gothenburg people`: http://codespeak.net/pypy/extradoc/sprintinfo/gothenburg-2005/people.html

1 0

This Week in PyPy 2
by Michael Hudson Nov. 11, 2005

Nov. 11, 2005

Introduction ============ This is the second of what will hopefully be many summaries of what's been going on in the world of PyPy in the last week. I'd still like to remind people that when something worth summarizing happens to recommend if for "This Week in PyPy" as mentioned on: http://codespeak.net/pypy/dist/pypy/doc/weekly/ where you can also find old summaries. There were about 100 commits to the pypy section of codespeak's repository this week. pypy-c py.py ============ Over the weekend (while I was being blown around Wales by the remnants of hurricane Wilma) Armin and a few others worked on trying to get a translated pypy-c to run the interpreted py.py. This resulted in a fixing a welter of small differences between CPython and pypy-c, though at the end of it all "we are still left in the dark of incomprehensible geninterplevel crashes caused by subtle differences between the most internal types of CPython and pypy-c." Multiple Spaces =============== In one of the reports we're currently writing for the end of phase 1 EU review: http://codespeak.net/pypy/dist/pypy/doc/draft-low-level-encapsulation.html we made this claim: The situation of multiple interpreters is thus handled automatically: if there is only one space instance, it is regarded as a pre-constructed constant and the space object pointer (though not its non-constant contents) disappears from the produced source, i.e. both from function arguments and local variables and from instance fields. If there are two or more such instances, a 'space' attribute will be automatically added to all application objects (or more precisely, it will not be removed by the translation process), the best of both worlds. And then we tried to do it, and had to tune the claim down because it doesn't work. This is because the StdObjSpace class has a 'specialized method' -- a different version of the wrap() method is generated for each type it is seen to be called with. This causes problems when there are genuine StdObjSpace instances in the translated pypy because of limitations in our tools. We looked at these limitations and decided that it was time to rewrite the world again, leading in to the next section... SomePBC-refactoring =================== One of the more unusual annotations produced by PyPy's annotator is that of 'SomePBC', short for 'SomePrebuiltConstant'. This annotation means that a variable contains a reference to some object that existed before the annotation process began (key example: functions). Up until now, the annotation has actually explicitly included which prebuiltconstants a variable might refer to, which seems like the obvious thing to do. Unfortunately, not all things that we'd like to annotate as a prebuiltconstant actually exist as unique CPython objects -- in particular the point of specializing a function is that it becomes many functions in the translated result. Also for 'external', i.e. not written in RPython, functions we want to be able to supply annotations for the input and exit args even if there is no corresponding CPython function at all. The chosen solution is to have the SomePBC annotation refer not to a CPython object but to a more abstracted 'Description' of this object. In some sense, this isn't a very large change but it affects most files in the annotation directory and a fair fraction of those under rpython/ and translator/. We're also cleaning up some other mess while we're there and breaking everything anyway. Draft-Dynamic-... ================= It's not linked from anywhere on the website (yet...) but the report that will become "Deliverable 05.1": http://codespeak.net/pypy/dist/pypy/doc/draft-dynamic-language-translation.… has been reviewed and re-reviewed in the last couple of weeks and is definitely required reading for anyone who has an interest in the more theoretical side of PyPy. Gtbg Sprint in December ======================= Hopefully very soon, we'll announce the next PyPy sprint... stay tuned! Cheers, mwh -- > I'm a little confused. That's because you're Australian! So all the blood flows to your head, away from the organ most normal guys think with. -- Mark Hammond & Tim Peters, comp.lang.python

1 0

pypy sync
by Carl Friedrich Bolz Nov. 10, 2005

Nov. 10, 2005

Hi pypy-devers! although the announcement is late, there will be a pypy-sync meeting today: Time & location: 1.00 - 1.30 pm (GMT+1) at #pypy-sync Regular Topics ==================== - activity reports (3 prepared lines of info, LAST/NEXT/BLOCKERS). - resolve conflicts/blockers Annotated Topics of the week ================================= New snake server ---------------- this is an informational topic: armin wants to tell us what is going on with the snake server. There are no further official topics today, you are free to come up with anything on short notice. Cheers, Carl Friedrich

1 0

will not be present at tomorrow pypy-sync meeting
by Samuele Pedroni Nov. 9, 2005

Nov. 9, 2005

I'm going to be on vacation, until wenesday/thursday of next week. I should be at the meeting then. Last I worked on reports (in particular 4.4) and helped with the PBC refactoring branch. Samuele.

1 0

even more import analysis
by Michael Hudson Nov. 6, 2005

Nov. 6, 2005

So I've beaten at my importfun tool even more and now it can produce an html report about the import (lack of) structure of PyPy; for an example start here: http://starship.python.net/crew/mwh/importfunhtml/pypy.interpreter.argument… You might not want to look at the directory itself -- there's about 10000 files in it :) I also think there are some bugs & omissions so I'll try to work on this a little more tomorrow, but I've run out of time tonight. Enjoy! Cheers, mwh PS: generation took about 5 minutes, so keeping it up to date shouldn't be that hard. -- Haha! You had a *really* weak argument! <wink> -- Moshe Zadka, comp.lang.python

3 5

Feature-comparatives benchmarks
by Armin Rigo Nov. 5, 2005

Nov. 5, 2005

Hi all, Even though it is short, I think that the draft-low-level-encapsulation would look complete enough if it had, for each described aspect: * a link to the doc (I'll add them); * some concrete benchmarks, in an "Experimental results" section, to see what would be the performance/memory impact of enabling each feature or leaving it out of the compiled PyPy. As far as I know our benchmark results are not nicely regrouped at some place, so I suggest we collect them on this mailing list. We need: * Stacklessness: what are the most up-to-date results? * Multiple Interpreters: I guess no one tried so far to compile in a mode where the 'space' variables don't go away. Let's. * Concurrency: not much data here apart from the massive unexplained overhead of enabling threads. * Memory Management: refcounting is veeery slow, which only shows that we could do much better there given sufficient efforts... * Evaluation Strategy: the thunk object space. Overhead? A bientot, Armin.

1 0

Document drafts
by Armin Rigo Nov. 5, 2005

Nov. 5, 2005

Hi all, People following only this mailing list may have missed some draft documents that we have been working on, as they are not linked to yet: http://codespeak.net/pypy/dist/pypy/doc/draft-dynamic-language-translation.… http://codespeak.net/pypy/dist/pypy/doc/draft-low-level-encapsulation.html A bientot, Armin

1 0

PyPy 0.8.0 just released
by Samuele Pedroni Nov. 3, 2005

Nov. 3, 2005

Thanks to all the people that contributed so far to PyPy and especially to the people that helped on this release. pypy-0.8.0: Translatable compiler/parser and some more speed ============================================================== The PyPy development team has been busy working and we've now packaged our latest improvements, completed work and new experiments as version 0.8.0, our third public release. The highlights of this third release of PyPy are: - Translatable parser and AST compiler. PyPy now integrates its own compiler based on Python own 'compiler' package but with a number of fixes and code simplifications in order to get it translated with the rest of PyPy. This makes using the translated pypy interactively much more pleasant, as compilation is considerably faster than in 0.7.0. - Some Speed enhancements. Translated PyPy is now about 10 times faster than 0.7 but still 10-20 times slower than CPython on pystones and other benchmarks. At the same time, language compliancy has been slightly increased compared to 0.7 which had already reached major CPython compliancy goals. - Some experimental features are now translateable. Since 0.6.0, PyPy shipped with an experimental Object Space (the part of PyPy implementing Python object operations and manipulation) implementing lazily computed objects, the "Thunk" object space. With 0.8.0 this object space can also be translated preserving its feature additions. What is PyPy (about)? ------------------------------------------------ PyPy is a MIT-licensed research-oriented reimplementation of Python written in Python itself, flexible and easy to experiment with. It translates itself to lower level languages. Our goals are to target a large variety of platforms, small and large, by providing a compilation toolsuite that can produce custom Python versions. Platform, Memory and Threading models are to become aspects of the translation process - as opposed to encoding low level details into a language implementation itself. Eventually, dynamic optimization techniques - implemented as another translation aspect - should become robust against language changes. Note that PyPy is mainly a research and development project and does not by itself focus on getting a production-ready Python implementation although we do hope and expect it to become a viable contender in that area sometime next year. PyPy is partially funded as a research project under the European Union's IST programme. Where to start? ----------------------------- Getting started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html PyPy Documentation: http://codespeak.net/pypy/dist/pypy/doc/ PyPy Homepage: http://codespeak.net/pypy/ The interpreter and object model implementations shipped with the 0.8 version can run on their own and implement the core language features of Python as of CPython 2.4. However, we still do not recommend using PyPy for anything else than for education, playing or research purposes. Ongoing work and near term goals --------------------------------- At the last sprint in Paris we started exploring the new directions of our work, in terms of extending and optimising PyPy further. We started to scratch the surface of Just-In-Time compiler related work, which we still expect will be the major source of our future speed improvements and some successful amount of work has been done on the support needed for stackless-like features. This release also includes the snapshots in preliminary or embryonic form of the following interesting but yet not completed sub projects: - The OOtyper, a RTyper variation for higher-level backends (Squeak, ...) - A JavaScript backend - A limited (PPC) assembler backend (this related to the JIT) - some bits for a socket module PyPy has been developed during approximately 16 coding sprints across Europe and the US. It continues to be a very dynamically and incrementally evolving project with many of these one-week workshops to follow. PyPy has been a community effort from the start and it would not have got that far without the coding and feedback support from numerous people. Please feel free to give feedback and raise questions. contact points: http://codespeak.net/pypy/dist/pypy/doc/contact.html have fun, the pypy team, (Armin Rigo, Samuele Pedroni, Holger Krekel, Christian Tismer, Carl Friedrich Bolz, Michael Hudson, and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html) PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a two year European Union IST research grant. The full partners of that consortium are: Heinrich-Heine University (Germany), AB Strakt (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany)

1 0