Mailman 3 January 2007 - pypy-dev

pypy 1.0 release meeting invitation
by Michael Hudson 01 Feb '07

01 Feb '07

Hi PyPy-developers! The release is drawing closer (the currently planned date is February, 15th) and it is time to meet and discuss the necessary work. Therefore we invite you all to a sync-meeting on Friday, February 2nd, 2007 14.00 UTC+1 #pypy-sync Regular Topics ============== * activity reports (LAST/NEXT/BLOCKERS, everybody posts prepared comma-separated lists of things) * resolving blockers/dependencies Release Goals ============= This topic is to get a common view on what the release goals are. A preliminary list of goals is: * object optimizations * pypy.net * more working modules * some performance * maturity * build tool * JIT preview * security prototype * js backend? * debian packaging? Task Planning ============= This topic is to collect tasks to be done for the release and divide them among people. Some tasks that we could think of: * work out a timeframe for jit usefulness * fix llvm(?) * test pypy-cs, run benchmarks. * windows testing! * documentation!!! * pypy.net * object optimizations (WP6) * release announcement * improve getting-started * transparent proxies * security prototype * make build-tool useable * polish js backend? Hope to see you all on friday! Cheers, mwh -- Python enjoys making tradeoffs that drive *someone* crazy <wink>. -- Tim Peters, comp.lang.python

6 6

Does/will pypy support cpython's c extension modules?
by Neal Becker 31 Jan '07

31 Jan '07

Subject says it all. Will I be able to use my existing c (c++ actually) modules?

10 15

RSession and py-trunk users
by Maciek Fijalkowski 29 Jan '07

29 Jan '07

I've implemented recently an addition, which allows to have cleanur conftest.py setup. Basically, conftest.py's dist_rsyncroots affects only directory where it is located, so in let's say pypy's parent dir you might have conftest.py with dist_rsyncroots = ['py', 'pypy', 'lib-python'] and in pypy's dir dist_rsyncroots = # all files except _cache and should work. There were different ideas how to solve that problem and from my POV this on is the simplest, but I'm open to different ones.

2 1

translation error in a fresh checkout
by Alexandre Fayolle 27 Jan '07

27 Jan '07

Hi, I'm experiencing problems when translating pypy from a fresh checkout (when testing the compilation of Debian packages) The translation crashes very early in the process: Loading grammar /home/alf/dev/pypy/dist/pypy/interpreter/pyparser/data/Grammar2.4 ******************** -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ "global name 'HMASK' is not defined" Happened at file /home/alf/dev/pypy/dist/pypy/lib/_classobj.py line 21 v += HMASK This comes from > /home/alf/dev/pypy/dist/pypy/objspace/flow/objspace.py(264)build_flow() -> raise FlowingError(format_global_error(ec.graph, ec.crnt_offset, str(a))) This used to work last tuesday, but with a checkout from yesterday, the bug occurs. I was puzzled at first because I could only see this during the translation for Debian packages, and not in my usual working checkout which I update on a regular basis. After blaming the Debian package building environment and not finding anything, it turns out that my working checkout has a pypy/_cache directory, which is used. If I rename this directory, translating will fail there too. Can other people confirm the problem ? If I'm correct it should affect the build tool too, since it builds in a fresh checkout. -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/

2 2

Machines at uni-duesseldorf
by Armin Rigo 25 Jan '07

25 Jan '07

Hi all, Here's an update on the computers that we have for PyPy at the HHU. Snake is back, but nothing wrong was found with its hardware, which means that it'll probably continue to hang randomly from time to time... Maybe we'll reinstall the machine from scratch. For all I know it could be corrupted system files. However, there are now two other machines. Just like snake they are machines whose performances can be a bit strange - hyperthreaded Xeon cores. Each core has roughly snake-like performance, but each machine has got 4 cores. The software setup is: SuSE Linux base, which I'm hiding as much as possible behind a Gentoo Linux installation in the /gentoo directory. When you log in, you reach a chrooted environment, so you should not see that at all unless you dig in special files. I've copied some accounts from codespeak, so you must use the same public key to log in (no password logins). Just ask me if you want to use the machines but I forgot to copy your account over. To login, you must use port 922 instead of 22. The latter would reach the SSH daemon of the outer SuSE installation (where you don't have an account). I recommend adding the following lines to your file ~/.ssh/config: Host wyvern HostName wyvern.cs.uni-duesseldorf.de Port 922 Host cobra HostName cobra.cs.uni-duesseldorf.de Port 922 Then you can log in with 'ssh wyvern' or 'ssh cobra'. Note that 'wyvern' is the fastest machine so feel free to prefer it. A great way to use them is for distributed testing. Put in your file ~/conftest.py: disthosts = ['wyvern'] * 4 + ['cobra'] * 4 Then you can do 'py.test --session=R'. Try also the --runbrowser option. A bientot, Armin

2 2

pypy trillke sprints Feb/March 2007
by holger krekel 22 Jan '07

22 Jan '07

========================================================================= PyPy Trillke "EU and beyond!" sprints (25-28th Feb, 1-5th March 2006) ========================================================================= .. image:: http://www.trillke.net/images/HausPanorama0304_113kb.jpg Some two years and some thousands of commits later, the EU project period of the PyPy (http://codespeak.net/pypy) project is about to close ... and a new period to begin: we are going for a sprint of three days of focusing on EU reports and administrative issues, and another three day sprint of happy hacking on the numerous interesting open ends of PyPy, the source code. We also intend to discuss and state our view on the time after the EU period (March 2007 is the last EU funded month), because clearly the project will not stop after our successful (knock knock!) completion of the EU project. It's already clear that some of us seriously plan for a relaxation time, i.e. one without having many obligations. But that should not keep us from thinking and envisioning the development from April on. So to the Sprint: we have two parts of the sprint, first the EU and second the public all-invited part:: 26th Feb - 28th Feb EU reports and adminstrative sprint 1st March break day / arrival for coding sprint 2nd March - 4th March public coding "beyond EU" sprint days All days are meant as full days, so please arrive 25th Feb / 1st March and leave 5th March if you can (this help us with pairing, introductions and logistical planning). --------------------------------------------- Possible Topics for the coding sprint --------------------------------------------- * working on .NET, Java and other backends * working on the Javascript, Prolog or a new frontend * Tuning the JIT, refining approaches `[1]`_ * Fixing PyPy-1.0 problems / improving it (PyPy-1.0 is scheduled for Mid February, btw) * improving the py lib/py.test, build environments, preparing for server administration efforts from April on * Work on/with prototypes that use the new features that PyPy enables: distribution `[2]`_ (based on transparent proxying `[3]`_), security `[4]`_, persistence `[5]`_, etc. `[6]`_. * discussion about the time after March, and how to organize/co-ordinate ourselves * all topics that are of interest otherwise (including maybe working on some particular EU related topics still!) .. _[1]: http://codespeak.net/pypy/dist/pypy/doc/jit.html .. _[2]: http://codespeak.net/svn/pypy/dist/pypy/lib/distributed/ .. _[3]: http://codespeak.net/pypy/dist/pypy/doc/proxy.html .. _[4]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#security .. _[5]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#persistence .. _[6]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html ----------------------- Location ----------------------- The sprint takes place at the Trillke Gut Steinbergstr. 42 Hildesheim, Germany http://www.trillke.net If you come to Hildesheim main station, take the Bus Number 3 (Hildesheimer Wald) and get out at "Waldquelle". Walking back a 100 meters gets you to a small street on the right leading to a big white building, the Trillke Gut. We'll have at least one larger room for sprinting, and a kitchen for our use. ----------------------- Accomodation ----------------------- We can probably arrange for some cheap or no-cost private accomodation, in private rooms located up in the house (and being part of "living groups" of 5-10 people). There also is a nice Guest house that has been used during recent events: http://www.gaestehaus-klocke.de/ and a four-star hotel 500 meters away from the Trillke: http://www.berghoelzchen.de/ ----------------------- Registration ----------------------- please subscribe to the pypy-sprint mailing list http://codespeak.net/mailman/listinfo/pypy-sprint and register by subversion: http://codespeak.net/svn/pypy/extradoc/sprintinfo/trillke-2007/people.txt or - if you have no checkin-rights - post to the pypy-sprint list above. -- merlinux GmbH Steinbergstr. 42 31139 Hildesheim http://merlinux.de tel +49 5121 20800 75 (fax 77)

1 0

means to get to the IRC Channel
by Leonardo Santagada 20 Jan '07

20 Jan '07

I'm finding it pretty hard to join the #pypy channel because everywere that has internet connection also has some sort of badly configured firewall. Do any one knows of a good way to make it work (like a tunnel, a web client or something?). If there is not can we put on codespeak.net a page with http://cgiirc.sourceforge.net/ for the #pypy channel? -- Leonardo Santagada (http://www.lomohomes.com/retype) Não se preocupe com o que os outros vão fazer. O melhor jeito de prever o futuro é inventa-lo. - Alan Kay

3 3

py lib WARNING / cleanup works
by holger krekel 17 Jan '07

17 Jan '07

Hi folks, this is a warning note that these days there are a number of changes and cleanups being done in py lib/py.test land. I'll do my best to keep pypy's usages in sync with those changes but it's obviously possible that i miss things. please report them on the py-dev(a)codespeak.net list (you need to be subscribed for getting your mails through immediately: http://codespeak.net/mailman/listinfo/py-dev) or here on the pypy-dev list. best, holger P.S.: Armin: i guess that some of your custom Session objects and other conftest customizations may need adaptation. -- merlinux GmbH Steinbergstr. 42 31139 Hildesheim http://merlinux.de tel +49 5121 20800 75 (fax 77)

1 0

Leysin sprint report
by Carl Friedrich Bolz 14 Jan '07

14 Jan '07

Hi pypy-dev! Unusually for the end of a sprint, everyone here is wide awake and fully alert here in the wonderfully pretty but remarkably snow-free ski resort of Leysin, in the Ermina B&B we have sprinted at three times before. Only joking, we are as tired as we always are. This sprint really is unusual in the sense that not very many new things were started. Slowly approaching the end of the EU-funded period, we are more in finalizing mode and try to get the results we promised (actually we had several EU related meetings to discuss some report writing plans and how to go about things in the next few weeks). It was no surprise that a lot of time went into making the JIT actually speed up code (a goal that is still not attained). There were two large subsections of this work: the relatively easy to explain task of improving the quality of the machine code the JIT produces and The Other One. Armin and Michael started the sprint with refactoring the interface to the code generators to make the lives of the backends easier, especially with regard to register allocation. This obviously broke all the backends, so the refactoring and the fixing of existing code took a day or so. Then Michael (with some later help from Niko), Armin and Eric worked on the PPC, i386, and LLVM backends respectively. They all ended up banging their heads fairly hard against various flat surfaces, especially Armin who had the wonderfully clean, regular and well designed i386 instruction set to work with. Arre and Samuele worked in the Terminology Production department, coming up with the wonderful concept of "virtualizable structures". Apparently it was much harder to implement them than to invent a term for them, so they mostly spent the whole week on it (and arrived only at "the easy part of the hard part"). To offer some hint of an explanation, virtualizable structures are data structures that live in the JIT world but can still be accessed by non-jitted code from the outside. Niko, who is funded as part of the Summer-of-PyPy program (so that's why we don't have snow: the Summer of PyPy is stronger than the Winter of Switzerland), was working with Anto on the Java backend he had started in Duesseldorf. By now a fair set of tests is passing and they managed to translate rpystone to Java bytecode (the result being roughly 20 times slower than the C version currently. although without many optimizations). After a few days Niko got fed up with Java and helped in the PowerPC efforts. The goal of Niko's project is to translate all of PyPy some time in the next few months and it seems to be well on course. A large amount of sprint energy was poured into making the Py-lib ready for release (but we all only believe it when we see it :-) ). Holger, Maciek, Guido, Carl Friedrich, Alexandre and Sylvain all worked on polishing, writing doc strings, documentation etc. for the various pieces of the package. Also a lot of functions were "unexported" to not have to deal with backward compatibility forever (which mostly broke Armin's large collection of strange conftest files in his deep and scary hack directory). Two sub-projects deserve special mention: Sylvain and Alexandre produced a debian package (which is confusingly called python-codespeak-lib, thanks to strange debian policies), including producing the mandatory man-pages for the various py-lib scripts. Otherwise the main thing the debian patches do is removing assorted cleverness from the py-lib to do with finding itself, as task which is obviously easier when you've been installed into a known location! After the success with packaging the py-lib, Sylvain and Alexandre set themselves the way scarier goal of packaging PyPy for Debian in a meaningful way. Almost disappointingly, they are dealing with the problem just fine and have a pretty good plan of how approach things: there will be a pypy-dev package which is for people developing in RPython, one or more binary packages that contain pre-translated PyPy versions (for example with stackless) and a pypy-lib package that will contain the common parts of both. An interesting point is that of dependencies: the pypy-dev package will "Depend" on those packages required to be able to run py.py, "Recommend" those packages that a developer wishing to work in RPython would need and "Suggest" some of the more eclectic packages PyPy sometimes uses. This will mean that if you set up a debian machine for working with PyPy, even if you plan to use the svn head version, installing pypy-dev will be an easy way to get all the dependencies right. The other py-lib sub-project that saw some serious polishing is the API documentation generator, "apigen". It's approach is very different from every other doc generator that we know of: instead of inspecting the source / AST / live objects and getting information from that, the tool runs all the tests of the project and observes what is happening using a trace hook. This means that it can present information about types that is inaccessible to other approaches (the information is not totally precise, of course). This also gives yet another reason for having thorough tests, since everything that is not tested will not be documented (as well as broken, as per "that which is not tested is broken"). A slightly out-of-date version of the resulting web pages can be seen at: http://johnnydebris.net/pyapi/ Another thing which was presented to the wider public for the first time is the build tool that Guido has been working on in the last months. The basic idea is that people can donate spare cycles on their machines by hooking them up to codespeak. Other people can then schedule PyPy translations which are distributed to one of the free machines. This has now reached the point where it basically works, but a lot of rough edges remain. So far it proved very useful to shine some, but not enough, light on a rather vicious and old annotation bug. We expect this tool to be useful for exploring the rather large configuration space of translation options, especially when it comes to inlining, which now has three more or less independent parameters. Samuele and Carl Friedrich ignored the fact that it was the breakday to work a bit more in the Terminology Production department. What they came up with is "shadow tracking" which does not have anything to do with OpenGL or real-time shading but is cool anyway. The idea is that every instance knows whether it shadows any attribute of any class in its MRO. This is useful information because it allows skipping the lookup in the instance dictionary if the attribute was already found in the class (a class lookup is already the first thing that is done, because of data descriptors). This led to a speedup of something like 10% for the richards benchmark, nearly no change for Pystone (which is hardly surprising since it is not an object oriented benchmark). Later they tried to implement method caching to save the lookup on the type too, at least for the most commonly called methods. This was a bit painful and lead to a rather small speedup of about 4% for richards. The hope is that this sort of optimization might also help the JIT later. Our other Summer-of-PyPy student (he is from Brazil, so it is summer for him), Leonardo, continued to explore the dark corners of the ECMAScript "specification". Antonio, Guido, Maciek and Armin all gave him moral support at various points in the sprint. A large success was that he managed (once) with Anto's help to translate the interpreter to .NET and C. It is still relying on Narcissus and Spidermonkey for parsing (parsing is boooring!). Leonardo is now working on increasing conformance and finding ambiguities in the specification. The subject of run-time modifiable syntax has been around for a long time in the PyPy world, and after a dormant period is waking up again. If you restrict yourself to py.py you can now for example add a do-while loop at runtime and during the sprint, a mixed on-/off-site team of Adrien, Michael and Sylvain worked on making this code translatable again, which involved the usual multi-layer confusions. Anto worked on improving the speed of pypy-cli by adapting and debugging various backend optimization to work with ootypes. The resulting fastest pypy-cli he got (involving --faassen and the Microsoft runtime) is roughly 10 times slower than CPython and 6 times slower than IronPython (again on the richards benchmark). Antoher step on the road to world domination. For the non-technical part, the sprint has also seen quite some skiing, long walks through the snowy landscape, star-gazing, several evenings of watching strange movies (polish karate being the strangest), lusting after Swiss Army knives with USB sticks built in and a whole night of "just one more game" of Durak (a russian card game Carl Friedrich wasted his school years with). Atenciosamente, Carl Friedrich & mwh

3 2