Mailman 3 November 2006 - pypy-dev

A new idea?
by Paul deGrandis 01 Dec '06

01 Dec '06

Hi everyone, I started to work on PyPy about a year to a year and a half ago, but had to quickly drop the project because of school and academic research I'm involved in. I really feel that even though PyPy is still at a stage of rapid development and evolution it's about time that people start really illustrating the true power of pypy. I'd like to be that guy for you. My idea is simple, I'll write small but useful applications that really illustrate the powers of pypy. These small applications can be loosely strung together as tutorials of language features or just examples of pypy's superiority. For some time now, I've developed small extensions to CPython in form of models, c extensions, and decorations to add features to the language that I find useful. However, Pypy's architecture naturally lends itself to these tweaks and in some cases supports them already. So basically, I want to know your thoughts on this. And if you have any ideas for applications or features you'd like to see illustrated please let me know! Regards, Paul

2 4

Some new first thoughts about pypy
by Edward K. Ream 30 Nov '06

30 Nov '06

The pypy tools promise to move programming (not just python programming) in new directions. They may show a way to realize an ancient dream of mine, viz., to eliminate the gc for some (most?, all?) objects. For details, see: http://webpages.charter.net/edreamleo/whitepapers.html#allocating-storage-u… Of course, nothing at all may come of this; it's just something to keep in the back of my mind. Your (gentle) comments are welcome. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo(a)charter.net Leo: http://webpages.charter.net/edreamleo/front.html --------------------------------------------------------------------

2 1

llvm backend issue on AMD64
by Alexandre Fayolle 30 Nov '06

30 Nov '06

Hi, While running translations with various options on an AMD64, I encountered issues with the llvm backend: the translation fails when llvm-as is called. The error message is:: [translation:ERROR] llvm-as: <stdin>:586398: Redefinition of function 'LL_os_fstat'! In entry_point.ll, I have a forward declaration of LL_os_fstat:: declare fastcc %structtype_tuple10* %LL_os_fstat(long) and indeed 2 definitions of the function. The first one is at line 449952, and it has a structure very similar to the definition I get when I translate in 32bit mode (but takes a long instead of an int as argument). The second one looks like:: internal fastcc %structtype_tuple10* %LL_os_fstat(long %tmp_13431) { %tmp_13433 = cast long %tmp_13431 to int %tmp_13432 = call fastcc %structtype_tuple10* %LL_os_fstat(int %tmp_13433) ret %structtype_tuple10* %tmp_13432 } If I comment out this second definition, I get a similar error regarding LL_os_lseek, LL_os_fork, etc. -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/

5 6

Crash starting pypy on XP
by Edward K. Ream 29 Nov '06

29 Nov '06

Hello all, Excellent documentation at: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#downloading-ru… Alas, when I execute python pypy/bin/py.py from C:\pypy-0.9.0 on XP I get: C:\pypy-0.9.0>python pypy/bin/py.py Traceback (most recent call last): File "pypy/bin/py.py", line 207, in <module> sys.exit(main_(sys.argv)) File "pypy/bin/py.py", line 78, in main_ space = make_objspace(Options) File "pypy/bin/py.py", line 66, in make_objspace compiler = cmdlineopt.compiler, File "C:\pypy-0.9.0\pypy\interpreter\baseobjspace.py", line 164, in __init__ self.initialize() File "C:\pypy-0.9.0\pypy\objspace\std\objspace.py", line 98, in initialize w_mod = self.setup_exceptions() File "C:\pypy-0.9.0\pypy\objspace\std\objspace.py", line 219, in setup_exceptions mod, w_dic = self.create_builtin_module('_exceptions.py', 'exceptions') File "C:\pypy-0.9.0\pypy\objspace\std\objspace.py", line 164, in create_builtin_module w_dic = PyPyCacheDir.build_applevelinterp_dict(fake, self) File "C:\pypy-0.9.0\pypy\interpreter\gateway.py", line 840, in build_applevelinterp_dict cls._setup() File "C:\pypy-0.9.0\pypy\interpreter\gateway.py", line 895, in _setup p = lp(pypy.__file__).new(basename='_cache').ensure(dir=1) File "C:\pypy-0.9.0\py\path\local\local.py", line 276, in ensure return p._ensuredirs() File "C:\pypy-0.9.0\py\path\local\local.py", line 265, in _ensuredirs if self.check(dir=0): File "C:\pypy-0.9.0\py\path\common.py", line 102, in check return self.Checkers(self)._evaluate(kw) File "C:\pypy-0.9.0\py\path\common.py", line 75, in _evaluate if bool(value) ^ bool(meth()) ^ invert: File "C:\pypy-0.9.0\py\path\local\local.py", line 35, in dir return stat.S_ISDIR(self._stat().mode) File "C:\pypy-0.9.0\py\path\local\local.py", line 29, in _stat self._statcache = self.path.stat() File "C:\pypy-0.9.0\py\path\local\local.py", line 285, in stat stat = self._callex(os.stat, self.strpath) File "C:\pypy-0.9.0\py\path\common.py", line 205, in _callex return func(*args) WindowsError: [Error 2] The system cannot find the file specified: 'C:\\pypy-0.9.0\\pypy\\_cache' Any help getting started would be appreciated. Thanks. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo(a)charter.net Leo: http://webpages.charter.net/edreamleo/front.html --------------------------------------------------------------------

7 9

JIT improvements for RISC processors
by Fabrizio Milo aka misto 15 Nov '06

15 Nov '06

Hi to everyone, I am really interested in code machine generation for risc processors. In order to have an idea of the work that can be done, and to create a proposal I will ask some questions: - What is already done? where is it? (paths, file names please) - What ideas are already defined and need just to be implemented? - Material that I have to / should read? I have a rough background on RISC assembler, the most comes from MIPS and IBM's PowerPC some material I am aware of is http://tinyurl.com/9dw43 Don't let me escape for the second (last?) time! :) ---------------------------- * P~ython Addicted * www.fabriziomilo.it

2 1

interesting paper on a object-oriented language for very constrained embedded systems
by Samuele Pedroni 09 Nov '06

09 Nov '06

http://compilers.cs.ucla.edu/virgil/virgil-oopsla06.pdf <http://compilers.cs.ucla.edu/virgil/virgil-oopsla06.pdf> RPython for different reasons or by different means does similar things already (devirtualisation, separate flexible load time) and could be adapted to implement other aspects/optimisations as well, under the no new allocations assumption. regards.

1 0

Summer of PyPy Call for Proposals
by Carl Friedrich Bolz 08 Nov '06

08 Nov '06

Last chance to join the Summer of PyPy! ======================================= Hopefully by now you have heard of the "Summer of PyPy", our program for funding the expenses of attending a sprint for students. If not, you've just read the essence of the idea :-) However, the PyPy EU funding period is drawing to an end and there is now only one sprint left where we can sponsor the travel costs of interested students within our program. This sprint will probably take place in Leysin, Switzerland from 8th-14th of January 2007. So, as explained in more detail at: http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html we would encourage any interested students to submit a proposal in the next month or so. If you're stuck for ideas, you can find some at http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html but please do not feel limited in any way by this list! Cheers, Carl Friedrich Bolz and the PyPy team

1 0

Duesseldorf sprint report
by Carl Friedrich Bolz 05 Nov '06

05 Nov '06

Hi all! it is with a familiar level of tiredness that we bring you these lines. We are again sitting in one of the rooms of the Institut für Informatik, after 6 days of sprinting. As usual, it has been a busy and productive (and sometimes strange) sprint. One of the new developments of the sprint was the work of Leonardo Santadaga, our "Summer" of PyPy student from Brazil. Leonardo proposed to write a JavaScript interpreter, had his proposal accepted and now gets his travel to sprints funded. This work has seen good progress every day, so that we now have an interpreter that handles simple snippet of JavaScript code. Leonardo had help from various other people changing over the course of the sprint such as Maciek, Guido (the reluctant Master of JavaScript) and Stephan. The parser is currently stolen from them Narcissus project, and the interpreter does not translate yet. For less than a weeks work though we think are doing pretty well (we are trying not to distract ourselves with crazy thoughts like translating a JS interpreter to JS or wondering how fast it would be after applying the magic JIT technology). Although Leonardo will be flying back to Brazil soon he will continue working on it (at least if he finds sufficient time between caring for his beloved new MacBook). The other PyPy sprint virgin was Niko Matsakis, a graduate student at ETH Zürich. To start with he worked with help of Antonio on the fledgling JVM backend. Antonio and Niko worked on moving code out of the CLI backend to be shared with other object-oriented backends. They got as far as supporting nearly everything except constants (which as usual turns out to be the hardest thing to support). The team was split up later in the week to work on Other Things. Niko only found out after he _left_ the sprint that Samuele is one of the authors of Jython, which he is interested in since some of his students are working on it. Another topic that received attention all week was the JIT (dramatic music). The architecture of the JIT has now crystallized enough to be able to split up the work into the (for mere mortals largely incomprehensible) "time shifting" front end and the (much more straightforward) code generating backends. Samuele and Arre started the week by adding some missing pieces for things called "portals" in the front end. A portal is the part of your code that marks the entry into JIT-land. For the PyPy interpreter it is expected to be the bytecode dispatcher (or in general the main loop of an interpreter). The JIT tests now take so long that they also had time to work on translating rsocket, which is an RPython-level socket library, and using it for implementing the socket module. For the first few days of the sprint Armin and Richard took a DFA engine (deterministic finite automaton) that Carl Friedrich wrote to lex Prolog code and adapted it to be jittable. On Wednesday Armin tried cast some light on the dark internals of the JIT with a short talk about this example. The DFA engine is viewed as interpreting the specification of the automaton and then turned into a compiler by the timeshifter. This compiler does not work in advance. Instead calls to the compiler are inserted into the transformed version of the DFA engine. If the DFA engine is used to match a string against the automaton, the compiler produces somewhat efficient machine code for doing that. Richard's work (with the help of Armin and Arre) for the rest of the sprint can be summarized as improving the timeshifter to be able to "un-adapt" this DFA engine until it at least resembled the original code. This involved adding a new piece of terminology to the ever-growing JIT jargon "deep-freezing" (something to do with structures that are so constant that even their contents are constant). Concurrently, and with its own set of problems, Michael worked on the JIT backend all week. He started together with Armin to document all the sometimes non-intuitively named methods of the backend API, which has only emerged in the last month or so. They renamed the most strange ones afterwards. After that Michael and Eric did a little polishing of the PowerPC backend and began to investigate writing an LLVM backend (C++ is hard). Eric had to leave mid-sprint so Michael continued with Niko (another Mac fanatic with insufficient funds to have a MacBook) to improve the PowerPC backend. This involved fights with the calling conventions, writing a greedy register allocator and lots of time waiting for tests (they didn't manage to find a nice side-project). At the end of the week the PPC backend can actually handle no more programs than it could at the beginning, but we are very much happier about its foundations. After the break-day Samuele and Maciek started working on the project that would occupy them for the rest of the sprint: adding "transparent proxy objects" to the PyPy interpreter. A transparent proxy is something that claims to be of a specific type (e.g. a list) but all operations are forwarded to a controller which can choose how to implement those in an arbitrary way, such as retrieving the values from a database, a remote host or similar. For most operations the multi-method based implementation of the standard object space made this relatively straightforward. For the black-sheep operations this was some more work but this is still something that, according to Samuele, "if you tried this with CPython it would take so long that you wouldn't want it anymore when you're finished". The motivation for writing these proxy thingies is to experiment with transparent remote objects, orthogonal persistence and security (there is also a plan to steal the pygame from CPython by using them). There was a little work on the side on the upcoming "apigen" tool, which is a new part of the py-lib. This is something Maciek, Guido and Brian Dorsey started before the sprint. It runs all the tests of a package (currently only the py-lib) and observes the behaviour of the running code. From this it generates API documentation with additional information like what types the arguments of the functions had, which attributes of instances were changed... Carl Friedrich spent quite some time on the deeply brain-taxing task of populating the new rlib directory by moving various files there and fixing all the imports. The rlib directory is meant to contain a library useable from RPython programs, so the code in there is all RPython and is supposed to make up in some way for the lack of a standard library in RPython. Such modules have been accumulating in various places where they don't really belong. Samuele, Anto and Carl Friedrich had a planning meeting for the integration of the .NET framework with the Python interpreter that Anto plans to do. This means being able to call .NET functions and use .NET classes from pypy.net. Samuele presented the two major approaches: The simpler, but more annoying one is to write wrapper classes that have the actual .net object as an attribute. For this you need to do conversions at the language boundaries and thus suffer a performance penalty. The other approach is to identify some of the classes of the PyPy interpreter with .NET classes so that you can pass them around freely. Guido and Carl Friedrich worked a small bit on the build tool, which supposedly gives people a way to build PyPy on remote hosts. This is now somewhat working modulo lots of annoying real-life details (error-handling being one of them). Together with Samuele they had a discussion about what to do in this area in the future. Various realistic ideas (such as a web frontend) and unrealistic ideas (having a quake configuration tool where you can hunt down and kill configuration settings that you don't want) were discussed. A very positive aspect of the sprint is that Christian who has been ill for quite a while is back with us and now finding all the things we have broken on windows in the meantime. Carl Friedrich with the help of Anto and Guido spent a big chunk of time near the end of the sprint in refactoring PyPy's file implementation. The first step was converting the stream implementation which was living at app-level before to be RPython and thus useable by other RPython programs (such as Prolog interpreters, hint hint). Afterwards they converted the file implementation to use the new facilities, which involved the usual fighting with the annotator to make it work. This should also make it possible to remove some horrendous hacks from the .NET backend, as the new file implementation should make it easier to move away from using a POSIX-like model (actually the .NET backend contained a POSIX compatibility layer written by Anto just for this purpose). Since we cannot seem to come up with a witty closing remark, we just stop. Ade, Carl Friedrich & mwh -- "... and the end result of all this is a dating site that matches people according to the sort of PyPy they compile." -- Samuele Pedroni

2 1

schedule()
by Armin Rigo 02 Nov '06

02 Nov '06

Hi Aurelien, It seems that the logic objspace has had now for a long time a schedule() function, instead of just doing the preemptive scheduling automtically. As we've always said, it is very easy to do the preemptive scheduling bit, but as you never followed-up on that, I guess I have to ask again. Do you want me to add a hook in the logic objspace that calls schedule() every Nth bytecode instruction? A bientot, Armin

3 3

IRC server unreachable
by Alexandre Fayolle 02 Nov '06

02 Nov '06

Hi, am I the only one unable to join any freenode IRC servers ? I get connection refused errors on servers listed on http://freenode.net/irc_servers.shtml Or has the IRC network for pypy changed ? -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/

2 1