Mailman 3 February 2003 - pypy-dev

ann: website eventually
by holger krekel 07 Feb '03

07 Feb '03

Hello, finally after quite some hacking here is the first incarnation of our web site: http://codespeak.net/pypy I hope it is stable although i did quite some things to moinmoin, roundup and mailman to make them all adhere to a consistent CSS among other stuff. The site currently focuses on the upcoming sprint but can easily be extended with new pages. Troy's summary is found under the 'lists' menu link. I hope you enjoy the new user interface to moinmoin although the UserPreferences aka login-screen is still the old one. Follow the advice on the "moin" entry page to cope with it. regards, holger -- "you can have it fast, cheap or high quality. pick two." (from the whiteboard at infrae, nl)

3 3

Re: [pypy-dev] pypy-dev List Summary - 06 Jan 2003 to 31 Jan 2003
by roccomoretti＠netscape.net 07 Feb '03

07 Feb '03

holger krekel <hpk(a)trillke.net> wrote: >Reading the summary i noticed that we really have a lot of loose >ends but i am probably not the first one to notice that. Would you (or anyone else, for that matter) mind posting a list to start people thinking/discussing? We don't need to have answers yet - knowing the right question is 80% of the battle. -Rocco __________________________________________________________________ The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/

3 3

pypy-dev List Summary - 06 Jan 2003 to 31 Jan 2003
by Troy Melhase 06 Feb '03

06 Feb '03

This is a summary of the discussions on the pypy-dev mailing list between 06 January and 31 January 2003. The full archive of the list can be found here: http://www.codespeak.net/pipermail/pypy-dev/ Bootstrapping ------------- Holger Krekel starts the discussion in earnest with a message regarding how the project came about, and how to get Minimal Python[1] bootstrapped. Robin Becker suggests implementing the ctypes module. Thomas Heller posts a working version using libffi, and the possibility of using ffcall is also mentioned. Christian Tismer would rather use the essential ideas of ctypes, rather than the module itself, to keep Minimal Python small: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000005.html The build environment is discussed in this thread; CPython, SCons, make and friends, and Builtool are mentioned. C Core, Macros -------------- Florian Schulze asks about plans for the C core. Christian says most objects will be borrowed at first. Andrew McGregor brings up macros, and a lively discussion ensues. The consensus is that macros have already been rejected by the BDFL, and implementing them would conflict with the project goal of maintaining conformance with the language definition: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000010.html Ouroboros --------- Damien Morton suggests using the ouroboros as the basis for the project logo. In his words: "In mythology, the Ouroboros is any image of a snake, worm, serpent, or dragon biting its own tail." He posts links and descriptions, and Holger wants to add a bit of cuteness and fun: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000017.html Intellectual Property --------------------- David Ascher wants the issues of IP and license resolved earlier rather than later. He suggests the IP be assigned to the PSF, and that the license be PSF or AFL: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000042.html Road Map --------- Bengt Richter suggests a project road map, and Guido responds with a quick list of Python-already-implemented-in-Python. Christian thanks him, and suggests that Python code devoid of trickery is more suitable to Psyco. Armin Rigo agrees: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000047.html You tricksters know who you are! PyVM ---- Paul Swartz posts a link to his PyVM, a Python Bytecode interpreter. http://www.codespeak.net/pipermail/pypy-dev/2003q1/000048.html Project Name ------------ Many suggestions for an official project name were posted to the list throughout the month. In lieu of listing them all (and surely omitting some), the summary is that there is no official name yet. "Minimal Python" and "PyPython" are used commonly. Compile.c ---------- Holger wonders aloud how to remove the need for compile.c, and talks about a multi-stage VM: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000097.html Psyco in Python --------------- Edward K. Ream asks how Psyco will fit in Minimal Python. Michael Hudson and Christian do their best to answer him, and mention the tantalizing possibility of having Psyco re-implemented in Python: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000091.html Lessons from Experience ----------------------- Rocco Moretti talks about his experience trying to write a Python interpreter in Python. He mentions the difficulty distinguishing between host and target, and that use of Exceptions in CPython gave him trouble. Christian advocates the generated-C approach, and Armin believes in separating application from interpreter Exceptions: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000104.html Low-Level Data Type Representation ---------------------------------- Christian, Samuele Pedroni and Paul Rubin discuss various approaches on representing low-level data types in at the much-higher Python level. Paul suggests tagged representation, Christian expresses the need to keep the types easily deduced by the interpreter, and Samuele suggests the boxed values approach. Guido chimes in with his first-hand knowledge of tagged representation in the ABC language: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000121.html From PyPy to Psyco ------------------ Armin revisits the first goal of the project, writing a Python interpreter in Python. From there, he suggests, the "interesting things" are possible. This interpreter can be statically analyzed, and from the analysis will come a minimal compiled interpreter, a bytecode checker, and Psyco: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000125.html Edward K. Ream responds with his interpretations and suggestions for making Armin's tools available sooner. Ported Psyco ------------ The subject of Psyco on non-x86 hardware comes up, and Armin says the newer code will be flexible and easy to port: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000218.html Leo Outlines and Questions -------------------------- Edward K. Ream posts links to Leo outlines he's made for Psyco and for a C tool suite. http://www.codespeak.net/pipermail/pypy-dev/2003q1/000137.html Edward also suggests that Psyco can't be faster than compiled C, and Armin reminds him that it's a matter of perspective. Agreement is quickly reached that the point isn't relevant to the project: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000153.html More Questions -------------- Edward is full of questions, and Armin has answers for him. Armin casts more light on Psyco, and his illumination deserves special note: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000155.html Bengt Richter brings up the idea of check-pointing Psyco code, and he and Edward discuss storing the code in .pyp files. Armin likes the idea, and thinks it might be done outside of Psyco: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000162.html Restricted Language ------------------- Christian poses many questions regarding the initial restricted language. Discussion turns quickly to types: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000160.html Armin clarifies the issue by drawing a distinction between Python at the interpreter-level and Python at the application-level: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000211.html Later in the thread, Scott David Daniels advises watching for boundary conditions, and advocates a check-pointing system: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000203.html Armin revisits the need for a different approach to exception handling: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000221.html Implementation Language Questions ---------------------------------- Stephan Diehl asks why the implementation could not be written in Objective C. Christian responds that the idea is to remove C, not to replace it: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000167.html Translating CPython Source -------------------------- Christian speaks from experience when he says translating CPython source and also maintaining concurrency with it is a loosing proposition. As an alternative, he suggests reusing portions of a C compiler package to provide a level of automated translation of CPython source: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000188.html Edward advises changing perspective such that the problem goes away. Thomas Heller suggests that folks pick up a CPython module or two and translate them into Python. Holger believes maintaining two sources, CPython and PyPython would be a nightmare, and Armin mentions Stackless and Jython as examples of the inherent difficulty of implementing Python from translated CPython. Armin believes the translation could be automatic for most changes to CPython given an association between custom .py files and the CPython source. Microthreads ------------ Sebastien Pierre asks if microthreads will be incorporated, and Armin says they would be easy enough, given a high-level description of the interpreter: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000219.html Minimal Sprint -------------- Armin asks if there are definite dates for the first sprint. Holger suggests the 17th to the 23rd of February, and lists a possible location in Hildesheim: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000237.html Holger later announces the Minimal Python Sprint. He cites the dates above, and the goals of the sprint: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000263.html C Emissions ----------- Logistix asks for clarification regarding the progression from byte code to C to assembly. Armin describes the two processes discussed so far: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000252.html CPython - PyPython ------------------ Holger recaps the topics decisions to date made regarding translating CPython sources. The Python language isn't to be modified, CPython takes lead wherever possible, and clean abstractions lead his list: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000239.html __builtin__, in Python ---------------------- Scott Fenton posts his replacement for the __builtin__ module: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000246.html The discussion turns to Scott's implementation of ord() and chr(), with Armin pointing out that these would be supplied to the interpreter, not by it: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000254.html Scott later says that his implementation wasn't meant to be definitive: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000291.html Bengt Richter tries to factor out some of the differences between type representation, and Scott and Samuele Pedroni discuss untyped bit vectors: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000319.html Ctypes News ----------- Thomas Heller posts news about the Ctypes module, noting that argument conversion was rewritten, and that libffi is integrated: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000310.html Python Bytecode from... GCC? ---------------------------- Thomas Fanslau asks generating Python bytecode from GCC would be possible, and Nathan Heagy says it would be difficult: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000259.html Compiler Package Notes ---------------------- Logistix posts his notes on the compiler package. Michael Hudson confirms most of his impressions: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000277.html Jeremy Hylton posts with his status on the package, and summarizes the work left to be done: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000286.html Jeremy wonders if Minimal Python would need to generate CPython-compatible bytecode, and Christian thinks it may not: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000288.html Armin posts an interesting solution to Jeremy's stack-depth computation problem. His solution is the main interpreter loop of Python in Python: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000302.html Miasma: x86 machine Code Generation in Python ---------------------------------------------- Darius Bacon posts his x86 code generator, and Holger asks him to explain the challenges and solutions: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000274.html Object-Derived -------------- Scott Fenton wonders how to implement classmethod, thinking it is only applicable to subtypes of object. Christian corrects the misunderstanding and suggests coding for old and new-style classes: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000292.html Christian also proposes an extension to the new class system. His extension would have __slots__ indicate their type and allow types to derive from nothing. Builtin Types ------------- Bengt Richter posts his ideas on "describing C structures in a canonical abstract way". Darius Bacon recognizes this as "First-Class Data-type Representation in SchemeXerox". Thomas Heller mentions these ideas are already present in ctypes: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000298.html Christian thinks of using Ctypes as the first target of a types implementation in Python. Samuele Pedroni brings up the issue of abstracting use of PyTypeObject and the interpreter internal inheritance and lookup mechanisms: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000313.html The issue of using Ctypes as the basis of a type system in Python is debated. Agreement is reached that the type system must describe types, but implementing the system is not urgent: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000351.html Holger posts his set of restrictions for the pypy-python-interpreter. Armin clarifies some of them and posts his additions: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000360.html __builtin__, Again ------------------- Scott Fenton posts his updated version of __builtin__: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000331.html The working URL for Scotts code comes a few messages later: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000333.html Holger thinks having this code pass the equivalent CPython tests would be nice, and that scripting doc string generation would be easy: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000336.html Pedro Rodriguez adds his Python implementation of the dir() builtin: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000343.html Bits with Types --------------- Bengt Richter posts initial ideas on a generic approach to evaluating machine code: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000355.html Modeled from Smalltalk ---------------------- Stephan Diehl suggests looking at Smalltalk and how it has been implemented in Smalltalk: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000356.html List Archive ------------ David Ascher sends a notice that ASPN is hosting an archive of the pypy-dev list: http://www.codespeak.net/pipermail/pypy-dev/2003q1/000371.html whew! -troy

3 4

smalltalk as a model?
by Stephan Diehl 05 Feb '03

05 Feb '03

Hello, during the last days I got a little bit interested in smalltalk (article in c't 2/2003). Since smalltalk is already about 30 years old and shares some similarities to Python, some aspects of smalltalk might ease the effort to implement Python in Python. Here are some smalltalk faclets that makes it interesting: - Everything is an object (even classes and code blocks for example) - All actions are performed by invoking an object method (even loops) - there are only 5 keywords and only a couple of "special" characters - the smalltalk VM is implemented in smalltalk - smalltalk comes with "batteries included" With smalltalk in mind, one could write a kind of ProtoPython that has much less overhead than the existing Python. Possible features of a ProtoPython (very incomplete): - smaller set of reserved words ("print","import",etc. shouldn't be reserved words) - no operators, '+','-',... will be methods - at least the for loop could be a method of a (to be defined) list class. Even if this approach is absolutely ridiculous, just looking at smalltalk might give some ideas/hints how to construct a self describing language. Some Links: Squeak (open source smalltalk runtime): http://www.squeak.org smalltalk 80 :The Language and Its Implementation (contains details about the VM): http://users.ipa.net/~dwighth/smalltalk/bluebook/bluebook_imp_toc.html [Python-Dev] Classes and Metaclasses in Smalltalk: Guido about a mail from Jim Althoff, inventor of smalltalk metaclasses http://mail.python.org/pipermail/python-dev/2001-May/014508.html smalltalkish python: http://squeak.cs.uiuc.edu/mail/squeak/msg04577.html Stephan

3 3

just joined
by Dean Hall 05 Feb '03

05 Feb '03

Hey guys, I just joined this list and thought I'd introduce myself. I'm Dean. How am I doing so far? I have a paper that I'm presenting at PyCon2003 about PyMite, my from-scratch flyweight python interpreter for 8-bit architectures (under development). The reviewer of the paper recommended that I contact this endeavor to share/exchange ideas. So, here I am. Rundown of PyMite (prepare to lower your expectations): Supports Integers, Strings, Lists, Dicts and Tuples [NOT float, long, complex]. Supports the useful 60+% of Python 2.0's bytecodes. Supports modules, classes, methods, functions [but not (yet) inheritance, varargs, or default args]. It's stackless; has a FAST inline native code interface; fits in 15KB of code on an Atmel atMega103 8-bit processor; runs fast enough to not be a pain. On the todo list: hybrid mark/sweep/coloring GC, exceptions. Source not yet available. I'll post it on SF by PyCon. It'll be open source, but I'm going to be very strict about code submissions because my size constraints are insanely small. Splinter development is OK by me (but I doubt anyone else is interested). !!Dean __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com

2 1

RE: [pypy-dev] Compiler Benchmarks
by Delaney, Timothy C (Timothy) 03 Feb '03

03 Feb '03

> From: Christian Tismer [mailto:tismer@tismer.com] > > I don't put any relevance into measurements, today, > and you shouldn't, either. > Some misbeliever muttered today, that Armin's brain has > shrunk from a planet down to an asteroid, by adopting this > PyPy project. Nice joke! Hey - I wasn't suggesting that it's because he's working on PyPy. He's in the real world now, so of course his brain is shrinking. That's why I think he needs to work on another doctorate ;) As to whether PyPy would be a suitable body of work ... Tim Delaney

1 0

List Summary Volunteer
by Troy Melhase 03 Feb '03

03 Feb '03

Hi Folks: I've been lurking on pypy-dev since the original announcement, and I'd like to help out in some way. I'm not a compiler writer, nor a C coder, but I think I can write a summary for the list traffic once a week. I assume that simply volunteering is enough to be given this responsibility. Sometime Saturday, I'll summarize the traffic from the last three weeks and send them to the list and to c.l.p for group review. Comments? -troy

4 7

Compiler Benchmarks
by logistix 03 Feb '03

03 Feb '03

As I've mentioned before, I've been goofing around with a parser written in pure Python. After getting sick of debugging the old one (like 70 stack frames down) I decided to hand-code one from scratch. I just ran some benchmarks and thought there might be some interest in the results. Something as processor intensive as compiling would also probably be a good benchmark for various incarnations of PyPython vs CPython. I compiled the contents of the Lib directory (but not subdirectories) with the builtin compile, the compiler module's compile, and a hacked compiler module that used my parser on a 866Mhz machine running WinXP. The builtin compile took an average of 2.9 seconds. The compiler module took an average of 105.3 seconds, or 36.3 times as long. The compiler module uses CPython's parser, but other than that it's in pure Python. The hacked compiler module took an average of 177.2 seconds or 61.1 times as long. It has a pure Python module, but relies on tokeize which leaves re as the only C dependancy left. Although I have no desire to write re in python (can you guess why?), I could hand-code a DFA for python tokens if people think there's a need. -logistix

4 3

Re: [pypy-dev] Compiler Benchmarks
by logistix 03 Feb '03

03 Feb '03

holger krekel <hpk(a)trillke.net> wrote: > > > Here's the code if you're interested. As I mentioned, you'll need at > > least 2.3a1. The instructions to tie it into the compiler modules are > > in the docstring: > > > > http://members.bellatlantic.net/~olsongt/python/pparser.py > > Sure looks interesting. You do a lot of function calls, right? > > > -logistix > > > > P.S. If anyone wants to patch this in and run Tools/Compiler/Regrtest.py > > against it, results would be appreciated. > > i ran the Lib/test/test_parser against your module but it choked mostly > (on missing totuple methods among other stuff). > > Regrtest is currently running with no problems. (it takes some time, > though :-) > My Regrtest is locking up on test_pickletest (or something like that) Also saw a bug in simple_stmt. > If this is really a quite complete parser then pparser.pyc > is a lot smaller than the equivalent parsermodule.o. > not that this means too much :-) > > does your pparser work similar to parsermodule.c ? > Actually parsermodule doesn't do any of the work. Here's my non-authorative explaination of how CPython parses, as I'm still figuring it out myself. There's a utility Pgen, which is Python's custom Parser Generator similar to Yacc or Bison. It uses grammar/grammar as input, and generates graminit.c. Graminit.c will hurt your brain if you try to figure out what it's doing (also highlights some of the perils of automatic code generation). Parsetok.c takes the generated grammar in conjunction with some tokenized text, and builds the AST tree based on these two inputs. Parsermodule.c just provides a Python interface to the above internals. The bulk of the code is validation code, so that if you try to compile a hand-built ast, it'll throw an error there instead of blowing up the compiler internals. It's actually been really handy for me to debug the stuff I've been writing. Hand-coding the parser, like I did, is more likely to result in subtile bugs creeping in and is harder to maintain than PGen if your source grammar is changing alot. Right now Python's grammar is reasonably stable so it's not too bad. I previously tried to write a Parser Generator in pure Python, but it had all kinds of bugs and I'm still not entirely comfortable doing serious low-level debugging in Python. I also noticed some comments in PGen that indicated they used a "unique" method of reducing the grammar.

1 0

Re: [pypy-dev] Compiler Benchmarks
by logistix 03 Feb '03

03 Feb '03

holger krekel <hpk(a)trillke.net> wrote: > > [Oren Tirosh Sun, Feb 02, 2003 at 01:26:57PM -0500] > > On Sat, Feb 01, 2003 at 07:34:44PM -0800, logistix wrote: > > > I compiled the contents of the Lib directory (but not > > > subdirectories) with the builtin compile, the compiler module's > > > compile, and a hacked compiler module that used my parser on a > > > 866Mhz machine running WinXP. > > > > > > The builtin compile took an average of 2.9 seconds. > > > > > > The compiler module took an average of 105.3 seconds, or 36.3 > > > times as long. The compiler module uses CPython's parser, but > > > other than that it's in pure Python. > > > > > > The hacked compiler module took an average of 177.2 seconds or > > > 61.1 times as long. It has a pure Python module, but relies on > > > tokeize which leaves re as the only C dependancy left. > > > > To get a (very rough) estimate of the performance we can expect from > > a PyPython compiler it would be interesting to know how fast this code > > runs with psyco. > > I agree that this would be interesting even though the PyPy-compiler > might run on a quite different PSYCO. I can imagine modifying the > compiler to implement the same restrictions that we might set on the > PyPy-Interpreter. I can't wait to begin hacking on this. > > logistix, is your parser/compiler code available anywhere? > > holger > I just wrote the parser, the compiler code is already in the base distribution. There were a few bugs that turned up in Python2.2, so that's why I didn't run psyco benchmarks. Just using the standard compiler module in 2.2 with psyco didn't yield much of an improvement (108 seconds vs 115). Here's the code if you're interested. As I mentioned, you'll need at least 2.3a1. The instructions to tie it into the compiler modules are in the docstring: http://members.bellatlantic.net/~olsongt/python/pparser.py -logistix P.S. If anyone wants to patch this in and run Tools/Compiler/Regrtest.py against it, results would be appreciated.

2 1