Mailman 3 January 2003 - pypy-dev

How much do we know?
by Edward K. Ream 18 Jan '03

18 Jan '03

Is there anyone out there (besides Armin) who believes he or she knows enough about psyco to: 1. Explain in detail how it works? 2. Modify the code? 3. Propose improvements? If so, would you consider sharing your view of psyco? What are the key ideas? If not, doesn't this suggest something that might forward the action of this project? Edward P.S. I've very much enjoyed the background links that various people have given out. EKR -------------------------------------------------------------------- Edward K. Ream email: edream(a)tds.net Leo: Literate Editor with Outlines Leo: http://personalpages.tds.net/~edream/front.html --------------------------------------------------------------------

1 0

Leo outlines & psyco
by Edward K. Ream 17 Jan '03

17 Jan '03

I have just uploaded to Leo's web site two outlines that may be of interest to this project. 1. pysco.leo (in pysco.leo.zip) is a Leo outline I have been using to study psyco. It contains the code for psyco, and the code for the main loop of the C interpreter. Notice how using sections makes the structure of both the C interpreter and the corresponding psyco compiler clear. It also contains a section allowing you to compare, on a case-by-case basis, the cases of the psyco "compiler" with the cases of the C interpreter's main loop. This shows just some of the power of clones...NOTE: This outline is based on code that is several months old. I include it mostly to show how Leo could be applied to this project. 2. CC2.leo (in CC2.leo.zip) contains most of the C source code for an optimizing C compiler, assembler, linker and loader written in the early 90's for the now-defunct Tuple corp. I am co-owner of this code (not that it has any commercial value). Again, I include this outline to show how useful Leo would be in a compiler project. Examples: a. The documentation section in CC2.leo contains all the documentation in a single place, and parts of the documentation are cloned and placed near the code to which the documentation pertains. b. If I were writing this code today I would use sections extensively to clarify the code. For the most part, this code is exactly as produced from Leo's import script, so it isn't nearly as clear as it could be. However, I have reworked the complex tokize routine (in the @file CCtokize.c tree in the "Tokens & preprocessor" section to show the kinds of things that can be done. c. CC2.leo contains a _huge_ amount of code and documentation. With Leo it is easy to use clones to focus on the task at hand and ignore everything else. You may find both pysco.leo.zip and CC2.leo.zip at: http://sourceforge.net/project/showfiles.php?group_id=3458 under the headings: Miscellaneous: CC2 & Psyco. In order to use these outlines (.leo files) you will need a copy of leo.py, which you can find at the url above under the headings: Leo2.py: 3.10. If you have Windows you can just download the single-click installer, leosetup.exe. Otherwise, please download leo-3.10.zip. I am no salesman, and I am always leery of touting Leo. However, I do think that Leo would really contribute to this project. I hope no one objects to this posting. If so, I apologize in advance :-) Edward P.S. I think using Leo for the primary source for a project like this makes sense. Leo is ideally suited to managing and clarifying complex code. Moreover, I would like to see Leo used (eventually, not now) to manage the entire Python project. However, at present there are issues with using Leo with CVS, so I think it makes sense to use Leo on a strictly experimental basis on an experimental project like this. It will be very easy to use Leo on this project. If there is any interest I shall explain the simple rules for doing so. I believe the cvs issues with Leo will largely be resolved in about two months when Leo 4.0 comes out. P.P.S For the best example of what Leo can do, look at Leo's own source code, LeoPy.leo, included in all distributions of Leo. In particular, look at the (Project Views) section of LeoPy.leo to see how to use Leo's clones to keep track of tasks (like bugs and other ongoing issues). EKR -------------------------------------------------------------------- Edward K. Ream email: edream(a)tds.net Leo: Literate Editor with Outlines Leo: http://personalpages.tds.net/~edream/front.html --------------------------------------------------------------------

1 0

Re: [pypy-dev] Re: [ann] Minimal Python project
by Paul Rubin 17 Jan '03

17 Jan '03

It can avoid lots of allocations, but also turns every object access into some macro processing. for a language that I have to code by hand, I don't like that so much. It's not really that bad. Most of the time, the macro doesn't do anything. In fact, wouldn't the pypy compiler take care of it automatically? The macro is only an issue in the low level C interpreter. For the bootstrap phase, I'd say hands off from any changes of the model. First we redefine the world identically "in place". When that works, we can try changing worlds. I'd think the object representation is pretty fundamental, so once you choose something, it will take a lot of rework to change it. Hopefully you can prepare for that by writing flexibly. > all other objects would be boxed, and the address left shifted by 1 > and ORed with 1. Humm. I would use the left shifted integer, or'ed with 1. After testing that bit, Arithmetic right shift gives the integer. That allows to use the address variant without any operation and has the nice debugging facility that addresses are already addresses, and object inspection works directly. Yes. Also, you can add the shifted integers together without having to unshift or mask them. You can also use 2 or 3 tag bits, and encode other types in a single word. For example, Python probably uses an awful lot of 1-character strings, so boxing those could do some good. Olin Shivers article about T is pretty inspirational--you might look at it: <http://www.paulgraham.com/thist.html>. p.s.: Paul, would you mind to participate in pypy-dev, then we can avoid crossposting. I guess I can live with this, though I'd rather use the newsgroup. Is crossposting so hard? If you want to put me on a mailing list, please use the address phr-pypy(a)nightsong.com so I can auto-route it. (Btw I don't like the name pypy much. There are several others in that thread which I like a lot better). There's another guy I'd also like to invite, a Lisp expert, if that's ok with you. He's been interested in writing a Python compiler for a while. I'll ask him if he wants to join, but he might not. How much traffic is on the list? Paul

3 2

Re: [pypy-dev] Concrete Syntax Tree
by Grant Olson 17 Jan '03

17 Jan '03

>Grant Olson wrote: >> I wrote some rough python code a few months ago that built Concrete >> Syntax Trees from the grammar file in the Python source, and also >> turned token streams into Abstract Trees based on the Concrete Trees. >> The parser module successfully compiled the Abstract Trees. Is this >> something you guys would be interested in or am I off base here? > >I would love to read that, of course! > >yours - chris Available at: http://members.bellatlantic.net/~olsongt/concrete.zip A little bit slower than the builtin parser ;) --------------------------------------- Get your free e-mail address @zworg.com

1 0

Fw: [pypy-dev] Re: [ann] Minimal Python project
by Samuele Pedroni 17 Jan '03

17 Jan '03

From: "Michael Hudson" <mwh(a)python.net> > > > integer in the range [-2**30,2**30-1] would be represented by their bit > > patterns shifted left by 1 bit > > > > all other objects would be boxed, and the address left shifted by 1 and ORed > > with 1. > > I seem to recall someone tried that with python quite recently and > found it was a pessimization (search python-dev, I guess). > > I'm not sure that's what Paul meant, though. he wrote tagged representation, and there was an e.g. in front of the example above, I don't claim it does wonder, even less together with psyco or with reference counting.

2 1

Re: [ann] Minimal Python project
by Christian Tismer 17 Jan '03

17 Jan '03

Paul Rubin wrote: > Christian Tismer <tismer(a)tismer.com> writes: > >>This leads to high-level descriptions of low-level >>fields of all structures. [...] > Do you have some other examples that really get used often? I'd think > f_lineno's don't get used that often. This was just an example for a common field (in fact it is used quite often at runtime, but almost only in the SET_LINENO opcode), that is embedded into some other structure. Everything else would to. The idea is to describe fixed data structures in a way that allows Python to easily deduce what the data type is, without any trial. See for example Thomas Heller's ctypes module, which has such descriptions for everything. A drawback of this is that we always need to access target variables in dotted notation. Otherwise we loose the static type info. If Python had type declarations, this would be no problem. Here an example from intobject.c: static int int_compare(PyIntObject *v, PyIntObject *w) { register long i = v->ob_ival; register long j = w->ob_ival; return (i < j) ? -1 : (i > j) ? 1 : 0; } Assuming that we try to model this in Python, the resulting code might look like this def int_compare(v, w): i, j = v.ob_ival, w.ob_ival if i < j: return -1 elif i > j: return 1 return 0 The above code only occours in contexts where integer objects are passed in. There is no type info in advance, but at the first execution of this code, v and w are passed in with their descriptions of all fields, and it is now absolutely clear that i and j are values of fixed sized integers. Code for directly accessing the ob_ival fields and doing the comparison can immediately be emitted when running the code first time. A remaining problem with the lack of declarations are local variables which are not members of a structure and it is not clear from the beginning what the primitive type should be. One ugly way would be to construct a structure for the local variables and to use dotted notation again. I hope this can be avoided by propagation of type info via __coerce__. Another snipped from intobject: for (i = 0, p = &list->objects[0]; i < N_INTOBJECTS; i++, p++) { if (PyInt_CheckExact(p) && p->ob_refcnt != 0) irem++; } Given a type object named c_int, this might translate to i = c_integer(0) p = list.objects while i < N_INTOBJECTS: # body not implemented here i += 1 p += 1 Here I use a known class as initialization of i. The data type is therefore known. p as a pointer field in the list structure is also known. The __coerce__ method of these classes can be written in a way, that they always propagate their own class to other operands, and especially in this case, the right operand is a constant. Given a definition like this: class c_integer(c_datatypes): def __coerce__(self, other): if type(other) == int: return self, c_integer(other) elif .... What I tried to express is that with little or no help of the programmer, primitive data types can be deduced quite easily, and unique code can be emitted on first execution of the code. > Will you give some thought to going to a tagged representation of > object references, and maybe a more traditional GC? That way you > avoid a lot of memory traffic, and also don't have to adjust reference > counts every time you touch anything. > > I think these would give a big performance boost, and also would also > simplify the C API. It might be possible to supply a compatibility > layer to make the old C API still useable. Can you give me a hint how this would be done? I have no experience with tagged representations, but if this can save so much, I should learn more about it. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

3 3

Pypy Roadmap
by Bengt Richter 16 Jan '03

16 Jan '03

Hi, this sounds interesting ;-) I was wondering how you were planning on identifying concrete tasks and assigning and tracking them, and keeping status visible to support cooperation. Do you already have CVS and bug/issue tracking methodology settled? I.e., such stuff is nuisance overhead or sanity preservation depending on the task, and the development phase, and scale/distribution of effort. Is there one single home page that leads to everything relevant, with some indication of historical vs current vs fire alarm links? Googling pypy got me (in two steps) to http://twistedmatrix.com/users/jh.twistd/moin/moin.cgi/PyPy Should that page be "claimed"? Here are some things that occur to me to think about in making a roadmap for your project. If you would make something like it to reflect your _actual_ project plans, it would help me (and maybe some other lurkers ;-) get an idea of what you are really up to ;-) 1.0 Historical background 1.1 Psyco 1.2 core group forms and evolves pypy idea 1.3 core group decides to go public, announces pypy-dev list 2.0 Initial pypy-dev discussions 2.1 Happening now ;-) 2.2 Preliminary expressions of interest, ideas 3.0 Straw-man roadmap 3.1 Decide hosting, bug/issue tracking methodology issues 3.1.1 Designation of primary info channels 3.1.1.1 Discussion - pypy-dev list, presumably 3.1.1.2 Current task/subproject/bugfix assignments/status 3.1.1.3 Single official page as comprehensive root of info tree 3.1.2 Sourceforge? 3.1.2.1 mirror/backup/file authentication issues? 3.1.3 pypy SDK? Recommended win32 & unix tool kits, compiler versions etc. 3.1.4 Windows vs unix issues 3.1.4.1 Maintain parallel MSVC++ project files like CPython's PCbuild directory? 3.1.4.2 Platform-independent builds? 3.2 Language issues 3.2.1 Decide on minimal core language bootstrap subset? 3.2.1.1 Identify needs for supporting 4.x below 3.2.1.2 Threading/locking/synchronizing issues? 3.2.2 Compiler hints/directives - what info & how? 3.2.2.1 command line opts? 3.2.2.2 config files? 3.2.2.3 source-embedded directives/pragmas/etc? 3.3.3.4 Preprocessing? 3.3 Other implementation issues? 3.3.1 Foreign function/C interface 3.3.2 New VM? Intermediate language representation? 3.3.3 Memory allocation/garbage collection/reference counting 3.3.4 Resource monitoring (guaranteed timely finalization?) 3.3.5 Security support 3.3.5.1 Sandboxing, restricted execution, resource quotas? 3.3.5.2 Special considerations for CGI/server contexts? 3.3.6 Checkpointable execution support, fastload images? 3.4 How to minimize platform C library dependencies for bootstrap core? 3.4.1 Strategy for weaning from temporary uses 3.5 Decide on initial build targets 3.5.1 Executables: Win32, unix, bootable images? 3.5.1 Libraries: DLLs, .so's 4.0 After the minimal core bootstrap language works 4.1 Re-implementation strategy using core boostrap language to write next level of language features. 4.1.1 How many levels of subset bootstrapping? Is it a hierarchy? 4.2 Special language requirements for writing what is now C in Cpython? Well, that's all the thoughts I have for now ;-) Regards, Bengt Richter

4 4

bootstrapping issues
by holger krekel 16 Jan '03

16 Jan '03

Welcome to the Minimal Python list, It appears (to me) that Armin and Christian are both busy for the weekend so i start some bootstrapping. First five paragraphs of background. The idea of actually starting to do "Minimal Python" arose from a thread in python-de (2003q1 first post). There soon was the idea of doing a sprint with the goal of releasing first bits. Armin, Christian and me realized early that opening up the discussion (instead of in private mails) is a very good idea as there is lots of experience and code out there. Nevertheless, i'd like the attendants of the planned sprint to decide which route they want to follow coding-wise eventually. We will propose some dates by the end of the month, btw. I think that - if in doubt - we follow the lead from CPython e.g. regarding coding style, language definition etc. Every deviation should be justified by the goals. I don't think we should follow a PEP process soon, though. During January i want to setup accounts and a cvs-repository. The involved policies will be liberal. ok. enough organizational things for now. Bootstrapping will be a hot topic for Minimal Python itself. We probably first want to use the CPython development environemnt to get going. We probably want to base it on the python-2.3 cvs tree. There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal. For avoiding the need of C-coded system modules there is interest to code a generalization of the "struct" module which allows *calling* functions at the C-level. This general C-"Extension" will be system-dependent. The gateway code between the "machine" and python probably needs to be coded in assembler: You can't "construct" a C-function call within ANSI-C. Both Microsoft and Apple have proprietary solutions for Java. (these insights courtesy of Jens-Uwe Mager and Christian Tismer who endorse the idea). When this succeeds we can start to code e.g. a POSIX-layer and wrap C-libaries (at runtime) from python. So much for now from my side. I am sure that as so many people inherit the good culture from c.l.py and python-dev we will have good discussions. Please note that there are some subscribers (e.g. Jens-Uwe Mager) who are experts but not neccessarily in core CPython. I hope/suggest that even naive sounding questions about Python and its core are welcomed. It certainly helps if more people understand the internals and the involved issues. I reserve the right to ask naive questions, myself :-) regards, holger krekel

9 19

getting rid of compile.c?
by holger krekel 16 Jan '03

16 Jan '03

Hi, i was wondering how/if to get rid of compile.c. ASFAIK it is currently needed for bootstrapping purposes. Wouldn't it be possible to deliver precompiled bytecode for the python compiler package and thus obsolete compile.c? The bytecode isn't platform dependent so this seems sensible to me. regards, holger

3 6

RE: Seeking Minimal Python Project Name
by Tim Jarman 15 Jan '03

15 Jan '03

How about: XPython as in: eXperimental Python or : eXtreme Python (but not ex-Python as in "This is an ex-parrot")? Helping-to-keep-this-thread-immortal-ly yrs, Tim J.

1 0