Mailman 3 January 2003 - pypy-dev

Re: [pypy-dev] Restricted language
by roccomoretti＠netscape.net 21 Jan '03

21 Jan '03

Armin Rigo <arigo(a)tunes.org> wrote: < excellent discussion of confusion between (host) Python and (interpreted) Python snipped > >Here we are using Python's __getitem__ protocol to implement Python's >__getitem__ protocol. I see nothing wrong in that, but it is easy to get >confused. To make things clearer I would way: > > self.valuestack.push(w.getitem(v)) ... > >Think about >the type() function; it could not return the real type of the implementing >object, because you couldn't implement lists or ints with a custom class. And >it cannot call a new method __type__() of the object, because you cannot add >such a new method to all already-existing built-in objects. You could hack >something that calls __type__() if it exists and returns the real type >otherwise, but you are running into trouble when interpreting programs that >define __type__() methods for their own purpose. This is the kind of >confusion we are bound to run into if we are not careful. > So if I understand you correctly, we should not use (either explicitly or implicitly) any of the special methods of the objects we create for the interpreter. The only acceptable use is member dereferencing (object.attribute). Only host objects are allowed to be part of special method use. (e.g. in function calls, in expressions such as a+b, etc.) This is not strictly a technical limitation in most cases, but good practice so that we avoid confusion between the two levels. Is this an accurate interpretation? -Rocco P.S. In your implementation of EPython exceptions, how would associated traceback objects be handled? __________________________________________________________________ The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/

2 1

Objective of minimal python
by David Roper 21 Jan '03

21 Jan '03

I've lurked on the periphery of the discussions so far and must admit to being somewhat lost as to what exactly is being proposed. Am I correct in believing the following? 1. That the objective of the project is to create a new implementation python that: i. Has a smaller memory footprint; and ii. runs applications faster than the current interpreter; and iii. is well suited to applications making significant use of lightweight threads and continuations. 2. To achieve this it is intended to rewrite the 'front-end' of the Python interpreter, specifically the language parser, lexical analyser and compiler / byte-code generator, in python itself. 3. The byte-code interpreter (is this the same as the virtual machine?) is a program that reads the byte-code stream as data and processes it to do 'real work'. In order to meet objective (iii), it is intended to write a new VM that employs structure and mechanisms seen in the 'stackless' implementation. There are three possible implementations of the VM: i. It could be written in 'C' and call the 'C' runtime library execute functions implied by the byte-codes, e.g. allocation of space, arithmetic functions etc. Such an approach could draw heavily upon the code base of the 'execution engine' (I don't know what it's really called) of the existing VM. ii. It could be written in any language (including, but not limited to, 'C') and convert the byte-code stream into 'C' (or, indeed, any other statically compiled language), to be passed off to a compiler for compilation, linking and subsequent execution. This advantage of this approach is that it can exploit the considerable advances that compiler writers have made in code optimisation. I presume that the reason that the byte code, rather than python source code, is converted to 'C' is to be able to reuse existing compiled classes for which one does not have access to the source. iii. I could emit machine specific assembly language instead of 'C'. If understand correctly, this is what Psyco does. The advantage to this approach is that the code emited can be modified as the execution of the program (byte-code stream) evolves, thus permitting optimisations to be performed that cannot be achieved in a statically compiled language. This disadvantage is that it would be a very considerable effort generally to produce assembly language that is as highly optimised as a good optimising compiler might achieve. I'd be most grateful for your comments. Regards, David Roper mailto:droper@lineone.net

4 3

Re: [pypy-dev] question about core implementation language
by roccomoretti＠netscape.net 21 Jan '03

21 Jan '03

logistix <logistix(a)zworg.com> wrote: >I think the point is that Python as a language provides absolutely no >access to the machine internals. You can't examine memory contents, >registers, I/O bus or call code directly. Can you do this all in C? (without library functions coded in assembly) >As part of the bootstrap >process the following four functions should probably be added to the >interperter (either as builtin functions or a module): > > GetMemoryFromOS() > ReturnMemouryToOS() > Peek() > Poke() > >ANd probably: > get/putRegister() > callCode() > >Other than that, everything could (eventually) be written in Python. If we were writing an Operating System, I would agree with you. (We can put down a Python based OS as another goal of the project. <while 1:wink()>) Fortunately, we aren't quite so low level. The standard OS functions tend to be higher level. As I understand it, much of libc.so on Unix systems is simply a very thin wrapper around OS functions which behave much like the C functions that call them. Since we don't want to reinvent the wheel, it would probably be best to reuse the standard interface provided. This is where the minimal C core comes in. We can probably limit it to basic input and output (PyBIOS?), plus a generic mechanism for discovering and calling dll/so functions. Or am I missing your point entirely? -Rocco __________________________________________________________________ The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/

1 0

Re: [pypy-dev] Questions for Armin
by roccomoretti＠netscape.net 21 Jan '03

21 Jan '03

On removing eval(), Nathan Heagy <nathanh(a)zu.com> wrote: >The only reason I bring it up, and I'm not really in any position to be >bringing things up, is that it seems to me that this dynamic >metacompiling stuff could be a real pain for making things fast and >optimized. Most fast languages don't have eval() and that may be part >of the reason they are fast. I'm sure Guido could jump in with a >wonderful reason why eval() is great but I think if it disappeared no >one would miss it, especially if it let Python compile to machine code. >In fact if that was the price of enabling Python to compile to machine >code I *guarantee* no one would miss it. Having eval() is no impediment to compiling Python to machine code in the common case. What's the difference between a Python program which doesn't use eval() and one where eval() is prohibited? To deal with eval() in compiling to machine code, you could do one of two things: (1) Halt with an error saying that eval() is not supported (use the Python interpreter instead), or (2) Tack on an additional function which replicates the function of the Python interpreter to handle the argument to eval(). As I understand it, (2) is close to the equivalent of what Psyco currently does for the "uncommon case" anyway. -Rocco __________________________________________________________________ The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/

1 0

Restricted language
by Christian Tismer 21 Jan '03

21 Jan '03

Hi Armin, in order to start considerations about how to implement Python-in-Python, here some initial questions. Yes, we should begin with the RISP processor :-) (Reduced Instruction Set Python). There are some issues where I have to navigate around in my little tests that I've done already. One thing is the lack of a switch statement in Python, which either leads to zillions of elifs, or to the use of function tables and indexing. We should come up with some "how to do this". Another thing is common for-loops in C. Almost all of them which I tried to translate into Python became while-loops. Is that ok? Data types. How do we model the data types which are used internally by Python? Somebody already showed a small Python interpreter for Python 2.0 (very sorry, I can't find who it was), which was implemented on top of basic Python objects. This was a nice attempt and makes sense to start with. But I gues, for a real Python in Python, we also need to re-build the data structures and cannot borrow lists, tuples and dicts and "Lift them up" into the new level. Instead, these need to be built from a minimum Python "object set" as well. How far should we go down? I was thinking of some basic classes which describe primitive data types, like signed/unsigned integers, chars, pointers to primitives, and arrays of primitives. Then I would build everything upon these. Is that already too low-level? Do you think this should be started using the builtin objects right now, and these should be replaced later, or from the beginning? How far should it go: Are we modelling reference counting as well? And since you said that whether to use reference counting at all, or a classical GC, or object/type tuples returned as register pairs, I'm asking how we should do the proper abstraction? This means clearly to me, that I should *not* repeat the Py_INCREF/Py_DECREF story from Python, but we need to do a more abstract formulation of that, which allows us to specify it in any desired way. So I guess we are not fine by just repeating the C implementation in Python, but we need some kind of "upsizing" the algorithms, away from a flat write-down in C, up to a more abstract defintion of what we want to do. I could even imagine that there are lots of other cases where the C source code already is much to verbose, over-specifying, and obfuscating what the code really is supposed to do. Would you propose to "start dumb", literally like the C source, and add abstractions after the initial thing works, or does this have to happen in the first place? Very many things are done in a specific way in C, just because of the fact that you have to use C. But when I know I have Python, I would most probably not use C string constants all around, but use Python strings. It is C that forces us to use PyString_FromString and friends. Do you think we should repeat this in the Python implementation? This is just the beginning of a whole can of worms to be considered. Just to start it somehow :-) all the best -- chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

4 8

Re: pypy-dev Digest, Vol 9, Issue 3
by Scott David Daniels 21 Jan '03

21 Jan '03

pypy-dev-request(a)codespeak.net wrote: Armin Rigo <arigo(a)tunes.org> wrote (among other things): > def my_function(a,b,c): > return a+b+c > > the emitted machine code looks like what you would obtain by compiling this: > > PyObject* my_function(PyObject* a, PyObject* b, PyObject* c) > { > int r1, r2, r3; > if (a->ob_type != &PyInt_Type) goto uncommon_case; > if (b->ob_type != &PyInt_Type) goto uncommon_case; > if (c->ob_type != &PyInt_Type) goto uncommon_case; > r1 = ((PyIntObject*) a)->ob_ival; > r2 = ((PyIntObject*) b)->ob_ival; > r3 = ((PyIntObject*) c)->ob_ival; > return PyInt_FromLong(r1+r2+r3); > } Here we need to be careful about boundary conditions and wierd cases. On a 16 bit machine (for easy reading), 0x7fff + 0x7fff should not go negative. While sign checks may work on two args, they certainly won't work for three: 0x70ff - 0x7fff + 0x0005. Also on Bengt Richter <bokr(a)oz.net>'s: > I'm also picking this place to re-introduce the related > "checkpointing" idea, namely some call into a builtin that > can act like a yield and save all the state that the > compiler/psyco etc have worked up. Perhaps some kind of .pyk > for (python checkpoint) file that could resume from where > the checkpoint call was. I believe it can be done if the > interpreter stack(s) is/are able to be encapsulated and a > little restart info can be stored statically and the machine > stack can unwind totally out of main and the C runtime exit, > so that coming back into C main everyting can be picked up > again. I did a checkpointing system for the SAIL language long ago and far away. The truly nasty part of a checkpoint is replicating the environment outside of the address space. Not only do you have exception state, but you also have * file state: Does the file even exist anymore? Do output files get re-opend in append mode, or re-created with copies of data? Is the input file really the same? Do you _know_ that? Do you simple seek and go? * system state: Which interrupts are armed, enabled, or suspended and pointed to what code in SAIL's case. I think Python's will be more about what callbacks are set and how to "passivate" things like Tkinter. * Hardware state I: Stop the program with a checkpoint after printing half a recipe. Two days later run the checkpoint. The printer is not longer ready to finish printing. This is not as nasty as: You have issued two bytes of a three-byte operation to an I/O device. Post-Checkpoint the second time could be quite entertaining. This is why I decided checkpoint would cause a stop and run the checkpoint: _slightly_ safer. * Hardware state II: Stop the program with a checkpoint after printing half a recipe. Install a new printer. Run the checkpoint. * Hardware state III: (Thank heaven for the old days). Stop the program with a checkpoint. Replace the CPU. Run the checkpoint. * Hardware state IV: Stop the program with a checkpoint. E-mail the checkpoint to a friend who runs it ont his zyglot-2000. Checkpoints are absolutely wonderful. You can build systems that had several development phases, each represented by a checkpoint. In fact IMSSS at Stanford built some large systems that way: We avoided elaborate macros to build data structures by building theim in the first checkpoint phase. We actually had three levels of authors, two of whom were used to starting at a particular checkpoint and building it to the next checkpoint. The final checkpoint was delivered as our product. In my experience, there is too much outside the language system's control to provide the nice straghtforward thing you (or really your users) want when they hear "checkpoint." Programs often only check environmental things once, assuming they can never change for the life of the process. The OS type, the display type, the current process id, .... Be careful to avoid someoner thinking of a checkpoint as a "secret sauce" that they can smear over their program to allow it to magically checkpoint itself every minute and hope to use any of the checkpoints multiple times. Chris probably has run into this with continuations, which can be thought of as a kind of "internal code checkpoint." -Scott David Daniels

1 0

SCons build tool
by Charles Crain 21 Jan '03

21 Jan '03

Hi. I have been lurking on the pypy list for a while, and I happened to see a message under the subject "bootstrapping issues" that caught my attention: There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. It just so happens that I'm part of the team developing SCons, which is a 100% pure Python make replacement based on the winning proposal for the Software Carpentry Project build tool. It's very full-featured, and includes many features that have to be hand-rolled with the traditional make/configure/automake solution. For more information, see www.scons.org. Oh, and it's free as in both beer and speech, of course. -Charles P.S. I don't subscribe to the email list, I only check the archives every once in a while, so please copy any replies directly to my email. Thanks.

1 0

How to translate 300000 lines of C
by Christian Tismer 21 Jan '03

21 Jan '03

Dear list, I already announced some concern in a recent message. [Edward, I need you for this, at least for advice!] Part One: Making you frightened about the code size --------------------------------------------------- Running the following command over the current Python CVS src/dist direcotry: wc $(find . -name '*.c' -or -name '*.h') gives this result today (Januaray 20, 2003, 2:31 (GMT+01.00) 319282 1132750 9397985 total Ok, this is about everything in the core distribution, may it be needed for Minimal Python (whatever it is) or not. Let's roughly shrink it down to 150.000 lines. This is 150.000 lines of well-written, tested, evolved, really good C code. Now, a crowd of maybe 5-10 people is going to meet in a sprint by end of February, trying to translate a relevant amount of this mountain of code into Python? Really working Python? Won't they get bored? I can do 1000 to 3000 lines per day, when re-coding into Python as a prototype. With debugging and code quality stuff, I'm down to 500 or less. Let's assume we have 10 people of comparable caliber. Given that we work for 10 days full-time, nobody being ill, not accounting for the parties we probably will have, everything being perfect, then we *might* have 50.000 lines of quality code done in that period. I don't really believe in such a great success, it will probably be much less, since programming in groups does not scale well, sorry. There will be lots of overhead, discussions, misunderstandings, personal problems, I will probably get shot, so let's expect 10.000 to 20.000 of good Python program lines. Now, think of code like ceval.c which is alone 3900 lines of code, and not the most simple code. Of course, we can create a serious new interpreter, with all "borrowed" objects wrapped in a proper way quite quickly, and I still think this is a good idea. But I think we can do much better. And most probably, people will not get bored to do the implementation, see part three. Part Two: Making you frightened about the C code ------------------------------------------------ No offense to the python-dev people (also, since I do belong to this group a little bit), the C code base is absolutely great. As a code base written in C, of course. But I would like to encourage everybody to pick some medium-sized C source file and try to translate it into Python. It is possible, and it isn't too difficult. But it makes you stumble and stumble and stumble. The more you look at it, you recognize that it is quite near to assembly language. Everything is written down, expanded in some rather efficient way, there is not much abstraction. There is no inheritance, but there are lots of repetitions of fimilar but not identical code. You are confronted with exceptions which need some mapping. You see all the primitive types being used all the time, and you'll wonder how to map them. (Yes, we can set up general directives how to do that). You will also find lots of stuctures which need to be implemented. Finally, you find myriads of builtin Py...stuff...() runtime functions which you need to emulate somehow. Then, looking at the frequency of python-checkins, you will find that your translation work will be voided in some near future. python-dev is improving things all the time, and you will be kept busy for a life time to adjust your Python version. This might come to an end, if the core developers might finally decide to drop the C implementation in favor of our new project. But this can only happen, if we are fast enough! Part Three: Proposing A Radical Consequence ------------------------------------------- I see no point in wasting manjears of coding to re-invent the whell by assembling piece-to-piece from C code to Python code. For sure, there are some very relevant modules which might need to be hand-coded. But, and this is driven by the summary of what I thought to re-code by hand today: I believe that it is possible to automate this translation process! We can set up some default mappings for the most frequent C constructs. There are a number of free-ware C compilers around, and also some C interpreters. My vision since today is now to augment such a compiler to become a Python extension, and then run this compiler over all the C code. The Python extension should then try to provide a re-write of the C code in Python! There are some simple rules to be obeyed, which come out of the top of my head and can be changed as needed, just to give an example: For every structure that appears in the source, emit an approporiate Class definition, based upon a base class that is designed to handle structures. For every switch statement, create an according number of local functions (indeed making use of the new scopes), and prepare a dispatcher table for all the functions. For every simple for loop, create an xrange construct. For every not simple for loop, create a while loop with a break condition. For every simple type instantiation, create a similar object that derives from a class that describes such simple types. For very macro constant, use a constant notation. for every macro function, provide a Python function. Remove every Py_INCREF and every py_DECREF. Instead, let's automate that, using more reference counts than necessary, since this can be deduced by a good code generator, later. The interpreter doesn't do it differently, anyway. This list is by far not complete. Addition: For every C module, provide an extra Python module that is able to override some of the automatic decision of above. Special example: For ceval.c, overwrite all the specialized opcode implementations which try to optimize integer operations. These should not be written by hand any longer, but they are the objective of Psyco's specializing features. That's what I'm saying today: Make the move from C to Python automatic, by 95 percent. Let's modify a C compiler to do most of the tedious tasks for us. Try to use pattern matching to remove more of specializations done for the sake of C. Remove C-specific optimizations and optimize for abstractions. Then, we can try to re-target, create C code or assembly from that. My proposal right now is: Let's write (or change) such a compiler which emits fairly good scripts, and then let's add modifications which make these into really good scripts. With some luck, these will withstand the hight frequency of python-dev's code changes, too. Whow, this was a lot of storm in my brain for today. I hope it made some sense -- cheers - chris

6 10

Re: [Fwd: Re: [pypy-dev] Re: psyco in Python was: Minimal Python project]
by Boyd Roberts 20 Jan '03

20 Jan '03

Nicolas Chauvat wrote: >Probably not a good idea. Ask google for "reply-to considered harmful". > I knew that response would turn up.

1 0

Proposal
by Stephan Diehl 20 Jan '03

20 Jan '03

Hi all, I'm a little bit confused at the moment about what you really try to achive and I'm not too sure if everybody is talking about the same thing. Anyway, since two of the founders of this pypy discussion are meeting in person this Friday I'd propose that I grill them a little bit and write up some paper about it (and post it here, of course). That's all Stephan

2 2