Mailman 3 February 2003 - pypy-dev

Re: Base Object library (was: stdobjspace status)
by Christian Tismer 09 Mar '03

09 Mar '03

Stephan Diehl wrote: Once again on this: > def __init__(w_self, rintval): > w_self.intval = rintval > > where rintval is already the restricted intval. > Am I right here? As said, I'm not sure if rintval will be an r_int already, it depends on the rest of the implementation which is quite a bit foggy to me, still. Maybe I define intval as a slot, and give it a setter method that always enforces the primitive r_int type. Also there will be a basic notation of primitive arrays. They might be built using the array module, but I think I like it better to implement it using derived list objects, with a restricted type attribute. And then, these objects are meant to describe just plain memory arrays, no Python objects. The are, but they don't behave like that. For instance, r_objectarray would be a list, which is restricted that it only can hold references to W_Object, or None which acts as NULL. r_objectarray would also override Python operators like "+" and "*", since these don't exist in RPython's arrays: An array can only be created once, with a fixed size. There is no concatenation or repetition. Instead, this has to be built upon that, in terms of the primitive arrays. So, W_ListObject would have some fields like self.objarray # r_objectarray self.maxlen # r_uint self.actlen # r_uint It has been suggested to not use maxlen, since we could use len(self.objarray), but I believe this is wrong to do. If we are modelling primitive arrays, then they don't support len() at all. Now, for example, the definition of concatenation for our new lists could read like so: class r_range(object): def __init__(self, *args): self.xr = xrange(*args) def __getitem__(self, idx): return r_int(self.xr[idx]) class W_ListObject(W_Object): __slots__ = ["objarray", "maxlen", "actlen"] def __init__(self, lng, data=None): self.objarray = newobjarray(lng) if data: for i in r_range(lng): self.objarray[i] = data[i] self.actlen = lng else: self.actlen = 0 self.maxlen = lng def list_list_add(space, w_lis1, w_lis2): lng1 = w_lis1.actlen lng2 = w_lis2.actlen lng = lng1 + lng2 res = W_ListObject(lng) for i in r_range(lng1): res.objarray[i] = w_lis1.objarray[i] for i in r_range(lng1, lng): res.objarray[i] = w_lis2.objarray[i] res.actlen = lng return res Maybe this is just very rough, but give a little an idea how things can work. Wile we are at it, Stephan, you wrote the longobject.py . You are probably aware of the fact that this implementation is slightly not what we need? You coded, for instance: def long_long_sub(space, w_long1, w_long2): x = w_long1.longval y = w_long2.longval try: z = x - y # <== ick! We don't have that! except Error,e: raise OperationError(Error, e) return W_LongObject(z) This is ok, if you rely on the existance of long objects. But for a real implementation, we need to use integer arrays like in longobject.c, and implement everything by hand. So how to fill longobject.py with real life? I think, we first need some basic structure like r_integerarray or maybe even better r_uintarray, and based upon that, we must build every and all the necessary loops. Anybody wants to try it? ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

7 16

stdobjspace summary?
by Armin Rigo 02 Mar '03

02 Mar '03

Hello everybody, As I won't read my mail during the whole next week I thought it might be helpful if I attempted some kind of summary. It collects numerous ideas posted here (recently or not) and a bit of personal feelings. I hope it might help in getting some distance over what we are actually trying to do with our "standard object space". I am afraid it is all very vague, but should be kept in mind before we actually design something new there. The std/ directory is the "space of compliant Python object implementations", as opposed to non-compliant object spaces which do some more funny things while the interpreter follows the bytecode. It collects various implementations for the same user types, which are based on some lower-level abstractions. The abstractions that are used are explicitely described in each file, for example: N-bits or machine-sized signed or unsigned integers; memory blocks with explicit management; bitfield objects; various hints (e.g. "this implementation should not be used with refcounts because it has many circularities"); or even which "RPython level" we use (if we develop more and more complex translators), including what basic type operations we allow. Then each low-level abstraction itself has a reference implementation in Python, like Christian's r_int; this should be put in a file that can easily be associated to alternate implementations of the same concept, the ones that can be used by the translator (like the description of the C 'int' type and the tedious complexities of overflow detection in C). These low-level implementations can also explicitely depend on other low-level implementations: for example, "C lists with stored length" could be implemented in term of malloc-style memory blocks -- as opposed to "lists as Java arrays" which don't go lower-level because the length is always stored in Java arrays. "Above" all these files, we have several compliant object spaces, whose purpose is to link a selection of implementations together. For testing purposes object spaces should be easy to build and run in Python, thanks to the Python reference implementations of low-level abstractions. On the other hand, the "C object space" links some or all the files which are (indirectly) based on C-implemented low-level abstractions only; this is what the translator will input. Which files an object space depends on should be specified declaratively, in such a way that we can actually use this information to implement the "wrap", "newlist", "newstring",... methods of the object space. These are the functions that build W_XxxObject instances of the basic types from scratch. Let me stress again that the maximum flexibility seems to be to allow each concept to be implemented in possibly several ways above lower-level concepts, including new ones if the existing ones don't suit you. The arguments on the mailing list show nicely enough that there is just no single better way to implement things, so we should just allow them all to coexist[1]. After that, *other* files are responsible for putting the pieces together -- explicitely (listing the files to include), implicitely (automatically searching for implementations that meet some requirements), or even dynamically at run-time to optimize the performances (not easy, but at least it should be *possible*)! [1] is a departure from one of Python's moto: "there is one and only one obvious way to do something". Well, I assume that I won't shock people around here if I say I never liked this one :-) A bientôt, Armin.

2 2

Re: [pypy-dev] Re: Base Object library (was: stdobjspace status)
by Armin Rigo 01 Mar '03

01 Mar '03

Hello Stephan, On Thu, Feb 27, 2003 at 05:23:44PM +0100, Stephan Diehl wrote: > About object instantiation: > Where excatly is the textual representation of a python object turned into an > object? Is it done by the parser? (for example "12345L" --> 12345L) Damn. I don't know. If we don't include any 'long' type among our allowed RPython types (and we probably don't), then we will have to rethink the way code objects are currently completely put into interpreter-space level. This cannot work if this code object contains constants of unknown types! So it seems that even code objects must be object-space dependent. They would be compiled by the application-level 'compiler' package inside of a given object space, and not be 'extractible' from there. The interpreter would only unwrap the bytecode string, but not the tuple of constants. If on the other hand we would prefer completely portable code objects, then they must contain "portable constant literals", including longs, and wrap() must support at least these constants. This could be nice, too, because we could then have object-space-independent bytecode repositories (like .pyc files). But then we must include in RPython the notion of longs -- at least their existence, not necessarily any specific operation on them. The same about complex numbers. Tough choice. Armin

4 6

RE: [pypy-dev] Release something?
by roccomoretti＠netscape.net 28 Feb '03

28 Feb '03

Armin Rigo <arigo(a)tunes.org> wrote: >Hello everybody, > >One more e-mail to remind you that one of the goal of the sprint was to >release something at the end of the week. In regards to release, we should keep in mind that *anyone* who wants to look at pypy can get subversion and check out the most recent version - so in a sense we have a continual release process. So the concept of a "release" only makes sense in regards to people who don't want to bother downloading svn or tracking changes - i.e. non- developers. Therefore it only makes sense to "release" pypy when it is worthwhile and interesting in and of itself. (Which may be now.) Therefore, I submit for approval the following (tentative) release milestone guidelines: ------------------ 0.0.1 - pypy passes all of CPython's regression tests under the trivial object space. (On the appropriate version of CPython.) 0.1.0 - pypy passes all of CPython's regression tests under a StandardObject Space. (Regardless of host Python.) 1.0.0 - pypy passes all of Python's (note generality) regression tests, and is stand-alone (needs no host Python implementation) -- Note that these are just suggestions to start discussion - feel free to ignore them. -Rocco __________________________________________________________________ The NEW Netscape 7.0 browser is now available. Upgrade now! http://channels.netscape.com/ns/browsers/download.jsp Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/

2 1

the truth about pypy
by holger krekel 27 Feb '03

27 Feb '03

Hi, everybody knows, that google is the modern "oracle" of today. So i asked http://www.googlism.com which uses google to find out the truth about something (usually a person). So i asked what "pypy" really is. The one and only answer was "pypy is the optimal initiator sequence" any questions? :-) holger

1 0

Re: [pypy-dev] Re: Base Object library (was: stdobjspace status)
by Laura Creighton 27 Feb '03

27 Feb '03

In a message of Thu, 27 Feb 2003 17:23:44 +0100, Stephan Diehl writes: >On Thursday 27 February 2003 16:53, you wrote: >> Hello Stephan, hello Laura, >> >> Thanks Laura for pointing out the potential language problems. > >Is there a possibility that Lauras mail didn't end up on the list? I coul >dn't >find it. Yes. I wrote this, but only sent it to Armin because I thought it was too long and maybe it was wrong too. It is Still too long. Apologies. Laura Is it possible that this misunderstanding is all about naming? I am getting the impression that Stephan and Christian are looking for some basic fundamental types, the foundation of the system, to build things out of. And Armin is saying that there aren't any types, just behaviours of object spaces. This is strongly reminding me of what happened to the physicists when they started looking for 'the elementary building blocks of matter'. Once you get to sub-atomic particles, you end up with a mess to think about. When Dirac discovered the positron, and anti-matter, suddenly you could no longer think of things as being made out of smaller things any more. A hydrogen atom is usually a proton and an electron. But it might temporarily be 1 proton, 2 electrons, and one positron. This made all the physicists go -eep-. What's Hydrogen? Something that behaves like Hydrogen, it seems :-). Time passes, and quantum dynamics is created/discovered. But the deep problem remains. Werner Heisenberg calls this the 'problem of physical content vs mathematical form'. He said that the problem is embedded in the language we use to think. Maybe we have a language problem too. We shouldn't have called it the StdObjSpace. Instead we should have called it the C_like_familiar_ObjSpace. It represents one way to implement an Object Space, and it has the desirable property that it is like the one we know. But we could eventually make as many of these as we like (though only a few of them are a good idea now, or we will confuse everybody even more than we are confusing ourselves now.) But I am now thiking of an ObjectSpace as, not a thing, but a collection of behaviours. And you can specify the behaviours. In particular, we can specify what behaviour is required in an Compliant Object Space, where Compliant is now defined as 'runs C python code and gets the same result as C python'. All of the spaces that do this will be equal to each other in some sense, though some will be more useful, and fun than others. We could make 2 that are exactly the same except for, for instance, the list implementation, which is one way to test that we have our ideas ok. Or one that does integers the way that we are used to, and one that does them this new way that Stephan proposes. Then, we can call the set of all Object Spaces that are Compliant, the StdObjSpaces.( <-- see the s) . But the idea is not to make too many of them, because what we need is a proof of concept that our idea of breaking out the object spaces into their own idea, and dispatching on that works, not the best StdObjSpace implementation. The best one will be determined by things like 'can Armin and Christian make it go _really fast_, which is something we aren't supposed to be thinking about now. (Though its hard not to sometimes. :-) ) (We can then make non-compliant objectspaces if we like. I mustn't think about that either :-). Thus what we need are experimental specifications, in the form of unit tests, to measure whether a given objectspace is 'standard' or not -- in other words, does it comply with C Python. We aren't looking for better bits to build our space out of, we are looking for better ways to know if our space produces the correct behaviours. Laura

3 2

Release something?
by Armin Rigo 27 Feb '03

27 Feb '03

Hello everybody, One more e-mail to remind you that one of the goal of the sprint was to release something at the end of the week. Did you realize that the interpreter (with the trivial object space) already works nicely enough to run the following: # python interactive.py Python 2.2.2 (#2, Nov 14 2002, 17:24:26) [GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.68mdk)] in pypy PyPyConsole / TrivialObjSpace >>> import dis >>> dis.dis(dis.dis) I particularly enjoy the slooooooowness of the resulting disassembly dump :-) Do you think, just for the fun, that this would be worth a "release"? It would show the "outside world" that we actually got something "serious" done. Of course "serious" needs quote, given the speed of the result, but at this point everybody should know that it was planned, and that we are dreaming about top speeds involving some rather obscure translations (probably invoking strange beasts from the theory of Computer Science). A bientôt, Armin.

4 4

Re: [pypy-dev] Re: Base Object library (was: stdobjspace status)
by Armin Rigo 27 Feb '03

27 Feb '03

Hello Stephan, hello Laura, Thanks Laura for pointing out the potential language problems. Your point of view on Compliant Object Spaces is very helpful. It seems to show that we will want different object spaces that share quite a lot of source files, e.g. two object spaces which are the same except for their list implementation. Moreover, some of these implementations might be easier to translate into efficient C code and some others might be better suited for translation to some other language. What we may thus need is a way to define several object spaces from the same set of files. The std/ directory would then contain all files that implement a bit of the "compliant" behavior for some specific kind of objects; it could contain e.g. both the standard and Stephan's decimal implementation of floats. What is left to be determined is thus how the objects are instantiated, as Stephan pointed out. This currently takes place in the class ObjSpace, in the methods wrap(), newlist(), newtuple(), newdict(), etc. Then what we will need at some point is to have several different subclasses of baseobjspace.ObjSpace in several files, possibly all inside std/, which import and instanciate different W_XxxObject classes. To make an object space which is the same as another one expect for its implementation of floats, just override its wrap() method. Sounds cool! A bientôt, Armin.

2 1

Test suite for basic types
by Armin Rigo 27 Feb '03

27 Feb '03

Hello everybody, I found in the standard Python lib a test, namely test_builtin, which tests all built-in functions. Is anybody aware of a similar test for the basic built-in types, testing ints, longs, lists, dicts, tuples, strings...? And every opcode? If not, this would probably be a good subproject. Besides, it is something that could be contributed back to CPython. It could also be turned into a much more accurate benchmark than that horrible PyStone ! A bientôt, Armin.

2 3

stdobjspace status
by Armin Rigo 27 Feb '03

27 Feb '03

Hello everybody, I've been progressing on putting the pieces together and will check-in my changes as soon as possible. The stdobjspace is getting ready to be tested together with the interpreter, but is not quite there yet. If someone (Michael?) can figure out a good solution for the END_FINALLY problem, I'd love to hear it. The problem is that in a try:except: block, when we enter the except: part, there are some control paths that reach the final END_FINALLY opcode (re-raising the exception) and some that don't (caught exception). It makes it hard to know when we must remove from the blockstack the description of the potentially re-raisable exception. I was also thinking about the explicit multimethod registration in the standard object space. For example, intobject.py contains the following code: def int_int_add(space, w_int1, w_int2): ... StdObjSpace.add.register(int_int_add, W_IntObject, W_IntObject) def int_int_sub(space, w_int1, w_int2): ... StdObjSpace.sub.register(int_int_sub, W_IntObject, W_IntObject) def int_int_mul(space, w_int1, w_int2): ... StdObjSpace.mul.register(int_int_mul, W_IntObject, W_IntObject) I am proposing that it could be written with some more declarative idiom abusing the 'class' keyword, something along these lines: class __multi__(W_IntObject, W_IntObject): def add(w_int1, w_int2, space): ... def sub(w_int1, w_int2, space): ... def mul(w_int1, w_int2, space): ... It looks much nicer, I think, and it addresses a point raised during the sprint (by Stephan if I remember right) about making the xxxobject.py files less dependent on the actual name of their object space. Indeed, the 'class __multi__' code no longer mentions StdObjSpace. This can be implemented using an original dispatching algorithm that has been as far as I know invented for the Slate language (http://slate.tunes.org/). The implementation of 'class __multi__(X, Y)' is defined to store the function objects defined in the body -- here add(), sub(), mul() -- into *both* the X and the Y class, into special-named attributes. Performing dispatch to 'space.add(x,y)' is done by looking into the class of 'x' and the class of 'y' to see if we can find the same function object in both. More specifically, after a definition like class __multi__(X, Y): def f(x, y, extra): ... we would have the function object 'f()' stored into each of the lists 'X.__dispatch_on_arg_1__' and 'Y.__dispatch_on_arg_2__'. Then a call like 'f(x,y,extra)' looks up the types of 'x' and 'y' and searches for a function object that would be present both in 'x.__class__.__dispatch_on_arg_1__' and 'y.__class__.__dispatch_on_arg_2__'. What is nice with this approach is the removal of the need for a central "repository" of functions, role which was currently held by StdObjSpace.add. BTW the new order of the arguments ('space' at the end) is such that single-dispatch is a particular case of Python's original dispatch semantics, with the argument we dispatch on being first (althought that original semantic is also extended by the delegation stuff, e.g. delegating from 'int' to 'long' for methods that fail to be implemented by 'int' for overflow reasons). A bientôt, Armin.

4 8