Re: Base Object library (was: stdobjspace status)
Stephan Diehl wrote: Once again on this:
As said, I'm not sure if rintval will be an r_int already, it depends on the rest of the implementation which is quite a bit foggy to me, still. Maybe I define intval as a slot, and give it a setter method that always enforces the primitive r_int type. Also there will be a basic notation of primitive arrays. They might be built using the array module, but I think I like it better to implement it using derived list objects, with a restricted type attribute. And then, these objects are meant to describe just plain memory arrays, no Python objects. The are, but they don't behave like that. For instance, r_objectarray would be a list, which is restricted that it only can hold references to W_Object, or None which acts as NULL. r_objectarray would also override Python operators like "+" and "*", since these don't exist in RPython's arrays: An array can only be created once, with a fixed size. There is no concatenation or repetition. Instead, this has to be built upon that, in terms of the primitive arrays. So, W_ListObject would have some fields like self.objarray # r_objectarray self.maxlen # r_uint self.actlen # r_uint It has been suggested to not use maxlen, since we could use len(self.objarray), but I believe this is wrong to do. If we are modelling primitive arrays, then they don't support len() at all. Now, for example, the definition of concatenation for our new lists could read like so: class r_range(object): def __init__(self, *args): self.xr = xrange(*args) def __getitem__(self, idx): return r_int(self.xr[idx]) class W_ListObject(W_Object): __slots__ = ["objarray", "maxlen", "actlen"] def __init__(self, lng, data=None): self.objarray = newobjarray(lng) if data: for i in r_range(lng): self.objarray[i] = data[i] self.actlen = lng else: self.actlen = 0 self.maxlen = lng def list_list_add(space, w_lis1, w_lis2): lng1 = w_lis1.actlen lng2 = w_lis2.actlen lng = lng1 + lng2 res = W_ListObject(lng) for i in r_range(lng1): res.objarray[i] = w_lis1.objarray[i] for i in r_range(lng1, lng): res.objarray[i] = w_lis2.objarray[i] res.actlen = lng return res Maybe this is just very rough, but give a little an idea how things can work. Wile we are at it, Stephan, you wrote the longobject.py . You are probably aware of the fact that this implementation is slightly not what we need? You coded, for instance: def long_long_sub(space, w_long1, w_long2): x = w_long1.longval y = w_long2.longval try: z = x - y # <== ick! We don't have that! except Error,e: raise OperationError(Error, e) return W_LongObject(z) This is ok, if you rely on the existance of long objects. But for a real implementation, we need to use integer arrays like in longobject.c, and implement everything by hand. So how to fill longobject.py with real life? I think, we first need some basic structure like r_integerarray or maybe even better r_uintarray, and based upon that, we must build every and all the necessary loops. Anybody wants to try it? ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[...]
Yes, I'm aware. If it is already anything, then only a proxy. My initial idea was, of course, just to build the interface of the objects first --- so they can be already used as long as everything is in an CPython environment --- and provide the "real" implementation later.
That's actually the place, where I'd really do it different to the CPython implementation. I'd factor out the r_longs to a library (as discussed several times in this thread) and would indeed write: ----------------------------------------------------------------------------------------- import basiclong # <== or something similar def long_long_sub(space, w_long1, w_long2): x = w_long1.longval y = w_long2.longval try: z = basiclong.sub(x , y) # except Error,e: raise OperationError(Error, e) return W_LongObject(z) --------------------------------------------------------------------------------------- As far as I'm concerned, the implementation of the basic/restricted types and the implementation of the W_XXXObjects should be clearly separated.
Anyway, we are probably not too far apart in our opinion about this topic :-) Cheers Stephan
-----Original Message----- Christian Tismer wrote:
Actual W_IntObject looks like: def __init__(w_self, intval): w_self.intval = r_int(intval) -- versus -- def __init__(w_self, rintval): w_self.intval = rintval Someone can help me to clarify my mind *I'm a bit confused*...?? W_IntObject is the implementation of *python int type*, a wrapped object (a). the *rint* is one of the possible *concrete* type that can be handled by the W_IntObject in its job of implementing the python int (b). Pypy must emit C/Assembler/ObjC/...restricted-python??.. In other words... is not the *W_IntObject with r_int* one of the *possible* choice that pypy can choose for representing (to target code emission!) a plain python int (c)? Instead of r_int, for example, I can choose to use a tagged-rapresentation of an int, and write a t_int that check the overflow and so problems (and where to check arithmetics?)...(d). So (d) lead to *how/where* replace the r_int with the t_int. Actual version suggest that this is done implementing different W_IntObject (with the same interface! All are python *int* object); second version of the init suggest that this is done *outside*, and that the W_IntObject is a *neutral* implementation of the W_IntObject dispate whatever the *compiler/optimizer/developer-of-pypy* decided to use as concrete *intval* (e). I think we need some idea and prototipe of the code *above* W_IntObject (Armin help ;) The compiler do something like.... ??? - Ok, folk! - Let's analize that application! - Umm in this hot-part of the program I can *try* to specialize and use a *W_IntObject with tagged int*! - Let's recompile this chunk of the app using *W_IntObject with t_int*, a pure python implementation of tagged int. - Simply *recompile* the hot-part (is all python! Not to do Armin tricks for just-in-time assembler compilation!) and check if all is ok! - No overflow, no problems, ok! - Emit *real* C/Assembler/Whatever tagged repr. What part of the my personal brainstorm is wrong? ;-( Paolo Invernizzi
Hello Christian, Whoow, I am afraid this thread is going in some wrong direction! As Paolo points out, this is getting confused. There are already several different levels at which things are working, and Stephan and you are adding yet another one. I don't say this is necessarily wrong, but things are not very clear. The W_IntObject class itself is supposed to be one possible implementation of Python's integers, described in a high-level point of view. The whole point is in that word "high-level". We (or at least I) want the objspace/std/* files to be translatable to whatever actual low-level language that we like, and not only C -- this is why the bitfield metaphor is not necessarily a good thing. When you are talking about r_int and similarly r_float or even r_long wrappers, you want to make it clear that you mean a "machine-level" plain object. But actually that was the whole point of RPython. The C translation of all the interpreter-level code will produce machine-level "int"s. In other words there is no deep reason to use a special wrapping mecanism for ints, floats, lists, whatever, to implement the W_XxxObject classes, because we will need *exactly the same* low-level objects to implement just *everything else* in the interpreter! For example, when the interpreter manipulates the current bytecode position, it manipulates what looks like Python integers and strings --- opcode = ord(self.bytecode[nextposition]) nextposition += 1 --- but we want this code to be translated into efficient C code, too. The least thing we want is to have to rewrite *all* this code with r_ints! So the *only* point of r_ints, as far as I can tell, is to have explicit control over bit width and overflow detection. There is no built-in Python type "integer with n bits" that generates the correct OverflowErrors. And there is no unsigned integer type, hence the need for r_uint. But apart from that, just use plain integers and it is fine! That is the whole purpose of writing our pypy in Python, isn't it? Creating r_long, r_float, r_list... looks like we are reinventing our own language. The confusion that may have led to this point is that we never explicitely said what objects we would allow in RPython. Remember, RPython is supposed to be Python with a few restrictions whose goal is *only* to ease the job of type inference and later translation. But the *existing* high-level constructs of Python must be kept! If we don't use them, we are back again at the C level and there is no point in writing pypy in Python. A couple of more specific points now... On Wed, Feb 26, 2003 at 07:46:12PM +0100, Christian Tismer wrote:
I feel exactly the opposite. Just *use* real Python lists all the way, including most of their normal operations. This is nice because they describe high-level operations. In my view a W_ListObject would have only two fields: * self.objarray # a real Python list of wrapped items * self.length # the actual list length with self.length <= len(self.objarray). The self.objarray list grows by more than one item at a time, to amortize the cost of adding a single element at a time, just like what the current CPython implementation does. All this works fine, is normal pure Python code, and can be expected to translate to an efficient C implementation in which lists are just translated into malloc'ed blocks of PyObject* pointers. The point is that we still have a nice Pythonic high-level implementation of W_ListObject but nevertheless can translate it into an efficient low-level implementation. Another point I would like to raise is that we don't have to give types to every variable in pypy, like enforcing that a slot contains an r_int. This is just unPythonic. *All* types are expected to be computed during type inference. So don't even say that W_ListObject.objarray is a list of wrapped objects or Nones -- this can be figured out automatically. ..from Paolo's reply:
This is right. Consider for example the case of *two* different implementations (that would both appear as having the 'int' type to users). Say that one is like Christian's W_IntObject+r_int, and the other one can only encode small, tagged integers. The choice to use one or both representations in an actual C implementation must be made by the RPython-to-C translator, and not in the object space. For example, if we want to produce something very similar to the current CPython, then we have no use for small, tagged integers. The question is thus, "how do we express things to allow for this?" Similarily, we may provide different implementations for lists, dictionaries, whatever; we may even consider that Python's "long" type is an unneeded hack, for long integers could be just another implementation for the 'int' type, which goes very much in the direction that Python seems to go with the recent automatic conversions of overflows to longs. The original intent of classes like W_IntObject was "one class, one implementation", and I think that we must stick to that idea because these classes are what are used for the multiple dispatch routines. I don't have a clear and complete answer for the rest of the question "how do we express things to allow for this?". I hope that this e-mail has clarified some points. Disagreement is welcome. I apologize to Christian and Stephan because it seems that we might have to reorganize the xxxobject.py sources, althought I'm not sure yet how. In an effort to go in that direction I'd like to add that nothing has been done yet about: * built-in methods (like list.append); the StdObjSpace.xxx.register() trick only works for built-in operators * non-built-in operators and methods, e.g. implementing something like long_long_add in application-space (longobject_app.py). A bientôt, Armin.
I admit for now I have just skimmed over the code. I can more or less see how something like classic-style classes can be implemented. Have you already thought on how to implement new-style classes and in particular built-in types subclassing like in: class myint(int): ... e.g. will the implemented multi-dispatching intepreted statically at PyPy static translation time ... or should the final low-level runtime support it dynamically? regards.
Samuele Pedroni wrote:
Speaking for myself, I always have wondered how the multimethod approach should be extended above the level of builtin types, since I don't understand how to extend it to derived types, without breaking compatibility to all the established __xxx__ rules in Python. I pointed to this problem on the sprint, but can't remember that this issue was solved. One vague idea could be that derived types by default inherit the multi-method dispatch, but when people override operators, we have to provide some compatible interpretation of things like __add__(self, other), or this would not be Python. Maybe we can find the necessary magic to embed this into multimethod dispatch dynamically, but I have really no clue how this should work. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Can someone knowledgeable spend a few moments explaining what people need to do to use subversion? Does it work on windows etc etc? -- Robin Becker
Robin Becker wrote:
Can someone knowledgeable spend a few moments explaining what people need to do to use subversion? Does it work on windows etc etc?
You can find a windows installer here: http://codespeak.net/~jum/ The current command line version is http://codespeak.net/~jum/svn-0.18.0-r4973-2-setup.exe Despite the fact that the wiki page does not list TortoiseSVN (should it do this now? I htink it is more stable), I'm very happy with it, since it sicely adds itself as a shell extension and shows the file status as you look into your folders with explorer: http://codespeak.net/~jum/TortoiseSVN-0.6.msi If you are using Windows 2000, you need to install service pack 3 for SVN. To check out the current source trunk, use something like svn co http://codespeak.net:8080/svn/pypy/trunk d:/pypy You can always update using Tortoise or the command line, being in d:/pypy and doing svn up cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
-----Original Message----- From: Armin Rigo
<snip>
<snip>
Can you point out some idea about the *work* of the RPython-to-C translator? I can only imagine what I pointed out in the last email... <FOGGY> - The RPython-to-C translator check the source code of the application for *static* *compile-time* optimization. - The RPython-to-C translator run the application under some objectspace. If I've understood well the idea, the application is the pypy-interpreter PLUS the application... - It performs introspection and optimize changing on-the-fly things. Some W_XXXObject is replaced with other, probably more *restricted or suited to the target the specific code emission*, ex the W_IntObject+r_int. Donno if this phase is to be done, I guess is an *additional* layer but remembering the discussion about the stack dept calculation, this can be the way to tell if all in the application is ok and aid the compiler... - loop analysis on even-more-restricted source - Finally emit C code for that specialized-parts, for example code that mimics current CPython functions for pypy-part of the interpreter, or code that recall the specialization done by psyco. </FOGGY> Paolo Invernizzi
Hello Paolo, On Thu, Feb 27, 2003 at 01:00:12PM +0100, Paolo Invernizzi wrote:
Can you point out some idea about the *work* of the RPython-to-C translator?
Your description is reasonable, althought I can imagine that we will start with only the following objective in mind: being able to re-emit C code that looks like CPython. The code written in RPython ("R"estricted) is only the interpreter-level stuff (and not absolutely everything, e.g. multi-dispatch should probably be rewritten by hand in C depending on the low-level object representation). I was targetting only this part for the RPython-to-C translator. In other words, this would (at first) be a completely static transformation of the pypy source to C. This would be application-independent, giving something that can be easily redistributed as an alternative to CPython. Armin
-----Original Message----- From: Armin Rigo
This is the case when the "application" is the pypyinterpreter, but conceptually, if the RPython-to-C translator works, it can translate every "application" written in RPython, or I'm missing something? Last question, I promise ;) I have not understood if *completely static* means that the code of pypy is analized *without* running pypy and the C sources are emitted. Or is it necessary for the translator to *execute* it for collecting information? Thanks a lot Armin for the kind responses! (and forget my bad english!) Paolo
Hello Paolo, On Thu, Feb 27, 2003 at 02:35:33PM +0100, Paolo Invernizzi wrote:
Yes, that's right.
It depends on your point of view. I would say that from some point of view it is just a static external transformation. From some other point of view, this analysis will certainly be implemented by actually *running* the pypy interpreter with a special object space, whose purpose is to collect a trace of everything going on and every variable's types, and at the end write C code. A bientôt, Armin.
Hello Christian, hello Stephan, Please, don't take my previous e-mail personally, I know that you have been working quite hard on the {int,float}objects. It was planned from the ground up that we would discover new issues and new ways of seeing them as the project kicks off. Also, it doesn't mean at all that anything you did will be trashed; it just means that (I feel) it is time to pause a bit and think about how, with the insight given by your code, we can build a coherent thing. At worst this will mean changing the interface to something more declarative than the StdObjSpace.xxx.register() calls -- if at all. A bientôt, Armin.
Hi Armin, On Thursday 27 February 2003 15:14, you wrote:
Don't worry. This whole discussion show at least that nothing is set already and freah ideas are still welcome :-)
At the moment, nothing needs to be trashed at all, since the "hard-to-implement" types are not implemented yet.
That shouldn't be a problem as well (could be done by a good editor :-) Nevertheless, I still don't see that my way woundn't be a valid one. As far as I've understood the issues so far, the implementation details of the W_XXXObjects are just that: details. The interpreter just sees them as black boxes with behaviour. The only place where we need to be carefull is the instanciation (and that was not discussed yet). The point of compiling RPython into C is a complete independent discussion. Cheers Stephan
Armin Rigo wrote: ...
We probably don't need bitfields now.
The is no r_float, since floats are just machine-level. There is also no r_long, since there are no longs at all.
Right, and I didn't hear me saying so. I just need the r_ints to make the emulation behave correctly, when we are running it in CPython!
Forget about r_long and r_float, I can't remeber that we talked about that, at least I didn't think of it. r_list is something where I'm not sure, because I don't believe that we can create all code by pure magic. Somebody has to write an implementation of the future list implementation (which has nothing to do with r_list but to use it as a building block). But I believe it is bad to use regular lists and use things like append(), or list1 + list2. This is not primitive and neither available in the C or Assembly targets, so why may we assume they exist? Only for that reason, I wanted to create a subtype of lists, which is just not capable to do certain things.
Then you should become clearer about what exactly we want to use from Python. ...
Sounds inconsequent to me. On the one hand you use the length attribute, so the thing cannot be translated into a pointer to a plain memory area, but it is more. On the other hand, you repeat the C implementation of piecewise allocation. Where do you set the fence?
Being explicit about actual length, but implicit about maximum length is a bit arbitrary for me. I thought to describe a piece of memory by using a list. It is a piece of memory, that malloc returns. So I thought I need to model length as well. If I don't model length, but rely on a code generator to figure this out, well, then I have the problem that this code generator does not exist yet, but I want to be able to generate code, as a proof of concept. For that reason, I liked an implementation that is so basic, that even I could write a code generator, without the need for any magic that I don't understand. Kinda bottom-up way, maybe this is the wrong way, but I need to be convinced. At the moment, I feel completely blocked.
You make the one false assumption that all the code is correct. The reason why I'd like to restrict the objarray, or especially the array used as the stack in frames, is better debugging. I was just told about the fact that we currently sometimes have "wrong" objects on the stack. Exactly that would be found easily, if I provide a modified array that checks for exactly that. I can't find how debugging is unpythonic.
This is a misunderstanding. I wasn't about using integer objects at all, the r_int emulated what will be a machine integer, later. This isn't about tagging. I don't thing to use integer /objects/ for implementing lists, for instance. I surely mean all integer objects to be turned into machine words. Again, this is probably since I thought that in /std/ we do what's needed for generatong C code.
I had the impression that the "std" objects space is meant to be the one to be translated into C. Therefore I was trying to write directly translatable code. But I see this goal disappearing... What I wanted to do is a 'real' implementation of basic types, whith 'real' algorithms, taken from the C sources, with some simplifications and clarifications by using Python, but not by using Python objects. If this is not the place to put this, then I'd like to ask for the place for that.
I agree. So are the implementations like W_IntObject implementations, or just interface definitions? ...
Yes. This doesn't happen, since we have no coherent design how this should be done. - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Christian Write...
This is what I was trying to clarify in my mind. I cannot figure out what is pertinence of the code generator and what is pertinence of "how we implement app-level python int in W_IntObject". IMHO we need at least some piece/ideas of this code-generator, to have a concrete case to discuss. And, I'm not convinced is a good idea to re-implement all CPython in pypy, passing all tests, release something and still have no code for C generation... the risk is to have to rewrite tons of code for adapting it to future requirement of the generator...
Again, this is probably since I thought that in /std/ we do what's needed for generatong C code.
Yup! My impression is that we are inverting the effort... if the generator is VERY smart I can write very pythonic code in implementations of W_XXXObj and compiler stuff, and delegate the analysis to it. If is pretty dumb, We can help it, with RPython for example but I think is time clarify this point.
I think the problem how much pythonic must be or must be not the algorithms for generator taste... Paolo Invernizzi
participants (7)
-
Armin Rigo -
Christian Tismer -
Florian Schulze -
Paolo Invernizzi -
Robin Becker -
Samuele Pedroni -
Stephan Diehl