Hi Armin, all, finally I realized that I'm doing way too much in my translation efforts. It happened to me when I turned md5.py into RPython using my prototypic genrpy transformer, that this is in many cases not needed at all. If you look at md5.py, you'll see that it *is* already almost restricted Python. I can just move almost all functions over to the interpreter level. In a sense, md5 and sha do not really belong to the appspace modules. Instead, they are almost ready internal implementations. Just the interface to the exported objects and methods needs to be created, to provide the importable builtin module. How do we handle such things? I guess such almost-interpreter level things should have a different folder. The published module would simply be filled with app2interp gateway calls to the internal stuff, or, well, directly writing down the wrappers would be fine as well, I guess. So my idea for cases like such is: Instead of transforming everything from appspace into interpreter level, we can use stuff almost as-is. Only non-trivial datatypes or unsupported features need to be translated from app to interp. Please correct me if I'm wrong. I was just shocked when I read the transformed MD5 source, like "hell, what am I doing here, this is no app code at all!" :-) Any ideas how to express the necessary actions? Do we move those files elsewhere? Do we use the __all__ attribute to name the interface stuff, or do we simply add the interface things by hand? thx & ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hi Christian, On Thu, Nov 25, 2004 at 09:04:18PM +0200, Christian Tismer wrote:
If you look at md5.py, you'll see that it *is* already almost restricted Python. I can just move almost all functions over to the interpreter level.
Yes, only some code is "essentially" application-level code. Some other code is quite "C-ish" already, and doesn't in any essential way need to correspond to space.xyz() calls. It can just "do its job" and return a result. In this respect md5.py and sha.py are similar to _formatting.py: they are all general Python code (so meant for app-level) that is almost RPythonic enough that we could put the code at interp-level with only some amount of uglifying. Didn't your mind go a full circle back to its starting point? I seem to remember that you started work on genrpy precisely to avoid to uglify _formatting.py by hand. Now you say that md5.py should better be put to interp-level manually... While this is of course possible, the idea of genrpy was to do it automatically. I know the generated code must look incredible right now, because it's just calls to space.xyz(). But the idea is to run the annotation stuff over the graphs -- which assumes some restrictions, which are generally fine for the kind of app-level code we are considering. Then genrpy can be extended to use the annotations, e.g. when it knows that a given variable is an integer, it should use a real integer in the produced code too, instead of a space.wrap(value), and real '+' '-' etc operations instead of space.add(), space.sub(), etc. In this way we produce interp-level code that will be: * unreadable * fast * automatically integrated with the rest of the interp-level code * still containing some space.xyz() operations if needed -- i.e. we can still use app-level code as a "macro language" to compactly write things like space.is_true(space.eq(space.getitem(w_list, space.wrap(i)), w_value)). A bientôt, Armin
Hi Armin,
If you look at md5.py, you'll see that it *is* already almost restricted Python. I can just move almost all functions over to the interpreter level.
But not by hand :-)
Yes, only some code is "essentially" application-level code. Some other code is quite "C-ish" already, and doesn't in any essential way need to correspond to space.xyz() calls. It can just "do its job" and return a result.
Yes
In this respect md5.py and sha.py are similar to _formatting.py: they are all general Python code (so meant for app-level) that is almost RPythonic enough that we could put the code at interp-level with only some amount of uglifying.
Yes
Didn't your mind go a full circle back to its starting point? I seem to remember that you started work on genrpy precisely to avoid to uglify _formatting.py by hand. Now you say that md5.py should better be put to interp-level manually...
No no. I'm getting further. I'm trying to figure out what can be just taken as is, as interpreter level code, and what needs to go into app space calls. I'm just not really decided what's right. I can start as I started: everything is a space.wrap of a space.whatever, and we need to have another simplification step. The alternative is to try to keep code as it is, if it is rpythonic, and just to identify the operations that RPython doesn't have, only wrap these, call the space, and immediately unwrap the result. This happened to me, when I saw that md5 is just already in shape togo into interp level, just a little class thing is needed.
While this is of course possible, the idea of genrpy was to do it automatically. I know the generated code must look incredible right now, because it's just calls to space.xyz().
Yes, and I have a simple problem running the stuff: What do I do over the space, and what do I directly? I think my generated functions should not be called via space,since the interface is interplevel,already. The things which I don't translate are those which use the space calls. Right?
But the idea is to run the annotation stuff over the graphs -- which assumes some restrictions, which are generally fine for the kind of app-level code we are considering. Then genrpy can be extended to use the annotations, e.g. when it knows that a given variable is an integer, it should use a real integer in the produced code too, instead of a space.wrap(value), and real '+' '-' etc operations instead of space.add(), space.sub(), etc.
SO what you are saying is that I don't careand just translate away? I thought it is a waste for stuff which is perfectly interplevel already, and I'd like to find this out and avoid translation.
In this way we produce interp-level code that will be:
* unreadable * fast * automatically integrated with the rest of the interp-level code * still containing some space.xyz() operations if needed -- i.e. we can still use app-level code as a "macro language" to compactly write things like space.is_true(space.eq(space.getitem(w_list, space.wrap(i)), w_value)).
Ok. But another thing that annoys me is that this macro language still has rpythonic restrictions, when I use the unmodified flow space. I think it should be full Python. I had the strong feeling that I did a whole mess with no real benefit. But I might be wrong. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
torsdagen den 2 december 2004 00.12 skrev Christian Tismer:
Ok. But another thing that annoys me is that this macro language still has rpythonic restrictions, when I use the unmodified flow space. I think it should be full Python. I had the strong feeling that I did a whole mess with no real benefit. But I might be wrong.
I think we need to define what RPython really is. Probably this is best done by maintaining a list of what limitations relative to Python there are. Initially, this list does not need to be exhaustive. Anything we know about restrictions will help. It is really hard to write any RPython if you don't know the limits and it is equally hard to know what to translate if you don't know which constructs that need the translation. If we can't nail anything down at the moment, I'd like to see an explanation why, as well as some sort of roadmap leading to a specification of what RPython is. Even if it is a highly moving target, we need to track the language with some sort of documentation. Jacob
Hi Jacob, On Thu, Dec 02, 2004 at 12:36:52AM +0100, Jacob Hallén wrote:
I think we need to define what RPython really is. Probably this is best done by maintaining a list of what limitations relative to Python there are. Initially, this list does not need to be exhaustive. Anything we know about restrictions will help.
But there is such a draft in the repository already: http://codespeak.net/svn/pypy/trunk/doc/objspace/restrictedpy.txt What is not really explained so far (and would be better put into some kind of glossary) is that R-Python is a set of "well-behaving promises" over the Python language. The name "RPython" puts maybe too much emphasis over it. Most "not-too-dynamic" programs are already R-Python or close. The concept of R-Python is by far not as important in PyPy as, say, the app-level vs. interp-level boundary. All our non-initialization-time interp-level code is R-Python by necessity, but a lot of our app-level code is also R-Python. Armin
Hi Jacob, hi all, [Jacob Hallén Thu, Dec 02, 2004 at 12:36:52AM +0100]
torsdagen den 2 december 2004 00.12 skrev Christian Tismer:
Ok. But another thing that annoys me is that this macro language still has rpythonic restrictions, when I use the unmodified flow space. I think it should be full Python. I had the strong feeling that I did a whole mess with no real benefit. But I might be wrong.
I think we need to define what RPython really is. Probably this is best done by maintaining a list of what limitations relative to Python there are. Initially, this list does not need to be exhaustive. Anything we know about restrictions will help.
Do you realize that in http://codespeak.net/pypy/index.cgi?doc/objspace/restrictedpy.html further down at "Object Restrictions" there is a basic list of rpythonic restrictions? I presume, the underlying restrictedpy.txt documentation could be enhanced from the experiences and enhancements at the last sprint, though. However, as long as we don't have a fully translated PyPy, we probably cannot fully complete the list.
It is really hard to write any RPython if you don't know the limits and it is equally hard to know what to translate if you don't know which constructs that need the translation.
With the above mentioned list we already should have most of what the restrictions on our Python usage are. Btw, IMHO we should always talk about restrictions to python usage, rather than talking about "RPython" as if it were some weird new language. One open topic regarding the restrictions is how to handle *args and **kwargs calls. We did a number of hacks at the last sprint to the flowobjspace and annotation to make them "work". But it's not satisfactory and produces quite a bit of the warnings that you see when running translate_pypy.py. Another open topic are dictionaries. We discussed an idea on IRC that dictionaries should only be statically prebuilt and not dynamically generated. But maybe this is not practical and we will eventually need to improve the annotator to handle dynamically built dictionaries. However, i am currently only aware of one truly dynamic usage of a dictionary which is in the Arguments class of the interpreter. Incidentally, this is connected to the other *args/**kwargs calling problem and thus we might try to solve both problems at once ... cheers, holger
hpk@trillke.net (holger krekel) writes:
One open topic regarding the restrictions is how to handle *args and **kwargs calls. We did a number of hacks at the last sprint to the flowobjspace and annotation to make them "work".
Uh, I don't think **args work in anyway at all.
Another open topic are dictionaries. We discussed an idea on IRC that dictionaries should only be statically prebuilt and not dynamically generated. But maybe this is not practical and we will eventually need to improve the annotator to handle dynamically built dictionaries.
Well, the SomeDict annotation is currently only used for the more dynamic variant of this, but is written as for the less dynamic variant. This is why translate_pypy crashes at the moment (eventually :). All the static dictionaries end up being SomePBCs, which makes sense, really.
However, i am currently only aware of one truly dynamic usage of a dictionary which is in the Arguments class of the interpreter. Incidentally, this is connected to the other *args/**kwargs calling problem and thus we might try to solve both problems at once ...
Extending SomeDict to cover uses as in argument.py probably isn't especially difficult, and is probably the first thing to try. Cheers, mwh -- Have you considered downgrading your arrogance to a reasonable level? -- Erik Naggum, comp.lang.lisp, to yet another C++-using troll
Hello Christian, On Thu, Dec 02, 2004 at 12:12:44AM +0100, Christian Tismer wrote:
If you look at md5.py, you'll see that it *is* already almost restricted Python. I can just move almost all functions over to the interpreter level.
But not by hand :-)
Well, that's a difficult problem, then: can we figure out automatically the proper interface to use md5.py as interp-level code? Maybe. This would be similar to the kind of hacks done in gateway.py: we could have a gateway function that inspects a class and generates an app-level class that is essentially a collection of built-in methods to access the original (now interp-level) methods. At a first glance this is independent from any translation stuff. It is independent from the other approach where we want to port code with some "essentially" app-level parts, which have to become space.xyz() calls. This was what I described: assume the app-level code is still fully RPython, pass it through the translator/annotator, and generate interp-level code from that. Now maybe there is a way to combine both approaches. Automatically? I don't know... We probably need a concrete example of both approaches before we can tell how to combine them for a 3rd concrete example. So probably both the inspect-to-generate-interface approach of gateway.py and the analyse-and-regenerate-everything approach of genrpy.py are worthwhile to pursue.
Ok. But another thing that annoys me is that this macro language still has rpythonic restrictions, when I use the unmodified flow space. I think it should be full Python.
Well, so far, a number of app-level helpers and modules are really full app-level Python, but there isn't much optimization we can do with that, at least without Psyco. The best I can imagine would be to turn the code in a series of space.xyz() calls and do some kind of local type inference to see what types can be known for sure -- but in my opinion that's a waste of time because very few types can be known for sure. A bientot, Armin.
participants (5)
-
Armin Rigo -
Christian Tismer -
hpk＠trillke.net -
Jacob Hallén -
Michael Hudson