Re: [pypy-dev] List and string in ootypesystem
Hi Antonio (and others) As you have probably seen, we started work on list support for ootypesystem here at the ongoing Leysin sprint. This cleared things up a bit for me on the conceptual level. Here's how OO lists work: - There is a new type constructor "List" in the ootype module. The low-level type of RPython list in the ootypesystem is a List instance with the item type of the list provided as an argument. E.g., an RPython list of integers has the low-level type List(Signed). - The interface of an OO list is defined by List._METHODS (see List.__init__). This interface is still incomplete, but it should be kept as small as possible, but large enough to support all operations permitted in RPython on lists. - When it comes to low-level operations, Lists are treated similarly to Instances. "new" is used to instantiate an (empty) list. Methods of the interface are rtyped as "oosend" operations with the name of the method. - Backends must special-case "new" with Lists and "oosends" to Lists. Basically, "new" can be mapped to the construction of a native list in the backend. The oosends can be mapped to native methods of the native list. All of this works for some basic list operations at this point (length, append, getitem, setitem), but there's still quite a bit of work to do for full list support in ootypesystem. After all this, the roadmap for bringing all other data structures to ootypesystem is now pretty clear to me: - dicts: Can be implemented analoguous to lists (type constructor, minimal interface etc.) - strings: Analoguous to lists, but no type constructor necessary - ranges, slices: Can be implemented as a special Instance (analoguous to tuples, in a way) I have two more days here at the Leysin sprint to work on lists. After that I again won't have much time to work on PyPy, so Antonio (or anyone else), you're very welcome to continue from here. I think the list code can serve as a good starting point for the other data structures. If there are any questions, I'm happy to answer them (if I can ;)). Cheers Nik
Hi Niklaus, you have just preceded me by about 2 hours... I would have sent a mail about rlist anyway. I've spent last days working on this topic: I tried to refactor the code for making rlist type-system specific as rtuple, rclass and others already was. It has been a bit difficult because it was my first hacking in the rpython directory: I had to read the sources carefully for understanding in deep how things works, and I'm not sure if I have completely understood the whole logic. I've just committed my work in the http://codespeak.net/svn/user/antocuni/pypy-antocuni directory. I'm not satisfied of it because it is a bit messy and I'm happy to know that now it's no longer needed because yours is surely better :-). By the way I think it has not been a waste of time because it let me to gain some knowledge of pypy's internals that will be useful in future. Niklaus Haldimann wrote:
I have two more days here at the Leysin sprint to work on lists. After that I again won't have much time to work on PyPy, so Antonio (or anyone else), you're very welcome to continue from here. I think the list code can serve as a good starting point for the other data structures. If there are any questions, I'm happy to answer them (if I can ;)).
Sure, I'll be happy to continue from here: it is much better than starting from scratch as I supposed to do! :-) I think you've saved me from a lot of headaches, thanks! Remind me to offer you a beer the first time we meet ;-). ciao Anto
Hi Antonio Antonio Cuni wrote:
I've spent last days working on this topic: I tried to refactor the code for making rlist type-system specific as rtuple, rclass and others already was. It has been a bit difficult because it was my first hacking in the rpython directory: I had to read the sources carefully for understanding in deep how things works, and I'm not sure if I have completely understood the whole logic.
I've just committed my work in the http://codespeak.net/svn/user/antocuni/pypy-antocuni directory. I'm not satisfied of it because it is a bit messy and I'm happy to know that now it's no longer needed because yours is surely better :-). By the way I think it has not been a waste of time because it let me to gain some knowledge of pypy's internals that will be useful in future.
Oops, I didn't intend to invalidate your work. ;) I actually checked your user directory yesterday, because you said in an earlier mail that you would work on the branch there. But since I didn't see any changes related to rlist I assumed you decided to postpone this ... What you did doesn't look so bad, I just looked at it. In general, I'm impressed that you found your way around the RTyper so easily. ;) The main difference to our work is that you created many new low-level operations for the list interface. Since there will also have to be a dict and string interface this would lead to an explosion of low-level operations. Our approach also makes testing of these data structures at a lower level easier (see test_oolist.py and test_oortype.py).
Sure, I'll be happy to continue from here: it is much better than starting from scratch as I supposed to do! :-) I think you've saved me from a lot of headaches, thanks! Remind me to offer you a beer the first time we meet ;-).
Hehe, looking forward to that. ;) Cheers Nik
Niklaus Haldimann wrote:
Oops, I didn't intend to invalidate your work. ;) I actually checked your user directory yesterday, because you said in an earlier mail that you would work on the branch there. But since I didn't see any changes related to rlist I assumed you decided to postpone this ...
No, I was working on that but I didn't commit because I wanted to get all things working before doing that... I've had some problem fixing all various modules that accessed rpython.rlist directly.
What you did doesn't look so bad, I just looked at it. In general, I'm impressed that you found your way around the RTyper so easily. ;)
Don't worry, it has not been so easy! :-)
The main difference to our work is that you created many new low-level operations for the list interface. Since there will also have to be a dict and string interface this would lead to an explosion of low-level operations. Our approach also makes testing of these data structures at a lower level easier (see test_oolist.py and test_oortype.py).
The reason beyond that is that I've found no other way to do this: I really wanted to create as few low-level ops as possibile, but I coudn't figure out how to do. I try to explain better what I did: my first attempt was to provide a low-level operation 'list_lenght' and then implement rtype_is_true in terms of 'list_lenght', so I wrote a thing like this: class ListRepr(AbstractListRepr): def rtype_len(self, hop): v_lst, = hop.inputargs(self) return hop.genop('list_len', [v_lst], resulttype=Signed) def rtype_is_true(self, hop): v_lst, = hop.inputargs(self) return hop.gendirectcall(ll_list_is_true, v_lst) def ll_list_is_true(lst): return lst is not None and len(lst) != 0 I hoped that the rtyper was smart enough to convert 'len(lst)' into my low-level op 'list_len', but it wasn't: indeed, it generated code that called len function for a generic PyObject* and that was not what I wanted. I tried to copy the implementation of lltypesystem.rlist.ll_list_is_true, but I couldn't because that can call the ll_lenght() method that I didn't have. Now it has no longer importance, but how could I do to get thing working? ciao Anto
Hi Antonio, On Tue, Apr 04, 2006 at 10:16:32AM +0200, Antonio Cuni wrote:
def ll_list_is_true(lst): return lst is not None and len(lst) != 0
I hoped that the rtyper was smart enough to convert 'len(lst)' into my low-level op 'list_len', but it wasn't: indeed, it generated code that called len function for a generic PyObject* and that was not what I wanted. I tried to copy the implementation of lltypesystem.rlist.ll_list_is_true, but I couldn't because that can call the ll_lenght() method that I didn't have. Now it has no longer importance, but how could I do to get thing working?
It is probably less confusing to call a method like .length() instead of using len() at this level again. But in both cases, you need to add support for this in the annotator and the rtyper again -- the ll_list_is_true() function is itself passed through both of them. I see you added SomeOOList to annoation.model.lltype_to_annotation(). There is already a generic 'def len()' in the base class SomeObject, so that's how the annotator is happy with your ll function's 'len(lst)'. Fine here. If you wanted a .length() method instead, you would need a 'def method_length()' in unaryop.py. On the rtyper side, you need something similar to rpython/rptr.py that maps SomeOOList back to its low-level type, with yet another Repr. It's this repr that must implement the operations you want to be able to use in low-level helpers; e.g. rtype_len() if you want len(lst) to work; or if instead you use .length() in low-level helpers, then you would need an rtype_method_length() in the repr corresponding to SomeOOList (by opposition to the rtype_len() in the repr corresponding to SomeList). Nik's approach is to map lists to reguar OO classes and instances, which are already supported in the annotator/rtyper with a similarly indirect approach: SomeOOClass/SomeOOInstance in the annotator, which rpython/ootypesystem/rootype.py maps back to the low-level OO types. Just like rptr.py, this rootype.py is only needed to support low-level helpers. A bientot, Armin
Hi Armin, hi Niklaus Armin Rigo wrote:
I see you added SomeOOList to annoation.model.lltype_to_annotation(). There is already a generic 'def len()' in the base class SomeObject, so that's how the annotator is happy with your ll function's 'len(lst)'. Fine here. If you wanted a .length() method instead, you would need a 'def method_length()' in unaryop.py.
On the rtyper side, you need something similar to rpython/rptr.py that maps SomeOOList back to its low-level type, with yet another Repr. It's this repr that must implement the operations you want to be able to use in low-level helpers; e.g. rtype_len() if you want len(lst) to work; or if instead you use .length() in low-level helpers, then you would need an rtype_method_length() in the repr corresponding to SomeOOList (by opposition to the rtype_len() in the repr corresponding to SomeList).
Nik's approach is to map lists to reguar OO classes and instances, which are already supported in the annotator/rtyper with a similarly indirect approach: SomeOOClass/SomeOOInstance in the annotator, which rpython/ootypesystem/rootype.py maps back to the low-level OO types. Just like rptr.py, this rootype.py is only needed to support low-level helpers.
Thank you for the great explanation: now I see things much more clearly, especially the role played by rptr.py and rootype.py, which I didn't understand very well before now. It seems that question by question I'm really getting into PyPy's logic... I hope I will be able to finish my apprenticeship soon :-) ciao Anto
Antonio Cuni <anto.cuni@gmail.com> writes:
It seems that question by question I'm really getting into PyPy's logic... I hope I will be able to finish my apprenticeship soon :-)
Unfortunately, I don't think you are Swiss, so you are probably doomed to being an apprentice forever like this rest of us :) More seriously, you seem to be doing very well! Hope you are having fun too :) Cheers, mwh -- M-x psych[TAB][RETURN] -- try it
participants (4)
-
Antonio Cuni -
Armin Rigo -
Michael Hudson -
Niklaus Haldimann