Mailman 3 June 2003 - pypy-dev

sprints goals / restricted python?
by holger krekel 18 Jun '03

18 Jun '03

Hello pypy and hello Armin :-) ASFAIK your idea of restricted python is to implement a TranslationObjSpace which emits C-code. We would e.g. run the interpreter (by running e.g. tests) against StdObjSpace and use the TranslationObjSpace to generate a C-file which is - when compiled and run - our restricted interpreter which can execute bytecode. I don't know but i think going the TranslationObjSpace way might turn into complicated code. For example how do you "detect" a loop from an objectspace's POV? It just sees a series of a series of ObjSpace-Method accesses and it seems hard to reverse engineer the loop. (I guess you say this loop-unfolding is basically a space/time trade-off :-) please correct me, if/where i am wrong. Anyway, I think the above approach is pretty ambitious for a three to four day sprint but this isn't the only "goal" criterium, of course. If i am not mistaken, we don't have any framework to generate C-code and make it "self-runnable" at the moment. That's why i started thinking about Pyrex which provides a framework (written in python) for generating C-code. The input is mostly python-compatible with some additional "type" information. It seems to me that if we (focus on) - make the interpreter record/annotate names with types (all names: function names, local names, global names, ...) - use this "annotation" information to generate Pyrex modules - use the pyrexc compiler to generate C-Modules from the generated Pyrex modules, we might get a working (partial-)C-interpreter module by the end of the sprint. To actually perform the annotation we could use an AnnotationObjSpace which proxies to the StdObjSpace but additionally annotates "names" with the types of the objects beeing referenced. Bascially we might just need to "intercept" all setattr__ANY_String_ANY calls so that for a given AnnotationObjSpace.setattr(x,y,z) call we roughly perform some hands-on annotation by storing it in "x.__pypytypemap__" which is a dictionary containing name-dictionary pairs. the latter dictionary then records all the types (or even values?) that have ever been associated with the name. Of course, the above generation series would still require CPython (because the generated C file compiles against C-Python). But later on, we could - hack up pyrex to generate C-files which don't require a full CPython anymore - hack up pyrex to be used with a TranslationObjSpace as a second means to generate the restricted interpreter. One of the positive points about this approach is that we use existing tools/frameworks (written in python) and we can probably split the group on various sub-tasks easily. But i am very open to your corrections ... cheers, holger

3 5

about to check in refactored test-framework
by holger krekel 18 Jun '03

18 Jun '03

hello pypy, since yesterday evening i cleaned up the all of our test stuff. I haven't checked it in, yet. I'd like to know if anybody has any strong feelings about the way it currently is... And here is how the refactored test stuff works: - all the testsupport, testcts, testwice and all the code duplications that came with it are gone. - not need to set any PYTHONPATH anymore to start a test or interpreter/interactive.py or interpreter/main.py - you can start any test by simply calling it directly via python test_XXX.py - you can run all tests by python test_all.py # in the root directory - you can run all tests below your current working dir say e.g. in interpreter/ python ../test_all.py # will start from the current dir - you can run all tests in one object space via python test_all.py -S # standard objectspace python test_all.py -T # trivial objectspace python test_all.py -ST # first standard, then trivial objectspace - you can pass some regular expressions python test_all.py pypy.interpreter # will only run tests whose path matches pypy.interpreter python test_all.py -S "%pypy.(obj|app)space" # will avoid objspace and appspace tests and otherwise run with # stdobjspace, the '%' inverts the match, btw - autopath configuration. every startable script (the tests among them, but also interpreter/interactive.py interpreter/main.py) first does a import autopath which configures the python path to include the directory which contains 'pypy'. moreover autopath provides some attributes pypydir the 'pypy' directory path itself this_dir the directory where your "autopath" lives useful for finding test data if you run pypy/tool/autopath.py then this will copy itself over all other 'autopath.py' instances and report "sync" status. No ugly shell scripting anymore. OTOH autopath.py will hardly need to be modified anymore because it just does path manipulation. - every test_*.py usually does import autopath from pypy.tool import test and derives from test.TestCase or test.AppTestCase and on __name__=='__main__' invokes test.main() to get to the "cmdline switched" objectspace you call test.objspace() to get to a specific objectspace you call test.objspace('std') or test.objspace('trivial') Some stuff regarding running tests successively in StdObjSpace and TrivialObjSpace (switch "-ST") is not completly right at the moment. But this is not so much a thing of the test framework, i think. As this checkin involves a lot of files i'd like to do the checkin ASAP to avoid conflicts. Any oppinions? regards, holger

2 2

Re: [pypy-dev] Oscon Paper
by Laura Creighton 18 Jun '03

18 Jun '03

In a message of Wed, 18 Jun 2003 15:42:42 +0200, Armin Rigo writes: >Hello Laura, > >On Tue, Jun 17, 2003 at 08:58:07PM +0200, Laura Creighton wrote: >> it is now in http://codespeak.net:8080/svn/pypy/trunk/doc/oscon2003-pap >er.txt > >The paper looks good -- I guess I will steal a couple of ideas for my own > >EuroPython presentation if you would allow me to do so :-) > > >Armin Of _course_ I don't mind! Let me know what else should go in there ... or modify it yourself, if you are so inclined. Laura

1 0

Oscon Paper
by Laura Creighton 18 Jun '03

18 Jun '03

Holger had a better idea. Share it in the doc directory. it is now in http://codespeak.net:8080/svn/pypy/trunk/doc/oscon2003-paper.txt and the wiki points to it. Oddness -- when i did an scn ci in the doc directory, after adding it, it said that builtins.txt changed. That is just the file where we listed whether the builtins were hard or messy or types and the like. I can't believe that I changed it since the sprint .... If somebody changed it, I guess there is a chance that I klobbered it with an old version, because it looks pretty much as I expect it to.. Laura

2 2

helps if i have a pypy-dev alias ....
by Laura Creighton 17 Jun '03

17 Jun '03

Date: Tue, 17 Jun 2003 20:12:23 +0200 To: pypy-dev(a)ratthing-b246.strakt.com cc: lac(a)ratthing-b246.strakt.com From: Laura Creighton <lac(a)ratthing-b246.strakt.com> Subject: OSCON paper Return-Path: <lac(a)ratthing-b246.strakt.com> I just put my first draft of the OSCON paper up on the Wiki. It is very rough. I am not happy with it. It contains lots of things from all over the Wiki, as well as some things that I wrote myself. I still don't know what it is that anybody wants me to say at OSCON, so I sort of threw this together, and it shows. We can edit this like crazy up until the day I give the talk (July 9), <well, we could afterwards, but what is the point>. Some of the paper is just ugly because I don't know how to make code look like code. Variable names look like they ought to be links, but I decided not to change that since I want the original version in the text I send to OSCON. If anybody wants to just hack it up and make it better, feel free. If you just want to send me suggestions on how _I_ can make it better, that is fine as well. If you want it on some different part of the wiki, less prominent -- that is probably a good idea as well, but i didn't know where would be a good place to put it. take care, Laura

1 0

Re: [pypy-dev] Re: Funding
by Laura Creighton 17 Jun '03

17 Jun '03

I am working on getting us European Union funding. I want to discuss this with you in person asap. The best way, I am told, to get funding is to build a coalition between universities and small and medium sized businesses. Do you have any experience with this? The Bang & Olufsen name will be incredibly useful, in itself. I am hideously busy today -- can this wait until we meet in person? I have the OSCON PyPy paper to submit today (although Vee, the Papers chair, is well aware that the whole thing will change based on next week). You all can tell me what I ought to have said instead ... I am really looking forward to meeting you. Laura

1 0

Funding
by Anders Lehmann 17 Jun '03

17 Jun '03

What is the status for the funding issue ? Are there some things that we could prepare before the sprint? (Like what programmes are we aiming at, size of the budget ...) I think I need to begin priming my boss this week, because the holiday season sets in after EuroPython, and then it will be august before we are up to speed again Kind regards Anders Lehmann

2 2

RE: [pypy-dev] App-space tests
by roccomoretti＠netscape.net 17 Jun '03

17 Jun '03

Michael Hudson <mwh(a)python.net> wrote: >Michael Hudson <mwh(a)python.net> writes: > >> roccomoretti(a)netscape.net (Rocco Moretti) writes: >> >>> Before I realized that metaclasses were involved, I first thought of >>> using a special subclass of TestCase - probably by overloading >>> TestCase.__call__() to appropriately wrap the function before it is >>> run. >> >> Yes, that would probably work. > >And it does, after a fashion. Well, it works fine as far as I can >tell, it's just that running the assertFoo methods in app-space *too* >(what the attached does) isn't ideal. OTOH, the fact that doing so >*works* is something we can be proud of. Glad to hear it works! Re: running method in appspace - I hoped that was something that was "fixed" with the addition of the PyByteCode Object. I'm assuming (as I wasn't at the sprint and haven't heard about it otherwise) that a regular bytecode object gets run on the CPython interpreter, and a PyByteCode object gets run on the PyPy interpreter. Although that's what I assume, I haven't been able to confirm this in the code, as I loose track of the control flow after calling ObjSpace.call(). (I started tracking from the CALL_FUNCTION opcode in opcode.py) >The only decision to be made is: do we distinguish between app-space >tests by a naming convention (as attached: any test starting test_app_ >runs in app space) or by inheriting from a different TestCase >subclass? I can't make my mind up which I prefer. I'd vote for seperate TestCases. App space should be suitably separated from interpreter level such that we avoid confusion. This also allows setup () and teardown() methods to work semi-sanely. It will also give a consistent interface to the assert*() methods (whether they are being called from interpreter level or app. level.) >In either case, I want to check something like this in, soon. The >implementation can be made more elegant/faster in due course. I'm +1 on checkin - as you say, it doesn't break anything, and we can always improve it later. -Rocco __________________________________________________________________ McAfee VirusScan Online from the Netscape Network. Comprehensive protection for your entire computer. Get your free trial today! http://channels.netscape.com/ns/computing/mcafee/index.jsp?promo=393397 Get AOL Instant Messenger 5.1 free of charge. Download Now! http://aim.aol.com/aimnew/Aim/register.adp?promo=380455

2 1

RE: [pypy-dev] Objects and types in the stdobjspace
by roccomoretti＠netscape.net 16 Jun '03

16 Jun '03

Michael Hudson <mwh(a)python.net> wrote: >I've had another idea: give unittest_w.TestCase_w a metaclass that >looks for methods called 'test_app_whatever' and arrange for such >methods to be called as objectspace methods. Sounds like a wonderful solution, but I have to confess that my eyes glaze over when trying to understand metaclasses - I probably should put more of an effort into learning about them one of these days... Is using metaclasses really nessasary? Before I realized that metaclasses were involved, I first thought of using a special subclass of TestCase - probably by overloading TestCase.__call__() to appropriately wrap the function before it is run. If we can use the subclass approach, we could potentially simplify things by having two seperate TestCase subclasses, one for interpreter level and one for application level tests. This would potentially require splitting of a single test case into two, one each for application and interpreter levels, whereas your implementation appears to allow both types of tests in the same class definition. But feel free to ignore this as I haven't tried putting it into code and have no clue what the problems with it may be. -Rocco __________________________________________________________________ McAfee VirusScan Online from the Netscape Network. Comprehensive protection for your entire computer. Get your free trial today! http://channels.netscape.com/ns/computing/mcafee/index.jsp?promo=393397 Get AOL Instant Messenger 5.1 free of charge. Download Now! http://aim.aol.com/aimnew/Aim/register.adp?promo=380455

2 2

W_XxxObjects
by Armin Rigo 16 Jun '03

16 Jun '03

Hello ! I have slowly come to think that wrapping every object we want to see at the application-level in a W_XxxObject class is conceptually a good idea but not necessarily pratically required. I used to think that it would be useful for the translator-to-C. Now I'm no longer convinced. Contrast these two implementations of integer comparison : def int_int_lt(space, w_int1, w_int2): i = w_int1.intval j = w_int2.intval return space.newbool( i < j ) vs. def lt__int_int(space, i, j): return bool(i < j) It seems to me that the first implementation is a typical example of the boilerplate kind of code that we hoped the project would remove, isn't it? How I see things right now would be that any type (or class) of restricted Python would be suitable for implementing Python objects. For complex objects nothing changes: we still need classes acting as structures, like a W_ListObject with the over-allocation machinery fields. But the most simple objects could be implemented just directly with simple interpreter-level objects instead of requiring them to be classes W_XxxObject. This idea comes from the fact that over the time the W_XxxObject class definitions themselves have become more and more skeletical, with all interesting functions defined as multimethods outside the class. The only relevant piece of information left in the W_XxxObject classes today is the 'statictype' attribute, which would be easily to implement in some other way. Yes I know I was the guy who pushed for explicit W_XxxObjects in the first place :-) This simplification could also avoid some problems with objects that are not properly wrapped, because the simple interpreter-level objects don't need any explicit wrapping any more. Actually "wrapping" may become a transparent operation: the translator-to-C will detect the need automatically. Example: if there is an (interpreter-level) variable 'x' containing an integer, and if at some point we use it in a call to a function that has been found to expect a wrapped object ( = any one of the available implementation of objects) then we must add some tag that says the int is really an int (as opposed to any of the other implementations of objects). The translator will have to do this kind of operations anyway for a lot of other cases. It just seems to me that it can do it pretty well for this case too. Wrapping would now be automatical. It would make the PyPy code slightly less strictly-typed, i.e. more Pythonic :-) and as we will need the translator at some point (probably soon) we may rest confident that typing bugs will eventually be caught by this analysis phase. In other words I am suggesting that we can simply use the basic types we allowed in restricted python to implement some of the xxxobject.py: * intobject.py - int (ints could be forced to r_ints before the operations for which it matter, e.g. "def lshift__int_int(self, i, j): ... return r_int(i) << j") * stringobject.py - str (this makes sense, as it enforces strings to be immutable once they're built; for mutable strings (e.g. during building) we can use rarray.CharArray or maybe just lists of characters?) * boolobject.py - bool (in Python 2.2 we could actually define a bool class, as long as we write "def lt__int_int(space, i, j): return bool(i < j)" with the explicit bool()) * floatobject.py - float * noneobject.py - guess * tupleobject.py - list (this is just a suggestion, a list being in restricted python the equivalent of allocated memory.) A bientôt, Armin.

6 11