Mailman 3 July 2005 - pypy-dev

bug in http://codespeak.net/moin/pypy/moin.cgi/FrontPage?action=show
by Laura Creighton 10 Oct '05

10 Oct '05

If you click on the 'documentation' link you get a traceback! ooops. I will look at this later unless somebody beats me to it. Laura

5 4

Build and test failures
by Ben.Young＠risk.sungard.com 09 Aug '05

09 Aug '05

Hi Everyone, Just synced to head this morning and noticed a couple of errors in the c translation process. This first is that on windows, a couple of headers are not available. I fixed this by patching locally with: Index: translator/c/extfunc_include.h =================================================================== --- translator/c/extfunc_include.h (revision 14483) +++ translator/c/extfunc_include.h (working copy) @@ -6,9 +6,9 @@ /******************************************************************/ -#include <unistd.h> +/*#include <unistd.h> #include <sys/types.h> -#include <sys/stat.h> +#include <sys/stat.h>*/ #include <fcntl.h> #include <time.h> #ifndef PATH_MAX Then, when the tests are run I get: $ ./test_all.py translator/c/ ============================= test process starts ============================= testing-mode: inprocess executable: c:\Python24\python.exe (2.4.1-final-0) using py lib: c:\Documents and Settings\YoungB\dist\py <rev unknown> translator\c\test\test_annotated.py[19] ................... translator\c\test\test_backendoptimized.py[39] ....................................... translator\c\test\test_database.py[17] ................. translator\c\test\test_exception.py[3] ... translator\c\test\test_extfunc.py[4] F... translator\c\test\test_genc.py[8] ........ translator\c\test\test_lltyped.py[2] .. translator\c\test\test_notype.py[33] ................................. translator\c\test\test_operation.py[1] . translator\c\test\test_support.py[7] ....... translator\c\test\test_symboltable.py[1] . translator\c\test\test_typed.py[38] ...................................... _______________________________________________________________________________ _________________________ entrypoint: test_time_clock _________________________ def test_time_clock(): def does_stuff(): return time.clock() f1 = compile(does_stuff, []) t0 = time.clock() t1 = f1() assert type(t1) is float t2 = time.clock() E assert t0 <= t1 <= t2 > assert 165.0 <= 2.5683609740123038e-005 [c:\Documents and Settings\YoungB\dist\pypy\translator\c\test\test_extfunc.py:16] - - - - - - - - - - - test_time_clock: recorded stderr - - - - - - - - - - - - [annrpython] (pypy.translator.c.test.test_extfunc:9) does_stuff -> SomeFloat() [annrpython] (pypy.rpython.rclass:575) ll_runtime_type_info__objectPtr -> SomePtr(ll_ptrtype=<* RuntimeTypeInfo (opaque)>) [annrpython] (pypy.rpython.module.ll_time:14) ll_time_clock -> SomeFloat() [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool(const=True) [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool() [annrpython] (pypy.rpython.rclass:565) ll_type__objectPtr -> SomePtr(ll_ptrtype=<* Struct object_vtable { parenttypeptr: * Struct objec t_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(const=<* struct object { typeptr=... }>, ll_ptr type=<* GcStruct object { typeptr: * Struct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opa que), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(ll_ptrtype=<* GcStruct object { typeptr: * Stru ct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:53) ll_raise_OSError__Signed -> SomeImpossibleValue() _______________________________________________________________________________ =========== tests finished: 171 passed, 1 failed in 263.86 seconds ============ I think this is because when clock is called in the extension it thinks it is the first time and returns the very small value. I don't really know though. I hope you all had fun at the sprint! Cheers, Ben

3 7

day 6: breakthrough!
by hpk＠trillke.net 31 Jul '05

31 Jul '05

hi folks, two and a half years later ... in the very same room where the first PyPy sprint happened ... we got to the point where PyPy basically runs on its own, after whole-program type inference and translation to C! The sixth day was a very productive day in various respects and we still had a long and nice lunch break. If you want to see the visual flow of events then revisit the pictures i took: http://codespeak.net/~hpk/hildesheim2-sprint-www/ Here is a more detailed overview of what happended on the sixth day: - Carl, Samuele and holger finished the Heidelberg sprint announcement, updated the website and the announcement was sent out. See the home page for more details and links. - Christian with initial help from Samuele went off to work on the interpreter level marshal module which we will need for better bootstrapping. Unfortunately, Christian had a bad stomach and missed much of the fun of day 6. - Armin and Holger as well as others continously restarted the whole translation process to fix small bugs and issues here and there. In between - they extended testing to allow applevel tests to easily access values setup_*'ed from interpreter level. - they fixed the posix module which was completely broken with respect to exception handling (you cannot let escape interpreter-level exception to pypy application level without carefully wrapping them in an OperationError). Samuele helped with making os.strerror available as an external (c-backend supported) function. - analysed and fixed with Samuele and Carl segmentation faults ... but note that GCC was behaving very fine and it turned out that our c-code generator currently consumes more RAM than GCC compiling the result at the moment :-) - they also started refactoring the translation driving interface (unfinished) - Carl Friedrich and Samuele used and extended/fixed the ll-interpreter which was sometimes able to discover possible segfaults and problems earlier (and clearer) than just running the C-backend. - Carl Friedrich started working on GC in PyPy in constant consultation with Samuele and did first working drafts in pypy/rpython/memory. Basically the whole memory simulator is working and gives nices errors if you do something strange like accessing freed or uninitialized. Furthermore they implemented annotation for memory addresses. - Richard communicated with Eric van Riet Paap (who has made good progress on the llvm-backend off-sprint!) and refactored, extended LLVM a bit driven by test suggestions from Armin. - the group discussed that there likely is a deeper exception handling problem lurking: implicitely exceptions are attached to the last (high level) operation in a block but when the RTyper converts this operation into possibly many low-level operations then it's unlikely that we can preserve the implicit assumption that only the last (low-level) operation possibly raises an exception. solution: unknown at this point. We should probably write down a few scenarios regarding exception handling describing the interactions/processing of exceptions in flowobjspace/annotation/rtyping - Armin communicated with Niklaus Heidimann who meanwhile has successfully implemented the _sre module as part of his SoC project. We'd like to first integrate this as an application level module but Niklaus wants to transform it to an interpreter-level one soon. - After we got the initial breakthrough run (goal/app_example.py running on top of translated non-faking PyPy) we continued a bit with various example and fixes. Running example code and modules showed very good results (and no segfaults). After having drunk champagne Samuele quickly mapped the good time we were having into a better 'time' module for PyPy. With that we could easily run the unmodified pystone.py on top of PyPy and reached some 337 pystones (as opposed to 9 pystones with PyPy on top of CPython). We are happy with this because translated PyPy seems to run stable and we are not using any optimizations yet. We quickly agreed that we only want to tackle optimizations at this point that result in at least factors-of-2 speedups rather than 5-10% speedups. Again please note that PyPy - as opposed to other python implementations - generally tackles compliancy, proper implementation and tests first before diving into optimizations. ok, so much so far. Today - on the last day - we are probably going to discuss on how to continue from here and also tell how you can actually run the translation and the translated PyPy yourselves (the current pypy-dist is able to but it currently requires a bit of manual tweaks at the very invocation-end, maybe you can already figure it out looking at the pictures and the entry_point code :-). cheers, holger

4 4

Re: [pypy-svn] r15405 - in pypy/dist/pypy: module/__builtin__/test module/__builtin__/test/impsubdir/compiled module/_codecs/test module/posix/test translator/goal translator/llvm2/module
by Eric van Riet Paap 30 Jul '05

30 Jul '05

Hi Holger, Is there some svn/vi setting I could use to prevent this eol problems? Eric hpk(a)codespeak.net wrote: >Author: hpk >Date: Sat Jul 30 12:54:42 2005 >New Revision: 15405 > >Modified: > pypy/dist/pypy/module/__builtin__/test/impsubdir/compiled/ (props changed) > pypy/dist/pypy/module/__builtin__/test/impsubdir/compiled/__init__.py (props changed) > pypy/dist/pypy/module/__builtin__/test/impsubdir/compiled/x.py (props changed) > pypy/dist/pypy/module/__builtin__/test/test_buffer.py (props changed) > pypy/dist/pypy/module/_codecs/test/ (props changed) > pypy/dist/pypy/module/_codecs/test/__init__.py (props changed) > pypy/dist/pypy/module/_codecs/test/autopath.py (props changed) > pypy/dist/pypy/module/_codecs/test/test_codecs.py (props changed) > pypy/dist/pypy/module/posix/test/test_posix2.py (props changed) > pypy/dist/pypy/translator/goal/targetsegfault.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_math.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_os.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_os_path.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_time.py (props changed) > pypy/dist/pypy/translator/llvm2/module/support.py (props changed) >Log: >FIXEOL > >_______________________________________________ >pypy-svn mailing list >pypy-svn(a)codespeak.net >http://codespeak.net/mailman/listinfo/pypy-svn > >

2 1

PyPy sprint announcement: Heidelberg 22nd - 29th August 2005
by Carl Friedrich Bolz 30 Jul '05

30 Jul '05

PyPy Sprint in Heidelberg 22nd - 29th August 2005 ================================================== The next PyPy sprint will take place at the Heidelberg University in Germany from 22nd August to 29th August (both days included). Its main focus is translation of the whole PyPy interpreter to a low level language and reaching 2.4.1 Python compliancy. To learn more about the new PyPy Python implementation look here: http://codespeak.net/pypy Goals and topics of the sprint ------------------------------ One of the main goals of the sprint is going to be a 0.7 release of PyPy. The 0.7 is meant to be the first self-contained PyPy version but still alpha with some C-extension modules missing and not meant for production use. Therefore the topics of the sprint will be: - translation efforts: it's not completely clear now what related task will be left at that time - work on 2.4.1 compliancy: there are some failing compliancy tests on which we plan to work - rewriting c-extension modules/integrating existing rewrites - all kinds of small release issues - possibly some more LLVM efforts Location & Accomodation ------------------------ The sprint will be held in a conference room of the Institute of Physics, University of Heidelberg. We have WLAN connection there and small kitchen to make tea and coffee. `See here`_ how to get there from the Heidelberg main station. There is a huge number of hotels_ in Heidelberg, all of which are unfortunately rather expensive. It should be no problem to reach the sprint venue from anywhere in Heidelberg. As an alternative you could also take a `hotel in Mannheim`_ which you can reach in 15 min from Heidelberg with the train. .. _`See here`: http://www.physi.uni-heidelberg.de/physi/front/anfahrt.en.php .. _hotels: http://www.cvb-heidelberg.de/index_eng.html .. _`hotel in Mannheim`: http://www.hotels.de/en/mannheim.htm Exact times ----------- The Pypy sprint is held Monday 22nd August - Monday 29th August 2005. Hours will be from 09:00 until people have had enough. It's a good idea to arrive a day before the sprint starts. Network, Food ------------- The Heidelberg Altstadt can be reached in approximately 10 min from the sprint venue. There you will find all kinds of restaurants and fast food places. You will probably need a wireless network card to access the network although we could set up a WLAN to cable bridge if necessary. Registration etc.pp. -------------------- Please subscribe to the `PyPy sprint mailing list`_, introduce yourself and post a note that you want to come. Feel free to ask any questions there! There also is a separate `Heidelberg people`_ page tracking who is already thought to come. If you have commit rights on codespeak then you can modify yourself a checkout of http://codespeak.net/svn/pypy/extradoc/sprintinfo/heidelberg-people.txt .. _`Heidelberg people`: http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/heidelberg-people.h… .. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint -- Carl Friedrich Bolz & the PyPy-Team

1 0

hildesheim2 day 2 + 3 report / status
by hpk＠trillke.net 29 Jul '05

29 Jul '05

Hi folks, the end of the world is near ... following up on Carl's day 1 report here are the main results of what happened up until now at the hildesheim2 sprint (we are in the break day currently which we spent in the city, at Wing-Tsun lessons or at family meetings). The main results so far are: - any faking can be removed now with python2.4 py.py --file --nofakedmodules so on the trunk you should see no 'faking' messages during startup anymore. It will take some time mostly because of the 'file' implementation living at app-level. Note that we have some problems running on top of Python2.3 at the moment and also some more problems with PYC file support. - Annotation can now complete without resorting to any SomeObjects anywhere (!!) - RTyping has only 2 Errors remaining. The next bigger problem is getting the compiler package integrated and bootstrapped properly. - note that we are not working from the snapshot but the trunk currently. After the sprint we probably return to the snapshot approach for translation. If you want to see more detailed daily-updated progress reports i suggest to read and follow the commits on http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/hildesheim2-plannin… Moreover, i shot a few pictures which i put here: http://codespeak.net/~hpk/hildesheim2-sprint-www and here is a picture of the surrounding place (Trillke-Gut, my living&working base): http://www.trillke.net/images/HomePagePictureSmall.jpg cheers, holger P.S.: It appears that we will have to continue our investigations regarding some of the current problems on the astral plane. Armin and Carl has already spend some time on assembler level yesterday to hunt down the weirdest TCC interactions. But there are other confusing events as well. um, we watched 'the nine lives of Thomas Katz' yesterday night, btw. Anyway, we are still taking bets if the first C-generated annotated RTyped PyPy is to produce a segmentation fault in the C-compiler or in the resulting binary.

4 5

Hildesheim sprint report 4 + 5
by Carl Friedrich Bolz 29 Jul '05

29 Jul '05

Hi PyPy-dev, somre more news from the Hildesheim sprint (I try to do this in some sort of chronological order although the days are kind of mixed in my mind): Thursday: - Holger and Richard worked on refactoring the mess that is the option handling of PyPy. Now it is possible to. Now the handling of options is unified in one place. It is now possible to tell PyPy exactly which modules to fake and for which to import our own implementation. - Armin and me completed our implementation of the math module, which is now completely translatable. - One of the big problems of the last days for the translation effort was that we had a bootstrap problem: our interplevel compiler and parser are not entirely done yet, so we need applevel version. These need to be imported using a parser and a compiler. Damn. At first we tried to get pyc-file support working, which involved finishing marshal. This approach was dropped because marshal used codecs which lives on applevel and therefore cannot be imported. Instead Armin came up with a hack: Basically we cheat during bootstrapping and use the underlying compiler and parser. Later (before translation) it is replaced by a real applevel implementation. Armin worked on this the basically the whole Thursday afternoon. In the evening it turned out that this approach leads to *exactly* the same problem. Armin, Samuele, Christian and me fixed Thursday evening. It kind of worked in the end but it introduced strange translation problems that Armin and Samuele fixed Thursday night (last checkin message: 3:45am). Friday: - Holger worked on py.test's stdout capturing because after some recent refactoring it didn't work any more for PyPy applevel tests. - Christian started reimplementing marshal on interplevel. - Richard tried once more to get LLVM prebuilt constants to work. - basically the day was spent trying to translate pypy (which involves watching PyPy: The Movie -- Two and a Half Years Later. We sit and watch the translator running on the beamer until it crashes). After a few typer errors and a bug in exception handling we actually managed to reach the point where the translator generated C code! To reach this you need quite a lot of RAM: something around 1GB just for the translator, annotator, rtyper and genc. The whole generated code is something like 120 Mb big (all in one file!). Surprisingly the compiler even manages to do something with it (which just means that it produces errors and does not just crash). While trying to actually translate PyPy we found various ommissions in various places: Some external functions were missing. While doing this the following compiler error was found: "testin_1.c:306344: error: 'Is_Unsigned_Division_Really_Useful' undeclared (first use this function)" well, obviously it is :-) - Armin worked on fixing an order problem: Some struct declarations have to be written to the C file in the right order because they depend on each other. - Samuele and me tried to run the flowgraph of PyPy on the llinterpreter (yay, yet another level!) which turned up some bugs in the RTyper and in the PyPy interpreter but mostly space operations that were still missing from the llinterpreter, mostly overflow and division by zero checking. Well, I'm pretty sure that I forgot some things but I'm too tired now. G'night, Carl Friedrich

1 0

Confused about RPython
by Niklaus Haldimann 29 Jul '05

29 Jul '05

Hi everyone I am currently seriously confused about the notion of RPython, and I hope someone can show me the light here. ;) I guess my basic problem is: How do I find out if some code is RPython? And especially in the context of the array module im currently working on, how do I find out if a module is RPython? One of my working hypothesis was that whatever can be geninterped is RPython. But by that definition, a *lot* more comes out as RPython than the documentation says. Is it just that the documentation is out-of-date or is geninterping not a good criterion? E.g., when Christian states that _codecs is now RPython, does that just mean it can be geninterped or is there something else to it? Armin just had a checkin where he replaced str.isalpha() with something else because string methods are not RPython. But I can geninterp and even C compile string methods just fine. Did he use the term "RPython" loosely, meaning something like "what can be usefully annotated" (ie, string methods always result in SomeObject, that's probably why they're frowned upon in the interpreter code)? Cheers Nik

3 5

Re: Strategy for array module?
by Armin Rigo 29 Jul '05

29 Jul '05

Hi Niklaus, Sorry for the delay. I've indeed been quite deeply into the PyPy sprint... At which level and with which style the array module should be implemented is indeed a bit unclear. If you are going for an app-level version, you should not worry too much about geninterp, I guess. The code you already started to write makes sense. Geninterp is most useful to solve bootstrapping issues, where we cannot import yet some app-level code but we can run its geninterp'ed version (e.g. for the _exceptions.py module). Of course geninterp gives some performance improvement too. The problem with array is that it is designed with the following model in mind: a bunch of memory out of which Python values can be read on demand. The other point of view -- a list of Python values that can be turned into a big string on demand -- make methods like byteswap() hard to implement, specially for arrays of floats or doubles. Not necessarily impossible, though. If this app-level approach works out, I wouldn't invest too much effort in geninterp compatibility, because at some point in the future we might have to think about a quite different approach: adding the notion of integers and floats of specific sizes, at least in the so-called "low-level" types. I guess that an efficient implementation of array should be written to take advantage of these. This will likely require as-yet-unspecified ways to interact between regular interp-level code and this low-level-typed code. In summary I guess that writing a pure app-level array module makes sense, with the buffer interface working for PyPy only. On Tue, Jul 26, 2005 at 06:15:01PM +0200, Niklaus Haldimann wrote: > As I probably told you, Lutz Paelike has sent me an incomplete array > implementation he wrote at the PyCon sprint. He uses StringIO to > represent the arrays internally as pure binary data. This is quite > sensible if you are after a quick pure Python implementation. But of > course it's useless if you want to be RPythonic. I'm not sure I understand this: using StringIO in this way can be sufficiently static to be geninterped. Both the direction you started and this StringIO-based version make sense. A bientot, Armin.

1 0

Re: [pypy-svn] r15225 - in pypy/dist/pypy: module/math rpython rpython/module
by hpk＠trillke.net 28 Jul '05

28 Jul '05

Hi Anders, On Thu, Jul 28, 2005 at 10:56 +0200, ale(a)codespeak.net wrote: > Author: ale > Date: Thu Jul 28 10:56:16 2005 > New Revision: 15225 > > Modified: > pypy/dist/pypy/module/math/__init__.py > pypy/dist/pypy/rpython/extfunctable.py > pypy/dist/pypy/rpython/llinterp.py > pypy/dist/pypy/rpython/module/ll_math.py we should try to add tests when adding new external function implementations. Otherwise we get frustrating errors in translation after running for >10 minutes. > Modified: pypy/dist/pypy/rpython/module/ll_math.py > ============================================================================== > --- pypy/dist/pypy/rpython/module/ll_math.py (original) > +++ pypy/dist/pypy/rpython/module/ll_math.py Thu Jul 28 10:56:16 2005 > @@ -2,6 +2,10 @@ > > import math > > +def ll_math_log(x): > + return math.log(x) > +ll_math_log.suggested_primitive = True 'log' actually takes two arguments (see pypy/module/math/interp_math.py or plain python's math.log). On a related note, ll-external function implementations cannot have default arguments. cheers, holger

2 1