Mailman 3 July 2005 - pypy-dev

bug in http://codespeak.net/moin/pypy/moin.cgi/FrontPage?action=show
by Laura Creighton 10 Oct '05

10 Oct '05

If you click on the 'documentation' link you get a traceback! ooops. I will look at this later unless somebody beats me to it. Laura

5 4

Build and test failures
by Ben.Young＠risk.sungard.com 09 Aug '05

09 Aug '05

Hi Everyone, Just synced to head this morning and noticed a couple of errors in the c translation process. This first is that on windows, a couple of headers are not available. I fixed this by patching locally with: Index: translator/c/extfunc_include.h =================================================================== --- translator/c/extfunc_include.h (revision 14483) +++ translator/c/extfunc_include.h (working copy) @@ -6,9 +6,9 @@ /******************************************************************/ -#include <unistd.h> +/*#include <unistd.h> #include <sys/types.h> -#include <sys/stat.h> +#include <sys/stat.h>*/ #include <fcntl.h> #include <time.h> #ifndef PATH_MAX Then, when the tests are run I get: $ ./test_all.py translator/c/ ============================= test process starts ============================= testing-mode: inprocess executable: c:\Python24\python.exe (2.4.1-final-0) using py lib: c:\Documents and Settings\YoungB\dist\py <rev unknown> translator\c\test\test_annotated.py[19] ................... translator\c\test\test_backendoptimized.py[39] ....................................... translator\c\test\test_database.py[17] ................. translator\c\test\test_exception.py[3] ... translator\c\test\test_extfunc.py[4] F... translator\c\test\test_genc.py[8] ........ translator\c\test\test_lltyped.py[2] .. translator\c\test\test_notype.py[33] ................................. translator\c\test\test_operation.py[1] . translator\c\test\test_support.py[7] ....... translator\c\test\test_symboltable.py[1] . translator\c\test\test_typed.py[38] ...................................... _______________________________________________________________________________ _________________________ entrypoint: test_time_clock _________________________ def test_time_clock(): def does_stuff(): return time.clock() f1 = compile(does_stuff, []) t0 = time.clock() t1 = f1() assert type(t1) is float t2 = time.clock() E assert t0 <= t1 <= t2 > assert 165.0 <= 2.5683609740123038e-005 [c:\Documents and Settings\YoungB\dist\pypy\translator\c\test\test_extfunc.py:16] - - - - - - - - - - - test_time_clock: recorded stderr - - - - - - - - - - - - [annrpython] (pypy.translator.c.test.test_extfunc:9) does_stuff -> SomeFloat() [annrpython] (pypy.rpython.rclass:575) ll_runtime_type_info__objectPtr -> SomePtr(ll_ptrtype=<* RuntimeTypeInfo (opaque)>) [annrpython] (pypy.rpython.module.ll_time:14) ll_time_clock -> SomeFloat() [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool(const=True) [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool() [annrpython] (pypy.rpython.rclass:565) ll_type__objectPtr -> SomePtr(ll_ptrtype=<* Struct object_vtable { parenttypeptr: * Struct objec t_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(const=<* struct object { typeptr=... }>, ll_ptr type=<* GcStruct object { typeptr: * Struct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opa que), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(ll_ptrtype=<* GcStruct object { typeptr: * Stru ct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:53) ll_raise_OSError__Signed -> SomeImpossibleValue() _______________________________________________________________________________ =========== tests finished: 171 passed, 1 failed in 263.86 seconds ============ I think this is because when clock is called in the extension it thinks it is the first time and returns the very small value. I don't really know though. I hope you all had fun at the sprint! Cheers, Ben

3 7

day 6: breakthrough!
by hpk＠trillke.net 31 Jul '05

31 Jul '05

hi folks, two and a half years later ... in the very same room where the first PyPy sprint happened ... we got to the point where PyPy basically runs on its own, after whole-program type inference and translation to C! The sixth day was a very productive day in various respects and we still had a long and nice lunch break. If you want to see the visual flow of events then revisit the pictures i took: http://codespeak.net/~hpk/hildesheim2-sprint-www/ Here is a more detailed overview of what happended on the sixth day: - Carl, Samuele and holger finished the Heidelberg sprint announcement, updated the website and the announcement was sent out. See the home page for more details and links. - Christian with initial help from Samuele went off to work on the interpreter level marshal module which we will need for better bootstrapping. Unfortunately, Christian had a bad stomach and missed much of the fun of day 6. - Armin and Holger as well as others continously restarted the whole translation process to fix small bugs and issues here and there. In between - they extended testing to allow applevel tests to easily access values setup_*'ed from interpreter level. - they fixed the posix module which was completely broken with respect to exception handling (you cannot let escape interpreter-level exception to pypy application level without carefully wrapping them in an OperationError). Samuele helped with making os.strerror available as an external (c-backend supported) function. - analysed and fixed with Samuele and Carl segmentation faults ... but note that GCC was behaving very fine and it turned out that our c-code generator currently consumes more RAM than GCC compiling the result at the moment :-) - they also started refactoring the translation driving interface (unfinished) - Carl Friedrich and Samuele used and extended/fixed the ll-interpreter which was sometimes able to discover possible segfaults and problems earlier (and clearer) than just running the C-backend. - Carl Friedrich started working on GC in PyPy in constant consultation with Samuele and did first working drafts in pypy/rpython/memory. Basically the whole memory simulator is working and gives nices errors if you do something strange like accessing freed or uninitialized. Furthermore they implemented annotation for memory addresses. - Richard communicated with Eric van Riet Paap (who has made good progress on the llvm-backend off-sprint!) and refactored, extended LLVM a bit driven by test suggestions from Armin. - the group discussed that there likely is a deeper exception handling problem lurking: implicitely exceptions are attached to the last (high level) operation in a block but when the RTyper converts this operation into possibly many low-level operations then it's unlikely that we can preserve the implicit assumption that only the last (low-level) operation possibly raises an exception. solution: unknown at this point. We should probably write down a few scenarios regarding exception handling describing the interactions/processing of exceptions in flowobjspace/annotation/rtyping - Armin communicated with Niklaus Heidimann who meanwhile has successfully implemented the _sre module as part of his SoC project. We'd like to first integrate this as an application level module but Niklaus wants to transform it to an interpreter-level one soon. - After we got the initial breakthrough run (goal/app_example.py running on top of translated non-faking PyPy) we continued a bit with various example and fixes. Running example code and modules showed very good results (and no segfaults). After having drunk champagne Samuele quickly mapped the good time we were having into a better 'time' module for PyPy. With that we could easily run the unmodified pystone.py on top of PyPy and reached some 337 pystones (as opposed to 9 pystones with PyPy on top of CPython). We are happy with this because translated PyPy seems to run stable and we are not using any optimizations yet. We quickly agreed that we only want to tackle optimizations at this point that result in at least factors-of-2 speedups rather than 5-10% speedups. Again please note that PyPy - as opposed to other python implementations - generally tackles compliancy, proper implementation and tests first before diving into optimizations. ok, so much so far. Today - on the last day - we are probably going to discuss on how to continue from here and also tell how you can actually run the translation and the translated PyPy yourselves (the current pypy-dist is able to but it currently requires a bit of manual tweaks at the very invocation-end, maybe you can already figure it out looking at the pictures and the entry_point code :-). cheers, holger

4 4

Re: [pypy-svn] r15405 - in pypy/dist/pypy: module/__builtin__/test module/__builtin__/test/impsubdir/compiled module/_codecs/test module/posix/test translator/goal translator/llvm2/module
by Eric van Riet Paap 30 Jul '05

30 Jul '05

Hi Holger, Is there some svn/vi setting I could use to prevent this eol problems? Eric hpk(a)codespeak.net wrote: >Author: hpk >Date: Sat Jul 30 12:54:42 2005 >New Revision: 15405 > >Modified: > pypy/dist/pypy/module/__builtin__/test/impsubdir/compiled/ (props changed) > pypy/dist/pypy/module/__builtin__/test/impsubdir/compiled/__init__.py (props changed) > pypy/dist/pypy/module/__builtin__/test/impsubdir/compiled/x.py (props changed) > pypy/dist/pypy/module/__builtin__/test/test_buffer.py (props changed) > pypy/dist/pypy/module/_codecs/test/ (props changed) > pypy/dist/pypy/module/_codecs/test/__init__.py (props changed) > pypy/dist/pypy/module/_codecs/test/autopath.py (props changed) > pypy/dist/pypy/module/_codecs/test/test_codecs.py (props changed) > pypy/dist/pypy/module/posix/test/test_posix2.py (props changed) > pypy/dist/pypy/translator/goal/targetsegfault.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_math.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_os.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_os_path.py (props changed) > pypy/dist/pypy/translator/llvm2/module/ll_time.py (props changed) > pypy/dist/pypy/translator/llvm2/module/support.py (props changed) >Log: >FIXEOL > >_______________________________________________ >pypy-svn mailing list >pypy-svn(a)codespeak.net >http://codespeak.net/mailman/listinfo/pypy-svn > >

2 1

PyPy sprint announcement: Heidelberg 22nd - 29th August 2005
by Carl Friedrich Bolz 30 Jul '05

30 Jul '05

PyPy Sprint in Heidelberg 22nd - 29th August 2005 ================================================== The next PyPy sprint will take place at the Heidelberg University in Germany from 22nd August to 29th August (both days included). Its main focus is translation of the whole PyPy interpreter to a low level language and reaching 2.4.1 Python compliancy. To learn more about the new PyPy Python implementation look here: http://codespeak.net/pypy Goals and topics of the sprint ------------------------------ One of the main goals of the sprint is going to be a 0.7 release of PyPy. The 0.7 is meant to be the first self-contained PyPy version but still alpha with some C-extension modules missing and not meant for production use. Therefore the topics of the sprint will be: - translation efforts: it's not completely clear now what related task will be left at that time - work on 2.4.1 compliancy: there are some failing compliancy tests on which we plan to work - rewriting c-extension modules/integrating existing rewrites - all kinds of small release issues - possibly some more LLVM efforts Location & Accomodation ------------------------ The sprint will be held in a conference room of the Institute of Physics, University of Heidelberg. We have WLAN connection there and small kitchen to make tea and coffee. `See here`_ how to get there from the Heidelberg main station. There is a huge number of hotels_ in Heidelberg, all of which are unfortunately rather expensive. It should be no problem to reach the sprint venue from anywhere in Heidelberg. As an alternative you could also take a `hotel in Mannheim`_ which you can reach in 15 min from Heidelberg with the train. .. _`See here`: http://www.physi.uni-heidelberg.de/physi/front/anfahrt.en.php .. _hotels: http://www.cvb-heidelberg.de/index_eng.html .. _`hotel in Mannheim`: http://www.hotels.de/en/mannheim.htm Exact times ----------- The Pypy sprint is held Monday 22nd August - Monday 29th August 2005. Hours will be from 09:00 until people have had enough. It's a good idea to arrive a day before the sprint starts. Network, Food ------------- The Heidelberg Altstadt can be reached in approximately 10 min from the sprint venue. There you will find all kinds of restaurants and fast food places. You will probably need a wireless network card to access the network although we could set up a WLAN to cable bridge if necessary. Registration etc.pp. -------------------- Please subscribe to the `PyPy sprint mailing list`_, introduce yourself and post a note that you want to come. Feel free to ask any questions there! There also is a separate `Heidelberg people`_ page tracking who is already thought to come. If you have commit rights on codespeak then you can modify yourself a checkout of http://codespeak.net/svn/pypy/extradoc/sprintinfo/heidelberg-people.txt .. _`Heidelberg people`: http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/heidelberg-people.h… .. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint -- Carl Friedrich Bolz & the PyPy-Team

1 0

hildesheim2 day 2 + 3 report / status
by hpk＠trillke.net 30 Jul '05

30 Jul '05

Hi folks, the end of the world is near ... following up on Carl's day 1 report here are the main results of what happened up until now at the hildesheim2 sprint (we are in the break day currently which we spent in the city, at Wing-Tsun lessons or at family meetings). The main results so far are: - any faking can be removed now with python2.4 py.py --file --nofakedmodules so on the trunk you should see no 'faking' messages during startup anymore. It will take some time mostly because of the 'file' implementation living at app-level. Note that we have some problems running on top of Python2.3 at the moment and also some more problems with PYC file support. - Annotation can now complete without resorting to any SomeObjects anywhere (!!) - RTyping has only 2 Errors remaining. The next bigger problem is getting the compiler package integrated and bootstrapped properly. - note that we are not working from the snapshot but the trunk currently. After the sprint we probably return to the snapshot approach for translation. If you want to see more detailed daily-updated progress reports i suggest to read and follow the commits on http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/hildesheim2-plannin… Moreover, i shot a few pictures which i put here: http://codespeak.net/~hpk/hildesheim2-sprint-www and here is a picture of the surrounding place (Trillke-Gut, my living&working base): http://www.trillke.net/images/HomePagePictureSmall.jpg cheers, holger P.S.: It appears that we will have to continue our investigations regarding some of the current problems on the astral plane. Armin and Carl has already spend some time on assembler level yesterday to hunt down the weirdest TCC interactions. But there are other confusing events as well. um, we watched 'the nine lives of Thomas Katz' yesterday night, btw. Anyway, we are still taking bets if the first C-generated annotated RTyped PyPy is to produce a segmentation fault in the C-compiler or in the resulting binary.

4 5

Hildesheim sprint report 4 + 5
by Carl Friedrich Bolz 30 Jul '05

30 Jul '05

Hi PyPy-dev, somre more news from the Hildesheim sprint (I try to do this in some sort of chronological order although the days are kind of mixed in my mind): Thursday: - Holger and Richard worked on refactoring the mess that is the option handling of PyPy. Now it is possible to. Now the handling of options is unified in one place. It is now possible to tell PyPy exactly which modules to fake and for which to import our own implementation. - Armin and me completed our implementation of the math module, which is now completely translatable. - One of the big problems of the last days for the translation effort was that we had a bootstrap problem: our interplevel compiler and parser are not entirely done yet, so we need applevel version. These need to be imported using a parser and a compiler. Damn. At first we tried to get pyc-file support working, which involved finishing marshal. This approach was dropped because marshal used codecs which lives on applevel and therefore cannot be imported. Instead Armin came up with a hack: Basically we cheat during bootstrapping and use the underlying compiler and parser. Later (before translation) it is replaced by a real applevel implementation. Armin worked on this the basically the whole Thursday afternoon. In the evening it turned out that this approach leads to *exactly* the same problem. Armin, Samuele, Christian and me fixed Thursday evening. It kind of worked in the end but it introduced strange translation problems that Armin and Samuele fixed Thursday night (last checkin message: 3:45am). Friday: - Holger worked on py.test's stdout capturing because after some recent refactoring it didn't work any more for PyPy applevel tests. - Christian started reimplementing marshal on interplevel. - Richard tried once more to get LLVM prebuilt constants to work. - basically the day was spent trying to translate pypy (which involves watching PyPy: The Movie -- Two and a Half Years Later. We sit and watch the translator running on the beamer until it crashes). After a few typer errors and a bug in exception handling we actually managed to reach the point where the translator generated C code! To reach this you need quite a lot of RAM: something around 1GB just for the translator, annotator, rtyper and genc. The whole generated code is something like 120 Mb big (all in one file!). Surprisingly the compiler even manages to do something with it (which just means that it produces errors and does not just crash). While trying to actually translate PyPy we found various ommissions in various places: Some external functions were missing. While doing this the following compiler error was found: "testin_1.c:306344: error: 'Is_Unsigned_Division_Really_Useful' undeclared (first use this function)" well, obviously it is :-) - Armin worked on fixing an order problem: Some struct declarations have to be written to the C file in the right order because they depend on each other. - Samuele and me tried to run the flowgraph of PyPy on the llinterpreter (yay, yet another level!) which turned up some bugs in the RTyper and in the PyPy interpreter but mostly space operations that were still missing from the llinterpreter, mostly overflow and division by zero checking. Well, I'm pretty sure that I forgot some things but I'm too tired now. G'night, Carl Friedrich

1 0

Confused about RPython
by Niklaus Haldimann 29 Jul '05

29 Jul '05

Hi everyone I am currently seriously confused about the notion of RPython, and I hope someone can show me the light here. ;) I guess my basic problem is: How do I find out if some code is RPython? And especially in the context of the array module im currently working on, how do I find out if a module is RPython? One of my working hypothesis was that whatever can be geninterped is RPython. But by that definition, a *lot* more comes out as RPython than the documentation says. Is it just that the documentation is out-of-date or is geninterping not a good criterion? E.g., when Christian states that _codecs is now RPython, does that just mean it can be geninterped or is there something else to it? Armin just had a checkin where he replaced str.isalpha() with something else because string methods are not RPython. But I can geninterp and even C compile string methods just fine. Did he use the term "RPython" loosely, meaning something like "what can be usefully annotated" (ie, string methods always result in SomeObject, that's probably why they're frowned upon in the interpreter code)? Cheers Nik

3 5

Re: Strategy for array module?
by Armin Rigo 29 Jul '05

29 Jul '05

Hi Niklaus, Sorry for the delay. I've indeed been quite deeply into the PyPy sprint... At which level and with which style the array module should be implemented is indeed a bit unclear. If you are going for an app-level version, you should not worry too much about geninterp, I guess. The code you already started to write makes sense. Geninterp is most useful to solve bootstrapping issues, where we cannot import yet some app-level code but we can run its geninterp'ed version (e.g. for the _exceptions.py module). Of course geninterp gives some performance improvement too. The problem with array is that it is designed with the following model in mind: a bunch of memory out of which Python values can be read on demand. The other point of view -- a list of Python values that can be turned into a big string on demand -- make methods like byteswap() hard to implement, specially for arrays of floats or doubles. Not necessarily impossible, though. If this app-level approach works out, I wouldn't invest too much effort in geninterp compatibility, because at some point in the future we might have to think about a quite different approach: adding the notion of integers and floats of specific sizes, at least in the so-called "low-level" types. I guess that an efficient implementation of array should be written to take advantage of these. This will likely require as-yet-unspecified ways to interact between regular interp-level code and this low-level-typed code. In summary I guess that writing a pure app-level array module makes sense, with the buffer interface working for PyPy only. On Tue, Jul 26, 2005 at 06:15:01PM +0200, Niklaus Haldimann wrote: > As I probably told you, Lutz Paelike has sent me an incomplete array > implementation he wrote at the PyCon sprint. He uses StringIO to > represent the arrays internally as pure binary data. This is quite > sensible if you are after a quick pure Python implementation. But of > course it's useless if you want to be RPythonic. I'm not sure I understand this: using StringIO in this way can be sufficiently static to be geninterped. Both the direction you started and this StringIO-based version make sense. A bientot, Armin.

1 0

Re: [pypy-svn] r15225 - in pypy/dist/pypy: module/math rpython rpython/module
by hpk＠trillke.net 28 Jul '05

28 Jul '05

Hi Anders, On Thu, Jul 28, 2005 at 10:56 +0200, ale(a)codespeak.net wrote: > Author: ale > Date: Thu Jul 28 10:56:16 2005 > New Revision: 15225 > > Modified: > pypy/dist/pypy/module/math/__init__.py > pypy/dist/pypy/rpython/extfunctable.py > pypy/dist/pypy/rpython/llinterp.py > pypy/dist/pypy/rpython/module/ll_math.py we should try to add tests when adding new external function implementations. Otherwise we get frustrating errors in translation after running for >10 minutes. > Modified: pypy/dist/pypy/rpython/module/ll_math.py > ============================================================================== > --- pypy/dist/pypy/rpython/module/ll_math.py (original) > +++ pypy/dist/pypy/rpython/module/ll_math.py Thu Jul 28 10:56:16 2005 > @@ -2,6 +2,10 @@ > > import math > > +def ll_math_log(x): > + return math.log(x) > +ll_math_log.suggested_primitive = True 'log' actually takes two arguments (see pypy/module/math/interp_math.py or plain python's math.log). On a related note, ll-external function implementations cannot have default arguments. cheers, holger

2 1

Hildesheim sprint progress report
by Carl Friedrich Bolz 26 Jul '05

26 Jul '05

Just some quick (and probably confused) notes about went on at the Hildesheim sprint day 1. Corrections welcome. As in Gothenburg we tried to write an evolving file that listed all the tasks: http://codespeak.net/svn/pypy/extradoc/sprintinfo/hildesheim2-planning.txt somehow we didn't get very far and decided that we just would all work on rtyper issues. The following people are here: Hogler, Armin, Samuele, Christian, Richard, Carl Friedrich. Holger and Christian paired and started removing some amount of faking in PyPy. They did this by removing the usage of file as well as stdout/stdin in the PyPy interpreter and resorting to the os primitives instead, which can be more easily translated to C. Armin and Christian worked on various rtyper issues. They worked on unicode, fixed hashing for numerical types, and worked on marshal. Samuele and me worked on external functions. We cleaned up some of the hooks that were there before the current external function mechanism and implemented some math and os function for the C backend so that you can now actually use some of the primitive file operations like fdopen, stat, write... without using the CPython API. Cheers, Carl Friedrich

1 1

translation and tests
by hpk＠trillke.net 26 Jul '05

26 Jul '05

Hello pypy-dev! today at the hildesheim2 sprint we decided that we'd like to annotate and translate from the trunk during the sprint. After the sprint we are probably changing back to work from a snapshot. This is more convenient than porting changes to the snapshost all the time as we are making a lot of changes. Also, please try to avoid checking in changes with failing tests. All of us are doing it sometimes, for example, when we change computers or locations and want to continue working somehwere else. However, even then it's best if you _temporarily_ inhibit the collection of the affected tests by placing a conftest.py file into an appropriate directory, containing this: import py class Directory(py.test.collect.Directory): def run(self): return [] This will tell py.test that there are no tests to be collected in the directory and all its sub directories. You can still run test files directly. If you want to checkin a failing test where you don't know yet how to fix it, then you can either add a 'py.test.skip("please help with this problem ...")' at the top of the failing test function or you can mark a whole file with 'inprogress_test_something.py' which will inhibit automatic collection while still allowing to run 'py.test inprogress_test_something.py'. Of course, the basic idea of all these suggestions is that we want to be able to run all PyPy tests to see if we messed up somehow. thanks and greetings, holger & samuele

1 0

Re: [pypy-svn] r15069 - pypy/dist/pypy/translator/llvm2/tool
by Christian Tismer 25 Jul '05

25 Jul '05

ericvrp(a)codespeak.net wrote: > Wrote simple tool to get an overview of the suggested_primitive functions that > still need to be implemented in the llvm backend. > The tool should be run from it's own directory for now! > This is the current output: (tail is what we want to know) Nice tool, very useful! Just a hint for further development: PyPy tends to avoid source code analysis wherever possible. And in this tool's case it would bhe quite easy to use introspecction: > + ll_modules_path = '../../../rpython/module' > + ll_files = [ll_modules_path + '/' + f for f in os.listdir(ll_modules_path) if f[:3] == 'll_' and f[-3:] == '.py'] > + ll_function = {} #XXX better use sets > + for ll_file in ll_files: instead of this, > + for s in file(ll_file): > + s = s.strip() > + if not s.startswith('ll_') or s.find('suggested_primitive') == -1 or s.find('True') == -1: > + continue > + ll_function[s.split('.')[0]] = True you could just import the file and inspect its objects. Then check for hasattr(func, 'suggested_primitive') and check if it is true. > + > + llvm_modules_path = '..' > + llvm_files = [llvm_modules_path + '/' + 'extfunction.py'] > + llvm_function = {} > + for llvm_file in llvm_files: > + t = 'extfunctions["%' Same here. Just import extfunctions from the file and look into it. This is just more robust about future changes and independent of formatting. It also survives stuff like programmatically adding bulks of stuff to extfunctions, which might happen at some point. Just my 0.2cent - ciao -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

next pypy sprints / some infos
by hpk＠trillke.net 20 Jul '05

20 Jul '05

Hi folks, first some quick notes regarding upcoming PyPy sprints. After the EuroPython 2005 sprint we decided that we are going for a sprint that i am organizing at my living and working place, Trillke-Gut in Hildesheim, Germany. It is going to start 25th July and last until 31st July 2005. Especially already scheduled participants please look at http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/Hildesheim2-sprint.… for some more information (and correct wrong dates!). The strong focus will be on getting a self-contained PyPy version working. Really. This time the sprint is only open for everybody who already knows PyPy. If you'd like to attend please send me a private mail. I am trying to find private accomodation for everyone. Please excuse the extremely short notice but the final decision was only taken end of last week and i have been somewhat busy. Most of the currently active developers knew about this coming for a longer time, though. For the end of August 2005 Carl Friedrich Bolz is organizing a sprint more open to new people. This is going to take place at the University of Heidelberg, Germany. More details will be announced soon. While we are at it: in the further away future, it's likely there will be a sprint in Paris around early October, organized by Logilab. And if you can imagine helping us to organize a PyPy sprint at your place, then please drop us a mail! For example, we haven't been to basically the whole of South Europe :-) For your information, we have started last thursday a practise of having all active pypy developers (and in particular those involved in the PyPy/EU project) meet on IRC for 30 minutes. This XP-style meeting is meant to synchronize our dev work better and give us a body to quickly decide about directions and issues. The first minutes of the pypy-sync meeting can be found here: http://codespeak.net/pypy/index.cgi?extradoc/minute/pypy-sync-07-14.html This "active developers" meeting will take place every thursday at 1pm (GMT+2/Berlin) for 30 minutes with a half-fixed agenda. It is not really open to spectators, i guess. Those have to wait for the minutes :-) If you have any questions or comments don't hesitate to share them here on the list. cheers, holger

1 0

Re: rtyper hook ++
by amaury.forgeotdarc＠ubitrade.com 19 Jul '05

19 Jul '05

Hello, Chritian Tismer wrote: > If we teach the specializer to not crash if something > does not work out, but to add the problem blocks to a list, > then we could have one report with all pending problems > and work through this in a more general way. > We also would get an estimate how much has still to be done. That's what I have done to follow the fantastic progress you make on the translation process. - In rtyper.py, there is already a crash_on_first_typeerror variable. Turn it to False, this will catch all TyperErrors. - There are some assertions and other exceptions that will stop the process. I changed all of them into TyperErrors. I suppose that some tests are not done at the right place, but they do the job... And the specialization goes to the end! There are 81 TyperErrors left for 20206 blocks. They were 230 last week. Good job! Then the "Back-end optimizations" fail, probably because of the incompletely specialized blocks. I hope this helps! Here are my diffs: Index: pypy/rpython/rstr.py =================================================================== --- pypy/rpython/rstr.py (revision 14755) +++ pypy/rpython/rstr.py (working copy) @@ -264,7 +264,8 @@ if curstr: r.append(curstr) curstr = '' - assert f in 'xdsrf' + if f not in 'xdsrf': + raise TyperError("String format not supported: %r" % (f,)) r.append((f,)) else: @@ -295,6 +296,9 @@ vitem, r_arg = argsiter.next() rep = inputconst(Void, r_arg) if code == 's' or (code == 'r' and isinstance(r_arg, InstanceRepr)): + if not hasattr(r_arg,'ll_str'): + raise TyperError("AttributeError: %s instance has no attribute 'll_str'" + % r_arg) vchunk = hop.gendirectcall(r_arg.ll_str, vitem, rep) elif code == 'd': assert isinstance(r_arg, IntegerRepr) Index: pypy/rpython/callparse.py =================================================================== --- pypy/rpython/callparse.py (revision 14755) +++ pypy/rpython/callparse.py (working copy) @@ -2,6 +2,7 @@ from pypy.interpreter.argument import Arguments, ArgErr from pypy.annotation import model as annmodel from pypy.rpython import rtuple +from pypy.rpython.rmodel import TyperError class CallPatternTooComplex(Exception): pass @@ -47,7 +48,7 @@ try: holders = arguments.match_signature(signature, defs_h) except ArgErr, e: - raise TypeError, "signature mismatch: %s" % e.getmsg(arguments, func.__name__) + raise TyperError, "signature mismatch: %s" % e.getmsg(arguments, func.__name__) assert len(holders) == len(rinputs), "argument parsing mismatch" vlist = [] Index: pypy/rpython/rbuiltin.py =================================================================== --- pypy/rpython/rbuiltin.py (revision 14755) +++ pypy/rpython/rbuiltin.py (working copy) @@ -107,9 +107,15 @@ def rtype_builtin_unichr(hop): assert hop.nb_args == 1 + if not hasattr(hop.args_r[0],'rtype_unichr'): + raise TyperError("AttributeError: %s instance has no attribute 'rtype_unichr'" + % hop.args_r[0]) return hop.args_r[0].rtype_unichr(hop) def rtype_builtin_list(hop): + if not hasattr(hop.args_r[0],'rtype_bltn_list'): + raise TyperError("AttributeError: %s instance has no attribute 'rtype_bltn_list'" + % hop.args_r[0]) return hop.args_r[0].rtype_bltn_list(hop) def rtype_builtin_isinstance(hop): Index: pypy/rpython/rtyper.py =================================================================== --- pypy/rpython/rtyper.py (revision 14755) +++ pypy/rpython/rtyper.py (working copy) @@ -31,7 +31,7 @@ log = py.log.Producer("rtyper") py.log.setconsumer("rtyper", None) -crash_on_first_typeerror = True +crash_on_first_typeerror = False class RPythonTyper: @@ -45,6 +45,7 @@ self.pbc_reprs = {} self.class_pbc_attributes = {} self.typererror = None + self.nb_typeerrors = 0 # make the primitive_to_repr constant mapping self.primitive_to_repr = {} for s_primitive, lltype in annmodel.annotation_to_ll_map: @@ -138,8 +139,8 @@ n = len(self.already_seen) if n % 100 == 0: total = len(self.annotator.annotated) - print 'specializing: %d / %d blocks (%d%%)' % ( - n, total, 100 * n // total) + print 'specializing: %d / %d blocks (%d%%, %d errors)' % ( + n, total, 100 * n // total, self.nb_typeerrors) # make sure all reprs so far have had their setup() called self.call_all_setups() @@ -200,7 +201,11 @@ def specialize_block(self, block): # give the best possible types to the input args - self.setup_block_entry(block) + try: + self.setup_block_entry(block) + except TyperError, e: + self.gottypererror(e, block, "block entry", None) + return # cannot continue this block # specialize all the operations, as far as possible if block.operations == (): # return or except block @@ -269,8 +274,9 @@ new_a1 = newops.convertvar(a1, r_a1, r_a2) except TyperError, e: self.gottypererror(e, block, link, newops) - if new_a1 != a1: - newlinkargs[i] = new_a1 + else: + if new_a1 != a1: + newlinkargs[i] = new_a1 if newops: if can_insert_here: @@ -343,13 +349,16 @@ """Record a TyperError without crashing immediately. Put a 'TyperError' operation in the graph instead. """ + self.nb_typeerrors += 1 e.where = (block, position) if crash_on_first_typeerror: raise + print "*** TyperError:",e if self.typererror is None: self.typererror = sys.exc_info() c1 = inputconst(Void, Exception.__str__(e)) - llops.genop('TYPER ERROR', [c1], resulttype=Void) + if llops: + llops.genop('TYPER ERROR', [c1], resulttype=Void) # __________ regular operations __________ @@ -377,7 +386,12 @@ def translate_op_contains(self, hop): r_arg1 = hop.args_r[0] r_arg2 = hop.args_r[1] - return pair(r_arg1, r_arg2).rtype_contains(hop) + p = pair(r_arg1, r_arg2) + try: + f = p.rtype_contains + except AttributeError,e: + raise TyperError(str(e)) + return f(hop) # __________ irregular operations __________ -- Amaury Forgeot d'Arc Ubix Development www.ubitrade.com

2 1

rtyper hook ++
by Christian Tismer 19 Jul '05

19 Jul '05

Hi Samuele, you are not online, therefore using email. """ pedronis: stakkars_: btw my new hook allows to get back to a given module or function, or to skip things we know specialize fine """ I like the idea. Can't we go even a bit further? If we teach the specializer to not crash if something does not work out, but to add the problem blocks to a list, then we could have one report with all pending problems and work through this in a more general way. We also would get an estimate how much has still to be done. The process we are doing by hand (walking PDB stack, looking at the block etc.) can be either automated or deferred. Yes, deferring would be the easiest: The list of problem blocks should be pickleable for later analysis. The pickling might not be strong enough to re-run stuff at the moment, but for the debugging, it would surely be sufficient. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

Re: [pypy-dev] Questions from a lurker
by Arthur Peters 18 Jul '05

18 Jul '05

I accidently replied to this off list, so the off-list emails (3 of them) are appended for postarity. Also I fixed the problem with test_all.py: I had created a __init__.py file directly in pypy-dist. I don't remember why I created it. Result was sure strange. Anyway it was my fault. Thanks all. -Arthur ---------------------------------------------------- On Sun, 2005-07-17 at 00:25 +0200, Carl Friedrich Bolz wrote: > Hi Arthur, > > some more notes -- again I might not be the perfect person to answer > this, corrections welcome. > > Arthur Peters wrote: > [snip] > > Makes sense, but it seems to me that there will be a lot of overlap > > between the translator and the JIT compiler. Couldn't large parts of > > the translator be reused? > > Yes and no. The translator (and especially the type inference which we > call annotation) works on special assumptions that don't apply to a > regular programs: For example that the objects bound to a name in a > certain place always have the same type. > > > > > BTW, I actually haven't heard for certain that there will be a JIT > > compiler. Will there be? I'd really like to see one. Python is a great > > language but runs slow sometimes and I think a JIT could really help. > > Though one of the main slow things about it is the function call > > overhead and for that to be reduced JIT compilation would have to happen > > at a level higher than functions and would need type inference and all > > that. However it would be really cool to have things like inlining of > > functions (it would require a decorator to mark the function as > > invariant so that the compiler knows that it cannot be changed at > > runtime like normal python functions can). I come from a C++ background > > so I'm used to having simple functions be as fast as macros. > > Yes, there will be a JIT compiler although probably a bit different from > other JIT compilers. The problem is that Python is so very dynamic that > you cannot really know /anything/ in advance. Which means that even > "just in time" -- e.g. right before the program is run -- is too early. > (This does not apply to the code our translator translates, because that > code, mainly PyPy's interpreter, was written with a lot of limitation > especially regarding the dynamism.) The first time you really know > anything for sure about a Python function is when you run it for the > first time (because then you have fixed values with fixed types to work > with). This is (heavily simplified) the idea driving Psyco: Psyco gets > information about a function when this function is being executed. > > I'm not totally sure whether the thing that makes Python slow is really > the function calls. One of the slower things might be dispatching (which > function will be executed for the code a + b) which is a thing that > Psyco addresses. And, ideally, the user would not need to take care of > telling the JIT which function to inline, the JIT should rather be > clever enough to realize that a function is worth inlining. I even think > that the JIT could do this better than the programmer. > > > > > Also it seems sad to have such a cool and difficult to create things as > > the translator end up only being used on one program. It seems like it > > could be useful elsewhere. > > > > Yesish. The problem is that it's rather hard to get the translator > working for a given Python program (or rather, get the program working > for the translator :-) ). At the moment you'd have to adhere to a lot of > restrictions which are sometimes not easy to explain. This will problem > only change to a certain degree. > > Regards, > > Carl Friedrich ------------------------------------------------- On Sat, 2005-07-16 at 17:40 -0500, Arthur Peters wrote: > On 7/16/2005, "Carl Friedrich Bolz" <cfbolz(a)gmx.de> wrote: > >Yes, there will be a JIT compiler although probably a bit different from > >other JIT compilers. The problem is that Python is so very dynamic that > >you cannot really know /anything/ in advance. Which means that even > >"just in time" -- e.g. right before the program is run -- is too early. > >(This does not apply to the code our translator translates, because that > >code, mainly PyPy's interpreter, was written with a lot of limitation > >especially regarding the dynamism.) The first time you really know > >anything for sure about a Python function is when you run it for the > >first time (because then you have fixed values with fixed types to work > >with). This is (heavily simplified) the idea driving Psyco: Psyco gets > >information about a function when this function is being executed. > > That makes sense. > > > > >I'm not totally sure whether the thing that makes Python slow is really > >the function calls. One of the slower things might be dispatching (which > >function will be executed for the code a + b) which is a thing that > >Psyco addresses. And, ideally, the user would not need to take care of > >telling the JIT which function to inline, the JIT should rather be > >clever enough to realize that a function is worth inlining. I even think > >that the JIT could do this better than the programmer. > > I agree. > > What I ment was a decorator that said that the method (or perhaps I > should say method name) would not be rebound. This would allow the > function to be embeded in the caller without fear of rebinding. This > would be an issue because this is valid: > > class A: > def f(self): > pass > > def call_f(a): > a.f() > > a = A() > call_f(a) # does nothing > > def new_f(): > print "Hi" > > a.f = new_f > > call_f(a) # prints "Hi" > > If A.f were inlined into call_f then the second call to call_f would > either run the wrong code or would have to trigger a recompile. > > -Arthur ---------------------------------------------------- On Sat, 2005-07-16 at 10:40 -0500, Arthur Peters wrote: > Answer interlaced. > > On 7/16/2005, "Carl Friedrich Bolz" <cfbolz(a)gmx.de> wrote: > >> - I read what some discussion of the GIL as it relate to pypy and I agree > >> that the GIL need to be implemented and that a different thread model > >> might be a good way to go. However I thought of the following: Would it > >> be possible to detect when a section of code only uses local variables > >> and unlock the GIL? This seems possible because local variables will > >> cannot be shared between threads anyway. In addition, local variables > >> likely be translated into more basic types than a python object (native > >> ints and floats for instance), that would not require any interaction > >> with the object-space to manipulate (not sure about that usage of the > >> term "object-space"). Thoughts? > > > >First a comment about the usage of the term "object space": I think you > >are mixing levels here (and I might misunderstand you). The PyPy > >interpreter (meaning the bytecode interpreter plus the standard > >objectspace) gets translated to low level code. This "interpreter level > >code" deals with the standard object space as a regular class instance > >that is in principle in no way different than any other class. The > >classes that appear on interpreter level are all translated to more > >basic types (what would be the alternative, there is no layer below that > >could deal with anything else), probably to something like a struct. > >Thus the operations of objects at this level never need to be done via > >the object space -- the object space is rather a regular object in itself. > > > >The object space /is/ used, if you interpret a Python program with > >PyPy's interpreter. The bytecode interpreter does not now how to deal > >with /any/ object (except for True and False), it has to delegate to the > >object space for every single operation. Even for basic types like ints > >and such -- at this level ("application level") there isn't any type > >inference or anything like that! > > Makes sense, but it seems to me that there will be a lot of overlap > between the translator and the JIT compiler. Couldn't large parts of > the translator be reused? > > BTW, I actually haven't heard for certain that there will be a JIT > compiler. Will there be? I'd really like to see one. Python is a great > language but runs slow sometimes and I think a JIT could really help. > Though one of the main slow things about it is the function call > overhead and for that to be reduced JIT compilation would have to happen > at a level higher than functions and would need type inference and all > that. However it would be really cool to have things like inlining of > functions (it would require a decorator to mark the function as > invariant so that the compiler knows that it cannot be changed at > runtime like normal python functions can). I come from a C++ background > so I'm used to having simple functions be as fast as macros. > > Also it seems sad to have such a cool and difficult to create things as > the translator end up only being used on one program. It seems like it > could be useful elsewhere. > > >Hope that helped a bit and wasn't too confused :-). > > I got most of it. ;-) Thanks a bunch. > > -Arthur

1 0

reconstructing structured flow graphs: pointers
by Samuele Pedroni 17 Jul '05

17 Jul '05

Our flowgraphs can be rendered easely in languages that support gotos. For some potential targets in would be nice* to be able to reconstruct structured graphs that can be rendered in terms of ifs/whiles etc. For (future) reference these are pointers to literature touching the subject and containing also further pointers to more literature: Assembly to High-Level Language Translation http://citeseer.ist.psu.edu/cifuentes98assembly.html Structuring Assembly Programs http://www.cs.uq.edu.au/~cristina/students/doug/doug.html the algorithms described use concepts and data structures common in the context of data-flow analysis. * targeting an underlying bytecode is also a possibility, or using some while/switch idiom possibly postprocessing the compilation result or adding a tagbody-like construct support to a compiler for the target.

1 0

Questions from a lurker
by Arthur Peters 16 Jul '05

16 Jul '05

Hello, I am Arthur. I've been lurking on the list for a month or so and I've had a good bit of fun playing with the code. My questions are: - Why did you start a second LLVM backend? Was the first one badly miss designed? - So I take it the initial goal will be to translate pypy into C (or LLVM) and create a binary executable. At that point will it be possible to translate any given program or will the translator still only work with a subset of the python language? Will the final goal be a JIT VM for python or will I at some point be able to statically compile python to machine code? - Will there be support for multiple specialized versions of functions like in psyco? I know this is a long way down the road but I'm curious what people think. - I read what some discussion of the GIL as it relate to pypy and I agree that the GIL need to be implemented and that a different thread model might be a good way to go. However I thought of the following: Would it be possible to detect when a section of code only uses local variables and unlock the GIL? This seems possible because local variables will cannot be shared between threads anyway. In addition, local variables likely be translated into more basic types than a python object (native ints and floats for instance), that would not require any interaction with the object-space to manipulate (not sure about that usage of the term "object-space"). Thoughts? - I have tried to run test_all.py on my desktop (a dual PIII running ubuntu breazy (development)) but I get the following exception: Traceback (most recent call last): File "pypy/test_all.py", line 6, in ? py.test.cmdline.main() File "/home/amp/source/pypy-dist/py/test/cmdline.py", line 8, in main config, args = py.test.Config.parse(py.std.sys.argv[1:]) File "/home/amp/source/pypy-dist/py/test/config.py", line 61, in parse config = bootstrapconfig(configpaths) File "/home/amp/source/pypy-dist/py/test/config.py", line 168, in bootstrapconfig config.loadconfig(defaultconfig).adddefaultoptions() File "/home/amp/source/pypy-dist/py/test/config.py", line 91, in loadconfig mod = importconfig(configpath) File "/home/amp/source/pypy-dist/py/test/config.py", line 208, in importconfig return configpath.pyimport() File "/home/amp/source/pypy-dist/py/path/local/local.py", line 381, in pyimport return __import__(modname, None, None, ['__doc__']) ImportError: No module named defaultconftest The cause of the problem seems to be a failure at /home/amp/source/pypy-dist/py/initpkg.py(179)__getattr__() while trying to get the attribute "__path__". This happens on both 2.3 and 2.4. This does not happen at all on my Ubuntu Hoary (stable) laptop. Any ideas? Thanks a bunch. I am very impressed by what pypy can do and I'm looking forward to seeing it progress. -Arthur PS: I am the amep that left a message on IRC.

2 1

llvm2 update
by Richard Emslie 15 Jul '05

15 Jul '05

Hi all, I've just added a dummy OpaqueType implementation and refactored the database wrt to pbc nodes. So pbcs are working for classes too now. Here is a brief overview of things that (I think) still need to me done. todo (probably in this order): * exception flow of control * any missing operations implementations (bit shifting for instance). These can be added when they are needed and should be straight forward. * review and update external function implementations for new way of doing things * overflows on ints/etc (more on that from Carl Friedrich...) refactoring: * should is_atomic() be a method? * grep "XXX"s :-) Of the above I am guess that exceptions are most interesting. The unwind / invoke operations of llvm fits this quite well, but still need somewhere to store the exception and do matching - unless I missed something? Any thoughts / takers? Cheers, Richard

3 3

targetpypy
by Ben.Young＠risk.sungard.com 15 Jul '05

15 Jul '05

Hi Everyone, I've been running translate_pypy -text targetpypy every couple of days recently, to see how things are progressing, but I always get the same error: ImportError: No module named _formatting Is this a known problem, or am I doing something wrong? Cheers, Ben

2 1

plans for os and math
by Samuele Pedroni 14 Jul '05

14 Jul '05

This mail is an attempt to summarize a bit our current thinking and plans for starting implementing os and math modules functionality. The idea is to implement them as mixed modules (like sys and __builtin__ are now), so to have, for starting, a module/posix and module/math packages in module. E.g. open would be implemented by posix at interpreter level, for example like: def open(w_fname, w_flag, w_mode=0777): fname = space.str_w(w_fname) flag = space.int_w(w_flag) mode = space.int_w(w_mode) # notice that an unwrap_spec attached to open could be used to the same effect fd = os.open(fname, flag, mode) return space.wrap(fd) For now constants in os.* would be reimported by such a module. (So the translation process and result are platform dependent, we have ideas on how to fix this but those are lower priority). Similary math would be delegating to math at interpreter level. The delegation is to get sort of placeholders for translation and to be able to run on top of CPython. We have started implementing in the toolchain from annotation to the backends functionality such that the above call to os.open at interpreter-level can be annotated and proper working functionality can be supplied at the end of the chain by the backend. (See rpython/extfunctable.py and the contents of rpython/module). So sorting out platform differences is intended to be a backend task. The first things we need in os.* are the functions and constants required by our app-level implementation of files in lib/_file.py and lib/_sio.py. Functions like os.stat return tuple-like objects that have also named attributes corresponding to the various fields. The idea is that the calls at interpreter-level to such functions return results that can only be numerically indexed, so os.stat(...)[0] is OK at interpreter-level but os.stat(...).st_mode is not. The mixed module would have OTOH to implement such result types (possibly at app-level) and wrap in them the results of interpreter level calls to placeholder functions. regards.

1 0

Fw: [pypy-svn] r14617 - pypy/dist/pypy/module/unicodedata[POSSIBLE SPAM ]
by Ben.Young＠risk.sungard.com 13 Jul '05

13 Jul '05

Hi, I was just looking over the subversion commit messages and had a couple of comments to make. Did you really want [0 * ... ] in some of these diffs, rather than [0] * ...? Also, don't lists resize to avoid this kind of behaviour? Sorry if you had already spotted them! Cheers, Ben pypy-svn-bounces(a)codespeak.net wrote on 13/07/2005 15:05:11: > Author: ac > Date: Wed Jul 13 16:05:10 2005 > New Revision: 14617 > > Modified: > pypy/dist/pypy/module/unicodedata/function.py > Log: > Avoid frequent reallocations when normalizing unicode. > > Modified: pypy/dist/pypy/module/unicodedata/function.py > ============================================================================== > --- pypy/dist/pypy/module/unicodedata/function.py (original) > +++ pypy/dist/pypy/module/unicodedata/function.py Wed Jul 13 16:05:10 2005 > @@ -100,6 +100,7 @@ > NCount = (VCount*TCount) > SCount = (LCount*NCount) > > + > def normalize(space, w_form, w_unistr): > form = space.str_w(w_form) > if not space.is_true(space.isinstance(w_unistr, space.w_unicode)): > @@ -121,7 +122,9 @@ > space.wrap('invalid normalization form')) > > strlen = space.int_w(space.len(w_unistr)) > - result = [] > + result = [0] * (strlen + strlen / 10 + 10) > + j = 0 > + resultlen = len(result) > # Expand the character > for i in range(strlen): > ch = space.int_w(space.ord(space.getitem(w_unistr, space.wrap(i)))) > @@ -132,33 +135,57 @@ > V = VBase + (SIndex % NCount) / TCount; > T = TBase + SIndex % TCount; > if T == TBase: > - result.extend([L, V]) > + if j + 2 > resultlen: > + result.extend([0 * (j + 2 - resultlen + 10)]) > + resultlen = len(result) > + result[j] = L > + result[j + 1] = V > + j += 2 > else: > - result.extend([L, V, T]) > + if j + 3 > resultlen: > + result.extend([0 * (j + 3 - resultlen + 10)]) > + resultlen = len(result) > + result[j] = L > + result[j + 1] = V > + result[j + 2] = T > + j += 3 > continue > - > - result.extend(decomposition.get(ch, [ch])) > + decomp = decomposition.get(ch) > + if decomp: > + decomplen = len(decomp) > + if j + decomplen > resultlen: > + result.extend([0 * (j + decomplen - resultlen + 10)]) > + resultlen = len(result) > + for ch in decomp: > + result[j] = ch > + j += 1 > + else: > + if j + 1 > resultlen: > + result.extend([0 * (j + 1 - resultlen + 10)]) > + resultlen = len(result) > + result[j] = ch > + j += 1 > > # Sort all combining marks > - for i in range(len(result)): > + for i in range(j): > ch = result[i] > comb = unicodedb.combining(ch) > if comb == 0: > continue > - for j in range(i, 0, -1): > - if unicodedb.combining(result[j - 1]) <= comb: > - result[j] = ch > + for k in range(i, 0, -1): > + if unicodedb.combining(result[k - 1]) <= comb: > + result[k] = ch > break > > - result[j] = result[j - 1] > + result[k] = result[k - 1] > else: > result[0] = ch > > if not composed: # If decomposed normalization we are done > - return space.newunicode(result) > + return space.newunicode(result[:j]) > > - if len(result) <= 1: > - return space.newunicode(result) > + if j <= 1: > + return space.newunicode(result[:j]) > > current = result[0] > starter_pos = 0 > @@ -166,8 +193,8 @@ > prev_combining = 0 > if unicodedb.combining(current): > prev_combining = 256 > - for j in range(1, len(result)): > - next = result[j] > + for k in range(1, j): > + next = result[k] > next_combining = unicodedb.combining(next) > if next_insert == starter_pos + 1 or prev_combining < next_combining: > # Combine if not blocked > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn >

2 1

Re: [pypy-svn] r14625 - pypy/dist/pypy/objspace/std
by Michael Hudson 13 Jul '05

13 Jul '05

tismer(a)codespeak.net writes: > Author: tismer > Date: Wed Jul 13 17:53:23 2005 > New Revision: 14625 > > Modified: > pypy/dist/pypy/objspace/std/strutil.py > Log: > another less urgent optimization from the train Kiel-Berlin: > > refactored string_to_float quite a lot. The issue was raised > by Python 2.4's test_long, which caused an overflow in strutil > instead of an 1.#inf. > I took the chance to rework this quite a little, with the result of > - less rounding errors > - smaller code > - much faster in extreme cases > - able to eval float('1.'+10000*'0'+'e-10000') and friends > - seems to produce exactly the same as builtin float as far as tested. Thank you for doing this! The original code was written in about 5 minutes flat... > + # Usage of long numbers is explicitly avoided, because > + # we want to be able to work without longs as a PyPy option. Unfortunately, the 'input problem for floating point numbers' cannot be solved using arithmetic of any fixed finite precision, see http://citeseer.ist.psu.edu/clinger90how.html for a proof of this. Also, I found a difference between CPython and your code: >>> strutil.string_to_float('0.099999999999999999') 0.099999999999999992 >>> float('0.099999999999999999') 0.10000000000000001 But still, a vast improvement, thanks! Cheers, mwh -- If comp.lang.lisp *is* what vendors are relying on to make or break Lisp sales, that's more likely the problem than is the effect of any one of us on such a flimsy marketing strategy... -- Kent M Pitman, comp.lang.lisp

2 2

pirate list
by Michal Wallace 13 Jul '05

13 Jul '05

Hello, PyPy list :) I've just started a new list for pirate, a python compiler for the parrot virtual machine. Pirate is written in python and currently uses the standard compiler package. However, as parrot supports native types, I would like to refactor it so that it can also work with PyPy's type annotation system (and possibly integrate with pypy in other ways) The pirate website is here: http://pirate.tangentcode.com/ Quite a bit of work has been done on pirate since I last updated the website back in 2003, mostly by Sam Ruby of http://intertwingly.net/ Sam is giving a python-on-parrot presentation at OSCON on August 4, so I'd like to get a few things cleaned up before then - for example, updating the site, creating a development roadmap, an fixing the code to work with the latest version of parrot, etc. If you're interested in helping out with any of these things, or just want to talk about pirate, then feel free to hop on board. The signup page is here: http://cornerhost.com/mailman/listinfo/pirate Thanks! Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal(a)sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ -------------------------------------

2 3