Mailman 3 August 2005 - pypy-dev

Re: genc to genllvm
by Armin Rigo Aug. 15, 2005

Aug. 15, 2005

Hi Richard! (CC to pypy-dev) On Sun, Aug 14, 2005 at 12:29:51AM +0100, Richard Emslie wrote: > Remember we talked about this at the end of the last sprint: > > def test_cast_to_int(): > def casting(v): > return int(ord(chr(v))) > ... being what caused the seg fault in the first few iterations of > genc'ing and pypy IIRC. Well I went hunting for the test again and didnt > find it (so the above test is mine, along with misunderstandings :-). It's in … [View More]

1 0

Re: [pypy-svn] r16054 - pypy/dist/pypy/translator/llvm2/test
by Christian Tismer Aug. 14, 2005

Aug. 14, 2005

rxe(a)codespeak.net wrote: > ASIDE - any pointers to any docs relating to acronyms used in the > transformation would be much appreciated. Before I read the docstring I > assumed it was doing the opposite of what it does. Sorry for my ignorance in > this matter. This article is quite useful: http://www.cl.cam.ac.uk/~jds31/research/ssi.html ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! … [View More]

1 0

Towards Milestone 1
by Armin Rigo Aug. 11, 2005

Aug. 11, 2005

Hi all, Here are some generally interesting comparisons between the current status of translation in PyPy and what we set as our Milestone 1. This is getting closer, both in term of work remaining to be done (good!) and deadline promized to the EU (hurry!). (For the latter, I am refering to http://codespeak.net/svn/pypy/funding/negotiations/part_b_2004_12_11.pdf pp 80-81.) Near the end of the Hildesheim sprint we acheived the first self-contained translated PyPy. Shortly thereafter we … [View More]could produce a standalone executable instead of an extension module for CPython. In this respect we have reached an important part of our first milestone. Note that this is still in heavy development; I'm not sure the current trunk successfully translates. It did, take my word :-) (BTW I propose that the pypy-translation-snapshot always contains the latest revision that is known to translate; we should have a script to automate taking a new snapshot.) There are some pieces missing from PyPy itself, most notably some C extension modules of CPython and a few specific features (zip-imports, weakrefs, ...). I will not discuss these here. The subject of this e-mail is to see what is missing translation-wise: * first it would be cool if LLVM could also translate PyPy. This seems to be very close as well! (Eric should tell us more about it after tomorrow's pypy-sync meeting) * memory management: the C back-end uses refcounting; LLVM uses the Boehm GC. It would be quite easy to have a flag to the C back-end to use Boehm as well instead of refcounting. Carl Friedrich is working on a more general GC framework; at the moment it is unclear (at least to me) when and how easily we will be able to use it to add custom GCs to our back-ends. (Carl should tell us more about it after tomorrow's pypy-sync meeting) * threading: implementing this requires a mixture of source-code-level changes and help from the translation process, depending on the approach taken. At the moment we have no threads at all. There are two threading models that are relatively easy to implement by now, and more to think about: 1) the Global Interpreter Lock (GIL), as in CPython. Only one thread interprets Python bytecodes at a time. All other threads are either blocked waiting for the GIL, or doing I/O. In CPython, around each I/O function call, there are hand-coded lines to release and re-acquire the GIL. In PyPy we can insert these lines automatically at translation time, whenever we call one of the hand-coded C functions in pypy/translator/c/src/ll_*.h. The source code of PyPy needs a minor extension so that every 10 or 100 bytecodes it calls a special function "now is a good time to release the GIL to give the other threads a chance to run". Should be fairly straightforward. 2) full Stackless. As long as some rather inefficient solution is good enough, this is not so difficult. We can modify the C back-end to generate functions differently, so that no C function calls any other C function directly. Instead, there is a short "main loop", along the lines of while (1) { next_fn = state->continuation_fn; state = next_fn(state); } Each generated function returns a new 'state' structure whose 'continuation_fn' member contains the function to call next. The 'state' structure also contains arbitrary data like the arguments we want to send to the next function. The net result is that the C stack is no longer used. Then we can have "tasklets" which each record a 'state' to run next, and switching to another tasklet is done by a special C function that returns the 'state' of the other tasklet. Getting the basics done should not be too difficult -- although it's an open door to endless involved optimization hacks, as Christian knows :-) 3) there are also other less well-thought ideas for threading, mostly along the lines of per-object locking. Here too, it should be possible to get a not-too-bad result without having to insert locks everywhere by hand. For example, we could reuse the proxy object space mechanism for a LockingObjSpace which first acquires the lock of each object involved in each space operation. * finally, let's consider the translation process itself. It is flexible in principle, but it's not really designed as a framework with a well-defined API or hooks to plug into. However, it is possible to hack here and there to change various translation aspects. This is, to some extent, the whole idea of PyPy's flexibility: a framework with hooks and APIs allows only so much experimentation; sooner or later the ability to directly code things differently is more powerful. Nevertheless I guess that some refactoring would help to make localized translation aspects easier to change, e.g. by providing more "policy" objects to control the process, or (for OOP enthusiasts) by designing the classes with subclassing in mind. For the short-term future, we have to draw priorities with this EU Milestone 1 in mind. We promized "hooks into the internals to alter translation aspects"; in my opinion (debate welcome!) spending time on this right now is not a really good idea. Better spend it on the more concrete issues of GC and threading, which in some sense already prove that we have enough hooks to do some variations. In particular, the Stackless version of PyPy should be an excellent and -- I believe -- quite reachable result. It's more than a minor variation: it's a completely different kind of generated code. It would definitely show that our translation process is capable of producing more than just a dummy translation -- which is the whole point. A bientot, Armin. [View Less]

3 3

pypy-sync meeting 11th Aug 1pm
by hpk＠trillke.net Aug. 10, 2005

Aug. 10, 2005

hi pypy-dev, tomorrow is the usual thursday-1pm active developers meeting at #pypy-sync on freenode. See http://codespeak.net/svn/pypy/extradoc/minute for minutes of previous meetings and below for the agenda of this one. If you can't come you may send me or someone else your activity infos by mail and we'll proxy them accordingly. cheers, holger ============================================= pypy-sync developer meeting 11th August 2005 =======================================… [View More]

1 0

Compilation errors
by Ben.Young＠risk.sungard.com Aug. 10, 2005

Aug. 10, 2005

Hi everyone, Is anyone else getting compilation errors on any generated c program at the moment? This is happening on Windows, and has been going on for a couple of days. The errors are: - - - - - - - - - - - test_set_attr: recorded stdout - - - - - - - - - - - - testing_1.c^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_os.h(92) : warning C4013: 'getcwd' undefined; assuming extern returning int^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_os.h(92) : … [View More] warning C4047: '=' : 'char *' differs in levels of indirectio n from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_os.h(160) : warning C4244: 'return' : conversion from '__int64' to 'long ', possible loss of data^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(26) : error C2143: syntax error : missing ';' before 'type'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(34) : error C2143: syntax error : missing ';' before 'type'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(35) : error C2065: 's' : undeclared identifier^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(35) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(36) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(37) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(38) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(39) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(40) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(42) : error C2065: 'buf_size' : undeclared identifier^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(42) : warning C4047: '=' : 'char *' differs in levels of indire ction from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(43) : warning C4047: 'function' : 'const char *' differs in lev els of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(45) : warning C4022: 'free' : pointer mismatch for actual param eter 1^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(48) : warning C4047: '==' : 'char *' differs in levels of indir ection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(53) : warning C4047: 'function' : 'const char *' differs in lev els of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(60) : error C2143: syntax error : missing '{' before '*'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(104) : warning C4133: 'return' : incompatible types - from 'RPy String *' to 'int *'^M Not sure exactly what's causing the error, as visual studio never does a file "traceback". Cheers, Ben [View Less]

1 0

Build and test failures
by Ben.Young＠risk.sungard.com Aug. 9, 2005

Aug. 9, 2005

Hi Everyone, Just synced to head this morning and noticed a couple of errors in the c translation process. This first is that on windows, a couple of headers are not available. I fixed this by patching locally with: Index: translator/c/extfunc_include.h =================================================================== --- translator/c/extfunc_include.h (revision 14483) +++ translator/c/extfunc_include.h (working copy) @@ -6,9 +6,9 @@ /*******************************************… [View More]***********************/ -#include <unistd.h> +/*#include <unistd.h> #include <sys/types.h> -#include <sys/stat.h> +#include <sys/stat.h>*/ #include <fcntl.h> #include <time.h> #ifndef PATH_MAX Then, when the tests are run I get: $ ./test_all.py translator/c/ ============================= test process starts ============================= testing-mode: inprocess executable: c:\Python24\python.exe (2.4.1-final-0) using py lib: c:\Documents and Settings\YoungB\dist\py <rev unknown> translator\c\test\test_annotated.py[19] ................... translator\c\test\test_backendoptimized.py[39] ....................................... translator\c\test\test_database.py[17] ................. translator\c\test\test_exception.py[3] ... translator\c\test\test_extfunc.py[4] F... translator\c\test\test_genc.py[8] ........ translator\c\test\test_lltyped.py[2] .. translator\c\test\test_notype.py[33] ................................. translator\c\test\test_operation.py[1] . translator\c\test\test_support.py[7] ....... translator\c\test\test_symboltable.py[1] . translator\c\test\test_typed.py[38] ...................................... _______________________________________________________________________________ _________________________ entrypoint: test_time_clock _________________________ def test_time_clock(): def does_stuff(): return time.clock() f1 = compile(does_stuff, []) t0 = time.clock() t1 = f1() assert type(t1) is float t2 = time.clock() E assert t0 <= t1 <= t2 > assert 165.0 <= 2.5683609740123038e-005 [c:\Documents and Settings\YoungB\dist\pypy\translator\c\test\test_extfunc.py:16] - - - - - - - - - - - test_time_clock: recorded stderr - - - - - - - - - - - - [annrpython] (pypy.translator.c.test.test_extfunc:9) does_stuff -> SomeFloat() [annrpython] (pypy.rpython.rclass:575) ll_runtime_type_info__objectPtr -> SomePtr(ll_ptrtype=<* RuntimeTypeInfo (opaque)>) [annrpython] (pypy.rpython.module.ll_time:14) ll_time_clock -> SomeFloat() [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool(const=True) [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool() [annrpython] (pypy.rpython.rclass:565) ll_type__objectPtr -> SomePtr(ll_ptrtype=<* Struct object_vtable { parenttypeptr: * Struct objec t_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(const=<* struct object { typeptr=... }>, ll_ptr type=<* GcStruct object { typeptr: * Struct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opa que), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(ll_ptrtype=<* GcStruct object { typeptr: * Stru ct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:53) ll_raise_OSError__Signed -> SomeImpossibleValue() _______________________________________________________________________________ =========== tests finished: 171 passed, 1 failed in 263.86 seconds ============ I think this is because when clock is called in the extension it thinks it is the first time and returns the very small value. I don't really know though. I hope you all had fun at the sprint! Cheers, Ben [View Less]

3 7

enabling marshal
by Christian Tismer Aug. 6, 2005

Aug. 6, 2005

Some small issues which came up when I enabled marshal: ------------------------------------------------------- a dict that never gets a value does not create an emptydict, but rtyper crashes. This happens easily, when code is disabled, temporally. os.write does not know that it returns a nonnegative int. I think we should have a way to express this instead of putting assertions. A funny observation on stringobject: The RTyper did not know how to handle mul_str_times. That should imply that … [View More]

1 0

pypy-sync meeting 4th August 1pm
by hpk＠trillke.net Aug. 4, 2005

Aug. 4, 2005

Hi pypy-dev, we've had three so called 'pypy-sync' meetings already on the last thursdays. The minutes are to be found under http://codespeak.net/svn/pypy/extradoc/minute However, they have so far been only announced on internal channels and only some of the minutes went to pypy-dev. If i remember correctly. This post is to announce the pypy-sync meeting for tomorrow (thursday 4th August) 1pm. The meeting is open for all active developers and closed to lurkers (which can read the … [View More]minutes afterwards). The duration is capped at 30 minutes. We did manage to conclude always in time so far. The agenda always has a fixed part and a variable part with "topics of the week". I guess we somewhat expect from EU funded developers to attend the meeting (or provide info off-line) but for all others it's a complete matter of their free choice. So what is the purpose of the pypy-sync meeting? It's an XP-style meeting that serves to synchronize development work and let everybody know who is working on what. It also serves as a decision board of the PyPy developers. If discussions last too long and decisions cannot be reached they are delegated to a sub-group or get postponed. Here is the agenda (and if you come, please try to come in time, there is no waiting and attendants are expected to not walk away during the 30 minutes). So below is the the agenda for this week's meeting. Developers wanting a topic on the list need to contact me until wednesday afternoon. This procedure is repeated every week. cheers, holger ============================================= pypy-sync developer meeting 4th August 2005 ============================================= Regular Topics ==================== - roll call. - activity reports (3 prepared lines of info). All Attendees submit activity reports in the form of three lines prepared before the meeting: LAST: work area1, work area2, ... NEXT: work area7, issue... BLOCKERS: the unresolved XXX blocked my work on issueX - resolve conflicts/blockers here we try to resolve any conflicts/blocks Topics of the week (brief discussion + quick decisions) ========================================================= regularly doing more closed sprints? ---------------------------------------- In a private conversation Bea asked Holger how the developer group thinks of the hildesheim2-sprint type (which was similar to the Pre-EP sprint in Gotenburg). A question is: do we think we want to regularly do sprints where we only invite developers who are at least somewhat up to speed with PyPy? Should sprints around conferences always be completely open and some of the sprints in between then possibly more closed? Note that Bea thinks of organizing a discussion about this at the upcoming Heidelberg sprint. 0.6.2 release: dropping the idea and just going for 0.7 ... ---------------------------------------------------------------------------- It has basically been decided by the hildesheim2 sprint group that Heidelberg should tackle a 0.7 self-contained PyPy version. (a previous pypy-sync meeting delegated this decision there,btw). It seems that we then probably want to drop the idea of a 0.6.2 release. But the release goal for 0.6.2, namely the compliancy goal of again reaching 90% compliancy for our 2.4.1 based tree, should then be shifted to 0.7. We should conciously decide this. Plans for fixing compliancy bugs ------------------------------------ It has been discussed by the hildesheim2 sprint group that we want to have a text file in lib-python/ that lists the problems with failing test modules so that we can share our analysis and have a lightweight process of fixing and tracking those issues. Moreover, Holger suggests (or even reminds?) to mandate adding small appropriate tests below pypy/ when fixing compliancy tests instead of just relying on the sometimes very slow-to-run compliancy tests. [View Less]

2 2

day 7: winding down
by Carl Friedrich Bolz Aug. 4, 2005

Aug. 4, 2005

Hi PyPy-Dev! Some quit notes about Sunday, the 7th and last day of the sprint. In general it was sort of a lazy day where we reveled in the success of the previous night ;-). The day was started with an extensive breakfast in the sun. After that Alex (from the Trillke-Gut) brought bows and arrows outside and gave us a short introduction to archery. We spent some time trying it (and lost four arrows in the proccess). - Holger and Armin worked on refactoring and improving the compiling … [View More]

2 1

PyPy broken! What's up?
by Christian Tismer Aug. 3, 2005

Aug. 3, 2005

Hi Friends, in Hildesheim, I struggled for many hours to get my new marshal module to compile and integrate. Finally, after removing all problematic things where the annotator or the rtyper complained about, I figured out that the current PyPy SVN version did not compile at all. Today, I tried again, after I brought marshal some quality levels further. Still, PyPy does not translate, neither on Windows ( with my patches of course) nor on Debian Linux. The property to be translatable to a C … [View More]

2 2