Mailman 3 December 2003 - pypy-dev

Andy Robinson wants to pen in a day for the ACCU Python-UK conference
by Laura Creighton Jan. 6, 2004

Jan. 6, 2004

14-17 April. Jacob and I cannot attend this, alas. Laura http://www.accu.org/conference/ ------- Forwarded Message Replied: "Andy Robinson" <andy(a)reportlab.com> Return-Path: andy(a)reportlab.com From: "Andy Robinson" <andy(a)reportlab.com> To: "Laura Creighton" <lac(a)strakt.com> Subject: RE: ACCU 2004 Date: Fri, 19 Dec 2003 09:59:46 -0000 We're trying to have a day on all the 'new implementations of python' at ACCU - Jython, .NET, PyPy, Psyco, Stackless etc. It occurs to me the relevant people are probably with you now. I'd LOVE to wrap it up with an overview talk on how they will all converge one day, if that's still the plan. Can you ask (a) if anyone could speak and (b) if they want a sprint in Oxford that week? I can look into sprint facilities in January if it helps... - - Andy ------- End of Forwarded Message

3 2

Pie-thon benchmark
by Seo Sanghyeon Jan. 2, 2004

Jan. 2, 2004

Happy new year, pypy-dev! Guido announced Pie-thon benchmark. http://mail.python.org/pipermail/python-dev/2003-December/041527.html I tried to run it against PyPy revision 2706. First, here is Python 2.3 performance on my system, measured by `make times'. b0.py: real 0m2.720s b1.py: real 0m1.069s b2.py: real 0m0.343s b3.py: real 0m2.137s b4.py: real 0m0.709s b5.py: real 0m2.100s b6.py: real 0m1.903s For PyPy, I used `time python py.py -S b?.py'. To make relative import work I prepended `PYTHONPATH=.' for b4.py and b6.py. b0.py: error __new__() takes exactly 4 arguments (1 given) b1.py: error maximum recursion depth exceeded b2.py: real 1m25.725s (250x slow) b3.py: error __new__ multimethod got a non-wrapped argument b4.py: error (this one imports and uses b0.py) b5.py: error global name complex is not defined b6.py: error unbound method must be called with instance as first argument More comments on b6.py: it took 22 minutes(!) to crash. It printed 42, 1000041, 999999, 49999950000, and finally crashed at `d = dict.fromkeys(xrange(1000000))'. Will investigate more.

3 2

documentation progress!
by holger krekel Dec. 27, 2003

Dec. 27, 2003

Hello PyPy, apart from the new "architecture" document the Amsterdam Sprint also produced a "goals" document which i now think should be rewritten to be (much) more higher level. See here for reference http://codespeak.net/pypy/index.cgi?doc/goals As it stands it's IMHO very confusing to understand from this document what PyPy aims at. I guess the current 'goals' document is more like an extensive "todo list" and maybe should just be renamed. (Because from a Todo-document one a) doesn't expect to understand everything at once and b) expects more lower level details). Apart from this little renaming we also concluded earlier that we want to have a "Status" document which should offer precise information about the implementation/concept status regarding our "components": interpreter, standard/flow/trace/trivial object spaces, annotation and translation. Each of these components should then have a list of main pieces wrong, missing, incomplete or (yes they exist!) complete. Btw, Armin has in the last days added/modified a chapter about Multimethods and Annotations (type inference): http://codespeak.net/pypy/index.cgi?doc/objspace/multimethod http://codespeak.net/pypy/index.cgi?doc/translation/annotation but don't forget to read the architecture document first in case you e.g. stumble on the term "object space" and don't know what it means :-) http://codespeak.net/pypy/index.cgi?doc/architecture cheers and some happy last days of the year, holger

1 0

Weird failure of test.py as testit.py
by Stefan Schwarzer Dec. 24, 2003

Dec. 24, 2003

To avoid a conflict between tool/test.py and a potential directory tool/test, I have renamed test.py to testit.py and made a corresponding change to test_all.py . Before I checked the change in, all tests, invoked from testit.py or test_all.py, ran. After checking in the changed files I noticed that the tests no longer ran. That is, testit.py would start but find no tests. Eventually I found out that everything worked as long as test.py or test.pyc remained in the tool directory. (So copying testit.py to test.py makes you again able to run the tests.) So I had checked my changes, finding everything working, checked in, and cleaned away test.py* . Thus, I didn't notice that something had gone astray before committing the changes. Now there's the question what's wrong with test.py/testit.py . I suspected an import of "test" from within the module but found no such import. I'm still searching for the cause of the problem, but if someone has a quick idea, it would be fine if he/she responded on the list or even fixed the problem. Stefan

2 4

Re: [pypy-dev] bug with vars() in a nested function
by Laura Creighton Dec. 23, 2003

Dec. 23, 2003

In a message of Mon, 22 Dec 2003 22:23:24 +0100, Alex Martelli writes: >The implementation of vars() in builtin.py relies on a call to getdictsco >pe() >to return the caller's locals as a dir, but unfortunately that doesn't wo >rk >when the caller of built-in vars() is a nested function -- nestedscope.py >overrides fast2locals to ensure setting in self.w_locals the free variabl >es >in addition to the locals, and that w_locals is what getdictscope returns >. > >Apparently this kind of bug must have bitten CPython at some point (or >at least be suspected!) since the unit-test for vars() looks for it prett >y >exactly (that's how I found it, too -- still striving to run as much of t >hat >CPython unit-test as feasible!) -- basically, it has something like: > > def test_vars(self): > def get_vars_f0(): > return vars() > def get_vars_f2(): > get_vars_f0() > a = 1 > b = 2 > return vars() > ... > self.assertEqual(get_vars_f2(), {'a': 1, 'b': 2}) > >where that 'useless' call to get_vars_f0 inside get_vars_f2 serves just >the purpose of "infecting" the latter with the former name -- and sure >enough the assertEqual fails because 'get_vars_f0' is reported as being >a key in the dict returned by get_vars_f2(). > >My instinct is to patch this with an optional switch to getdictscope to >ask for "return the TRUE locals only please" -- but since a vaguely >analogous patch I had tried to remedy the issue with generators raising >StopIteration (i.e. "directly attacking the symptom rather than going for >a more general solution") was criticized, I thought I'd better ask first >rather than committing anything yet -- this one IS a rather marginal and >obscure side issue after all, so for now I'll just mark it as 'XXX TODO' >and comment it away in the builtin_functions_test.py file (what's one >more...). > >Which reminds me: do we have a "known bugs" file/summary, or are >the infos about known bugs, limitations &c spread around in comments >and docstrings? 'cause the XXX TODO in the builtin_*_test.py files >are quite a nifty list of known current bugs and limitations for built-in >s, >so I wondered if I should collect them in some documentation file... Please put them someplace and have goals refer to them. I wonder if this is what we should be using the issue tracker for.... As per your real question -- fast2locals strikes me as a hack, that needs to be more general, and our understanding of what a scope would be has been significantly mangled since getdictscope was written. Something in me says that we want to do scoping in some more cleaner way, but I cannot quite envision what it will be after builtins changes _again_ to be more like a regular module. I am now looking at our current crop of builtins, and thinking ... 'the only reason you lot are there is because CPython had you there'. Now that we have hacked the architecture one more time again, grin, we have something a lot cleaner (for now at any rate) ... I am pushing for a more elegant definition of builtin, based on a pragmatic idea of 'you have to be built in becauswe we cannot make you any other way' What of ours make it? Which cannot be made in app space and why, given that interp space is dead? I think that we have gloriously moved to a place where most of our builtins really do not have to be. Some sort of execfile sort of thing is all we need. This is probably wrong, and indicates that my vision is too idealised, and I miss very real practical problems. I await more enlightenment. But I still think that most of our 'builtins' are more 'reimplentations of CPython builtins, done because CPython doesn't have them so we couldn't have a working Python without them'. And we could implement them as a module. When we have leftovers, real problems in converting one object space to another, then things like getdictscope -- or a more general thing that does this and more actually -- strikes me as a thing that belongs __there__. Scoping rules strike me as object space 'required things to leave a hook out for'. But perhaps I am just confused again, and oversimplifying in my mind. I await enlightenment. Laura > > >Alex > >_______________________________________________ >pypy-dev(a)codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev

4 5

bug with vars() in a nested function
by Alex Martelli Dec. 23, 2003

Dec. 23, 2003

The implementation of vars() in builtin.py relies on a call to getdictscope() to return the caller's locals as a dir, but unfortunately that doesn't work when the caller of built-in vars() is a nested function -- nestedscope.py overrides fast2locals to ensure setting in self.w_locals the free variables in addition to the locals, and that w_locals is what getdictscope returns. Apparently this kind of bug must have bitten CPython at some point (or at least be suspected!) since the unit-test for vars() looks for it pretty exactly (that's how I found it, too -- still striving to run as much of that CPython unit-test as feasible!) -- basically, it has something like: def test_vars(self): def get_vars_f0(): return vars() def get_vars_f2(): get_vars_f0() a = 1 b = 2 return vars() ... self.assertEqual(get_vars_f2(), {'a': 1, 'b': 2}) where that 'useless' call to get_vars_f0 inside get_vars_f2 serves just the purpose of "infecting" the latter with the former name -- and sure enough the assertEqual fails because 'get_vars_f0' is reported as being a key in the dict returned by get_vars_f2(). My instinct is to patch this with an optional switch to getdictscope to ask for "return the TRUE locals only please" -- but since a vaguely analogous patch I had tried to remedy the issue with generators raising StopIteration (i.e. "directly attacking the symptom rather than going for a more general solution") was criticized, I thought I'd better ask first rather than committing anything yet -- this one IS a rather marginal and obscure side issue after all, so for now I'll just mark it as 'XXX TODO' and comment it away in the builtin_functions_test.py file (what's one more...). Which reminds me: do we have a "known bugs" file/summary, or are the infos about known bugs, limitations &c spread around in comments and docstrings? 'cause the XXX TODO in the builtin_*_test.py files are quite a nifty list of known current bugs and limitations for built-ins, so I wondered if I should collect them in some documentation file... Alex

2 1

amsterdam sprint reports (20th december 2003)
by holger krekel Dec. 23, 2003

Dec. 23, 2003

Hello PyPy, the Amsterdam sprint has just finished and here is a report and some surrounding and outlook information. As usual please comment/add/correct me - especially the sprinters. I also wouldn't mind some discussion of what and how we could do things better at the next sprint. First of all, big thanks to *Etienne Posthumus* who patiently helped organizing this sprint even though he had to deal with various other problems at the same time. Before i start with details i recommend reading through the new Architecture document at http://codespeak.net/pypy/index.cgi?doc/architecture.html in case you don't know what i am talking about regarding the Amsterdam sprint results :-) Originally, we intended to go rather directly for translation and thus for a first release of PyPy. But before the sprint we decided to go differently about the sprint not only because Michael Hudson and Christian Tismer had to cancel their participation but we also wanted to give a smooth introduction for the new developers attending the sprint. Therefore we didn't press very hard at translation and type inference and major suprises were awaiting us anyway ... fixing lots and lots of bugs, adding more builtins and more introspection ------------------------------------------------------------------------- On this front mainly Alex Martelli, Patrick Maupin, Laura Creighton and Jacob Hallen added and fixed a lot of builtins and modules and made it possible to run - among other modules - the pystone benchmark: on most machines we have more than one pystone with PyPy already :-) While trying to get 'long' objects working Armin and Samuele realized that the StdObjSpace multimethod mechanism now desparately needs refactoring. Thus the current "long" support is just another hack (TM) which nevertheless allows to execute more of CPython's regression tests. In a releated effort, Samuele and yours truly made introspection of frames, functions and code objects compatible to CPython so that the "dis.dis(dis.dis)" goal finally works i.e can be run through PyPy/StdObjSpace. This is done by the so called "pypy_" protocol which an object space uses to delegate operations on core execution objects (functions, frames, code ...) back to the interpreter. redefining our standard type system at application level -------------------------------------------------------- Originally we thought that we could more or less easily redefine the python type objects at application level and let them access interpreter level objects and implementations via some hook. This turned out to be a bootstrapping nightmare (e.g. in order to instantiate classes you need type objects already but actually we want our first classes to define exactly those). While each particular problem could be worked around somehow Armin and Samuele realized they were opening a big can of worms ... and couldn't close it in due time. The good news is that after lots of discussions and tossing ideas around we managed to develop a new approach (see the end of the report) which raised our hopes we can finally define the types at application level and thus get rid of the ugly and hard to understand interpreter level implementation. Improving tracing and debugging of PyPy --------------------------------------- With PyPy you often get long tracebacks and other problems which make it hard to debug sometimes. Richard Emslie, Tomek Meka and me implemented a new Object Space called "TraceObjSpace" which can wrap the Trivial and Standard Object Space and will trace all objectspace operations as well as frame creation into a long list of events. Richard then in a nightly hotel session wrote "tool/traceinteractive.py" which will nicely reveal what is going on if you execute python statements: which frames are created which bytecodes are executed and what object space operations are involved. Just execute traceinteractive.py (with python 2.3) and type some random function definition and see PyPy's internals at work ... It only works with python 2.3 because we had to rewrite python's dis-module module to allow programmatic access to dissassembling byte codes. And this module has considerably changed from python 2.2 to 2.3 (thanks, Michael :-) "finishing" the Annotation refactoring -------------------------------------- That should be easy, right? Actually Guido van Rossum and Armin had started doing type inference/annotation in Belgium just before EuroPython and we have since refactored it already at the Berlin sprint and in between the sprints several times. But it turned out that Annotations as we did them are *utterly broken* in that we try to do a too general system (we had been talking about general inference engines and such) thus making "merging" of two annotations very hard to do in a meaningful way. But after beeing completly crushed on one afternoon, Samuele and Armin came up with a new *simpler* approach that ... worked and promises to not have the same flaws. It got integrated into the translator already and appears to work nicely. I think this is the fourth refactoring of just "three files" and, of course, we already have the 'XXX' virus spreading again :-) refactoring/rewriting the test framework ---------------------------------------- PyPy has an ever increasing test-suite which requires a lot of flexibility that the standard unittest.py module just doesn't provide. Currently, we have in tool/test.py and interpreter/unittest_w.py a collection of more or less weird hacks to make our (now over 500) tests run either at interpreter level or at application level which means they are actually interpreted by PyPy. Tests from both levels can moreover be run with different object spaces. Thus Stefan Schwarzer and me came up with a rewrite of unittest.py which is currently in 'newtest.py'. During my train ride back to germany i experimentally used our new approach which let our tests run around 30% faster (!) as a side effect. More about this in separate mails as this is - as almost every other area of PyPy - refactoring-in-progress. Documentation afternoon ----------------------- We (apart from Richard who had a good excuse :-) also managed on Friday to devote a full afternoon to documentation. There is now an emerging "architecture" document, a "howtopypy" (getting started) and a "goals" document here: http://codespeak.net/pypy/index.cgi?doc Moreover, we deleted misleading or redundant wiki-pages. In case you miss one of them you can still access them through the web by following "Info on this page" and "revision history". We also had a small lecture from Theo de Ridder who dived into our flowgraph and came up with some traditional descriptions from compiler theory to describe what we are doing. He also inspired us with some other nice ideas and we certainly hope he revisits the projects and continues to try to use it for his own purposes. There of course is no question that we still need more higher level documentation. Please don't use the wiki for serious documentation but make ReST-files in the doc-subtree. I guess that we will make the "documentation afternoon" a permanent event on our sprints. Towards more applevel code ... ------------------------------ As mentioned before the approach of defining the python types at application level didn't work out as easy as hoped for. But luckily, we had - again in some pub - the rescuing idea: a general mechanism that lets us trigger "exec/eval" of arbitrary interpreter level code given as a string. Of course this by itself is far too dynamic to be translatable but remember: we can perform *arbitrarily dynamic* pythonic tricks while still *initializing* object spaces and the interpreter. Translation will start with executing the initialized interpreter/objspace through another interpreter/flowobjspace instance. Some hacking on the last day showed that this new approach makes the definition of "builtin" modules a lot more pythonic: modules are not complicated class instances anymore but really look like a normal module with some "interpreter level escapes". It appears now that in combination with some other considerations we will finally get to "types.py" defining the python types thus getting rid of the cumbersome 10-12 *type.py files in objspace/std. There are still some XXX's to fight, though. Participants ------------ Patrick Maupin Richard Emslie Stefan Schwarzer Theo de Ridder Alex Martelli Laura Creighton Jacob Hallen Tomek Meka Armin Rigo Guenter Jantzen Samuele Pedronis Holger Krekel and Etienne Posthumus who made our Amsterdam Sprint possible. outlook, what comes next? ------------------------- On the Amsterdam sprint maybe more than ever we realized how strongly refactoring is the key development activity in PyPy. Watch those "XXX" :-) Some would argue that we should instead think more about what we are doing but then you wouldn't call that extreme programming, would you? However, we haven't fixed a site and date for the next sprint, yet. We would like to do it sometime February in Suitzerland on some nice mountain but there hasn't emerged a nice facility, yet. Later in the year we might be able to do a sprint in Dublin and of course one right before EuroPython in Sweden. Btw, if someone want to offer helping to organize a sprint feel free to contact us. Also there was some talk on IRC that we might do a "virtual sprint" so that our non-european developers can more easily participate. This would probably mean doing screen-sessions and using some Voice-over-IP technology ... we'll see what will eventually evolve. After all, we might also soon get information from the EU regarding our recent proposal which should make sprint planning easier in the long run. We'll see. For now i wish everyone some nice last days of the year which has been a fun one regarding our ongoing pypy adventure ... cheers, holger (who hopes he hasn't forgotten someone or something ...)

2 1

Re: [pypy-dev] Re: [pypy-svn] rev 2674 - pypy/trunk/doc
by Laura Creighton Dec. 22, 2003

Dec. 22, 2003

In a message of Mon, 22 Dec 2003 23:48:18 +0100, holger krekel writes: >Hi Bengt, > >[Bengt Richter Mon, Dec 22, 2003 at 01:17:03PM -0800] >> At 18:07 2003-12-22 +0100, you (holger krekel) wrote: >> >Hi Alex, >> > >> >[Alex Martelli Mon, Dec 22, 2003 at 05:52:40PM +0100] >> >> Is there a way, with subversion, to have a simple conformance test f >or this >> >> kind of formatting parameters be run automatically, whenever a textf >ile is >> >> committed? If files breaking these conventions (or whatever convent >ions >> >> we can all agree on) were never committed, this would minimize the n >eed >> >> for "changes that are purely related to formatting", I think. >> > >> >Indeed. A pre-commit hook is able to perform checks like this although >> >the details might get hairy if you go for "indentation of four-spaces" > >> >enforcement (e.g. considering docstrings) . I guess i'd go for simpli >city >> >and just enforce "line-length < 80" and svn:eol-style==native for all >*.py >> >and *.txt files (unless their svn:mime-type is explicitely non-text or > some >> >such). Non-conforming commits would simply be refused and nothing wou >ld be >> >changed on the fly. Opinions? >> >> Comments from left field (since I wound up a mere lurker after all ;-) > >Your are welcome and that can change, anyway :-) > >> 1. I could see automated refusal of commits being a nuisance unless the >re >> was an override available. > >If we choose transparent and consensual rules and can provide nice error >messages i think they should help more then do harm. Often have people >checked in files just to find out they did wrong and had to checkin >another time ... > >> 2. What about a separate pypytidy.py tool that people could use like >> a standards-enforcer/beautifier/spell-checker _before_ committing? >> Seems like an enforcer-hook would contain much of the logic anyway, >> and done separately a useful tool could evolve as a side effect. > >we have pypy/tool/fixeol.py although this mainly deals with line >endings. Our tools sections is an ever expanding branch of PyPy :-) > >cheers, > > holger >_______________________________________________ >pypy-dev(a)codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev I am thinking that we need a way to say 'this line purposely longer'. Big URLS, and things like: 'Don't write code like this: <blatt code that won't fit on one line>' Laura

1 0

Re: [pypy-svn] rev 2674 - pypy/trunk/doc
by holger krekel Dec. 22, 2003

Dec. 22, 2003

Hi Alex and others, i think we should try to minimize the number of formatting changes not only in program code but also in ReST files as it makes reading the diffs really hard because you basically have to reread the whole document in order to see the differences. As this has happended now several times i think it makes sense to think of some convention which allows the diffs to stay readable. Maybe (especially) ReST-changes should be done with a two-step approach: first the one where you can read the content diff and then the one (if neccessary at all) which has purely formatting changes and says so in the commit message. And we might also think about a common line width so that not everybody reformats the text again and again :-) cheers, holger [alex(a)codespeak.net Mon, Dec 22, 2003 at 04:26:21PM +0100] > Author: alex > Date: Mon Dec 22 16:26:20 2003 > New Revision: 2674 > ...

3 5

help fully fixing generators...?
by Alex Martelli Dec. 22, 2003

Dec. 22, 2003

enumerate currently does not work -- and it does not work because generators (which enumerate, implemented at app level, uses) don't behave correctly when a StopIteration is raised -- they propagate it (propagate the OperationError, at interpreter level) rather than turning it into a "clean return from the generator" (a SGeneratorReturn, at interpreter level). I easily fixed it for a simple "raise StopIteration" in the generator itself, and committed that change (as well as the unit-test showing the bug existed), but I'm not sure where to edit to fix it when the StopIteration is being PROPAGATED from a call to something.next() in the generator, which is what enumerate needs. I haven't checked in a unit test showing the problem, but, basically, the issue is something like: def agen(): it = iter('ciao') while 1: yield it.next() print list(agen()) in CPython this works fine, but in pypy it currently doesn't (yet). Suggestions, anybody? Alex

2 2