Mailman 3 November 2004 - pypy-dev

SHA 256, 384, 512: Index of ftp://ftp.philosophysw.com/pub/software/
by Christian Tismer Nov. 19, 2004

Nov. 19, 2004

Hi Jacob, Laura, while you are at it... I'd really love to have these as well. Just in case you want to do more hashing stuff :-) <ftp://ftp.philosophysw.com/pub/software/> ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

autopath obsolete?
by Christian Tismer Nov. 19, 2004

Nov. 19, 2004

Hi colleagues, I was wondering why we still need stuff like autopath? I simply use a pypy.pth file in my standard python folder which contains the path to pypy, and things work fine. Why do we really need autopath? -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

5 7

Re: [pypy-svn] r7364 - in pypy/trunk/src/pypy: appspace appspace/test module/test
by hpk＠trillke.net Nov. 18, 2004

Nov. 18, 2004

[arigo(a)codespeak.net Thu, Nov 18, 2004 at 12:07:01PM +0100] > Author: arigo > Date: Thu Nov 18 12:07:01 2004 > New Revision: 7364 > > [... wrestling commits with appspace-modules ... ] > > Directly testing the class 'complex' buried in __builtin__module.py is funny, > beacuse it's difficult to import this __builtin__module.py directly into > CPython. Well, it's not impossible, given sufficiently obscure hacks. Yadda > yadda boum. Committing applevel_in_cpython.py for that purpose, i.e.: testing > directly in CPython some of the app-level code written in the > module/*module.py files. > > That's obviously a wrong solution. We need something more like splitting the > module/*module.py into several files (e.g. class complex:...) and most of them > could just be imported in CPython too. Such modules should probably be allowed to be a directory which can have multiple files, all of which will be put into module's namespace. This should also make it easier to test them from CPython ... holger

1 0

Re: [pypy-dev] explicit documentation (re)organization
by Laura Creighton Nov. 8, 2004

Nov. 8, 2004

In a message of Sun, 07 Nov 2004 12:03:19 +0100, holger krekel writes: >Hi Laura, >I don't think that the "pypy documentation for and from >developers" should primarily strive to have this side-function >of preserving bits and pieces of formatting or other meta-info >that might be helpful to particular users. > >IMHO the pypy documentation serves two purposes: > >- letting the developers easily organize up-to-date > documentation, by integrating hacking at ReST files > into the coding process as seemless as possible We share this goal. > >- letting others easily see this current view of the developers > regarding the implementation and architecture of PyPy But we don't share this one. I want us to write documentation primarily for us. I think that there can be a place for 'documentation for others, and introductory material' but that is not, primarily, what I am interested in doing when I am writing documentation. I write documentation first for me, and then for the people I work with, and last and by far the least for other interested parties. This changes, of course, when you are writing an 'Introductory Manual for New Users' or some such, where the explicit goal of the work is to communicate with new users. I am not opposed to the writing of such things, but I don't see it as having much to do with the ongoing documentation of a project, except coincidentally. The ongoing documentation that we write for ourselves is constantly becoming obsolete as what we are doing differs from what we thought we would do when we wrote something. There is nothing wrong with this. Efforts to keep what we wrote in sync with what we actually are doing, in my experience have always resulted in people documenting less and less and less. I don't want the responsibility of keeping what I wrote current with what we are doing. If I am stuck with this, I will document to the minimum extent possible. >That being said i definitely see the point of wanting to >look at an earlier revision of a file, especially if it >has been deleted from the repo. To date there is no nice >tool i know off that provides you a complete search functionality >over ancestor revisions (not even trac at the moment although it >goes some way). Now you know why I wanted to write a CAPS system at Strakt. :-) Some day real soon now, it will even do this properly. >So we might move documents into some 'attic' instead of deleting them. >But note that most often files will be modified and not deleted. >And we certainly don't want to keep all older revisions of modified >files in some 'attic' folder as this would lead to a plethora of files >and would undermine the whole point of a version control system. Yes. Outlining the cases when you want to undermine the whole point of a version control system is precisely what I am trying to do. When documents change a lot over their lifetime, there is often a case to be made for having different versions of the document (mar 2000, oct 2000, dec 2001) and the like available, as reference documents. Deciding which documents deserve such treatment is an art, more than a science. Without such a history, you end up presenting your current ideas without the context of how they evolved to what they are at present. In some cases, this 'dis-in-history'-ing is the actual goal, and what you are trying to do. But in other cases, it is the historical development of the roadmap, that is of interest, not wherever the arrow 'We are Here' currently points. They are contradictory goals, and extremely hard to realise at the same time. What actually happens, wehn documenting, in my experience, is that somebody writes a document, and it gets modidied quite a bit while it is the focus of some work that is being hacked on right now, or will be hacked on in the immediate future. Then we finish hacking that, and move on to other goals and make other documents. Time passes. The code that the original doc referred to gets refactored and modified in the light of new ideas we have in the context of our new goals. The original doc does not get touched, because it is no longer interesting in the context of where development is happening now, or next week, or next month. Eventually somebody decides that the document is so obsolete that they delete it. > >Oh, and btw, if you say "i know that at EuroPython 2004 there was >this nice ReST-thingie i did" you can issue > > svn up -r "06-10-2004" > >in the doc directory which gets you right back to the complete >view at that time. Taking this single-step indirection seems >reasonable enough to me for most cases. Not for me. I am not looking for something to read, but something to scan and to search. Which means I want it as a webpage I can glance at, and that Google can index. Also, I don't know about you, but when you remember a phrase from a book or a webpage, don't you remember stuff like 'about 2/3rds of the way through the book, on a right-facing page, second to last sentence before a new paragraph' ? People who remember information independently of their presentation medium have different searching and retreival needs than those who remember their information with lots of embedded context. It's a neat hard problem in cognative science and human-computer interfaces. > >cheers, > > holger take care, Laura >_______________________________________________ >pypy-dev(a)codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev

4 3

More on optimization
by Armin Rigo Nov. 7, 2004

Nov. 7, 2004

Hi! Sorry for focusing the next sprint so much on translation. This might have put some people off. Well, lesson learned. It doesn't mean we should stop talking about translation :-) Pushing the previously dicussed ideas to their conclusion, we get an interesting point of view... First, for the context, the current situation in the repository: RPython code can be turned into a flow graph. Then the annotation pass on the flow graph infers types, with the side effect that it infers which functions call which functions, so it helps to build the complete list of functions that should be translated. Finally, genc.py produces naive C code using PyObject* only. The inferred types are not really used; the annotation phase is currently useful only to build the complete call graph. (Let's ignore the Lisp and Pyrex generators for the moment.) Here is an example with the flow graph's operations on the left and the corresponding C code (after macro expansion) on the right: v3 = add(v1, v2) v3 = PyNumber_Add(v1, v2); Quite obvious, and quite slow. Imagine instead that there is a C-ish low-level language, whose input "syntax" is of course flow graphs, with only operations like the following ones: * llcall(func, arg1, arg2...) # calls another function * lladd_int(x, y) # adds two ints and returns an int * lleq_ptr(x, y) # compares two pointers for equality * llstruct('name') # creates an empty structure in the heap * llget_int(x, 'key') # reads the field 'key' of struct object x * llset_int(x, 'key', value) # writes the field 'key' of struct object x * llget_ptr(x, 'key') # \ * llget_ptr(x, 'key', value) # / the same with fields containing a pointer The only data types would be "pointer to structure" and the atomic types like "int" and "char". A structure would be essentially like a dictionary, with either strings or integers as keys (a dict with int keys is much like a list, without the ability to insert or remove elements easily). This very limited set of operations allows interesting global analysis and optimizations, like removing the allocation of structures in the heap altogether, or not writing some fields of some structures when they have a known constant value, or (most commonly) not using a hash table but just a 'C struct' with fields when all the keys are known. It is easy to modify our regular flow graphs until they only use the above operations: we replace high-level operations like 'add' with calls to (or inlined copies of) support functions like: def rpyhon_add(v, w): if type(v) is int and type(w) is int: a = llget_int(v, 'ob_ival') b = llget_int(w, 'ob_ival') x = llstruct() llset(x, 'ob_type', int) llset(x, 'ob_ival', lladd_int(a, b)) return x else: return llcall('PyNumber_Add', v, w) Only such support functions can use the ll*() functions, which stand directly for the corresponding ll* operation. The ll*() functions themselves are not executable in the above Python source (actually they don't have to be callable objects at all). They are only meant as placeholders. If you wish, it's just an easy way to write down a complicated flow graph using ll* operations: instead of inventing a syntax and writing a parser for it, we use the Python syntax and the flowobjspace to build our flow graphs. Note that the type() used above is a function defined as: def type(v): return llget_ptr(v, 'ob_type') (Actually, we might want to directly execute such code, for testing, with Python dictionaries in most variables and (say) llget_ptr() defined as dict.__getitem__(), but that's not the main goal here.) Let's come back to PyPy. We can now replace each high-level operation with a call to a support function like the above one. Initially these support functions could contain only the llcall() to the CPython API; then we can progressively add more cases to cover all the RPython behavior. The goal would be that eventually the support functions and the optimizer become good enough to remove all calls to the CPython API. The optimizer can do that using global analysis: e.g. in the above rpython_add(), if both 'v' and 'w' come from a previous llstruct() where 'ob_type' was set to 'int', then we know that the first branch is taken. This is a kind of generalized type inference using global constant propagation; it would superscede half of the annotation pass' job. Also, the incremental nature of this approach is nice (and very Psyco-like, btw). It is good for testing along the way, and also means that when we reach the goal of supporting enough for RPython, the result is still nicely useful for non-RPython code; in this case, when types cannot be inferred, there are remaining calls to the CPython API. I think that this would be a widely useful tool! I also like a lot the idea of a low-level language and optimizer which is entierely independent from Python and PyPy. We could even (still) devise an assembly-like syntax and a parser for it, and make it a separate release. Such an optimizer is obviously not easy to write but I think it's quite worth the effort. Most of it would be independent on the target language: it would e.g. replace operations with still more precise operations describing the usage of the structs, like llheapalloc('C struct name') llheapget_int(x, 'field', 'C struct name') Also, we should look closely for related work. It looks like it *has to* have been done somewhere before... But the kind of global optimizations that we need might be different from the ones any reasonable language would need, because no one would reasonably write code like the rpython_add() above when he actually means to do a simple integer addition :-) What's essential here, after constant propagation, is to do liveness and alias analysis for the heap structures, so that they can be not allocated in the heap at all as often as possible. Also, if we want to do explicit memory management (i.e. we would have to use a 'llfree' operation) with reference counters, then the above example of rpython_add() becomes even longer -- but of course it should still be optimized away. A bientôt, Armin.

4 18

Re: [pypy-dev] explicit documentation (re)organization
by Laura Creighton Nov. 7, 2004

Nov. 7, 2004

In a message of Sat, 06 Nov 2004 12:34:31 +0100, holger krekel writes: >Hi Laura, > >[Laura Creighton Sat, Nov 06, 2004 at 12:19:04PM +0100] >> I would like all the documents produced to cotinue to be under svn sour >ce code >> control. > >sure, using subversion for holding the basic documentation is certainly >not going to go away :-) > >> And I really do not want obsolete things deleted, only moved. At >> some point, one of us is likely to want to do a paper on the developmen >t >> process, and it will be very useful to have things that document what >> we used to think, and when we thought it for that purpose. Also, the f >act >> that we have decided against a certain approach does not make it theref >ore >> uninteresting. I'd like to be able to go back and think about roads no >t >> taken at some point. > >Yes, I see the point. However, when we delete things in a certain revisi >on >they don't go away from the repository. It is always possible to look a >t >former revisions and see what happened. I completly agree that this >is something we want to have. Also Jum and me are working on the >new codespeak setup which will feature a "trac" environment which >allows to search for older commits and files ... The trac sounds interesting. Requiring that I look things up on the repository might be ok for documents that are current, but have changes in them, but I think is inappropriate for entire documents that are obsolete. We've run into this problem before. I've written something, and it becomes obsolete. Somewhere along the line, in an effort to not confuse people with old stuff, you take the what seems reasonable step of removing this. What you don't know is that that particular document was the one and only time I had ever managed to make ReST do <something>. Or I thought up a neat way to express something, or I had to look up the spelling of a word. All in all, this means that the document continues to have present and lasting value to me beyond its value to you -- or anybody else. But I don't notice that it is gone until the next time I need something from it, and then I expect to be able to find it by clicking on something. So what I would prefer is a quasi-mirror of the pypy site, with a header of 'obsolete'. If it isn't in the current, then I can find old docs in the same location as they would be if they were still current. I need to be able to do 'click-click-click -- aha, there is the table I was looking for --' and then get on with what I want. <snip> Laura ps - updated the page, and bonus hero points for working over one weekend.

3 4

Re: [pypy-dev] explicit documentation (re)organization
by Laura Creighton Nov. 7, 2004

Nov. 7, 2004

In a message of Sun, 07 Nov 2004 10:16:59 +0100, Anna Martelli Ravenscroft writes: >Laura and Holger - >is there a way I can help with this? I miss working with you guys, and >this seems like somewhere that I could be useful, whether or not I fully >comprehend the ins and outs of where the pypy project is at the moment. > >Please let me know - offlist if you like. > >Anna I miss working with you too. But I leave tomorrow to host a anti-sw-pat conference at EU Parl in Brussels. Then I fly to Santa Cruz, USA, and then I fly directly to the Vilnius Sprint. So it will be a while before I can continue this discussion, by which time I expect something to already happen. Laura

1 0

explicit documentation (re)organization
by hpk＠trillke.net Nov. 6, 2004

Nov. 6, 2004

hi folks, i may not be alone in thinking that the current documentation organization in PyPy leaves an lot to be desired. IMO the main problem: we don't have a clear and maintained entry point for documentation for developers (tm) and we don't cleanly link all the _relevant_ and somewhat up-to-date documentation. Also we link to a lot of half-way/not decent/wrong documentation. So i plan to move towards i model that is already used with the upcoming py lib (http://codespeak.net/py ): - there would be a directory which holds an "index.txt" which will be ResT-translated in order to serve as the main developer documentation entry point - this index.txt will link to the various important other documentation chapters _explicitely_ - linking to the chapters happens with relative links so that you can rest-convert and check links before checking in. With the py lib there even is a test that checks that the documentation does not produce warnings. Currently, in PyPy we are using an implicit scheme which basically rest-converts everything that was dropped into some directories. IMO the result is confusing, unordered and didn't really work out. I believe that getting the documentation in a better shape is crucial to involve new developers (and even the current ones :-) Comments? I'd like to do the reorganization over the weekend and would afterwards appreciate any help in bringing the documentation up to date, especially deleting or moving all the obsolete stuff. This is real work but it really makes a difference so please help ... cheers, holger

3 2