Mailman 3 November 2005 - pypy-dev

Pascal backend
by Sanghyeon Seo 02 Dec '05

02 Dec '05

I started Pascal backend for a while now. Have a look at: http://sparcs.kaist.ac.kr/~tinuviel/pypy/ Resulting Pascal source can be compiled with FPC(Free Pascal Compiler), available from http://www.freepascal.org/ What do you think? Also, if I understood correctly, translation part is undergoing much change in the branch. What do I need to change? Seo Sanghyeon

2 1

Full Python Annotation
by Adam Olsen 01 Dec '05

01 Dec '05

I'm a bit disorganized here, but I don't think delaying will help much as my brain is turning to mush. Nothing ventured... Notes for description of value annotator Iterative. Allows examination of individual steps Stable. Produces result in finite amount of time Order Insensitive. Always produces the same result, no matter what order in which it processes the sourcecode Specialization of builtins. All functions are given a "space depth". Multiple calls (at different lines of code) to the same function will, if it has a deeper depth, create separate copies of the function in their own "space". If those functions then call a function of a shallower depth they will use a shared function (rather than continuing to specialize it). The result is that separate calls to builtin functions (such as the plus operator) will get separate annotations, but without risk of an infinite loop in the annotation process. Values. A Value is a set (in the mathematical sense) of possible values, be they integer, float, string, or whatever. In practice it must store an approximation so that a set of 0..2**64-1 can be represented. It is acceptable for the set to be expanded to include values that are not really possible (as the program will still behave correctly), but not acceptable to be smaller. Note however that as an iterative process is used to determine values, it will initially be to small, and later replaced with new Values that cover the full range. Hypothetical value expansions. A mechanism is needed to expand values to their full range quickly, rather than iterate it one step at a time. To do this an integer_add will notice on the second pass (not the first!) that it had a previous possible output, and that the new output appears to create a sequence from it. A new value set is created as normal (increased by only a single step), except a "hypothetical" part is added to it. The hypothetical contains a Value set with the predicted sequence of values (not including any already known), as well as a reference to the node/space that created it. The value is then set as the new output for the integer_add node and is propagated through the rest of the code. Other nodes will modify the hypothetical as if it was a normal value set, including putting a cap on the the highest value in the case of an if branch, but they will not explore new code as a result of it. If the hypothetical reaches the node/space that created it then this indicates there is a loop which allows the value to expand over the whole range, and the node will convert it into a normal value (including any caps that were discovered). Branches and bool replacement Values. To effect a cap on the range of a value the bool value produced by lessthan and similar operations will contain a pair reduced Value associated with the input variables. If the bool is used for a branch then either the "True" or "False" version of the reduced Value associated with the bool's original input variable will be used to replace the variable thereafter. Note that a form of Single Static Assignment is used to allow a variable to have different values at different points in a function. Comment on SSA and aliases. As the branch that reduces a variable's range may happen outside the function itself, it is necessary to make the SSA alias lists globally available and allow modification at any point. This means they must be copied to every node, with the resulting cost being quadratic (O(n**2)). However, as most of the copies would have identical contents I believe it would be possible to optimize this and get performance that is closer to linear (O(n)). Finally, a note on what sorts of code this is aimed at. Anything where all the code can be determined at compiletime, ie no eval/exec/__import__/reload. Some analysis of code that does use those features would still be possible however. Spaces diagram: http://members.shaw.ca/rhamph/python-annvalue/spaces.pdf Values diagram: http://members.shaw.ca/rhamph/python-annvalue/values.pdf Partially annotated test "program": http://members.shaw.ca/rhamph/python-annvalue/annotation.pdf My sourcecode (requires autopath.py): http://members.shaw.ca/rhamph/python-annvalue/basicblock.py -- Adam Olsen, aka Rhamphoryncus

2 1

Re: [pypy-svn] r20364 - in pypy/branch/somepbc-refactoring/pypy/translator/c: . test
by Samuele Pedroni 29 Nov '05

29 Nov '05

mwh(a)codespeak.net wrote: > Author: mwh > Date: Mon Nov 28 18:27:39 2005 > New Revision: 20364 > > Modified: > pypy/branch/somepbc-refactoring/pypy/translator/c/extfunc.py > pypy/branch/somepbc-refactoring/pypy/translator/c/genc.py > pypy/branch/somepbc-refactoring/pypy/translator/c/pyobj.py > pypy/branch/somepbc-refactoring/pypy/translator/c/test/test_annotated.py > pypy/branch/somepbc-refactoring/pypy/translator/c/wrapper.py > Log: > test_annotated passes again, though this is pretty much a 'hack-until-it- > passes' approach. > > I see, > Modified: pypy/branch/somepbc-refactoring/pypy/translator/c/test/test_annotated.py > ============================================================================== > --- pypy/branch/somepbc-refactoring/pypy/translator/c/test/test_annotated.py (original) > +++ pypy/branch/somepbc-refactoring/pypy/translator/c/test/test_annotated.py Mon Nov 28 18:27:39 2005 > @@ -1,7 +1,7 @@ > import autopath > import py, sys > from pypy.translator.tool.cbuild import skip_missing_compiler > -from pypy.translator.translator import Translator > +from pypy.translator.translator import TranslationContext > > from pypy.translator.test import snippet > > @@ -12,7 +12,7 @@ > class TestAnnotatedTestCase: > > def getcompiled(self, func): > - t = Translator(func, simplifying=True) > + t = TranslationContext(simplifying=True) > # builds starting-types from func_defs > argstypelist = [] > if func.func_defaults: > @@ -20,9 +20,15 @@ > if isinstance(spec, tuple): > spec = spec[0] # use the first type only for the tests > argstypelist.append(spec) > - a = t.annotate(argstypelist) > + a = t.buildannotator() > + a.build_types(func, argstypelist) > a.simplify() > - return skip_missing_compiler(t.ccompile) > + from pypy.translator.c import genc > + builder = genc.CExtModuleBuilder(t, func) > + builder.generate_source() > + skip_missing_compiler(builder.compile) > + builder.import_module() > + return builder.get_entry_point() > > def test_set_attr(self): > set_attr = self.getcompiled(snippet.set_attr) > I missed this subtle point, indeed before going on writing more code like this, we need to think where and how much of translator.Translator to keep working and where instead to switch using TranslationContext

2 1

automatic generation of report pdfs
by Carl Friedrich Bolz 27 Nov '05

27 Nov '05

Hi pypy-dev! I set up a script that automatically generates all the report-pdfs (in eu-tracking/deliverable) every ten minutes and puts them into my home directory: http://codespeak.net/~cfbolz/deliverable_pdf The way I do it is quite hackish at the moment and might change soon. I hope that I can at least put error files in the directory if the rest is not valid. Cheers, Carl Friedrich

1 0

[Fwd: Re: [pypy-dev] Re: [pypy-svn] r20092 - pypy/dist/pypy/rpython]
by Christian Tismer 27 Nov '05

27 Nov '05

[this message^did not appear on pypy-dev]

1 0

Re: [pypy-dev] Re: [pypy-svn] r20092 - pypy/dist/pypy/rpython
by Christian Tismer 27 Nov '05

27 Nov '05

Bengt Richter wrote: > At 12:55 2005-11-21 +0100, Armin Rigo wrote: ... >>That's a possibility. For PyPy we should be more explicit, though, and >>use a "class r_longlong" or "class r_int64". It should be part of a >>general design that would allow integers of various precisions. The >>design decision is to choose if we want classes with explicit sizes >>(r_int64) or C-like names (longlong)... > > > Please pardon my jumping in, but IMO a collection of _unambigous_ names like r_int32 > is in one sense just a small notch better than a collection of _ambiguous_ names > for various integer implementations (like int, long, etc.). Yes, I think at a certain primitive level I'd prefer to have an explicit way to spell the size of my numbers. I don't know yet if this should be done via names, or if there are better ways. For my application during the US trip for instance, I made a little data description language, a very tiny set of Python classes which allow to build record descriptions for data. I didn't use type names at all, but used notations like Char = StrDataDef("X", 1) Str16 = StrDataDef("some stuff", 16) Numeric = StrDataDef(123, 9) Price = StrDataDef(99999999999999, 14) Num1 = StrDataDef(9, 1) StrVar = StrDataDef("varying", 1, 255) The idea is to give an example value instead of a data type. See for instance the Price field. The maximum possible value is intentionally not written as a long. It just says how many digits I need. This will be a long in most cases, but on a 64 bit machine, it would be a simple integer, and my application would produce faster code. > If you want a "general design that would allow integers of various precisions," > perhaps one could think about some way of parameterizing the precision info (e.g., > exact or min bit widths) rather than just cleaning up the ambiguity in C's way > of defining an evolving series of increasingly ridiculous (what next, extralonglong?) > platform-dependent names. I.e., when would it be time to introduce r_int128 ? I would like to have two things: - a way to express what precision I need for something - a way to ask which precisions are native data types This is because depending on whether there is a native type or if it must be emulated, I might want to choose different implementations. From my work on the long integer implementation, I know that explicit types are quite error-prone and make the code less readable. The nicest thing I could imagine was a way to track the necessary bit sizes automatically in the annotator. But that's a difficult thing, because our current assumption is that overflows are ignored. In order to do explicit bit tracking, we would probably need an extra r_someint, which is handled specially to track the needed preceision? > If various integer subtypes are being generated, perhaps one could specify > minimum or exact bitwidths etc as parameters seen by a metaclass, like __SLOTS__ > only relevant to integer representation, or have an integer class factory function > that takes specs as parameters and raises an exception if the required class > can't be generated implementing the specs a particular platform. > > Just a thought. Interesting, too! We also could go further and produce an implementation for every small fixed bit size that is needed, even if it is not that efficient. This would be done in situations where there is no choice, like the 64 bit long file pointers. For things like the general long implementation, I would ask the systtem for the most efficient native types and build upon them. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

This Week in PyPy 4
by Michael Hudson 26 Nov '05

26 Nov '05

Introduction ============ This is the fourth of what will hopefully be many summaries of what's been going on in the world of PyPy in the last week. I'd still like to remind people that when something worth summarizing happens to recommend if for "This Week in PyPy" as mentioned on: http://codespeak.net/pypy/dist/pypy/doc/weekly/ where you can also find old summaries. There were about 50 commits to the pypy section of codespeak's repository since the last summary (not quite a week). SomePBC-refactoring =================== We attacked the RTyper quite a lot, which meant staring at some of the most obscure code in the codebase, and made substantial but incomplete progress (currently about 60% of the rtyper tests pass). We're optimistic that the majority of work is done on the branch, but there may be many strange details to cope with before translate_pypy runs again. Sprint Preparation ================== The next sprint is less than two weeks away -- it's definitely time to be buying flights and booking accomodation if you're going to be there :) LLVM progress ============= Richard implemented threading in the LLVM backend, bringing another feature that was previously pypy-c only in. Stacklessness next? PyPy spreads ============ Christian returned from America where he'd been consulting for a company implementing some systems in RPython which had been implemented in Java, and after some effort beating the Java versions for performance. This company had found out about PyPy and RPython from reading our mailing lists -- a nice example of how open development processes can work (and even make you money!). Resource consumption ==================== As part of being EU-funded, we have to keep track of the resources we use and have a slightly unusual problem: we haven't spent enough time or money in the first half of the project, and have to find something to do about this... PyPy at conferences =================== All three talks on PyPy that we submitted to PyCon were accepted, so there will be talks from - Michael and Christian on the current state of PyPy (whatever that may be at the time :), - Holger and Armin on the architecture and future of PyPy, and - Bea and Holger on the methodology of PyPy and the issues around being EU funded. Further to that, two papers were accepted for the Chaos Communication Congress in Berlin over the new year were accepted: http://events.ccc.de/congress/2005/fahrplan/events/585.en.html http://events.ccc.de/congress/2005/fahrplan/events/586.en.html Again, one talk is on the technology of PyPy and the other on methodology/business issues. So if you're going to a Python or hacker conference any time soon, you're likely to hear about PyPy :) -- Any form of evilness that can be detected without *too* much effort is worth it... I have no idea what kind of evil we're looking for here or how to detect is, so I can't answer yes or no. -- Guido Van Rossum, python-dev

1 0

[lalo.martins@gmail.com: Re: Document drafts]
by Armin Rigo 25 Nov '05

25 Nov '05

Hi, Forwarded for Lalo, who cannot seem to post to pypy-dev, answering the following: > Hi all, > > People following only this mailing list may have missed some draft > documents that we have been working on, as they are not linked to yet: > > http://codespeak.net/pypy/dist/pypy/doc/draft-dynamic-language-translation.… > http://codespeak.net/pypy/dist/pypy/doc/draft-low-level-encapsulation.html ----- Forwarded message from Lalo Martins <lalo.martins(a)gmail.com> ----- I'm quite impressed with them, technically. The only suggestions I have about them are non-technical. I find that many sentences are too long, without break. It's a vice I have myself, as a non-native writer. Native readers often tell me, this makes my text too heavy, too hard to parse. Of course, the primary target audience (in academia) will have higher tolerance to that, but it's still somewhat bad style. I'd sprinkle commas generously, at least. Try reading them out loud :-) (the sentences, not the commas) best, Lalo Martins -- So many of our dreams at first seem impossible, then they seem improbable, and then, when we summon the will, they soon become inevitable. -- http://www.laranja.org/ mailto:lalo.martins@gmail.com GNU: never give up freedom http://www.gnu.org/ ----- End forwarded message -----

1 0

pypy-sync
by Adrien Di Mascio 24 Nov '05

24 Nov '05

Hi there, I won't be able to attend the pypy-sync meeting today. I did nothing the last week for PyPy, except preparing our trip to Gothenburg. Next week, I will probably have a look at some Logic-related stuff. Cheers, Adrien.

1 0

This week in PyPy 3
by Michael Hudson 22 Nov '05

22 Nov '05

Better late than never... ======================= This Week in PyPy 3 ======================= Introduction ============ This is the third of what will hopefully be many summaries of what's been going on in the world of PyPy in the last week. I'd still like to remind people that when something worth summarizing happens to recommend if for "This Week in PyPy" as mentioned on: http://codespeak.net/pypy/dist/pypy/doc/weekly/ where you can also find old summaries. There were about 60 commits to the pypy section of codespeak's repository this week. SomePBC-refactoring =================== Work on the branch continued, to the point that the annotator now works but the scary mess of the RTyper still remains. We're still pleased with the ideas behind the branch -- the new annotator code has a good deal fewer hacks than the old (though it still has quite a few, of course). Backend progress ================ There was a fair bit of light refactoring on the LLVM backend this week, including a recommendation to upgrade to the newly released LLVM 1.6. This gives slightly better performance, meaning that a new pypy-llvm is the closest to CPython performance we've gotten yet (still about 8 times slower, mind). The main change is that the list of operations that can raise exceptions is now shared with the genc backend, reducing duplication and maintence overhead. Basically this means that the LLVM backend is and should remain compatible with the default pypy-c build (no threads, only the Boehm GC and no stackless features). There was also progress on the JavaScript backend, mainly focussed on adding some of the stackless features currently sported by the C backend. Cheers, mwh -- > It might get my attention if you'd spin around in your chair, > spoke in tongues, and puked jets of green goblin goo. I can arrange for this. ;-) -- Barry Warsaw & Fred Drake

1 0

pypy-dev November 2005