Mailman 3 July 2004 - pypy-dev

iron python was released
by holger krekel Aug. 2, 2004

Aug. 2, 2004

Hi pypy-dev, just in case you missed it (i know i did so far), IronPython has been released: http://www.ironpython.com/ on 28th of July. It is an (incomplete) port of Python to the Common Lanuage Runtime (CLR) of Microsoft's .NET. And the author, Jim Hugunin, will begin to work for Microsoft on 2nd of August (read more about it on the web site above). If anyone has tried it out already feel free to share your impressions. So far, I only skimmed the source code (c#) a bit because it doesn't seem to be easily runnable on linux. cheers, holger

4 3

Re: [pypy-dev] iron python was released
by holger krekel July 31, 2004

July 31, 2004

[Bob Ippolito Fri, Jul 30, 2004 at 06:26:28PM -0400] > > On Jul 30, 2004, at 11:24 AM, holger krekel wrote: > > >just in case you missed it (i know i did so far), IronPython > >has been released: http://www.ironpython.com/ on 28th of > >July. It is an (incomplete) port of Python to the Common > >Lanuage Runtime (CLR) of Microsoft's .NET. And the author, Jim > >Hugunin, will begin to work for Microsoft on 2nd of August > >(read more about it on the web site above). > > > >If anyone has tried it out already feel free to share your > >impressions. So far, I only skimmed the source code (c#) a > >bit because it doesn't seem to be easily runnable on linux. > > Well it's easily runnable on OS X if you have mono installed.. so I > would have to imagine it's easier on Linux. yip, it does run with mono 1.0 on linux. However, as Robin mentioned, the Console easily gives errors and segmentation faults, even. I first thought that this indicated that it wasn't running properly at all. But running some scripts and pystone works. has anyone actually re-compiled the c# files into an executable on linux? I just used "mint bin/IronPythonConsole.exe". cheers, holger

1 0

Ann: Revival of the BytecodeHacks
by Christian Tismer July 30, 2004

July 30, 2004

Dear community, I could not resist to do this announcement, although this project belongs to Michael Hudson. Mike, please forgive me. I owe you a beer. In a single day-and-night session, I hacked against bytecodehacks, to upgrade it from Python 1.5.2 to Python 2.3 . This was quite some work, although not so very much, due to the excellent basic layout which Michael created. What are the Bytecodehacks? --------------------------- The bytecodehacks allow you to do certain modifications to compiled Python code objects. There are lots of applications included, like macro expansions and function inlining, things which Python does not provide and also is not supposed to provide. This kind of madness exists for people who ask for it, without worring those who don't care. Again, this great package was written by Michael Hudson in early 2000, and to my knowledge, it was never ported to the more recent Python versions. Michael told me that this is a package he is no longer too fond of, since it was written in his early days. He also told me that he is not keen on supporting it so much, because he would be tempted to give it a whole rewrite. Now, I'm thinking differently. I got the package to work, after about 12 hours of hacking, and it simply works. Since I didn't write the initial version, I have a different relationship to it. In other words: It is easier to maintain a foreign package than your own, since you are not married with it. Why do I dig into foreign areas? -------------------------------- Well, I have enough work with my Stackless package. Stackless is almost ready. (Almost, like your toy railway gets really ready; it really never will.) I just built a minimum Psyco support into it, because I'm basically always after as much speed as I can get. But there are limits with the regular Python interpreter. So my idea is to use these crazy other projects to get more performance, and to support them directly. My first idea to accelerate Psyco using Stackless was to provide Stackless with extra hardware stacks, which can be switched at light-speed. I still have this idea in mind, but the implementation is not so trivial. Comparatively, replacing generators (yield calls) with a couple of save/restores of tuples *is* almost trivial, as I'm probably going to show tomorrow. In Python, these "fake-generators" would be reasonably slower. But, by the fact that these are then Psyco-enabled, makes them really, really fast, and also completely inlineable. I think to name the module "renegate". :-) Why do I want to revive this package ------------------------------------ Well, I am a pragmatic guy, and I have a really good reason why I need the bytecodehacks. I am writing a sophisticated package which involves parsing of PDF files, and I want to do it all in Python. In order to get this PDF processor to almost C speed, I used Armin Rigo's wonderful Psyco package. Unfortunately, Psyco has a few limitations, which act as a show-stopper: - generators are not supported. That means, whenever I use a generator, Psyco will not accelerate it, but will act as a small slow-down. - Psyco is great at optimizing simple structures like lists, tuples, numbers and strings. It is less able to enhance things like object properties. Using self frequently disables almost all of Psyco's capabilities. - Psyco has difficulties with inlining. Simple functions *are* inlined, but when they contain a conditional branch or they exceed some limit, inlining is disabled. This *could* be changed, but with a lot of effort by changing C code. This is not going to happen, because all of this stuff will be enhanced and re-implemented during the PyPy project. Now, by combining the re-animated bytecodehacks project with Psyco, I am almost sure that I can remove certain restrictions from Psyco, by turning problematic Python structures into simple ones, which the current Psyco can handle natively. Poor man's PyPy --------------- Although I am a member of the PyPy project, and I do belong to the people who initiated the PyPy project, I am impatient, and I want to get a few of the expected PyPy results right now. Psyco is phantastic but not perfect, and it needs some help to gather maximum performance. By adding Bytecodehacks in the right manner, I think I can fill this gap. With BCH, I can replace generators by ordinary methods of a class (plus a few bytecode instructions which have no real Python equivalent, like goto). By inspecting the data flow of a self.attribute, I can prove that it is invisible outside and replace it by a simple local variable in many cases. By using Bytecodehacks for proper inlining of functions, I can deliver Psyco from this difficult task. The expected result ------------------- By consequently applying the methods I sketched above, I expect that I can make almost every existing application reasonably faster. I will provide this as a service for customers and charge them for relevant acceleration. The software will stay open-sourced. This is just a few add-ons to Psyco and Bytecodehacks, and I'm not the author of these. I just found out how nicely they can fit together. My guess is an overall acceleration of at least a factor of five for almost any native Python application. There is no proof yet, this is all Vodoo from my stomach. But this stomach tends to be quite reliable Mike, please forgive me this announcement. You should have written it, but I was so very inspired. Getting the bytecodehacks for Python 2.3 ---------------------------------------- The current source code is available at cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/bytecodehacks login cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/bytecodehacks co bytecodehacks cheers -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

Ann: Revival of the BytecodeHacks
by Christian Tismer July 30, 2004

July 30, 2004

Dear community, I could not resist to do this announcement, although this project belongs to Michael Hudson. Mike, please forgive me. I owe you a beer. In a single day-and-night session, I hacked against bytecodehacks, to upgrade it from Python 1.5.2 to Python 2.3 . This was quite some work, although not so very much, due to the excellent basic layout which Michael created. What are the Bytecodehacks? --------------------------- The bytecodehacks allow you to do certain modifications to compiled Python code objects. There are lots of applications included, like macro expansions and function inlining, things which Python does not provide and also is not supposed to provide. This kind of madness exists for people who ask for it, without worring those who don't care. Again, this great package was written by Michael Hudson in early 2000, and to my knowledge, it was never ported to the more recent Python versions. Michael told me that this is a package he is no longer too fond of, since it was written in his early days. He also told me that he is not keen on supporting it so much, because he would be tempted to give it a whole rewrite. Now, I'm thinking differently. I got the package to work, after about 12 hours of hacking, and it simply works. Since I didn't write the initial version, I have a different relationship to it. In other words: It is easier to maintain a foreign package than your own, since you are not married with it. Why do I dig into foreign areas? -------------------------------- Well, I have enough work with my Stackless package. Stackless is almost ready. (Almost, like your toy railway gets really ready; it really never will.) I just built a minimum Psyco support into it, because I'm basically always after as much speed as I can get. But there are limits with the regular Python interpreter. So my idea is to use these crazy other projects to get more performance, and to support them directly. My first idea to accelerate Psyco using Stackless was to provide Stackless with extra hardware stacks, which can be switched at light-speed. I still have this idea in mind, but the implementation is not so trivial. Comparatively, replacing generators (yield calls) with a couple of save/restores of tuples *is* almost trivial, as I'm probably going to show tomorrow. In Python, these "fake-generators" would be reasonably slower. But, by the fact that these are then Psyco-enabled, makes them really, really fast, and also completely inlineable. I think to name the module "renegate". :-) Why do I want to revive this package ------------------------------------ Well, I am a pragmatic guy, and I have a really good reason why I need the bytecodehacks. I am writing a sophisticated package which involves parsing of PDF files, and I want to do it all in Python. In order to get this PDF processor to almost C speed, I used Armin Rigo's wonderful Psyco package. Unfortunately, Psyco has a few limitations, which act as a show-stopper: - generators are not supported. That means, whenever I use a generator, Psyco will not accelerate it, but will act as a small slow-down. - Psyco is great at optimizing simple structures like lists, tuples, numbers and strings. It is less able to enhance things like object properties. Using self frequently disables almost all of Psyco's capabilities. - Psyco has difficulties with inlining. Simple functions *are* inlined, but when they contain a conditional branch or they exceed some limit, inlining is disabled. This *could* be changed, but with a lot of effort by changing C code. This is not going to happen, because all of this stuff will be enhanced and re-implemented during the PyPy project. Now, by combining the re-animated bytecodehacks project with Psyco, I am almost sure that I can remove certain restrictions from Psyco, by turning problematic Python structures into simple ones, which the current Psyco can handle natively. Poor man's PyPy --------------- Although I am a member of the PyPy project, and I do belong to the people who initiated the PyPy project, I am impatient, and I want to get a few of the expected PyPy results right now. Psyco is phantastic but not perfect, and it needs some help to gather maximum performance. By adding Bytecodehacks in the right manner, I think I can fill this gap. With BCH, I can replace generators by ordinary methods of a class (plus a few bytecode instructions which have no real Python equivalent, like goto). By inspecting the data flow of a self.attribute, I can prove that it is invisible outside and replace it by a simple local variable in many cases. By using Bytecodehacks for proper inlining of functions, I can deliver Psyco from this difficult task. The expected result ------------------- By consequently applying the methods I sketched above, I expect that I can make almost every existing application reasonably faster. I will provide this as a service for customers and charge them for relevant acceleration. The software will stay open-sourced. This is just a few add-ons to Psyco and Bytecodehacks, and I'm not the author of these. I just found out how nicely they can fit together. My guess is an overall acceleration of at least a factor of five for almost any native Python application. There is no proof yet, this is all Vodoo from my stomach. But this stomach tends to be quite reliable Mike, please forgive me this announcement. You should have written it, but I was so very inspired. Getting the bytecodehacks for Python 2.3 ---------------------------------------- The current source code is available at cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/bytecodehacks login cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/bytecodehacks co bytecodehacks cheers -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

Interesting project currently underway: libjit
by VanL July 29, 2004

July 29, 2004

I just saw this, and thought it might be of interest to this group: [From http://www.dotgnu.org/] The libjit <http://www.southern-storm.com.au/libjit.html> library implements Just-In-Time compilation functionality. Unlike other JIT's, this one is designed to be independent of any particular virtual machine bytecode format or language. The hope is that Free Software projects can get a leg-up on proprietary VM vendors by using this library rather than spending large amounts of time writing their own JIT from scratch. This JIT is also designed to be portable to multiple architectures. If you run libjit on a machine for which a native code generator is not yet available, then libjit will fall back to interpreting the code. This way, you don't need to write your own interpreter for your bytecode format if you don't want to. [end] VanL

1 0

Re: [pypy-svn] r5649 - pypy/trunk/src/pypy (fwd)
by Richard Emslie July 26, 2004

July 26, 2004

Sorry, really ought to learn how to use email! Obviously none of the intended parties got this. :-( ---------- Forwarded message ---------- Date: Sat, 24 Jul 2004 19:39:46 +0100 (BST) From: Richard Emslie <rxe(a)ukshells.co.uk> To: mwh(a)codespeak.net Cc: pypy-svn(a)codespeak.net Subject: Re: [pypy-svn] r5649 - pypy/trunk/src/pypy Hi Michael! On Sat, 24 Jul 2004 mwh(a)codespeak.net wrote: > Author: mwh > Date: Sat Jul 24 18:39:01 2004 > New Revision: 5649 > > Modified: > pypy/trunk/src/pypy/TODO > Log: > Random updates. > > In my working tree I've commented out the code that prints interpreter- > level tracebacks and have yet to miss them. Should I check this in? Think I got rid of also in traceinteractive.py... if that counts for my vote ;-) > > > Modified: pypy/trunk/src/pypy/TODO > ============================================================================== > --- pypy/trunk/src/pypy/TODO (original) [schnip] > > * improve traceobjectspace! currently it only catches > calls from outside the space into the space but not > the ones where space operations involve calling more space > - operations (the latter are not traced)! > + operations (the latter are not traced)! (fixed?) AFAIK yes! I've got some changes to checkin still (testit.py was broken and changing verbosity from traceinteractive.py) Should we combine interactive.py and traceinteractive.py ? (Since trace one was a blatant copy and paste hack :-) - also a TODO I believe). We could make interpreter level tracebacks a verbosity option? Cheers, Richard

2 1

Re: [pypy-svn] r5628 - pypy/trunk/src/pypy/objspace/std
by Michael Hudson July 24, 2004

July 24, 2004

mwh(a)codespeak.net writes: > Author: mwh > Date: Fri Jul 23 12:31:52 2004 > New Revision: 5628 > > Modified: > pypy/trunk/src/pypy/objspace/std/stringobject.py > Log: > the reason faked types shouldn't be specific to an instance: > turn on string to unicode delegation I'd like to change this message to the reason faked types shouldn't be specific to an instance: turn on string to unicode delegation also, kill some dead code and use what appears to be the modern way to detect mappings (hasattr(thing, "keys"), believe it or not) but "svn pe --revprop -r 5628 svn:log" didn't work. Ideas? Cheers, mwh

2 2

pypy vs parrotbench
by Michael Hudson July 23, 2004

July 23, 2004

I've tried running the parrotbench code with pypy and (eventually :-) got this far: b0 -- after 52 minutes: OverflowError: long int too large to convert to int b1 -- infinite recursion b2 -- passes b3 -- failed, after an hour with "(application-level) AttributeError: __cmp__" b4 -- ENO__subclasses__ b5 -- b6 -- Of the errors, I've only made the faintest effort to understand the b3 one, and only got confused there :-/ (quiz: what is object.__cmp__? not sure this is relavent, but it might be). I might try b5 & b6 tonight. Cheers, mwh -- Never meddle in the affairs of NT. It is slow to boot and quick to crash. -- Stephen Harris -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html

2 2

Re: [pypy-svn] r5606 - in pypy/trunk/src/pypy: module objspace/std
by Michael Hudson July 22, 2004

July 22, 2004

benyoung(a)codespeak.net writes: > Added some more of the the __r...__ functions to complex. Made the > objectsapce return complex object rather than cpython wrap them. I > think this is the correct behaviour. I don't understand how you managed to check this in... for one thing, I'd already done it, and for another, cpythonobject is dead on the trunk! Cheers, mwh -- I also fondly recall Paris because that's where I learned to debug Zetalisp while drunk. -- Olin Shivers

3 5

Re: [pypy-svn] r5369 - in pypy/trunk/src/pypy: scripttest scripttest/test tool
by holger krekel July 15, 2004

July 15, 2004

Hi Rocco, [rocco(a)codespeak.net Sun, Jun 27, 2004 at 10:53:28PM +0200] > Author: rocco > Date: Sun Jun 27 22:53:28 2004 > New Revision: 5369 > > Added: > pypy/trunk/src/pypy/scripttest/ (props changed) > ... > Log: > * tool/app_level_diff.py > > I hacked together a little tool for running a script at app level, and checking > its output against an expected output file. (Sort of like regrtest.py) I added > a feature where the files can be sectioned off into multiple parts, to reduce > disk clutter. > > * scripttest directory I guess this at least should not live directly in src/pypy but in some subdirectory (e.g. tool) or at 'src/scripttest' or something similar. And btw, it's preferable if you talk to some other developer or best to pypy-dev before adding a whole bunch of things :-) > The start of using app_level_diff as a testing tool - I've converted the > CPython regression test test_grammar to it's use. Sectioning is incomplete - > the parts that are sectioned are mainly to isolate failures > > I've also started to do a script made from the examples given in Guido's > tutorial. Incomplete as of now. Can you give some examples of usage? I am a bit of a loss what exactly you are trying to do here. cheers, holger

2 3