Guido van Rossum wrote: >

Yes. It surprises me that you think so lightly of something that if it was easy obviously would have been done already.

I think it's clear noone really knows why this hasn't been done yet. Even on this list, whose users are presumably well clued-in about Python, I've heard: ``you're welcome to volunteer.'' ``Nobody cared enough to do it.'' I'm sure writing a compiler isn't easy, and I wasn't suggesting that. However, head-way (Psycho) has been made in a very short period of time, so it hardly seems impossible given enough time and effort. I think such an effort would be well worth it. ===== Regards, Graham http://movies.yahoo.com.au - Yahoo! Movies - What's on at your local cinema?

Graham Guttocks wrote:

I think such an effort would be well worth it.

Talk is cheap. Neil

...

Talk is cheap.

What, so I can't inquire about or request a potential feature unless I can implement it myself? Please.

Graham, you have worn out your welcome here. Python is free -- if you don't like it, you can walk away, or pay somebody to add the feature you want. You can't just go about requesting features and expect them to be implemented if no-one else thinks they are needed. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Graham Guttocks]

Guido van Rossum wrote:

...

Graham, you have worn out your welcome here.

Well, I apologize for bringing up a subject you don't enjoy talking about. Has anyone ever told you guys that you don't handle constructive criticism well? I've seen it as a lurker on this list as well as c.l.p, and now firsthand.

OK, now I have been basically ignoring this thread and purposely staying out of this whole mini-flame war that is starting. But I have to disagree with the suggestion that this list, as a whole, cannot handle constructive criticism. I have to read every single piece of email that comes through this list and yes, some people can get a little snappy. But first of all most people here are volunteering their time here and when something that is stupid or has a hint of insult it flips a switch (I know it does in me). And second, we are all human (except Timbot, effbot, and Alexbot when he pops in from time to time). When this list gets a question about something that is asked constantly it just wears on some of us and we just want to ignore it. And yet people still responded to your question, which is more than some lists will do. But this list's integrity is very high. Saying the people here cannot take criticism is, to quote Guido, "bullshit". Every point here is argued and people still manage to keep it "clean" as long as you don't come in here making accusations. When someone comes in and suggests something hasn't been done because it might be "too difficult", most of us take it personally (I would suspect Guido especially). It is a suggestion that the knowledge of this list is not high enough to do this. As has been stated, though, it just has not been a priority of python-dev to worry about this and has nothing to do with ability. If you are that worried about performance, code it in C and hook into Python. It is going to take someone caring enough to actually write a solution and no one here cares enough. But you have been "trolling" this list, so you know all of this by now. I-can't-believe-I-managed-to-prevent-msyelf-from-yelling-ly yr's, Brett

Graham Guttocks writes:

Well, I apologize for bringing up a subject you don't enjoy talking about. Has anyone ever told you guys that you don't handle constructive criticism well? I've seen it as a lurker on this list as well as c.l.p, and now firsthand.

This list has no problem with constructive criticism. The list has a problem with anything being posted to the list that doesn't have to do with the current development of Python. These posts fall in two major catagories: 1) Features or bugs that somebody wants written, but isn't willing to write. This is the Python Developers list not the Python Feature Request List. It is intended for you to discuss the changes you are making or proposing to make to the language, not to request that your pet project be written by the other readers. A bug reporting mechanism exists, and this isn't it. 2) Questions that have been asked a million times and are already on the web page or in the FAQ. You've managed to fall in both catagories. Merriam-Webster defines "constructive" as "promoting improvement or development" and "promoting" as "to contribute to the growth or prosperity of." It's hard to see how bringing up a question that was answered in the FAQ over a year ago is contributing to the growth of anything. Yet you got not one, but two replies from Guido who both told you exactly why native code generation hasn't been done and that the rest of the answer was in the FAQ. If it's so easy to do, and so worthwhile, do it. I have more tractable things to spend my time on.

Have you forgotten that without users and potential users Guido, Python would cease to exist?

Why? It certainly came into being with only developers. I think if only they ever used it, they would still consider it a worthwhile effort. -- Christopher A. Craig ed is the standard text editor

Graham Guttocks wrote:

...

Talk is cheap.

What, so I can't inquire about or request a potential feature unless I can implement it myself? Please.

wow. I look away for two hours (damn the french!), and when I get back, some kiwi troll has successfully taken control over python-dev. amazing. where's 2.3 alpha 2, btw? </F>

At 9:45 AM +1300 1/31/03, Graham Guttocks wrote:

I think such an effort would be well worth it.

Then go for it. It's actually not all that tough, but badgering people who've decided not to for a variety of reasons won't get you that far... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

"Graham Guttocks" wrote in message news:20030130211504.810.qmail@web10302.mail.yahoo.com...

Dan Sugalski wrote:

...

It's actually not all that tough

Oh? Guido seems to disagree with you.

No contradiction or disagreement. It is not all that tough to do something of limited benefit, such as pre-interpreting Python byte codes into a series of function calls in some compilable-to-machine-code language. It is *must* harder to do something much more useful, which is all Guido would be interested in. A a grad student, Armin Rigo conceived and implemented a clever idea which works with a large subset of Python. Now that he has graduated to the real world, I am sure all of us hope that he finds the means to continue. PyRex and Weave are other projects that help certain Python applications. Terry J. Reedy

At 10:15 AM +1300 1/31/03, Graham Guttocks wrote:

Dan Sugalski wrote:

...

It's actually not all that tough

Oh? Guido seems to disagree with you.

That's fine. Guido and I have a different set of skills. It's not that tough with the right set. (In the same way that designing languages isn't that tough with the right set of skills)

...

but badgering people who've decided not to for a variety of reasons won't get you that far...

Badgering?

Look, it's obvious that I've hit a nerve here, but that wasn't my intention. The FAQ is not clear as to what those ``variety of reasons'' are, and I guess I'm learning why this is.

Because it freaks everyone out. :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

At 12:58 AM +0100 1/31/03, Fredrik Lundh wrote:

Dan Sugalski wrote:

...

...

Oh? Guido seems to disagree with you.

That's fine. Guido and I have a different set of skills. It's not that tough with the right set.

you mean you've finished Parrot, and you have a working Python compiler for it? can we see some benchmarks, please?

It's not done yet, no. Doesn't mean that it isn't sufficiently complete to draw some conclusions, and we do have a working cross-platform JIT, which is the part that has the most direct bearing on the original subject of this little firestorm. Native code compilation isn't difficult, nor is expecting a gain of at least 20% unreasonable over straight interpretation. (Expecting a factor of 10-12 speedup not unreasonable in some circumstances) Ignoring parrot, I'll point you at some of the Scheme and Smalltalk setups. (Andyou'd be hard-pressed to find something to beat smalltalk for runtime dynamism) If you want to bet, I'll put up $10 and a round of beer (or other beverage of your choice) at OSCON 2004 for all the python labs & zope folks that says parrot beats the current python interpreter when running all the python benchmark suite programs (that don't depend on extensions written in C, since we're shooting for pure engine performance) from bytecode. Game? -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

At 9:12 AM +0100 1/31/03, Fredrik Lundh wrote:

Dan Sugalski wrote:

...

Ignoring parrot, I'll point you at some of the Scheme and Smalltalk setups. (And you'd be hard-pressed to find something to beat smalltalk for runtime dynamism)

Not on python-dev.

The original topic was native-code compilers, and it was put forth that writing one for a dynamic language was either very hard or impossble, and if you did there wasn't much win. The counterexamples of the Scheme and Smalltalk compilers are relevant--both languages are more dynamic at runtime, and both languages have native compilers that provide significant speed benefits, and both languages have implementations with relatively little man-time and brainpower that provided significant speed benefits without semantic changes. Unless I'm misreading you, and you're positing that python's more dynamic at runtime than smalltalk, but I presume that's not the case.

...

If you want to bet

Just show me the code.

That was the point of the bet.

...

I'll put up $10 and a round of beer (or other beverage of your choice) at OSCON 2004 for all the python labs & zope folks that says parrot beats the current python interpreter when running all the python benchmark suite programs (that don't depend on extensions written in C, since we're shooting for pure engine performance) from bytecode. Game?

If you're talking about simple stuff like pystone, people have already done *interpreters* that can run that kind of bench- mark 5-25 times faster than CPython. All it takes is a fast GC, call-site caching, some low-overhead primitives, and a fast calling mechanism.

Then pick one you consider more indicative. I don't much care, honestly--as long as it doesn't depend on C extensions it's fine.

But if you want to run real Python programs, you still have to solve all the hard problems. Judging from the Parrot material I've seen, it's not obvious that you're even aware of what they are.

I have a list of hard problems. Luckily for me, python doesn't hit any of them, but that may as much be because of what I consider a "hard problem" than anything else.

After all, if you know how to solve them, wouldn't it be easier to tell us what you've done (or plan to do), instead of pointing to non-Parrot implementations of non-Python languages done by others?

If parrot was finished, sure. It's not, hence the 2004 date on the challenge (while I could meet the deadline for this year's OSCON, enough other things would get shuffled aside that it wouldn't be a good allocation of resources) and the Scheme and Smalltalk examples, which *do* exist, and have for quite some time. C'mon Fred, you *know* that if I just told you how I planned to solve the problems I'd get ripped to shreds (and rightly so) for handwaving past the implementation issues, and nothing I have now will satisfy, as it's not finished so there's always that "well, it isn't done yet" rejoinder. (Which, again, is both fine and appropriate) Still, the challenge stands if you or anyone else wants to take me up on it. Catch me before OSCON 2003 and we'll go from there. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

[Michael Hudson]

It's unclear to me that scheme is more dynamic than Python.

I don't think Scheme is more dynamic either. The language has set types that you just don't happen to have access to or see directly. But there is no clear cut way of defining new types short of writing it in C. It was bad enough reading Paul Graham calling Python "water-down Lisp with infix syntax and no macros", but having Scheme being considered more dynamic is blasphemy in my book (but that might be because Scheme was forced upon me in school =). -Brett

Brett Cannon :

It was bad enough reading Paul Graham calling Python "water-down Lisp with infix syntax and no macros", but having Scheme being considered more dynamic is blasphemy in my book (but that might be because Scheme was forced upon me in school =).

Yo. Don't you be dissin' my homeboy Paul. He is flyyyy. Actually, I had a long lunch with Paul recently and am in a position to state confidently that he thinks Python is pretty nifty. Yes, Paul has some LISP-inspired criticisms. I agree with some (yes, Python would be a better language if it were a little more expression-oriented) and I disgree with others (I have regretfully concluded that macros are a bad idea). On the other hand, he concedes that dict as a first-class type is a very good idea, and that the Python OO system is right in style even if arguable in detail. Paul's website recommends Python as the best alternative if you can't do LISP; see http://www.paulgraham.com/faq.html. Under the circumstances (which include the fact that he's one of the two or three influence leaders of the LISP community) that's about the strongest endorsement one could expect. Guido, if you ever get a chance to hang with Paul, take it. He's an very intelligent and capable person, well-spoken, a natural leader, and no mean language designer himself. Knowing both of you, I predict you would get along famously and learn a lot from each other. In fact, you could do a lot worse for a keynote speaker at a future Python conference than to invite him. And then listen to his talk carefully. Very few people would be better qualified than Paul as a friendly critic from a different tradition. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

At 8:44 AM +0100 2/1/03, Fredrik Lundh wrote:

Brett Cannon wrote:

...

...

It's unclear to me that scheme is more dynamic than Python.

I don't think Scheme is more dynamic either.

from a language runtime perspective, I don't think Smalltalk is more dynamic either.

I'm still waiting for the Parroteers to prove me wrong.

You'll probably be waiting a long time--it's generally off topic for the list, most people who aren't convinced won't believe me anyway, and I don't really care enough about this to put in the effort to convince you. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

At 12:00 PM +0100 2/3/03, Christian Tismer wrote:

Dan Sugalski wrote: ...

...

If you want to bet, I'll put up $10 and a round of beer (or other beverage of your choice) at OSCON 2004 for all the python labs & zope folks that says parrot beats the current python interpreter when running all the python benchmark suite programs (that don't depend on extensions written in C, since we're shooting for pure engine performance) from bytecode. Game?

Do you mean the current Python interpreter of 2004? That could be quite a different beast...

While it could be, that's unlikely. Still, that'd be part of the ground rules we'd work out if someone takes me up on the challenge. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

At 1:15 PM -0500 2/3/03, Guido van Rossum wrote:

...

At 12:00 PM +0100 2/3/03, Christian Tismer wrote:

...

Dan Sugalski wrote: ...

...

If you want to bet, I'll put up $10 and a round of beer (or other beverage of your choice) at OSCON 2004 for all the python labs & zope folks that says parrot beats the current python interpreter when running all the python benchmark suite programs (that don't depend on extensions written in C, since we're shooting for pure engine performance) from bytecode. Game?

Do you mean the current Python interpreter of 2004? That could be quite a different beast...

While it could be, that's unlikely. Still, that'd be part of the ground rules we'd work out if someone takes me up on the challenge.

I'm in. Given how low you set your stakes, I don't think you're very confident, so I'd like to call your bluff. :-)

The stakes are low because I have to pony up the cash if I lose--my budget's limited, and regardless of how confident I am of winning, if I put up a lot, my wife will kill me dead for making the challenge and, well I think I'd rather avoid that. :) Shall we, then, arrange the details at OSCON 2003, if you'll be there? I expect we can twist the conference organizer's arm into letting us make a tiny announcement near the end... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

At 1:41 PM -0500 2/3/03, Guido van Rossum wrote:

...

The stakes are low because I have to pony up the cash if I lose--my budget's limited, and regardless of how confident I am of winning, if I put up a lot, my wife will kill me dead for making the challenge and, well I think I'd rather avoid that. :)

Especially as you'd automatically lose if your wife were to kill you. :)

Well, we might still beat you, though I'd *definitely* lose personally. :)

...

Shall we, then, arrange the details at OSCON 2003, if you'll be there? I expect we can twist the conference organizer's arm into letting us make a tiny announcement near the end...

Sure. We should do this as a lightning talk in the Python lightning track.

Works for me. I doubt we'll need that much time to work things out. OTOH, as conferences love animosity, real or imagined, it wouldn't surprise me if Tim O'Reilly'd be happy to announce this if we can work the details out ahead of time. Fine with me either way... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Guido van Rossum wrote:

...

I'm in. Given how low you set your stakes, I don't think you're very confident, so I'd like to call your bluff. :-)

Any opinion on where to spend effort? I was thinking of dusting off the register VM code (aka rattlesnake). OTOH, perhaps the non-local namespace optimizations would give more bang for the buck.

Neil

I was thinking it might be most effective to have a little conversation with Dan's potential sponsors. But maybe that's because my first name is Guido... ;-) Seriously, I expect Dan to lose without any effort on our part, but if we want to make an effort, I think that there could be some low-hanging fruit in inlining certain builtins (e.g. len, range, xrange) that as far as we know aren't shadowed by globals. Is that what you call non-local namespace optimizations? A new VM design would be a major upheaval, but if we can pull it off it would certainly not be a bad idea. John Aycock and some of his students have a design of a new VM that might be worth looking into. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

Seriously, I expect Dan to lose without any effort on our part,

Really! Well, then, shall we raise the stakes, and add a cream pie (of the loser's choice) at ten paces to the bet? :)

I'd be delighted to deliver it in person. :) --Guido van Rossum (home page: http://www.python.org/~guido/)

Dan> Really! Well, then, shall we raise the stakes, and add a cream pie Dan> (of the loser's choice) at ten paces to the bet? :) Having had personal experience with the inverse relationship between geek quotient and athletic prowess I suspect you might want to make that five paces. ;-) Skip

...

Having had personal experience with the inverse relationship between geek quotient and athletic prowess I suspect you might want to make that five paces. ;-)

Nah, that's OK--I'll be practicing my aim before then. :-P

That was my line. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

At 3:25 PM -0500 2/3/03, Eric S. Raymond wrote:

Dan Sugalski :

...

Really! Well, then, shall we raise the stakes, and add a cream pie (of the loser's choice) at ten paces to the bet? :)

Oh, *goddess*. I want to emcee this event in the worst way. And I know exactly how to do it -- like a herald reading cartels of defiance at a joust.

Gentlemen, you've got yourself a master of ceremonies and a frame of comic patter, if you want it. For this I might even put on a tuxedo...

Argh! The very thought of that is making me want to gouge my eyes out. :) Seriously, this is something we'll have to work out with the conference folks. Wouldn't surprise me if they may have someone in mind, or if not we go with some semi-independent person. (Perhaps some bitter Lisp hacker that hates all of us because everyone's more successful than Lisp... :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

At 5:41 PM -0500 2/3/03, Eric S. Raymond wrote:

Dan Sugalski :

...

Seriously, this is something we'll have to work out with the conference folks. Wouldn't surprise me if they may have someone in mind, or if not we go with some semi-independent person. (Perhaps some bitter Lisp hacker that hates all of us because everyone's more successful than Lisp... :)

If you suggest Paul Graham I'm going to have to hurt you.

Hey, if we can pitch a pie at him regardless of who wins, I'm OK with it. Come to think of it, if you want to host it...: ) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Guido van Rossum wrote:

I think that there could be some low-hanging fruit in inlining certain builtins (e.g. len, range, xrange) that as far as we know aren't shadowed by globals. Is that what you call non-local namespace optimizations?

I meant continuing work started by Ping, Jeremy, and you on non-local namespaces (something other than dicts). Tim's suggestion of optimizing function calls sounds like another good one. Neil

Guido> And I suggest that we analyze pystone (or whatever benchmark we Guido> decide to use). If you decide to use pystone, I'd argue you incorporate psyco into the distribution and stick the following code right after the definition of Proc8: try: import psyco except ImportError: pass else: psyco.bind(Proc8) (I believe that's the correct psyco call and the correct function to optimize. It's been awhile.) ;-) Skip

Neil> Any opinion on where to spend effort? I was thinking of dusting Neil> off the register VM code (aka rattlesnake). OTOH, perhaps the Neil> non-local namespace optimizations would give more bang for the Neil> buck. I'd help play around with rattlesnake. Skip

>> I'd help play around with rattlesnake. Brett> Any links on rattlesnake? I tried googling and came up with only Brett> a brief mention by Ping and Skip about it. You just didn't look far enough back. ;-) Rattlesnake was something I started in 1998 (based on 1.5.1 as I recall) to come up with an alternate, register-based virtual machine for Python. The observation that Tim made which got me headed in that direction was that three-register machines are much easier to optimize than stack-based architectures. The plan was to mechanically translate the bytecode for the stack architecture to the bytecode for the register architecture. (I used my peephole optimizer as the basis for that translator. Conversion to the new architecture was thus a rather substantial "peephole" operation.) The result should be relatively easy to optimize (just by eliding many of stack motion, if nothing else). A few of us got a reasonable start on it, but it eventually fizzled. There were a couple things that didn't ever work right, and starting with 1.5.2 the virtual machine started moving faster than I could keep up in the odd spare moment. It sat idle until a little over a year ago when Neil began working on it. To the best of my knowledge, he's never released anything, but whatever he's got laying about is bound to be more current than anything I could point you at. Still, here are some pointers for those interested: * Here's a pointer to a paper about the peephole optimizer: http://www.musi-cal.com/~skip/python/spam7/optimizer.html * Here's the latest stuff I bundled up for Neil in 2001: http://www.musi-cal.com/~skip/python/rattlesnake20010813.tar.gz * Here's the long inactive mailing list: http://groups.yahoo.com/group/rattlesnake * There's a python-dev thread from February 2002: http://mail.python.org/pipermail/python-dev/2002-February/thread.html (search for "rattlesnake"). * Finally, here's a python-dev thread about rattlesnake from 1998: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=7399hf%24scb%241%40nnrp1.dejanews.com&rnum=1&prev=/groups%3Fq%3Drattlesnake%2Bmore%2Bbite%2Bfor%2Bpython%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3D7399hf%2524scb%25241%2540nnrp1.dejanews.com%26rnum%3D1 Skip

As Guido mentioned, inline builtins. I have a real simple patch which does this. ie, LOAD_GLOBAL(name) -> LOAD_CONST(builtin_function). However, there is a problem with the patch: the resulting code can't be marshalled properly. I haven't tried to fix that yet.

That's because you can't marshal references to built-in function objects. I was thinking of adding appropriate new opcodes for a few builtins that are called a lot, like len. This would be implemented using something like this: case BUILTIN_LEN: { long n; v = POP(); n = PyObject_Size(v); Py_DECREF(v); if (n >= 0) { x = PyInt_FromLong(n); if (x != NULL) { PUSH(x); continue; } } else { err = n; } break; }

Right now JUMP_IF_(TRUE, FALSE) keep their computed value on the stack. They are always followed by POP_TOP. If the JUMP_IF_* removed the value, it would be one less trip through eval_frame loop (no POP_TOP). I've got a patch for this which fixes 5 of the 8 cases where JUMP_IF_* are generated. The problem with the remaining 3 cases is that a jump to a jump occurs. By removing the jump to a jump, that should also help performance.

Yes. I think there's also an inefficiency in the bytecode generated for 'not': instead of generating a UNARY_NOT opcode, it could switch the sense of the test (changing JUMP_IF_FALSE into JUMP_IF_TRUE and vice versa).

I have a crazy idea that removing the switch and making our own jump table in the eval_frame loop could improve performance. But I've never tried this because it's a lot of work. And it could hurt, not help performance.

The only way to find out is to try it. ;-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> I was thinking of adding appropriate new opcodes for a few Guido> builtins that are called a lot, like len. This would be Guido> implemented using something like this:

Guido> case BUILTIN_LEN: ...

Would you special case those calls so that, in effect, __builtin__.len couldn't be overridden by a "len" object in the globals or locals?

No, I would only generate a BUILTIN_LEN opcode if there was no local or global variable 'len'. A few days ago I proposed a restriction on assigning to an attribute of a module that stores a new name in that module that is the same as the name of a built-in; that was meant to outlaw people doing ugly stuff like import random random.len = lambda x: len(x)-1 print random.choice([1,2,3]) and expecting to affect the implementation of choice(). I don't think anybody would do this on purpose (or even by accident), but the possibility exists, and I'd prefer to live without lying awake about that case. (We could make random.__dict__ read-only, like the new-style class __dict__, if you worry about other ways of stuffing unexpected variables inside it. We could also avoid the optimization if there's an exec statement anywhere in a module, since one could exec code that says import __builtin__ global len len = lambda x: __builtin__.len(x)-1 or one could find another way of outlawing such abominations (including simply declaring that one shouldn't do that). --Guido van Rossum (home page: http://www.python.org/~guido/)

...

...

Guido> I was thinking of adding appropriate new opcodes for a few Guido> builtins that are called a lot, like len. This would be Guido> implemented using something like this:

Guido> case BUILTIN_LEN: ...

Would you special case those calls so that, in effect, __builtin__.len couldn't be overridden by a "len" object in the globals or locals?

No, I would only generate a BUILTIN_LEN opcode if there was no local or global variable 'len'.

A few days ago I proposed a restriction on assigning to an attribute of a module that stores a new name in that module that is the same as the name of a built-in; that was meant to outlaw people doing ugly stuff like

import random random.len = lambda x: len(x)-1 print random.choice([1,2,3])

and expecting to affect the implementation of choice().

I don't think anybody would do this on purpose (or even by accident), but the possibility exists, and I'd prefer to live without lying awake about that case.

(We could make random.__dict__ read-only, like the new-style class __dict__, if you worry about other ways of stuffing unexpected variables inside it.

do you mean specifically random.__dict__ or any modules dict?

Any module's dict.

If the latter there would be quite some breakage. It is at least used for monkey patching modules to make them "unittestable" which is a valid use case IMO.

Why would this be done by patching the module's __dict__ rather than assigning to attributes of the module?

Maybe module's dicts could be "frozen" by default and this could be explicitely turned off (via a sys-hook).

What I proposed was only closing off write access to the __dict__ so that the setattr code for type module can implement certain restrictions (e.g. no assignment to attributes named "len"). --Guido van Rossum (home page: http://www.python.org/~guido/)

...

...

...

...

"GvR" == Guido van Rossum writes:

...

do you mean specifically random.__dict__ or any modules dict?

GvR> Any module's dict.

...

If the latter there would be quite some breakage. It is at least used for monkey patching modules to make them "unittestable" which is a valid use case IMO.

GvR> Why would this be done by patching the module's __dict__ rather GvR> than assigning to attributes of the module?

...

Maybe module's dicts could be "frozen" by default and this could be explicitely turned off (via a sys-hook).

GvR> What I proposed was only closing off write access to the GvR> __dict__ so that the setattr code for type module can implement GvR> certain restrictions (e.g. no assignment to attributes named GvR> "len"). I'd certainly like that feature, except for the times I'd want to circumvent it. On many occasions I have externally patched a module's namespace for debugging. It's been really convenient to replace some function with a wrapper that does some logging or updates a table tracking currently used resources. I've done the same with builtin open/file. As you noted, it seems less useful to override other builtins for this purpose. I'd suggest that frozen module namespaces by a feature of -O, except that I never use -O. Jeremy

...

...

...

...

"BAW" == Barry A Warsaw writes:

...

...

...

...

"JH" == Jeremy Hylton writes:

JH> On many occasions I have externally patched a module's namespace JH> for debugging. It's been really convenient to replace some JH> function with a wrapper that does some logging or updates a JH> table tracking currently used resources. I've done the same JH> with builtin open/file. As you noted, it seems less useful to JH> override other builtins for this purpose. BAW> I've done the same, and when I've needed, it's been incredibly BAW> handy. Rare, but useful. Then again, I've also only done this BAW> with open(), so maybe something like an open() hook would BAW> suffice. I can imagine some of the other built-ins /might/ be BAW> useful to debuggingly replace, but that's just speculation on BAW> my part, and may be yagni. But it is useful for to replace modules. For example, if you want to debug a module that uses the thread module, you can replace it's "thread" global with something that hooks "start_new_thread." I think that pattern is not uncommon. Jeremy

...

...

...

...

"GvR" == Guido van Rossum writes:

...

But it is useful for to replace modules. For example, if you want to debug a module that uses the thread module, you can replace it's "thread" global with something that hooks "start_new_thread." I think that pattern is not uncommon.

GvR> My proposal is to allow rebinding existing globals, just not to GvR> allow binding *new* globals *if* they shadow certain builtins. Oh! +1 on that. just-add-a-future-statement-ly y'rs, Jeremy

My recollection from Skip's rattlesnake paper and from a similar paper about peephole optimizations in Interlisp is that we'll see perhaps a 5-10% speedup by focusing on the small stuff. I expect that the builtin-specific opcodes will do more than that, but the module-level namespace still ends up being expensive to use and arbitrary module globals are probably called as often as any other global. It'd be interesting to measure how many of the LOAD_GLOBAL calls are for builtin names using some simple benchmarks. Jeremy

Wasn't there also a suggestion (I think Christian made just before Minimal Python was announced) to have the parser know what built-ins were and thus just inline the code?

Most builtins don't have an inline version unless you add new opcodes that invoke e.g. PyObject_Size. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Neil Schemenauer]

Any opinion on where to spend effort?

For real life boosts, function/method calls. Making "len" a builtin wouldn't help so much for saving a couple dict lookups as it would for avoiding the call machinery, and for callees coded in Python that's heavy work.

IIRC the largest expense in function calls is setting up the frame object for the new function to use. I don't think there are any low-hanging fruit here, because the frame model is pretty deeply embedded in the interpreter. It seems like all the work in PyEval_EvalCodeEx() has to get done sometime.

Then that's where we should have a clean fresh look. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Graham Guttocks]

... I think such an effort would be well worth it.

Indeed, I spent the first 15 years of my computer career writing optimizing compilers for a living, and it's *so* worth it that in 10+ years of Python development, nobody has offered to pay me so much as an insignificant fraction of how much it's worth <0.5 wink>. These kinds of things get done by legions of grad students in search of a thesis, or by serious funding; they don't get done by part-time volunteers with demanding jobs.

[Tim]

These kinds of things get done by legions of grad students in search of a thesis ...

[Samuele Pedroni]

in which case then typically portability, long term maintainability and absence of rough edges are quite irrelevant.

At first, yes, but it's not irrelevant that *something* gets done. For example, countless now-defunct Lisp companies were based on the founder(s)'s grad work, and a huge (relative to the size of the community) amount of Lisp compiler knowledge is still available for the grabbing. Even if you don't look at a lick of that code, the heritage of published papers is priceless. Or worthless, to judge from how often it's referenced <0.1 wink>.

[Graham Guttocks]

I see. Again, I didn't realize the enormity of this, because of how fast Psycho has progressed (from a part-time volunteer).

Not any part-time volunteer. Armin did Psycho while finishing his doctoral dissertation, and applied a brain the size of a planet to the job, with passion. That's the way intriguing compilers for dynamic languages get written in the absence of realistic funding (Dan notwithstanding <wink>). For a recent idea of what "realistic funding" means, dig thru now-old but not-yet-ancient stories about Sun's efforts on Java JIT technology. Of course everyone's an expert on everything they don't do, so it's natural to assume it's just a lack of will, or foresight, or something. Since O'Caml is in the Subject line here, note that one person reimplemented Python in O'Caml, with the goal of running much faster than CPython: http://vyper.sourceforge.net/ They gave up some time ago, with the last word on the topic there being: The interpreter Pystone benchmark is currently about 10 times slower than CPython 1.5.2. Considerable improvement is expected in the future. At least CPython (what we call the C implementation of Python) has gotten faster since then, but I'm not sure that's the considerable improvement the author had in mind <wink>.

You might consider adding the above to the FAQ to make it more clear that it is theoretically possible, but not without an enormous amount of work and/or funding.

The question doesn't upset me, but I prefer to limit the FAQ to frequently asked questions.

Tim Peters writes:

[Graham Guttocks]

...

I see. Again, I didn't realize the enormity of this, because of how fast Psycho has progressed (from a part-time volunteer).

Not any part-time volunteer. Armin did Psycho while finishing his doctoral ^ nope.

dissertation, and applied a brain the size of a planet to the job, with passion.

It could be pointed out that Armin's thesis was in purest-of-the-pure mathematics and had nothing to with Python. Last I heard, he'd got a postdoc in something that could related to dynamic compilation, but I don't know if that amounts to being paid to work on psyco. Cheers, M. -- Screaming 14-year-old boys attempting to prove to each other that they are more 3133t than j00. -- Reason #8 for quitting slashdot today, from http://www.cs.washington.edu/homes/klee/misc/slashdot.html

From: "Guido van Rossum"

...

Really? It surprises me that after 10 years, this isn't something that has been given more priority. Is the problem simply too difficult?

Yes. It surprises me that you think so lightly of something that if it was easy obviously would have been done already.

a "data" point from Towards a Universal Implementation Substrate for Object-Oriented Languages, Mario Wolczko, Ole Agesen, David Ungar, paper presented at the OOPSLA 99 workshop on Simplicity, Performance and Portability in Virtual Machine Design, OOPSLA '99, Denver, CO, Nov 2, 1999. http://research.sun.com/kanban/oopsla-vm-wkshp.pdf 1 Building virtual machines is hard Building a high-performance virtual machine (VM) for an object-oriented language such as Smalltalk or JavaT 2 is a major undertaking. ... In summary, building a high-performance VM typically requires several man-years of effort by those experienced in the field (e.g., the Self VM, in its present form, is the result of some 25 man-years of work), and results in a large, complex program, which is difficult to modify and adapt.

...

...

...

...

"GG" == Graham Guttocks writes:

GG> Serious wasn't a good choice of word, but I think you know GG> what I mean. Only that Python isn't the appropriate tool for every programming task. No news there. But it is perfectly appropriate for a vast array of programming tasks. GG> e.g, what if I wanted to write an httpd daemon that could GG> scale as well as Apache (millions of hits/day)? Python would GG> not be up to the task. A language like CMUCL or OCaml would GG> be, because they offer a native code compiler. You mean like Twisted? http://twistedmatrix.com/documents/howto/faq#auto8 -Barry

[Graham Guttocks]

e.g, what if I wanted to write an httpd daemon that could scale as well as Apache (millions of hits/day)?

More than 625 hits per second average? If you really have this need, and see Python developments that could help you, I would guess that Python developers might be interested in them. If you do not really have this need, than this discussion is merely discursive and academical, and might be better held elsewhere than on `python-dev'.

Python would not be up to the task.

I would be tempted to consider many architectural avenues, before bluntly deciding that the choice of Python as an implementation language is _the_ problem. With millions of hits per day, I would most probably be big enough to have a flurry of other problems of all kinds, and then, Python would be more on the side of many solutions than the source of my problem. -- François Pinard http://www.iro.umontreal.ca/~pinard

On Fri, 31 Jan 2003, [iso-8859-1] Graham Guttocks wrote:

"Barry A. Warsaw" wrote:

...

Really? How many of these 2100+ projects (just to pick a few at random <wink>) aren't serious?

Serious wasn't a good choice of word, but I think you know what I mean.

e.g, what if I wanted to write an httpd daemon that could scale as well as Apache (millions of hits/day)? Python would not be up to the task. A language like CMUCL or OCaml would be, because they offer a native code compiler.

Depending on your type of traffic, a server may be network bound long before it's CPU bound, so compiling doesn't really help much. Millions of hits per day isn't a big deal, BTW. At the company I work for we have a Python server that easily handles 200 req/sec on sub-1GHz Intel boxes. At constant load that's about 20 million per day. While not as fast as Apache, it's far higher performance than we or many other sites need. -Dave