On Mon, 2 Aug 2004, Jeremy Hylton wrote:

On Mon, 2 Aug 2004 14:30:39 +0100 (GMT Daylight Time), John J Lee <jjl@pobox.com> wrote: [...]

...

Would somebody mind checking this in:

http://www.python.org/sf/997626

It fixes a bug in httplib which badly breaks urllib2 now that urllib2 is using httplib.HTTPConnection instead of httplib.HTTP (POSTs don't work).

It includes patch and a new test, and does not require doc updates, but nobody has reviewed it yet.

I'll take a look, probably tomorrow morning.

Jeremy [...]

It don't think it's in especial need of reviewing by an expert. It just needs to get checked in! John

On Mon, 2004-08-02 at 10:40, Phillip J. Eby wrote:

At 04:32 PM 8/2/04 +1000, Anthony Baxter wrote:

...

The @decorator patch has been landed on the trunk, after getting the go-ahead from the BDFL. I'll update PEP-0318 with the final syntax in the next day or so.

@decorator won? When did that happen?

First I heard about it was when I saw Anthony's checkin messages. Maybe it was throwing that pie that inspired Guido -- an '@' does kind of look like a pie. I think Anthony's checkin message was accurate enough though -- it's the syntax everyone can hate equally. But I'm glad /something/ made it in! print>>-set-the-precedent-ly y'rs, -Barry

On Mon, 2004-08-02 at 11:14, Anthony Baxter wrote:

But I like Barry's idea - I think it should be referred to at the 'pie decorator syntax'.

:) I'm going to convert MM3 to use pie decorator syntax to play with this (in my CST). -Barry

On Aug 2, 2004, at 12:55 PM, Phillip J. Eby wrote:

At 01:14 AM 8/3/04 +1000, Anthony Baxter wrote:

...

Barry Warsaw wrote:

...

On Mon, 2004-08-02 at 10:40, Phillip J. Eby wrote:

...

@decorator won? When did that happen?

First I heard about it was when I saw Anthony's checkin messages. Maybe it was throwing that pie that inspired Guido -- an '@' does kind of look like a pie. I think Anthony's checkin message was accurate enough though -- it's the syntax everyone can hate equally. But I'm glad /something/ made it in! print>>-set-the-precedent-ly y'rs, -Barry

Hey, I was just going off Guido's decision (in email - looking back, it wasn't cc'd to python-dev, which probably explains the lack of 400 followups <wink>)

Channelling the BDFL, I think the idea was to put it in 2.4a2, and see how it works for people. If it turns out that it's really really really hated, we can try something else in a3. Guido can of course follow up to this (hint hint) and give his own answers.

I would think the fact that the '[decorators]' syntax can be implemented in pure Python (no changes to the interpreter) for existing Python versions would give more weight to it. That is, if someone wants to implement a decorator that's forwards and backwards-compatible, that's possible with the list syntax, but not the @ syntax.

... but that also means you can still make the [decorators] syntax work in 2.4, if you want compatibility or don't like @syntax. -bob

Is anyone else seeing a stack overflow assertion error in test_pyclbr.py (WinMe, MSVC++6.0)? Raymond

[Raymond Hettinger]

Is anyone else seeing a stack overflow assertion error in test_pyclbr.py (WinMe, MSVC++6.0)?

BTW, which version of Python are you talking about? Current CVS can't compile under MSVC 6.0 anymore, because PC/pyconfig.h #defines HAVE_UINTPTR_T and HAVE_INTPTR_T now, which aren't correct for VC6, and then pyport.h goes nuts -- virtually nothing compiles anymore unless you use VC7.1.

[Raymond Hettinger]

Too bad about 6.0. I had thought the plan was to support both 6.0 and 7.1. I'm not sure what we gained by dropping 6.0 right at the end :-(

There's no plan to support MSVC 6.0. Martin routinely checks things in that break it, and that's fine. The only reason it *has* worked until recently is that people sometimes volunteer to patch up the 6.0 part of the tree. I gave some days to that at the start, because I didn't have a machine at the time that *could* run 7.1. Now I do, so I pay a lot less attention to 6.0 (and Win98SE) now. IOW, 6.0 never had, and still doesn't have, a champion (== someone willing to keep it working) for 2.4. I would have volunteered, but your report of a stack overflow in test_pyclbr scared me away <wink>.

Phillip J. Eby wrote:

But then why not just make that the default syntax, so that no migration is necessary, and only one syntax has to be learned/explained to people?

For a very simple reason: The time for discussion is now over. There is little point in going over all the same arguments again and again. For this alpha release, no amount of arguing will change what has been committed. You really have to organize a public outcry if you want the syntax changed before 2.4. Regards, Martin

At 12:09 PM 8/2/04 -0700, Guido van Rossum wrote:

...

...

...

I would think the fact that the '[decorators]' syntax can be implemented in pure Python (no changes to the interpreter) for existing Python versions would give more weight to it.

Can it? I must've missed that. It sure sounds like an incredible hack -- how to you prevent the default behavior that the list of decorators is thrown away by the interpreter?

By using sys.settrace (and a careful tracer implementation to avoid interfering with debuggers or other active tracers): def as(*decorators): if len(decorators)>1: decorators = list(decorators) decorators.reverse() def callback(frame,k,v): for d in decorators: v = d(v) frame.f_locals[k] = v add_assignment_advisor(callback) def add_assignment_advisor(callback,depth=2): """Invoke 'callback(frame,name,value)' on the next assignment in 'frame' The frame monitored is determined by the 'depth' argument, which gets passed to 'sys._getframe()'. Note that when 'callback' is invoked, the frame state will be as though the assignment hasn't happened yet, so any previous value of the assigned variable will be available in the frame's locals. (Unless there's no previous value, in which case there will be no such variable in the frame locals.) """ frame = sys._getframe(depth) oldtrace = [frame.f_trace] old_locals = frame.f_locals.copy() def tracer(frm,event,arg): if event=='call': # We don't want to trace into any calls if oldtrace[0]: # ...but give the previous tracer a chance to, if it wants return oldtrace[0](frm,event,arg) else: return None try: if frm is frame and event !='exception': # Aha, time to check for an assignment... for k,v in frm.f_locals.items(): if k not in old_locals: del frm.f_locals[k] break elif old_locals[k] is not v: frm.f_locals[k] = old_locals[k] break else: # No luck, keep tracing return tracer # Got it, fire the callback, then get the heck outta here... callback(frm,k,v) finally: # Give the previous tracer a chance to run before we return if oldtrace[0]: # And allow it to replace our idea of the "previous" tracer oldtrace[0] = oldtrace[0](frm,event,arg) # Unlink ourselves from the trace chain. frm.f_trace = oldtrace[0] sys.settrace(oldtrace[0]) return oldtrace[0] # Install the trace function frame.f_trace = tracer sys.settrace(tracer) class X(object): [as(classmethod)] def foo(cls): .... Of course, 'as' is a hack to let you use arbitrary callables as decorators, rather than having to use add_assignment_advisor directly.

...

...

...

That is, if someone wants to implement a decorator that's forwards and backwards-compatible, that's possible with the list syntax, but not the @ syntax.

.. but that also means you can still make the [decorators] syntax work in 2.4, if you want compatibility or don't like @syntax.

But then why not just make that the default syntax, so that no migration is necessary, and only one syntax has to be learned/explained to people?

Because that syntax received significant boohs when I presented it at EuroPython.

And "@" *didn't*??? Ah well. C'est la vie.

>> Can it? I must've missed that. It sure sounds like an incredible >> hack -- how to you prevent the default behavior that the list of >> decorators is thrown away by the interpreter? Phillip> By using sys.settrace (and a careful tracer implementation to Phillip> avoid interfering with debuggers or other active tracers): I suspect Guido might class that as an "incredible hack". ;-) Skip

Guido van Rossum wrote:

...

...

Can it? I must've missed that. It sure sounds like an incredible hack -- how to you prevent the default behavior that the list of decorators is thrown away by the interpreter?

By using sys.settrace (and a careful tracer implementation to avoid interfering with debuggers or other active tracers):

Ah, yuck. Not an acceptable solution. And it doesn't let you write

[classmethod]

-- you have to wrap the 'classmethod' in something relatively ugly.

Here's a brief test for a syntax-change-less implementation of this feature, not as complete as test_decorators, but a good start, I believe: def test(): try: from test import test_decorators except ImportError: test_decorators = None class DecoratorTest(object): __metaclass__ = DecoratableType def foo(self): print 'undecorated foo' decorate(staticmethod) def bar(x): print x decorate(classmethod) def baz(cls, y): print cls, y if test_decorators: counts = {} decorate(test_decorators.countcalls(counts), test_decorators.memoize) def quux(self): print 'quux called' o = DecoratorTest() o.foo() o.bar('static method on instance object') o.baz('class method on the instance') DecoratorTest.bar('static method on the class') DecoratorTest.baz('class method on the class') if test_decorators: print 'Calling quux once' o.quux() print 'Calling quux twice' o.quux() print 'Called quux', DecoratorTest.counts['quux'], 'times' And here's the implementation, without using settrace, and without requiring wrappers for individual decorators: import inspect MAGIC_NAME = '__internal_decorators_list__' class Decorator(object): def __init__(self, firstCallable, *callables): cf = inspect.currentframe() self.callables = [firstCallable] + list(callables) self.decoratesLine = cf.f_back.f_lineno + 1 cf.f_back.f_locals.setdefault(MAGIC_NAME, []).append(self) def __call__(self, f): i = iter(self.callables) f = i.next()(f) for c in i: f = c(f) return f decorate = Decorator class DecoratableType(type): def __new__(cls, name, bases, attrs): decorators = attrs.get(MAGIC_NAME, []) if decorators: del attrs[MAGIC_NAME] lines = {} for (k, v) in attrs.items(): try: source, lineno = inspect.getsourcelines(v) except: pass else: lines[lineno] = k for d in decorators: if d.decoratesLine in lines: k = lines[d.decoratesLine] attrs[k] = d(attrs[k]) return super(DecoratableType, cls).__new__( cls, name, bases, attrs) There are clear drawbacks to this approach. Metaclass required, no obvious ways to support free functions, and it depends _slightly_ hackishly on the inspect module. I think at least one of these problems can be solved (and line number handling can be made smarter to deal with intervening comments, whitespace, etc). What are the advantages? * It works with Python 2.2 and Python 2.3. * It requires no interpreter changes. It can be distributed with the standard library, as a cookbook recipe, or in the official documentation as a hint regarding more "advanced" decorator usage. * It introduces no new syntax and uses up no operator character. * It supports arbitrary expressions. * It's pure python. I realize there is little or no chance of '@decorator' being pulled from 2.4a2. I hope that something along the lines of the above will be considered, instead, for the next alpha, unless there is widespread community support for '@decorator', as opposed to the ridiculously faint support ("it's better than nothing") currently behind it. Jean-Paul Calderone

On Tue, 2004-08-03 at 06:36, Jp Calderone wrote:

I realize there is little or no chance of '@decorator' being pulled from 2.4a2. I hope that something along the lines of the above will be considered, instead, for the next alpha, unless there is widespread community support for '@decorator', as opposed to the ridiculously faint support ("it's better than nothing") currently behind it.

Personally I think it's not just better than nothing, it's better than any of the alternatives I've seen, including the do-nothing option. I added a comment to your blog post in defence of @decorator -- http://www.livejournal.com/users/jcalderone/3913.html?thread=2121#t2121 but lets not restart a syntax debate here. My guess is that the syntax will grow on people as they use it. If not there is the alternative of using one of the pure-python versions or just carrying on using the existing way of doing things. Mark

At 01:36 AM 8/3/04 -0400, Jp Calderone wrote:

There are clear drawbacks to this approach. Metaclass required, no obvious ways to support free functions, and it depends _slightly_ hackishly on the inspect module. I think at least one of these problems can be solved (and line number handling can be made smarter to deal with intervening comments, whitespace, etc).

But at least *two* of those drawbacks are critical, IMO: the metaclass requirement and the absence of free function support.

What are the advantages?

* It works with Python 2.2 and Python 2.3.

* It requires no interpreter changes. It can be distributed with the standard library, as a cookbook recipe, or in the official documentation as a hint regarding more "advanced" decorator usage.

* It introduces no new syntax and uses up no operator character.

* It supports arbitrary expressions.

* It's pure python.

If those are the only advantages, one might as well use the settrace-based hack, which doesn't have the problems that this one does. Note, by the way, that your 'decorate' function can be implemented by renaming my 'as' function to 'decorate' and dropping the 'decorators.reverse()' line. (i.e. 'as(x,y,z)' is the same as 'decorate(z,y,x)'). Although my examples wrapped such calls in brackets, there's no actual need for them. (Note also that the settrace hack works for class decorations as well as functions.) (FYI: 'as' and 'add_assignment_advisor' are available in the PyProtocols CVS head, in the 'protocols.advice' module.)

I realize there is little or no chance of '@decorator' being pulled from 2.4a2. I hope that something along the lines of the above will be considered, instead, for the next alpha, unless there is widespread community support for '@decorator', as opposed to the ridiculously faint support ("it's better than nothing") currently behind it.

Actually, I'm not sure I agree that it's better than nothing. For one thing, I don't think that there's any ambiguity of interpretation when decorators are written as function calls, be it 'as(classmethod)' or 'decorate(spam)' -- and I doubt that the point was argued at EuroPython, given that Guido missed the thread about it. But... this is moot, since it clearly has been pronounced upon. That pretty much means folks who want decorators right away are left with using the hacks, and not having a syntax-compatible upgrade path. However, that's *still* better than where they were before, because none of the original syntax proposals were expected to be backward-compatible. In practical terms, all this means is that I'll just use the hacked syntax until I no longer need to support Python 2.3.

On Tue, 2004-08-03 at 10:32, Guido van Rossum wrote:

I don't understand why you prefer your hack over the established way to do decorators pre-2.4, which is

def foo(): ... foo = staticmethod(foo)

This works across releases (including IronPython), doesn't require any magic, is documented, etc. So if your main constraint is that it be implementable pre-2.4, you already have a solution. Isn't that much better than spending effort on hacks based on sys.settrace (which are surely going to produce bafflement from users who aren't familiar with that implementation hack)?

+1 -Barry

I think this is a great point. It makes me wonder if:

@staticmethod def foo(): ...

if sufficiently better than:

def foo(): ... foo = staticmethod(foo)

to justify the language change. FWIW, It isn't to me.

Sigh. This discussion is going around in pointless circles; we're *months* past that point. You're wasting your time (and mine). --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, 03 Aug 2004 08:24:46 -0700, Guido van Rossum <guido@python.org> wrote:

Sigh. This discussion is going around in pointless circles; we're *months* past that point. You're wasting your time (and mine).

Is there a PEP that summarizes the current state of the decorators design? One of the goals of PEPs was to prevent design discussions from going in circles. Another goal was to keep core developers abreast of changes without having to read every message on python-dev. As you mentioned in an earlier email, I think the work on 2.4 would have been smoother if we had used the PEP process more effectively. We never had a decent PEP for the decorator syntax discussion. PEP 318 describes a particular design choice, but it's weak on rationale. There's no PEP that summarizes the various syntax options and the reason for the eventual choice. Facundo's PEP for decimal arithmetic (PEP 327) looks like an example of the right thing. I haven't followed the work closely, but it looks like the PEP describes the current state of the design and the rationale for it. The generator PEP is in better shape than it used to be. Jeremy

I made the proposal in response to exactly the same instinct that Jim is reflecting here. If people really need a prefix syntax for decorators it would be best to at least _start_ with an implementation that doesn't involve core language changes, because the whole area looks quite perilous from a language design point of view.

We *had* an implementation without core language changes, but it was a postfix syntax. A prefix syntax should be designed with future usability in mind exclusively, unconstrained by "implementability without language changes". That's a much more stifling requirement than the requirement that existing code shouldn't be broken (which is stifling enough in itself but IMO unavoidable). --Guido van Rossum (home page: http://www.python.org/~guido/)

At 07:32 AM 8/3/04 -0700, Guido van Rossum wrote:

...

In practical terms, all this means is that I'll just use the hacked syntax until I no longer need to support Python 2.3.

I don't understand why you prefer your hack over the established way to do decorators pre-2.4, which is

def foo(): ... foo = staticmethod(foo)

For the same reason that PEP 318 was written in the first place: to improve readability.

This works across releases (including IronPython), doesn't require any magic, is documented, etc.

True; and most of my existing decorator-using code works this way, and I don't plan on rushing out to change any of it any time soon.

Isn't that much better than spending effort on hacks based on sys.settrace (which are surely going to produce bafflement from users who aren't familiar with that implementation hack)?

Well, up until recently, I thought the [] syntax was still a shoo-in and that the settrace hack was just going to be a way to write backwards-compatible code. In the meantime, I've implemented a predicate-dispatch generic function system using the settrace hack for PyProtocols 1.0, and don't want to have to put off using it until 2.4. (FWIW, I still support 2.2.2, so "until 2.4" means maybe 1-2 years for me.) It looks like this: [when("isinstance(x,Foo) and y>23")] def do_something(x,y): # code for when x is a Foo and y>23 [when("isinstance(x,Bar) and y<99")] def do_something(x,y): # code for when x is a Bar and y<99 Rendered without any decorator syntax, it would have to be something like: def do_something_x_Foo_y_gt_23(x,y): # code for when x is a Foo and y>23 do_something.add("isinstance(x,Foo) and y>23", do_something_x_Foo_y_gt_23) def do_something_x_Bar_y_lt_99(x,y): # code for when x is a Bar and y>99 do_something.add("isinstance(x,Bar) and y>23", do_something_x_Bar_y_lt_99) ...which is a significant loss in readability. Now, it's true that this will just become '@when("whatever")' in 2.4, and I'm *fine* with that, just disappointed that the syntax won't be forward compatible. Indeed, what I'd rather focus on is making sure that the *semantics* will be forward compatible. Which I think they will be. What I'll probably do is make 'when()' and other decorators return a callable that deactivates the trace hack as a side effect of performing the decoration. That way, [when()] and @when() can coexist in a code base during the migration period.

At 08:23 AM 8/3/04 -0700, Guido van Rossum wrote:

...

Now, it's true that this will just become '@when("whatever")' in 2.4, and I'm *fine* with that, just disappointed that the syntax won't be forward compatible.

Implementability in 2.2 and 2.3 was never on the list of requirements for decorators. You're adding this constraint yourself. I know it was brought up before but I never agreed with that constraint (and the previous community favorite, [...]-after-args, wasn't compatible either).

All understood and agreed. I was only trying to answer the question that you asked; I was emphatically *not* trying to reopen the discussion. My "disappointment" here is on a par with being disappointed that a restaurant is out of my favorite dish: a trifling matter that doesn't deserve the time I've already spent on it, except that I hate being judged poorly for something that I didn't actually *do*. You seem to think that I'm saying "@" is a bad thing, or that "[]" is better for the language or community than "@". But I'm most emphatically *not* saying those things, and don't believe I ever have.

I presume your framework is already full of hacks,

Well, I don't know if I'd call 4 hacks "full", especially since two of those are to implement decorators (class level and function level). The other two are to implement 'call-next-method' for generic functions, and to implement microthreads with generators. Out of nearly ninety thousand of lines of Python, I don't think that "full" is an accurate characterization here.

which is fine by me, but I don't want to use it as a deciding factor for the decorator syntax.

I'm not asking you to, and I don't believe I ever have. I only replied to J.P. Calderone's suggestion of a less-functional hack (i.e. if you're going to hack, you might as well make it do the whole thing), and to *your* emails inquiring why I and other people thought a call-oriented syntax was better than bare "[]". With regard to the actual language change, I conceded the point when you said "let's move on" the other day. I understand your frustration with this issue, and I can also understand that you are probably feeling pretty second-guessed right now, but *I'm* not second-guessing you! So there's no call for the Spanish Inquisition here, and certainly not for personal insults because I answered questions that *you* asked. (The Python standard library actually has *six* modules or packages that use sys._getframe(), not including tests; shall we call it "full of hacks", too?)

At 07:04 AM 8/3/04 -0700, Guido van Rossum wrote:

...

Here's a brief test for a syntax-change-less implementation of this feature, not as complete as test_decorators, but a good start, I believe:

[fast forward to syntax example]

...

decorate(staticmethod) def bar(x): print x

decorate(classmethod) def baz(cls, y): print cls, y

I'm speechless. If the ambiguous

[classmethod] def foo(x): ...

is rejected because it doesn't look like it does something to foo, how come there's suddenly a crop of solutions that have the same problem being proposed?

Because '[foo]' is absolutely a list with no side effects in today's Python. But '[foo()]' clearly *can* have some side effect, so if you're reading the code you have to look up 'foo' to understand what's happening there. This isn't at all sudden, btw: David Abrahams made the first 'decorate' proposal in June, and suggested the idea of using the debugger hook to implement it. I thought that one of the best parts of David Abrahams' idea was that requiring a function call made the [] syntax less ambiguous.

What you write looks like a call to the function decorate(), followed by a function method definition. The "action-at-a-distance" that is presumed by the decorate() call is difficult to explain and a precedent for other worse hacks.

I agree, which is why I wrap my decorators in [], even though it has no effect on the actual operation. It's pure "eye candy" sprinkled with syntactic sugar. :)

Its only point in favor seems to be that it doesn't use '@'.

Well, the other point is that it allows you to use C# syntax. Note that if it were spelled: [classmethod()] def foo(x): ... this would also resolve the ambiguity (IMO).

Just a question, when "as" become a keyword in Python 3, can we expect a new syntax using that keyword instead (or additionally)? There's multiple possible syntaxes using "as", even replacing simply @ by "as" would be an improvement IMHO, since that would be english-readable (as static method define foo), like most of Python. Guido van Rossum wrote:

Not IMO, that would just add more mystery (and would require yet another hack in classmethod to allow it to be called without args).

Regards, Nicolas

Guido van Rossum wrote:

[snip]

I'm speechless. If the ambiguous

[classmethod] def foo(x): ...

is rejected because it doesn't look like it does something to foo, how come there's suddenly a crop of solutions that have the same problem being proposed? What you write looks like a call to the function decorate(), followed by a function method definition. The "action-at-a-distance" that is presumed by the decorate() call is difficult to explain and a precedent for other worse hacks. Its only point in favor seems to be that it doesn't use '@'.

In my view, the strongest point in favor of a solution that involves calling functions rather than changing syntax is that the functions involved can be placed in the standard library rather than the interpreter. I believe a widely held view is that features can be supported by the stdlib do not merit language changes? Moreover, I have the impression that many people are clamoring for this feature, no matter how it ends up looking, because they simply *must have it*. Well, if they must have it, why wait for 2.4, when clearly they can have it right now? Jp

Guido van Rossum wrote:

No viewpoint ought to be held to the exclusion of all others. Using functions with "action-at-a-distance" through sys.settraceback may be okay for an obscure feature that can't be had any other way yet doesn't merit changes to the language, but that's not the situation for decorators. The widely held view here is that decorators need to be added as a syntactic feature to avoid the problems with the postfix notation used in 2.2 and 2.3. Decorators are slated to be an important new language feature and their design needs to be forward-looking, not constrained by what can be implemented in 2.3.

Just to be clear, the implementation I shared did not use sys.settrace(). It did rely on inspect.currentframe(), but so do a handful of other stdlib modules (as PJE pointed out).

One important feature of @deco (and of [deco] prefix as I proposed it) is that it is a compile-time feature. This has all sorts of advantages, since future compilers and tools will be able to know more about methods. Solutions based on sys.settrace fail badly here.

...

Moreover, I have the impression that many people are clamoring for this feature, no matter how it ends up looking, because they simply *must have it*. Well, if they must have it, why wait for 2.4, when clearly they can have it right now?

Sure, but that's not an argument against improving their situation in 2.4.

If it is truly an improvement. Here we (and many others) disagree. I like Python(*) because it is simple and generally unsurprising. I think that '@deco' adds complexity, both for beginners and experienced programmers alike, so I do not see it as an improvement. I am content with "foo = deco(foo)" and various metaclass-based automatic decorations, and see no need for them to be moved into the core. I may be free to not use the '@deco' syntax in 2.4, but that will not prevent me from having to read it in the code of others, or from explaining it to other Python programmers. None of these are new points, so I don't expect a response to them.

If I had any hope that something better than @deco was possible but hasn't been discovered yet, I'd be holding off too. But I don't think that's the case. Every solution proposed has its own set of drawbacks, and the drawbacks aren't getting resolved by waiting.

How many features that everyone hates can Python support? The Zen of Python might state that practicality beats purity, but if it continues to do so, time after time, at some point we're going to look up and wonder where that clean, readable, maintainable programming language we all loved has gone. Jp (*) And I really do like Python. A lot. Of the software I now write, at least 99.9% of it is in Python. The developers of this language _almost_ always show more good sense, intelligence, and skill than most other groups I can think of. You all deserve more praise than you're given.

On Tue, Aug 03, 2004 at 08:25:19PM -0400, Terry Reedy wrote:

Apparently, the question of whether decos are an improvement has at least temporarily been decided in the affirmative.

That's not apparent to me. It seems to me that most people just got sick of the endless debate and tuned out. The people who had urgent need for the feature were probably a bit more persistent. Personally, I don't see the cost vs. benefit of the feature to be compelling. It seems there is not a lot you can do with a decorator without resorting to ugly hacks (e.g. messing with bytecode). returning-to-tuned-out-mode-ly yr's Neil

[Andrew McNamara]

It's also entirely possible that "most people" were happy with the way things were going and chose not to post...

I was indeed "happy enough".

My personal opinion is that the pie syntax is acceptable,

Ditto. It does simple things quite well, and it makes it obvious they're being done. Then again, I like "print >> fileobj" too <0.7 wink>.

and the continuing debate is doing harm

We're in no danger of running out of pixels.

and providing no enlightenment.

That's for sure. In the old days, Guido would Pronounce, and we'd all bite our tongues (although not necessarily each his own). The less time Guido can make for Python, the more important becomes graceful capitulation.

In general, I predict most Python code will continue to be blissfully unadorned with decorators.

Right. Outside the test suite, there are very few uses of classmethod, staticmethod and property in the standard library. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

An advantage of the @ syntax is than it makes the complexiity separable and ignorable for someone learning/teaching basic Python,

I don't agree that they are ignorable. People will see them in code and will *have* to understand what they mean. Given sme examples I've seen here, this will sometimes be a significant chalenge.

There are a number of "advanced" features in python that could be a barrier to understanding for new users, but which are rarely seen in practice - metaclasses being the classic example. I imagine these features are rarely seen because most people don't understand them, and those that do have the good taste not to use them gratuitously. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

...

There are a number of "advanced" features in python that could be a barrier to understanding for new users, but which are rarely seen in practice - metaclasses being the classic example.

I imagine these features are rarely seen because most people don't understand them, and those that do have the good taste not to use them gratuitously.

This feature won't be rarely used. Class and static methods and properties are created fairly often, certainly far more often than meta classes.

Sorry, what I meant was advanced uses of decorators will be rare. Certainly "staticmethod" and "classmethod" will be common. It's a personal opinion, but I find @staticmethod easier to parse visually than the current situation. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

Bob Ippolito wrote:

...

Your proposal would make decorators nearly as bad as the current pre-2.4 situation. This is REALLY ugly:

foo = decorator(....) @foo def fun(....): ...

I agree, that's really ugly. I wasn't proposing that. I proposed not allowing decorator arguments. Without decorator arguments, a much simpler syntax can be used, as in: def classmethod foo(...): ... Or, alternatively, I'm proposing allowing the above simpler syntax when arguments are not needed and allowing the pie syntax to handle more complex cases. The original motivation for decirators was to deal with things like properties and specialized methods. I'd like to see these cases handled well. I think these are going to be the most common cases and the cases that new Python programmers are most likely to come accross. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

On Wed, 2004-08-04 at 09:46, Jim Fulton wrote:

I don't agree that they are ignorable. People will see them in code and will *have* to understand what they mean. Given sme examples I've seen here, this will sometimes be a significant chalenge.

I think most of the cognitive challenge for decorators will be in understanding the side-effects of the decorator, not in understanding the syntax for wrapping a function in a decorator. That won't change by using some other (or no new) syntax -- the Python 2.3. spelling would be just as hairy. -Barry

On Wed, 04 Aug 2004 09:46:55 -0400, Jim Fulton <jim@zope.com> wrote:

Terry Reedy wrote:

....

...

Decorators will add complexity, regardless of syntax.

Especially in their current over-powered form. If decorators didn't have to take arguments, then you could use a syntax like:

def classmethod foo(cls, ...): ...

This does add complexity, but I think it enhances understandability.

I understand why some people want to be able to pass arguments to decorators, but frankly, I find many of the examples of doing so down right scary.

I wonder if we should have add a note to the Python style guide that forbids such things from the standard library <0.2 wink>.

...

An advantage of the @ syntax is than it makes the complexiity separable and ignorable for someone learning/teaching basic Python,

I don't agree that they are ignorable. People will see them in code and will *have* to understand what they mean. Given sme examples I've seen here, this will sometimes be a significant chalenge.

It certainly depends on the use. I don't think @classmethod will be any more or less confusing than the current practice of calling foo = staticmethod(foo) after the def. I agree that more elaborate uses may be confusing; then again, I suspect that more elaborate code will be confusing regardless of how it uses decorators. Jeremy

Guido van Rossum <guido@python.org> writes:

...

...

...

I would think the fact that the '[decorators]' syntax can be implemented in pure Python (no changes to the interpreter) for existing Python versions would give more weight to it.

Can it? I must've missed that. It sure sounds like an incredible hack -- how to you prevent the default behavior that the list of decorators is thrown away by the interpreter?

See http://article.gmane.org/gmane.comp.python.devel/60542 I'm really surprised you missed it, and was likewise disappointed that it wasn't included in the weekly (?) summary. It seemed like a pretty significant development to me. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

The one thing I do dislike about the new addition is the arbitrary (?) restriction on expression allowed in a decorator. From Grammar.. decorator: '@' dotted_name [ '(' [arglist] ')' ] That seems quite odd to me. Is there some reason it couldn't be: decorator: '@' test ?

The first patch on SF actually had '@' test, and I requested that it be changed. Most things that are 'test' but not 'dotted_name' optionally followed by an argument list don't make sense as decorators; if you really have a need to write @ foo or bar def f(): ... you can write deco = foo or bar @deco def f(): ... --Guido van Rossum (home page: http://www.python.org/~guido/)

Skip Montanaro wrote:

I'll be happy to update PEP 318 to conform to the pie decorator syntax. Can someone describe the application order? If I have

@ decorator1 @ decorator2 def func(...): ...

Which decorator is applied first to the function object, decorator1 or decorator2?

func = decorator1(decorator2(func)) I hope to check in a revised 318 this afternoon. Anthony

On Aug 2, 2004, at 9:05 PM, Guido van Rossum wrote:

The first patch on SF actually had '@' test, and I requested that it be changed. Most things that are 'test' but not 'dotted_name' optionally followed by an argument list don't make sense as decorators; if you really have a need to write

@ foo or bar def f(): ...

you can write

deco = foo or bar @deco def f(): ...

An even better workaround is to write: def d(arg): return arg @d(foo or bar) def f(): ... However, it seems as if this restriction creates a new class of expression for no gain. It is true enough that *most* python expressions aren't useful to write after a @, but that's also true of a lot of other places an expression can be used (e.g. before a () of a function call. A list comprehension can never result in a callable object. An arithmetic operation usually won't.). The only real necessary restriction on the @ operator is that its argument be callable and take a single argument. Many expressions could return a callable object. Why not let them? Is it really worth having a special case just to SyntaxError expressions that sometimes won't result in an appropriate callable? Things someone might want to do, ordered roughly from most reasonable to least reasonable ;) @foo().bar() @foo or bar @mydecorators['foo'] @lambda f: foo(f) or bar(f) Why disallow these forms? It seems quite difficult, especially, to explain why the first one does not, or should not, work. James

James Y Knight wrote:

[snip]

An even better workaround is to write: def d(arg): return arg

@d(foo or bar) def f(): ...

However, it seems as if this restriction creates a new class of expression for no gain. It is true enough that *most* python expressions aren't useful to write after a @, but that's also true of a lot of other places an expression can be used (e.g. before a () of a function call. A list comprehension can never result in a callable object. An arithmetic operation usually won't.).

The only real necessary restriction on the @ operator is that its argument be callable and take a single argument. Many expressions could return a callable object. Why not let them? Is it really worth having a special case just to SyntaxError expressions that sometimes won't result in an appropriate callable?

Things someone might want to do, ordered roughly from most reasonable to least reasonable ;) @foo().bar() @foo or bar @mydecorators['foo'] @lambda f: foo(f) or bar(f)

Why disallow these forms? It seems quite difficult, especially, to explain why the first one does not, or should not, work.

James

For what it's worth, I agree with James completely. Jp

On Aug 2, 2004, at 12:54 PM, Raymond Hettinger wrote:

...

...

The only real necessary restriction on the @ operator is that its argument be callable and take a single argument. Many expressions could return a callable object. Why not let them? Is it really worth having a special case just to SyntaxError expressions that sometimes won't result in an appropriate callable?

Things someone might want to do, ordered roughly from most reasonable to least reasonable ;) @foo().bar() @foo or bar @mydecorators['foo'] @lambda f: foo(f) or bar(f)

Why disallow these forms? It seems quite difficult, especially, to explain why the first one does not, or should not, work.

James

For what it's worth, I agree with James completely.

Jp

I concur.

Raymond

Ditto. -bob

On Tue, 2004-08-03 at 05:17, James Y Knight wrote:

Things someone might want to do, ordered roughly from most reasonable to least reasonable ;) @foo().bar() @foo or bar @mydecorators['foo'] @lambda f: foo(f) or bar(f)

Why disallow these forms? It seems quite difficult, especially, to explain why the first one does not, or should not, work.

I don't think any of these seem that likely in practice, and if you do want them, it is not hard to say: # Hopefully some comment here explaining the reason why you're # doing this. And a more meaningful name than foo_or_bar! foo_or_bar = foo or bar @foo_or_bar def func(): pass I would say leave it like this for now. It is easy to relax the restriction later if there's a demand. Mark

BTW, here's another use case I haven't seen suggested yet: def substitution_for(regexp): sub = re.compile(regexp).sub return lambda func: lambda text: sub(func,text) e.g. (example from some recent code of mine, redone as a decorator) @substitution_for('[a-z][A-Z]') def camelspace(m): """Insert space into camelcase regexp match.""" return m.group(0)[0]+' '+m.group(0)[1:] ... info.title = camelspace(info.basefilename) Instead my current code defines a separate variable for the re.compile().sub object, then later has to put this variable together correctly with the undecorated camelspace function to perform the substitution. Using decorators allows the sub and replacement function to become merged into a single object with a more intuitive API. (Also, my actual code uses more than one of these substitutions...) I don't generally prefer to use explicit lambdas but expanding them out with names makes the definition of substitution_for a lot longer without much readability gain in my eyes. -- David Eppstein Computer Science Dept., Univ. of California, Irvine http://www.ics.uci.edu/~eppstein/

On Aug 3, 2004, at 10:40 AM, Guido van Rossum wrote:

I have a gut feeling about this one. I'm not sure where it comes from, but I have it.

I guess I have the opposite gut feeling: special classes of expressions in the grammar are always bad. Michael Chermside's message said this better than I can. Heck, even 'except' can take an arbitrary expression as an argument! Now I'm damn sure I'd never write code that says except FooError and f() or MyErrors[5]: because that would be insane, but I'm glad python doesn't restrict the argument to some strange subset of expressions that someone decided were approved! Consistency of grammar is a key feature, even when it allows users to write insane programs.

It may be that I want the compiler to be able to recognize certain decorators.

If I understand what you're saying, it seems that you're thinking the @ form should be more a static declaration, not a dynamically evaluated expression, so that the compiler can reason about it without running code. However, that isn't the case even now, and if it was, it wouldn't fit in with the rest of the python language very well. Definitely an interesting idea, but I think it would require the @'s argument to be a compile-time macro, instead of a run-time expression. And python doesn't have compile-time macros. With @ taking a runtime expression, I don't see how the compiler could possibly do anything with its argument. Nothing can be guaranteed. As a postscript, I must say I am alarmed by some comments others have used to justify the current restricted state. In particular (paraphrased), "it's good because it keeps people from using lambda", and "it's good because it keeps people from doing '@1+1' by mistake. For the first: you hate lambdas so much? They are in python -- arbitrarily forbidding their use in certain expressions because you don't like them is pretty strange. For the second: You want to do type checking via syntax error?! *That* is insanity. And it doesn't even work well (of course). James

David Eppstein wrote:

I'm actually pretty neutral on restricted decorators vs @expression. But @lambda(func): body def func(...): ... is (I think) much more likely to be an abuse of language than the Pythonic way of writing something, so I don't see a lot of harm in preventing it.

And, as someone pointed out, *lifting* a restriction is a heck of a lot easier than imposing one later on (even after only 1 alpha). YouAintGonnaNeedIt and all that. . . Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268

James Y Knight <foom@fuhm.net> writes:

Now I'm damn sure I'd never write code that says except FooError and f() or MyErrors[5]: because that would be insane,

I think class C(random.choice([dict, list])): pass is my favourite example of this :-) Cheers, mwh -- That being done, all you have to do next is call free() slightly less often than malloc(). You may want to examine the Solaris system libraries for a particularly ambitious implementation of this technique. -- Eric O'Dell, comp.lang.dylan (& x-posts)

Greg Ewing wrote:

Well, I'm not. It seems like a decision made in haste.

Maybe I'll get used to it in a few decades, but right now I feel very disappointed. We were *so* close to agreeing on something nice, and then for some reason it all went pear-shaped. :-(

PEP 318 is now nearly 14 months old. There's been a truly staggering number of posts on the subject - and every time it's raised, the same points come up. The only things to have happened recently were the new @ syntax, and PJE's settrace hack. If not now, when? This could drag on for another 14 months, otherwise - and I really don't want to hold up 2.4 final for that long. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

On Mon, 2004-08-02 at 23:51, Guido van Rossum wrote:

And yes, I'm not 100% comfortable with it either. But I know one thing: in this particular case, now is better than never. So I'm picking a solution everybody can hate equally.

(What's wrong with pear-shaped, anyway? It's one of my favorite shapes. :-)

So I conducted a little experiment, converting a bunch of modules that were using Python 2.3 compatible "post-processing" decorator syntax, to pie decorator syntax, so that I could get a more visceral reaction to the decision. For me, the short answer is: +1, and not just because "it's better than nothing". For longer functions, it's a clear win; the '@' stands out nicely, especially with the python-mode.el font-locking support I just added. :) It certainly exhibits all the benefits that motivated adding special syntax for decorators in the first place. For shorter functions, e.g. read-only accessors, my only (very minor) lament is that I can't compact the enter thing down to a single line. My fingers want to do something like: @property def get_bacon(self): return self._bacon but then I tell my fingers to do this instead: @property def get_bacon(self): return self._bacon Or, we'll compromise, my fingers and I, on: get_bacon = property(lambda self: self._bacon) So it's at least a wash there. (Really, pie decorators are better here too, IMO.) My only other minor lament has nothing to do with this specific syntax. I still wish there was a way to decorate multi-arg property()'s, but oh well. All in all, I'm quite happy with this syntax. Thanks Guido! -Barry

On Tue, 03 Aug 2004 14:01:42 -0400, Barry Warsaw <barry@python.org> wrote:

For longer functions, it's a clear win; the '@' stands out nicely, especially with the python-mode.el font-locking support I just added. :) It certainly exhibits all the benefits that motivated adding special syntax for decorators in the first place.

I think this is also a strong point for the '@' syntax. How should an editor know that a list before a method is a decoration and should be highlighted differently? The backwards compatibility to python < 2.4 doesn't count in my opinion, because other features like generators and list-comprehensions also can't be used in older pythons, and the old syntax with the decoration after the method body is still available, so this is no problem. +1 for the pie syntax from my little voice as a user of python Regards, Florian Schulze

On Mon, Aug 02, 2004 at 06:06:09PM +0100, Paul Moore wrote:

Can someone have a look at http://www.python.org/sf/870382 ? I'm not at all sure how to address Martin's comments, and I'd like to see this in Python 2.4, in some form or other.

If you'd like to encourage examination of a bug or patch at the next bug day, you can add them to the 'bugs for examination' section of http://www.python.org/moin/PythonBugDayStatus. No guarantees, of course -- if the bug can only be reviewed by one person, and that person isn't around for the bug day, nothing much can be done -- but at least it'll draw some attention to them. (Will send an announcement for the next bug day shortly...) --amk

Guido van Rossum wrote:

Did the relative import syntax that was approved ever get checked in? I thought it was on the list of things to land in 2.4?

Nope. The 2.4 release schedule PEP now lists this as lacking someone to drive it. I won't/can't spend time on it. So, if one of the people who wants it could step forward, that'd be excellent. Even if it's just to do the multi-line import syntax of from foo import ( bar, baz, bar2, baz2, bar3, baz3 ) -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Dima Dorfman wrote:

I took a stab at implementing this (the easy part of PEP 328--parens around import lists). Are the exact semantics of what's allowed documented somewhere?

I think this syntax was originally proposed here: http://mail.python.org/pipermail/python-dev/2003-December/041072.html which mentions that since "a, b = c" is already equivalent to "(a, b) = c", so " in mind, I'd hand out the following scores (0=don't really care):

import (os, sys)

+0

from sys import (stdin, stdout, stderr)

+1

import (os)

+0

from sys import (*)

-0

from (sys) import stdin

-0

import (os), (sys)

-1

import (os,)

-1 </F>

...

The 2.4 release schedule PEP now lists this as lacking someone to drive it. I won't/can't spend time on it. So, if one of the people who wants it could step forward, that'd be excellent. Even if it's just to do the multi-line import syntax of

from foo import ( bar, baz, bar2, baz2, bar3, baz3 )

I took a stab at implementing this (the easy part of PEP 328--parens around import lists).

Great!

Are the exact semantics of what's allowed documented somewhere?

Not unless it's in the PEP -- it usually takes a stab at implementing something to find what's left unspecified!

PEP 328 mostly talks about relative and absolute imports, and it doesn't specify the exact semantics of where parentheses should be allowed. My patch (attached) accepts

import (os, sys) from sys import (stdin, stdout, stderr) import (os) from sys import (*)

but rejects

from (sys) import stdin import (os), (sys) import (os,)

Should any of those be allowed? Anything that I missed?

This suggests (given the recent criticism of how we've been following the PEP process) that the PEP needs to be updated first. My own gut feeling is that it's not important to allow parentheses on "bare" (from-less) imports, since you can easily repeat the 'import' keyword on a new line; but that it is important on the from...import form, and then only after the "import" keyword. So that would accept your 2nd and 4th example but reject the 1st and 3rd. I guess one can argue about (*); I don't care about it and suggest that it be dropped.

The patch is incomplete in other ways; the docs haven't been updated and neither have the parser module and compile package. If it's decided that it would be okay to include this separately from relative/absolute import support (they're different features, really), I'll complete the patch.

Please do, and please use the SF patch manager to upload the patch. --Guido van Rossum (home page: http://www.python.org/~guido/)

I'd suggest that only ``from foo import (...)`` be permitted (where "..." does not include ``*``). But I don't care much, and I'll update the PEP as soon as it's clear which precise versions are allowed.

This seems to be the consensus so far, so why don't you add all that detail to the PEP. --Guido van Rossum (home page: http://www.python.org/~guido/)

My own gut feeling is that it's not important to allow parentheses on "bare" (from-less) imports, since you can easily repeat the 'import' keyword on a new line;

It might not be strictly necessary, but for consistency I think it should be alllowed. By the way, rather than parentheses, I would prefer to be able to simply end the line with a comma and continue on to the next line: from Bubble import foo, spam, tomato, grail, arthur, camembert Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

On Mon, Aug 02, 2004, Guido van Rossum wrote:

Did the relative import syntax that was approved ever get checked in? I thought it was on the list of things to land in 2.4?

Sorry, I've been slacking on driving it. What it really needs right now is someone to implement it. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." --reddy@lion.austin.ibm.com

Jim Fulton <jim@zope.com> writes:

...

In the mean time, can anyone give a pointer to where this was proposed or discussed?

There are two slides in my EuroPython keynote (http://www.python.org/doc/essays/ppt/euro2004/euro2004.pdf) (pages 10 and 11) that mention it. It was discussion during and after this keynote that made me change my mind -- first I accepted that [deco] before def was unacceptable, then I found that @deco was not universally hated (Fredrik Lundh even liked it). --Guido van Rossum (home page: http://www.python.org/~guido/)