RE: [Python-Dev] Re: decorators and 2.4
Jeff Bone writes:
IMHO, making decorators functions is a bad idea. Decorators are metadata about the function to which they refer.
Bob Ippolito responds:
Uh, the WHOLE POINT we want this is to have side-effects. If it doesn't make the function act in a different way, it might as well live > in the doc string or something.
The most wanted use cases are all function transformations, not the setting attributes on function objects.
And back to Jeff:
Declarative vs. imperative. Design-time *definitional* modification of > behavior vs. runtime. I don't really think you want side-effects in the literal sense. E.g. "classmethod" isn't a side-effect, it's a definitional thing. If you really want side-effecting operations on functions, you've already got that given higher-order / first-class functions.
Just one more opinion here, but I'm with Bob on this one. Sure, many of the uses I envision are "declarative" in the sense that they just associate metadata with the function. But many other uses I envision would alter the function's behavior. Yes, of COURSE it can all be achieved by applying higher-order operations to Python's first-class functions -- in my mind, that's what decorators ARE... a cleaner syntax allowing easier use of higher order functions. -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it.
Just one more opinion here, but I'm with Bob on this one. Sure, many of the uses I envision are "declarative" in the sense that they just associate metadata with the function. But many other uses I envision would alter the function's behavior.
I'm really curious what these uses might be. Folks keep saying this kind of thing, but when you dig in IMHO most of these actually seem to be design-time things, statements *about* the intent or use of the function (relative to the language runtime itself, e.g. classmethod and so forth, or relative to a framework, such as registering a function, or relative to some tool, e.g. for extended documentation purposes or type-checking, etc.) rather than e.g. runtime modifications to its behavior. I wouldn't keep pushing on this except that I'm entirely puzzled about what these legitimate "runtime" applications might be that aren't already more easily and clearly accomplished with existing mechanisms: arguments, lexical scope, and higher-order functions. I'm all for decorations as a mechanism for declaring things *about* the function as above, but I'm at a loss re: rationale for "decorations" that do more than declaratively and immutably *decorate* the function. Given that my distinctions between declarative and imperative uses of decorators and design- vs. run-time seem to have confused more than they've illuminated, perhaps a series of questions, scenarios, and thoughts might help clarify the areas of my concern. (1) Does a function have access to its own decorators? I expect the answer here should be "yes." We should be careful, though, that this does not introduce a mutable container for state. We've already got one such thing: object instances. We don't need another. (2) Do decorators define literal constant values at design-time or do they define a scope of mutable bindings? The PEP gives this example: def mymethod(f) [attrs(versionadded="2.2", author="Guido van Rossum")]: ... That seems fine, but what about this: x = "Guido van Rossum" def mymethod(f) [attrs(versionadded="2.2", author=x)]: Danger, Will Robinson! ;-) Even worse: x = "Guido van Rossum" def mymethod(f) [attrs(versionadded="2.2", author=x)]: oldAuthor = someHandleToMyDecorators.atts.author someHandleToMyDecorators.atts.author = "Eric Idle" IMHO, the RHS of an "assignment" in a decorator should only be some value that can be known at "compile" time, which probably means it's either a reference to a singly-assigned constant value or a literal. If you allow arbitrary variables or attributes in decorators to be assigned, then you limit the utility of decorators for many of their intended purposes such as e.g. type-checking. Furthermore, if you can "assign" a decorator's attribute more than once --- particularly if the decorated thing (function, method, object, whatever) can access those bindings --- then you've simply re-invented objects with an alternate and idiosyncratic syntax. (3) Should it be possible to conditionally evaluate decorators based on run-time state? The following, IMHO, is truly scary and inhibits many of the intended uses of decorators. This shouldn't be legal, ever: if foo = 3: [someDecoratorThatModifiesBarsRuntimeBehavior] else: [someOtherDecorator] def bar(...): ... If you're going to do that, just use callables, nested scopes, first-order functions, currying, etc. jb
On Mon, 2004-06-28 at 12:50, Jeff Bone wrote:
Just one more opinion here, but I'm with Bob on this one. Sure, many of the uses I envision are "declarative" in the sense that they just associate metadata with the function. But many other uses I envision would alter the function's behavior.
I'm really curious what these uses might be.
Here's one of my use case. In the Mailman3 implementation of the BerkeleyDB backend storage, I need to set up Berkeley transactions around various calls. It's programmatically possible to set up an explicit transaction to group several operations together, but it's also possible to wrap a single operation in a single transaction. Also, the Berkeley transactions must be wrapped in try/excepts so that if any Python level exception occurs, the transaction can be properly aborted. There's enough complexity in the implementation, and similarity in the use, that I wanted to encapsulate all that in a 'txnprop' decorator. I currently use the 'call-after-def' wrapper idiom that's possible in Python 2.3 today, but I want to use decorators for this, especially because I sometimes want to combine the txnprop wrapper with other decorators. Here's a pointer to some of the code: http://cvs.sourceforge.net/viewcvs.py/mailman/mailman3/src/mailman/berkeleydb/bdblist.py?rev=3.2&view=auto The txnprop() descriptor guarantees that self.txn is bound to a transaction inside the method call. -Barry
Okay, this is a great and meaty example, thanks Barry. It's the first one that I've seen that is clearly not a purely declarative thing at least in the abstract. What would the proposed decorator version of this look like? On Jun 28, 2004, at 12:19 PM, Barry Warsaw wrote:
Here's a pointer to some of the code:
http://cvs.sourceforge.net/viewcvs.py/mailman/mailman3/src/mailman/ berkeleydb/bdblist.py?rev=3.2&view=auto
The txnprop() descriptor guarantees that self.txn is bound to a transaction inside the method call.
-Barry
On Mon, 2004-06-28 at 13:27, Jeff Bone wrote:
Okay, this is a great and meaty example, thanks Barry. It's the first one that I've seen that is clearly not a purely declarative thing at least in the abstract. What would the proposed decorator version of this look like?
Depends on which syntax Guido chooses <wink>. I'd be able to remove all the "func = txnprop(_func)" trailers, and do something like: [txnprop] def create_list(self, listid): # ... or def create_list(self, listid) [txnprop]: # ... or @txnprop def create_list(self, listid): # ... All of which seem much more clear than what I currently have to do. -Barry
On Jun 28, 2004, at 12:48 PM, Barry Warsaw wrote:
On Mon, 2004-06-28 at 13:27, Jeff Bone wrote:
Okay, this is a great and meaty example, thanks Barry. It's the first one that I've seen that is clearly not a purely declarative thing at least in the abstract. What would the proposed decorator version of this look like?
Depends on which syntax Guido chooses <wink>.
C'mon, Barry, work with me here. ;-) I'm not as interested in the specific syntax as what the syntax implies about the semantics.
I'd be able to remove all the "func = txnprop(_func)" trailers, and do something like:
[txnprop] def create_list(self, listid): # ...
All of which seem much more clear than what I currently have to do.
Granted. But let's look at the above. Now, you're intimately familiar with the current implementation of txnprop, so you think of the above in terms of its literal side-effecting behavior. But looking at it as a consumer of the interface --- simply as a user reading the code, or as some automated tool which doesn't understand and doesn't necessarily NEED to understand the effect of txnprop, that looks awfully declarative --- doesn't it? You're in essence saying "the following is a transactional function." So on closer examination it looks like this application can still be given purely declarative semantics from the API consumer's perspective. That's not going to screw up e.g. typing --- particularly not if e.g. the implementation of txnprop is further decorated to indicate what exceptions are thrown, etc. It doesn't so much *modify* the behavior of create_list from the API consumer's (human or otherwise) perspective as it does *declare* what it is. jb
On Mon, 2004-06-28 at 14:06, Jeff Bone wrote:
Granted. But let's look at the above. Now, you're intimately familiar with the current implementation of txnprop, so you think of the above in terms of its literal side-effecting behavior. But looking at it as a consumer of the interface --- simply as a user reading the code, or as some automated tool which doesn't understand and doesn't necessarily NEED to understand the effect of txnprop, that looks awfully declarative --- doesn't it? You're in essence saying "the following is a transactional function."
Which is why early on, I favored decorator-after-def syntax, like so: def [txnprop] create_list(self, listid): because it read more English-like: "Define a txnprop called create_list with arguments self and listid". But I'm definitely not going to give that poor deceased equine even one more whack.
So on closer examination it looks like this application can still be given purely declarative semantics from the API consumer's perspective. That's not going to screw up e.g. typing --- particularly not if e.g. the implementation of txnprop is further decorated to indicate what exceptions are thrown, etc. It doesn't so much *modify* the behavior of create_list from the API consumer's (human or otherwise) perspective as it does *declare* what it is.
Perhaps, but I haven't followed this thread close enough to know what point you want to make. For me, I want the feature with semantics that are described in the PEP, which are basically equivalent to what the explicit wrappers can do in Python today. I think there's fairly widespread agreement on the semantics, and the only questions open are which syntax Guido will chose, and what version of Python he'll want to add this to. Given the contenders still standing, I don't care about the former, but I want the latter to be "Python 2.4". that's-just-me-though-ly y'rs, -Barry
At 11:50 AM 6/28/04 -0500, Jeff Bone wrote:
I wouldn't keep pushing on this except that I'm entirely puzzled about what these legitimate "runtime" applications might be that aren't already more easily and clearly accomplished with existing mechanisms: arguments, lexical scope, and higher-order functions.
I've already said this several times, and PEP 318 says it too: PEP 318 seeks a *syntax* that allows those "existing mechanisms" to be invoked at the *beginning of a function definition*. That is all. Period. Finito. End of story. So, to talk about applying restrictions to what behavior is allowed for decorators, is to completely miss the point of PEP 318. Please re-read PEP 318's "Motivation" section.
On Jun 28, 2004, at 3:11 PM, Phillip J. Eby wrote:
So, to talk about applying restrictions to what behavior is allowed for decorators, is to completely miss the point of PEP 318. Please re-read PEP 318's "Motivation" section.
Phil, with all due respect, there's no need to get peevish. I've read PEP 318 several times, and I think you're missing my point *about the PEP.* You'll note that I'm not disagreeing with the desirability of decorators nor any of the use cases in the PEP nor any given proposed syntax. I'm concerned that treating this as simple syntactic sugar w/o a bit more precision about impact on evaluation and on scope might actually impair the ability to use decorators for (at least some of) their intended purposes. The PEP only implies semantics (by saying "it's just syntactic shorthand for things like...") --- and that implied semantics is in fact inconsistent with at least one of the implied desiderata. Note too that an explicit design goal of the PEP is: * not make future extensions more difficult If you allow conditional evaluation or parameterization of decorators w/o some consideration about their impact on static type analysis --- as in the nightmare scenario I offered previously --- AND if you believe that adding some optional static typing and type analysis to Python in the future is at least possibly desirable, then it leads one to consider whether additional constraints *might* be in order. E.g. if evaluation of decorators proceeds "at runtime" then we can forget all ideas of using them for any sort of static code analysis which, if you noticed, *is* one of the examples from the PEP. (That would be the "Examples" section, ahem. ;-) jb Footnote, aside, more of an anecdotal cautionary tale than anything else: Today the following happens:
def bar(): ... x = 3 ... foo() ... def foo(): ... print x ... bar() Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 3, in bar File "<stdin>", line 2, in foo NameError: global name 'x' is not defined
This is as it should be. Decorators shouldn't have an impact on variables that occur free in decorated bodies. This is an example of the lack of clarity in the spec; you can argue that it's implicit that this kind of behavior should be preserved, but IMHO that's more a function of some implied understanding of the implementation than it is any specification of something that the language should or should not do. (And such problems lead early Lisp to have effed up scoping rules for so long that old Lisp hands can't even agree on whether it was a feature or a bug. ;-)
At 04:29 PM 6/28/04 -0500, Jeff Bone wrote:
If you allow conditional evaluation or parameterization of decorators w/o some consideration about their impact on static type analysis --- as in the nightmare scenario I offered previously --- AND if you believe that adding some optional static typing and type analysis to Python in the future is at least possibly desirable, then it leads one to consider whether additional constraints *might* be in order.
No, it doesn't. You can already make weird decorators that disassemble the bytecode of the function and create a new function object, for example. PEP 318 does nothing to change that fact. Thus, attempting to define restrictions for them is moot, as far as anybody attempting to create program analysis tools -- they already have to deal with this dynamism today, and restricting PEP 318 simply won't help.
E.g. if evaluation of decorators proceeds "at runtime" then we can forget all ideas of
For all practical purposes, Python doesn't have any time *other* than runtime. Any analysis tool that doesn't recognize this is going to have a hard time making sense out of Python code. As a simple example: try: from lib1 import SomeClass except ImportError: from lib2 import SomeClass class MyClass(SomeClass): """My stuff here""" So, what class does 'MyClass' inherit from, here? Note that this is perfectly legal Python, and I've certainly written the equivalent of it more than once or twice. So if this be the stuff of nightmares, please don't try to wake me up. :)
using them for any sort of static code analysis which, if you noticed, *is* one of the examples from the PEP. (That would be the "Examples" section, ahem. ;-)
To which example are you referring? Type constraints? That's not static code analysis, that's code generation. Interfaces? That's registration. Neither has anything to do with static code analysis, AFAICT.
This is as it should be. Decorators shouldn't have an impact on variables that occur free in decorated bodies. This is an example of the lack of clarity in the spec;
If that's true, then the Python language reference is unclear because it doesn't say that you mustn't use higher-order functions to change the behavior of other functions.
On Jun 28, 2004, at 9:05 PM, Phillip J. Eby wrote:
At 04:29 PM 6/28/04 -0500, Jeff Bone wrote:
If you allow conditional evaluation or parameterization of decorators w/o some consideration about their impact on static type analysis --- as in the nightmare scenario I offered previously --- AND if you believe that adding some optional static typing and type analysis to Python in the future is at least possibly desirable, then it leads one to consider whether additional constraints *might* be in order.
No, it doesn't.
Yes, it does --- as the examples offered indicate.
You can already make weird decorators that disassemble the bytecode of the function and create a new function object, for example. PEP 318 does nothing to change that fact.
True, but irrelevant unless you happen to be strategically myopic and slightly programming-language illiterate. Tell me, Phil --- what is your damage here? I believe we both want the same outcome. So why are you getting so aggressive (or defensive, I can't tell which) about the route to the particular, and shared, outcome? As surely as I'm convinced about the need for and worthiness of certain considerations about static type-checking and the future of this language, I'm willing to grant that you're equally concerned about something else, even though I don't yet know what. Why can't we have the discussion on that factual basis instead of bogus third-hand references to the questionable intent (though somewhat clear, but still questionable relative to desiderata) of a spec you didn't write? The only legit explanation I can think of is that *you know how to implement the PEP as you understand it now, but fail to understand the longer-term implications of said interpretation of PEP / implementation." Take the time to connect w/ me or others on these concerns; I think you will find them not entirely out of sync. I may not have posted here much over the years, but I'm sure both I and countless others have put in the time to get the hearing for our concerns w/o being swatted down w/ an inconclusive spec like and ignorant school-marm swatting an overly-inquisitive 2nd-grader.
Thus, attempting to define restrictions for them is moot,
No, it's not -- as demonstrated previously.
as far as anybody attempting to create program analysis tools -- they already have to deal with this dynamism today, and restricting PEP 318 simply won't help.
What a completely BS response. Yet --- true, but only if one accepts a particular and rather simple-minded and overly-literal interpretation of the ***GOALS*** of PEP 318, driven by hackers that are inclined to hack certain changes into THE (and that in itself is an indictment) implementation of the language --- changes that that are very lightly implied by the PEP into the interpreter, rather than considering the longer-term goals and objectives of both the PEP and the language. And THAT IS MY POINT. jb CTO, Deepfile
On Tuesday 2004-06-29 08:04, Jeff Bone wrote: [Philip Eby:]
You can already make weird decorators that disassemble the bytecode of the function and create a new function object, for example. PEP 318 does nothing to change that fact.
[Jeff:]
True, but irrelevant unless you happen to be strategically myopic and slightly programming-language illiterate. Tell me, Phil --- what is your damage here? I believe we both want the same outcome. So why are you getting so aggressive (or defensive, I can't tell which) about the route to the particular, and shared, outcome?
Excuse me, but when you're implying that Philip is "strategically myopic and slightly programming-language illiterate" and asking "what is your damage?" and calling his comments "bullshit" and likening him to "an ignorant school-marm", you are not in a great position to take the moral high ground about being either "aggressive" or "defensive".
As surely as I'm convinced about the need for and worthiness of certain considerations about static type-checking and the future of this language, I'm willing to grant that you're equally concerned about something else, even though I don't yet know what. Why can't we have the discussion on that factual basis instead of bogus third-hand references to the questionable intent (though somewhat clear, but still questionable relative to desiderata) of a spec you didn't write?
This topic has already been extensively discussed in python-dev. It is not, in fact, questionable whether the intent of the spec is to permit a broader range of operations on functions than you think desirable. It is.
The only legit explanation I can think of is that *you know how to implement the PEP as you understand it now, but fail to understand the longer-term implications of said interpretation of PEP / implementation."
Please consider the possibility that other people may understand your concerns but disagree with them. Being unable to read Philip's mind, I don't know whether that's his situation, but it certainly seems plausible to me. * Let's rewind a bit here. Unless my mail archives are broken, your first entry into this discussion was when you declared that "Decorators are metadata about the function to which they refer.". Not "it would be preferable to restrict decorators to [etc]" -- not, that is, a statement about what sort of decoration ought to be done, or ought to be permitted, but a statement about what decorators *are*. Now, doubtless analytic philosophers could spend hours arguing about what it actually means to make such a statement when decorators aren't currently part of the language spec. But it seems reasonable to me to take their meaning to be defined by some combination of (1) the PEP that proposes their introduction, (2) the discussion that preceded and followed the writing of that PEP, and (3) the actual implementations people have produced. And, according to all three of those, your statement about what decorators are is, simply, not true. And as long as you're making statements about what decorators *are* it seems entirely appropriate for Philip, or anyone else, to point you to the nearest thing we have to a definition of decorators in Python when you say something that just ain't so. Now, certainly, it's also reasonable to discuss what decorators *should* be, and I think that's what you actually intend in this thread. So, your main argument is that decorations should be "declarative". Unfortunately, it isn't clear to me quite what you mean by "declarative"; from your response to Michael Chermside's and Barry Warsaw's examples, it appears that you want to define it at a rather high level of abstraction -- but at a high enough level of abstraction (doubtless higher than you have in mind) *anything* can be construed as "declarative" :-), and it's not obvious to me where your boundaries are. So, let me make a proposition that I haven't yet seen anything to refute: It is good for decorators to be, in some sufficiently high-level sense, "declarative". However, at that level, the meaning of "declarative" is broad enough that any mechanism that enables the full range of desirable, "declarative" decorators will also enable the "non-declarative" uses that you deplore. There is no Python-level restriction (such as "no functions" or "nothing that looks like functions") agree are legitimate, and prevent (or even discourage) the uses of which you disapprove. I'm willing to guess that you don't agree with that proposition. Would you care to offer some evidence against it? Here are a couple of more specific comments you've made: "Using the same syntax for decorators as for functions conceptually weakens this distinction and encourages misuse of decorators, even to the point of potentially encouraging (or implying) side-effects as a result of decoration." It's not clear to me what you mean by "using the same syntax for decorators as for functions", and the context in the thread doesn't help. Could you explain in what way the existing proposals use the same syntax for decorators as for functions, and how you think that should be changed? "To the extent that decorators either (a) are evaluated conditionally based on runtime state, or (b) modify and are in turn modified by runtime state, decorators as mutable environments or with true side effects make a mess of that." So presumably you would like the decoration mechanism to prevent, or at least discourage, decorators that "modify and are in turn modified by runtime state". But at the implementation level, *everything* in Python is runtime state. Changing a function's docstring is a modification of runtime state. Presumably, then, you have some more abstract notion of what is "runtime state" and what isn't. But I can't tell what it is. Can you explain? -- g
On Jun 29, 2004, at 5:50 AM, Gareth McCaughan wrote:
Excuse me, but when you're implying that Philip is "strategically myopic and slightly programming-language illiterate" and asking "what is your damage?" and calling his comments "bullshit" and likening him to "an ignorant school-marm", you are not in a great position to take the moral high ground about being either "aggressive" or "defensive".
Sorry, yes, you're correct. I apologize to Phil and the list. It's just rather frustrating to get beat all to hell (and told patronizingly to "read" a freakin' PEP I've read, repeatedly, before even speaking up) for no apparent reason just for raising what IMHO are rather legitimate concerns. My bad, I'll go back to lurk and if this particular feature turns out to be the giant hairball it appears to be I won't say anything. :-/ jb
At 07:55 AM 6/29/04 -0500, Jeff Bone wrote:
Sorry, yes, you're correct. I apologize to Phil and the list. It's just rather frustrating to get beat all to hell (and told patronizingly to "read" a freakin' PEP I've read, repeatedly, before even speaking up) for no apparent reason just for raising what IMHO are rather legitimate concerns.
I merely suggested that you review the PEP and discussions, because the concerns you were trying to bring up have been rehashed on Python-Dev about six or seven times over the lifetime of the PEP, to the best of my recollection. Also if I recall correctly, Guido weighed in on at least one or two of those threads pointing out that there was not going to be any practical restriction put on decorators, even if some uses might be considered a bit gauche. I'm sorry if you took my suggestion as implying anything more than that. However, such was not my intention.
Jeff Bone <jbone@place.org> writes:
On Jun 29, 2004, at 5:50 AM, Gareth McCaughan wrote:
Excuse me, but when you're implying that Philip is "strategically myopic and slightly programming-language illiterate" and asking "what is your damage?" and calling his comments "bullshit" and likening him to "an ignorant school-marm", you are not in a great position to take the moral high ground about being either "aggressive" or "defensive".
Sorry, yes, you're correct. I apologize to Phil and the list. It's just rather frustrating to get beat all to hell (and told patronizingly to "read" a freakin' PEP I've read, repeatedly, before even speaking up) for no apparent reason just for raising what IMHO are rather legitimate concerns.
It's probably easy to underestimate the level of frustration around this issue. There have been THOUSANDS of posts to python-dev on this or close-by topics in the last 18 months. People who appear to be trying to take the discussion back to areas where there was perceived consensus aren't going to be popular.
My bad, I'll go back to lurk and if this particular feature turns out to be the giant hairball it appears to be I won't say anything.
This is why the subject is so delicate: it requires seeing into the future, and that's hard. If all decorators end up being used for is classmethods then it almsot doesn't matter which of the proposed syntaxes get used. If they become common, then it's a bigger issue. FWIW, I think the idea of limiting what a descriptor can be or do is insane (as well as possibly impossible). If the issue is that scary, then we shouldn't add any syntax at all. Cheers, mwh -- Just getting something to work usually means writing reams of code fast, like a Stephen King novel, but making it maintainable and high-quality code that really expresses the ideas well, is like writing poetry. Art is taking away. -- Erik Naggum, comp.lang.lisp
participants (6)
-
Barry Warsaw -
Chermside, Michael -
Gareth McCaughan -
Jeff Bone -
Michael Hudson -
Phillip J. Eby