Has anyone else ever thought that it might be useful to access a function object from within the call? I've come across this situation a few times recently, and thought it would be very useful to be able to do something like, for example: def a_function() as func: print(func.__doc__) It would be useful, for example, to record state between calls, or if the function wants to reuse its own properties (like in the above example). Consider a function which should print the number of times it has been called on every call def counter(add) as func: if not hasattr(func, 'count'): func.count = 0 func.count += 1 print(func.count) This could also be implemented using classes, i.e. class Counter: def __call__(self, add): if not hasattr(func, 'count'): func.count = 0 func.count += 1 print(func.count) counter = Counter() But this is much more clumsy, results in an extra object (the Counter class) and will be quite complicated if counter is a method rather than a function. The reason I've used "as" syntax is that it is consistent with other python statements (e.g. "with" and "except"), wouldn't require a new keyword and is backwardly compatible. Any thoughts? David
On Sat, Aug 6, 2011 at 1:10 AM, David Townshend <aquavitae69@gmail.com> wrote:
Has anyone else ever thought that it might be useful to access a function object from within the call?
Yes: http://mail.python.org/pipermail/python-list/2009-May/1203977.html Cheers, Chris
On Sat, Aug 6, 2011 at 1:16 AM, Chris Rebert <pyideas@rebertia.com> wrote:
On Sat, Aug 6, 2011 at 1:10 AM, David Townshend <aquavitae69@gmail.com> wrote:
Has anyone else ever thought that it might be useful to access a function object from within the call?
Yes:
And also: [Rejected] PEP 3130: Access to Current Module/Class/Function http://www.python.org/dev/peps/pep-3130/ Cheers, Chris
On Sat, 6 Aug 2011 01:19:58 -0700 Chris Rebert <pyideas@rebertia.com> wrote:
On Sat, Aug 6, 2011 at 1:16 AM, Chris Rebert <pyideas@rebertia.com> wrote:
On Sat, Aug 6, 2011 at 1:10 AM, David Townshend <aquavitae69@gmail.com> wrote:
Has anyone else ever thought that it might be useful to access a function object from within the call?
Yes:
And also: [Rejected] PEP 3130: Access to Current Module/Class/Function http://www.python.org/dev/peps/pep-3130/
The new magic super() uses a similar, hidden, compiler-activated hack to work properly:
class C: ... def f(): super ... C.f.__closure__ (<cell at 0x7fe2bb402e98: type object at 0x14ec5e0>,) C.f.__closure__[0].cell_contents <class '__main__.C'>
sup = super class D: ... def f(): sup ... D.f.__closure__
Regards Antoine.
On 8/6/2011 4:19 AM, Chris Rebert wrote:
On Sat, Aug 6, 2011 at 1:16 AM, Chris Rebert<pyideas@rebertia.com> wrote:
On Sat, Aug 6, 2011 at 1:10 AM, David Townshend<aquavitae69@gmail.com> wrote:
Has anyone else ever thought that it might be useful to access a function
[Rejected] PEP 3130: Access to Current Module/Class/Function http://www.python.org/dev/peps/pep-3130/
The first problem with this is that it is three proposals in one. Each should have been considered separately on its own merits. The second problem stems from the first: there is a blanket rejection of all three with no reference to the relative merits of the three different proposals. The __module__ proposal is particularly weak as the only use case given is replacing if __name__ == '__main__': ... with if __module__ is sys.main: ... I would reject the collective proposal just to reject this. The __class__ proposal seems to have been dealt with partly by revisions to super. I have not read it enough to know if anything more is left, but there is not much, if any, demand for more that I have seen. The proposal for access to a function from within the function has two important use cases, mentioned in the PEP. First is to make truly recursive functions. Second is to dependably access function attributes from within the function. (Without that, there are hardly used even a decade after their introduction.) In both cases, the idea is to make the function operate as desired independently of external namespace manipulations (even from outside the module) that the function and its author literally have no control over. Except purely for speed, function attributes could then replace pseudo-parameters with default args. This proposal/desire comes up constantly on python-list, generally with support. I believe it was part of two recent threads. I would like to know if the rejection of the idea so far is a rejection in principle (and if so, why) or a rejection of specifics. -- Terry Jan Reedy
On Sat, Aug 6, 2011 at 2:28 PM, Terry Reedy <tjreedy@udel.edu> wrote:
On 8/6/2011 4:19 AM, Chris Rebert wrote:
On Sat, Aug 6, 2011 at 1:16 AM, Chris Rebert<pyideas@rebertia.com> wrote:
On Sat, Aug 6, 2011 at 1:10 AM, David Townshend<aquavitae69@gmail.com> wrote:
Has anyone else ever thought that it might be useful to access a function
[Rejected] PEP 3130: Access to Current Module/Class/Function http://www.python.org/dev/peps/pep-3130/
The first problem with this is that it is three proposals in one. Each should have been considered separately on its own merits. The second problem stems from the first: there is a blanket rejection of all three with no reference to the relative merits of the three different proposals.
The __module__ proposal is particularly weak as the only use case given is replacing if __name__ == '__main__': ... with if __module__ is sys.main: ... I would reject the collective proposal just to reject this.
The __class__ proposal seems to have been dealt with partly by revisions to super. I have not read it enough to know if anything more is left, but there is not much, if any, demand for more that I have seen.
The proposal for access to a function from within the function has two important use cases, mentioned in the PEP. First is to make truly recursive functions. Second is to dependably access function attributes from within the function. (Without that, there are hardly used even a decade after their introduction.) In both cases, the idea is to make the function operate as desired independently of external namespace manipulations (even from outside the module) that the function and its author literally have no control over. Except purely for speed, function attributes could then replace pseudo-parameters with default args.
This proposal/desire comes up constantly on python-list, generally with support. I believe it was part of two recent threads. I would like to know if the rejection of the idea so far is a rejection in principle (and if so, why) or a rejection of specifics.
+1 Maybe a more straightforward effort would be appropriate if the other ideas sank the function part. Here's my 2c on how to make it work. Of the three code blocks, functions are the only ones for whom the resulting object and the execution of the code block are separate. So a code object could be executing for the original function or a different one that is sharing the code object. Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"? To finish things off, bind to every new code object the function for which it was created, perhaps as "co_func". That way you will always know what function object was called and which one the code object came from originally. No new syntax. One new attribute on code objects. One new implicit name in locals(). Code to add the function object to the code object at definition time. Code to add the [other] function object at execution time. -eric
-- Terry Jan Reedy
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
Eric Snow wrote:
Of the three code blocks, functions are the only ones for whom the
[grammar police] "Whom" is for people. You want "which". [/grammar police] (I don't normally correct people's grammar, but to me, this error broke the flow of the sentence like being hit in the face with a sock with half a brick in it.)
resulting object and the execution of the code block are separate. So a code object could be executing for the original function or a different one that is sharing the code object.
Fairly unusual, but I grant it could happen. I've done it myself :)
Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"?
I'm afraid I can't interpret this (which may be my ignorance rather than your fault). The only guess I can make is based on what you say later: "One new implicit name in locals(). so I presume you mean that the function should see a local variable (perhaps called "me", or "this"?) that is bound to itself. Presumably if a function wants to use that same name as a local, nothing bad will happen, since the local assignment will just override the implicit assignment. But what about code that expects to see a nonlocal or global with the same name? What happens when two functions, sharing the same code object, get called from two threads at the same time? Are their locals independent? For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed. And I say that as somebody who does want a way for functions to know themselves. I don't think that use-case is so important that it should be implicitly added to every function, on the off-chance it is needed, rather than explicitly on demand.
To finish things off, bind to every new code object the function for which it was created, perhaps as "co_func". That way you will always know what function object was called and which one the code object came from originally.
What benefit will this give? Have you ever looked at a code object and said, "I need a way of knowing which function this is from?" If so, I'd love to know what problem you were trying to solve at the time! Code objects don't always get created as part of a function. They can be returned by compile. What should co_func be set to then? Finally, if the function has a reference to the code object, and the code object has a reference to the function, you have a reference cycle. That's not the end of the world now as it used to be, in early Python before the garbage collector was added, but still, there better be a really good use-case to justify it. (Perhaps a weak reference might be more appropriate?) -- Steven
On Sat, Aug 6, 2011 at 11:36 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Eric Snow wrote:
Of the three code blocks, functions are the only ones for whom the
[grammar police] "Whom" is for people. You want "which". [/grammar police]
(I don't normally correct people's grammar, but to me, this error broke the flow of the sentence like being hit in the face with a sock with half a brick in it.)
I aim to please! <wink>
resulting object and the execution of the code block are separate. So a code object could be executing for the original function or a different one that is sharing the code object.
Fairly unusual, but I grant it could happen. I've done it myself :)
Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"?
I'm afraid I can't interpret this (which may be my ignorance rather than your fault).
No, the fault is likely mine. But it seems so clear to me. :)
The only guess I can make is based on what you say later:
"One new implicit name in locals().
so I presume you mean that the function should see a local variable (perhaps called "me", or "this"?) that is bound to itself.
One called __function__ (or the like). A "dunder" name is used to indicate its special nature and limit conflict with existing code. The function that was called would be bound to that name at function execution time. Keep in mind that I am talking about the frame locals, not anything stored on the code object nor on the function object. Not to overdramatize it, but it would happen at the beginning of every call of every function. I don't know what that overhead would be.
Presumably if a function wants to use that same name as a local, nothing bad will happen, since the local assignment will just override the implicit assignment. But what about code that expects to see a nonlocal or global with the same name?
What happens when two functions, sharing the same code object, get called from two threads at the same time? Are their locals independent?
I'm afraid I don't know. I expect that each would get executed in separate execution frames, and so have separate frame locals.
For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed. And I say that as somebody who does want a way for functions to know themselves. I don't think that use-case is so important that it should be implicitly added to every function, on the off-chance it is needed, rather than explicitly on demand.
For me the use case involves determining what function called my function. Currently you can tell in which execution frame a function was called, and thereby which code object, but reliably matching that to a function is not so simple. I recognize that my case is likely not a general one.
To finish things off, bind to every new code object the function for which it was created, perhaps as "co_func". That way you will always know what function object was called and which one the code object came from originally.
What benefit will this give? Have you ever looked at a code object and said, "I need a way of knowing which function this is from?" If so, I'd love to know what problem you were trying to solve at the time!
You caught me! :) I don't already have a use case for this part. I had only considered that without this you could not determine where a code object came from, or if a function had borrowed another's code object. This is certainly only useful in the case that one function is using the code object of another, which we have all agreed is not that common. However, with a co_func I felt that all the bases would be covered.
Code objects don't always get created as part of a function. They can be returned by compile. What should co_func be set to then?
None, since there was no function object created along with the code object. Same with generator expressions.
Finally, if the function has a reference to the code object, and the code object has a reference to the function, you have a reference cycle. That's not the end of the world now as it used to be, in early Python before the garbage collector was added, but still, there better be a really good use-case to justify it.
(Perhaps a weak reference might be more appropriate?)
Good point. Mostly I am trying to look for an angle that works without a lot of trouble. Can't fault me for trying in my own incoherent way. :) -eric
-- Steven
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Sun, Aug 7, 2011 at 3:46 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Sat, Aug 6, 2011 at 11:36 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Eric Snow wrote:
Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"?
I'm afraid I can't interpret this (which may be my ignorance rather than your fault).
No, the fault is likely mine. But it seems so clear to me. :)
The only guess I can make is based on what you say later:
"One new implicit name in locals().
so I presume you mean that the function should see a local variable (perhaps called "me", or "this"?) that is bound to itself.
One called __function__ (or the like). A "dunder" name is used to indicate its special nature and limit conflict with existing code.
Without thinking too much about this I like it.
The function that was called would be bound to that name at function execution time. Keep in mind that I am talking about the frame locals, not anything stored on the code object nor on the function object. Not to overdramatize it, but it would happen at the beginning of every call of every function. I don't know what that overhead would be.
It could be made into a "cell", which is the same way all locals are normally represented. This is very fast. Further the code for it could be triggered by the appearance of __function__ (if that's the keyword we choose) in the function body. I don't really care what happens if people use locals() -- that's inefficient and outmoded anyway. (Note that!)
Presumably if a function wants to use that same name as a local, nothing bad will happen, since the local assignment will just override the implicit assignment. But what about code that expects to see a nonlocal or global with the same name?
That's why a __dunder__ name is used.
What happens when two functions, sharing the same code object, get called from two threads at the same time? Are their locals independent?
I'm afraid I don't know. I expect that each would get executed in separate execution frames, and so have separate frame locals.
The frames are completely independent. They all point to the same code object and under the proposal they will all point to the same function object. I see no problems here except self-inflicted, like using __function__ to hold state that can't be accessed concurrently safely; note that recursive invocations have the same issue. I see it as a non-problem.
For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed.
Right. If it were expected that people would start writing recursive calls using __function__ routinely, in situations where a name reference works, I'd be very unhappy with the new feature. (And if someone wants to make the argument that recursive calls using __function__ are actually better in some way I am willing to filibuster.)
And I say that as somebody who does want a way for functions to know themselves. I don't think that use-case is so important that it should be implicitly added to every function, on the off-chance it is needed, rather than explicitly on demand.
For me the use case involves determining what function called my function. Currently you can tell in which execution frame a function was called, and thereby which code object, but reliably matching that to a function is not so simple. I recognize that my case is likely not a general one.
But it is a nice one. It solves some issues that pdb currently solves by just using a file/line reference.
To finish things off, bind to every new code object the function for which it was created, perhaps as "co_func". That way you will always know what function object was called and which one the code object came from originally.
What benefit will this give? Have you ever looked at a code object and said, "I need a way of knowing which function this is from?" If so, I'd love to know what problem you were trying to solve at the time!
You caught me! :) I don't already have a use case for this part. I had only considered that without this you could not determine where a code object came from, or if a function had borrowed another's code object. This is certainly only useful in the case that one function is using the code object of another, which we have all agreed is not that common. However, with a co_func I felt that all the bases would be covered.
Ah, but you can't do that! There are many situations where a single code object is used to create many different function objects. E.g. every time you have a nested function. Also the code object is immutable. This part is all carefully considered and should be left alone.
Code objects don't always get created as part of a function. They can be returned by compile. What should co_func be set to then?
None, since there was no function object created along with the code object. Same with generator expressions.
Just forget this part.
Finally, if the function has a reference to the code object, and the code object has a reference to the function, you have a reference cycle. That's not the end of the world now as it used to be, in early Python before the garbage collector was added, but still, there better be a really good use-case to justify it.
(Perhaps a weak reference might be more appropriate?)
Good point.
Mostly I am trying to look for an angle that works without a lot of trouble. Can't fault me for trying in my own incoherent way. :)
On the rest of that rejected PEP: - I'm not actually sure how easy it is to implement the setting of __function__ when the frame is created. IIRC the frame creation is rather far removed from the function object, as there are various cases where there is no function object (class bodies, module-level code) and in other cases the function is called via a bound method. Someone should write a working patch to figure out if this is a problem in practice. - The primary use case for __function__ to me seems to access function attributes, but I'm not sure what's wrong with referencing these via the function name. Maybe it's when there's a method involved, since then you'd have to write <classname>.<methodname>.<attrname>. - It seems that the "current class" question has already been solved for super(). If more is needed I'd be okay with extending the machinery used by super() so that you can access the magic "current class" variable explicitly too. - For "current module" I've encountered a number of use cases, mostly having to do with wanting to define new names dynamically. Somehow I have found: globals()[x] = y # Note that x is a variable, not a literal cumbersome; I'd rather write: setattr(__this_module__, x, y) There are IIRC also some use cases where an API expects a module object (or at least something whose attributes it can set and/or get) and passing the current module is clumsy: foo(sys.modules[__name__]) On the whole these use cases are all fairly weak though and I would give it a +0 at best. But rather than a prolonged discussion of the merits and use cases, I strongly recommend that somebody tries to come up with a working implementation and we'll strengthen the PEP from there. -- --Guido van Rossum (python.org/~guido)
On Sun, Aug 7, 2011 at 10:10 PM, Guido van Rossum <guido@python.org> wrote:
On the rest of that rejected PEP:
- I'm not actually sure how easy it is to implement the setting of __function__ when the frame is created. IIRC the frame creation is rather far removed from the function object, as there are various cases where there is no function object (class bodies, module-level code) and in other cases the function is called via a bound method. Someone should write a working patch to figure out if this is a problem in practice.
With a PEP 3135 closure style solution, the cell reference would be filled in at function definition time, so that part shouldn't be an issue.
- The primary use case for __function__ to me seems to access function attributes, but I'm not sure what's wrong with referencing these via the function name. Maybe it's when there's a method involved, since then you'd have to write <classname>.<methodname>.<attrname>.
That does raise an interesting question, though: in a function wrapped via decorators, should __function__ refer to the innermost function or the outermost one? Reference by name lazily accesses the outermost one, but doesn't care how the decorators are applied (i.e. as part of the def statement or via post decoration). A __class__ style cell reference to the result of the 'def' statement would behave differently in the post decoration case. While referencing the innermost function would likely be wrong in any case involving function attributes, having the function in a valid state during decoration will likely mandate filling in the cell reference before invoking any decorators. Perhaps the best solution would be to syntactically reference the innermost function, but provide a clean way in functools to shift the cell reference to a different function (with functools.wraps doing that automatically). This does seem like an area ripe for subtle decoration related bugs though, especially by contrast with lazy name based lookup.
- It seems that the "current class" question has already been solved for super(). If more is needed I'd be okay with extending the machinery used by super() so that you can access the magic "current class" variable explicitly too.
No need to extend it, that info is already available for explicit reference:
class C: ... def f(self): ... print(__class__) ... C().f() <class '__main__.C'>
(see postscript for more details)
- For "current module" I've encountered a number of use cases, mostly having to do with wanting to define new names dynamically. Somehow I have found:
globals()[x] = y # Note that x is a variable, not a literal
cumbersome; I'd rather write:
setattr(__this_module__, x, y)
There are IIRC also some use cases where an API expects a module object (or at least something whose attributes it can set and/or get) and passing the current module is clumsy:
foo(sys.modules[__name__])
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted: import sys _this = sys.modules[__name__] (or using dict-style access on globals()) Cheers, Nick. P.S. More details on the magic __class__ closure reference:
dis.show_code(C.f) Name: f Filename: <stdin> Argument count: 1 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS Constants: 0: None Names: 0: print Variable names: 0: self Free variables: 0: __class__ C.f.__closure__[0].cell_contents <class '__main__.C'>
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Aug 7, 2011 at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Aug 7, 2011 at 10:10 PM, Guido van Rossum <guido@python.org> wrote:
On the rest of that rejected PEP:
- I'm not actually sure how easy it is to implement the setting of __function__ when the frame is created. IIRC the frame creation is rather far removed from the function object, as there are various cases where there is no function object (class bodies, module-level code) and in other cases the function is called via a bound method. Someone should write a working patch to figure out if this is a problem in practice.
With a PEP 3135 closure style solution, the cell reference would be filled in at function definition time, so that part shouldn't be an issue.
Yes, I was thinking of something like that (though honestly I'd forgotten some of the details :-).
- The primary use case for __function__ to me seems to access function attributes, but I'm not sure what's wrong with referencing these via the function name. Maybe it's when there's a method involved, since then you'd have to write <classname>.<methodname>.<attrname>.
That does raise an interesting question, though: in a function wrapped via decorators, should __function__ refer to the innermost function or the outermost one?
IMO there is no doubt that if __function__ were to exist it should reference the innermost function, i.e. the thing that was created by the 'def' statement before any decorators were applied.
Reference by name lazily accesses the outermost one, but doesn't care how the decorators are applied (i.e. as part of the def statement or via post decoration).
What do you mean here by lazily?
A __class__ style cell reference to the result of the 'def' statement would behave differently in the post decoration case.
Oh you were thinking of making it reference the result after decoration? Maybe I know too much about the implementation, but I would find that highly confusing. Do you even have a use case for that? If so, I think it should be a separate name, e.g. __decorated_function__.
While referencing the innermost function would likely be wrong in any case involving function attributes, having the function in a valid state during decoration will likely mandate filling in the cell reference before invoking any decorators. Perhaps the best solution would be to syntactically reference the innermost function, but provide a clean way in functools to shift the cell reference to a different function (with functools.wraps doing that automatically).
Hm, making it dynamic sounds wrong. I think it makes more sense to just share the attribute dict (which is easily done through assignment to the wrapping function's __dict__).
This does seem like an area ripe for subtle decoration related bugs though, especially by contrast with lazy name based lookup.
TBH, personally I am in most cases unhappy with the aggressive copying of docstring and other metadata from the wrapped function to the wrapper function, and wish the idiom had never been invented.
- It seems that the "current class" question has already been solved for super(). If more is needed I'd be okay with extending the machinery used by super() so that you can access the magic "current class" variable explicitly too.
No need to extend it, that info is already available for explicit reference:
class C: ... def f(self): ... print(__class__) ... C().f() <class '__main__.C'>
(see postscript for more details)
Awesome.
- For "current module" I've encountered a number of use cases, mostly having to do with wanting to define new names dynamically. Somehow I have found:
globals()[x] = y # Note that x is a variable, not a literal
cumbersome; I'd rather write:
setattr(__this_module__, x, y)
There are IIRC also some use cases where an API expects a module object (or at least something whose attributes it can set and/or get) and passing the current module is clumsy:
foo(sys.modules[__name__])
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted:
import sys _this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a distraction and an unwarranted jump into the implementation. It may not even work: there are some recipes that replace sys.modules[__name__] with some wrapper object. If __this_module__ existed it would of course refer to the "real" module object involved. -- --Guido van Rossum (python.org/~guido)
On Sun, 7 Aug 2011 09:07:02 -0400 Guido van Rossum <guido@python.org> wrote:
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted:
import sys _this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a distraction and an unwarranted jump into the implementation. It may not even work: there are some recipes that replace sys.modules[__name__] with some wrapper object. If __this_module__ existed it would of course refer to the "real" module object involved.
I would also add a ton of reference cycles and might create new juicy problems in the module shutdown procedure :) Regards Antoine.
On Sun, Aug 7, 2011 at 11:07 PM, Guido van Rossum <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
With a PEP 3135 closure style solution, the cell reference would be filled in at function definition time, so that part shouldn't be an issue.
Yes, I was thinking of something like that (though honestly I'd forgotten some of the details :-).
I'd forgotten many of the details as well, but was tracking down some super() strangeness recently (to answer a question Michael Foord asked, IIRC) and had to look it up.
IMO there is no doubt that if __function__ were to exist it should reference the innermost function, i.e. the thing that was created by the 'def' statement before any decorators were applied.
Yeah, I'd mostly realised that by the time I finished writing by last message, but figured I'd record the train of thought that got me there.
Reference by name lazily accesses the outermost one, but doesn't care how the decorators are applied (i.e. as part of the def statement or via post decoration).
What do you mean here by lazily?
Just the fact that the reference isn't resolved until the function executes rather than being resolved when it gets defined.
A __class__ style cell reference to the result of the 'def' statement would behave differently in the post decoration case.
Oh you were thinking of making it reference the result after decoration? Maybe I know too much about the implementation, but I would find that highly confusing. Do you even have a use case for that? If so, I think it should be a separate name, e.g. __decorated_function__.
The only reason I was thinking that way is that currently, if you do something like [1]: @lru_cache() def fib(n): if n < 2: return n return fib(n-1) + fib(n-2) then, at call time, 'fib' will resolve to the caching wrapper rather than to the undecorated function. Using a reference to the undecorated function instead (as would have to happen for a sane implementation of __func__) would be actively harmful since the recursive calls would bypass the cache unless the lru_cache decorator took steps to change the way the reference evolved: @lru_cache() def fib(n): if n < 2: return n return __func__(n-1) + __func__(n-2) # Not the same, unless lru_cache adjusts the reference This semantic mismatch has actually shifted my opinion from +0 to -1 on the idea. Relying on normal name lookup can be occasionally inconvenient, but it is at least clear what we're referring to. The existence of wrapper functions means that "this function" isn't as clear and unambiguous a phrase as it first seems. (I think the reason we get away with it in the PEP 3135 case is that 'class wrappers' typically aren't handled via class decorators but via metaclasses, which do a better job of playing nicely with the implicit closure created to handle super() and __class__)
While referencing the innermost function would likely be wrong in any case involving function attributes, having the function in a valid state during decoration will likely mandate filling in the cell reference before invoking any decorators. Perhaps the best solution would be to syntactically reference the innermost function, but provide a clean way in functools to shift the cell reference to a different function (with functools.wraps doing that automatically).
Hm, making it dynamic sounds wrong. I think it makes more sense to just share the attribute dict (which is easily done through assignment to the wrapping function's __dict__).
Huh, I hadn't even thought of that as a potential alternative to the update() based approach currently used in functools.wraps (I had to jump into the interactive interpreter to confirm that functions really do let you swap out their instance dict). It's interesting that, once again, the status quo deals with this according to ordinary name resolution rules: any wrapping of the function will be ignored, *unless* we store the wrapper back into the original location so the name resolution in the function body will see it. Since the idea of implicitly sharing state between currently independent wrapper functions scares me, this strikes me as another reason to switch to '-1'.
This does seem like an area ripe for subtle decoration related bugs though, especially by contrast with lazy name based lookup.
TBH, personally I am in most cases unhappy with the aggressive copying of docstring and other metadata from the wrapped function to the wrapper function, and wish the idiom had never been invented.
IIRC, I was the one who actually committed the stdlib blessing of the idiom in the form of 'functools.wraps'. It was definitely a hack to deal with the increasing prevalence of wrapper functions as decorators became more popular - naive introspection was giving too many wrong answers and tweaking the recommended wrapping process so that 'f.__doc__' would work again seemed like a better option than defining a complex introspection protocol to handle wrapped functions. I still think it was a reasonable way forward (and better than leaving things as they were), but it's definitely an approach with quite a few flaws.
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted:
import sys _this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a distraction and an unwarranted jump into the implementation. It may not even work: there are some recipes that replace sys.modules[__name__] with some wrapper object. If __this_module__ existed it would of course refer to the "real" module object involved.
Some invocations of runpy.run_module also cause the 'sys.modules' based idioms to fail, so there may be a case to be made for this one. I suspect some folks would use it to avoid global declarations as well (i.e. by just writing '__module__.x = y'). It might cause the cyclic GC some grief, though,so the implementation consequences would need to be investigated if someone wanted to pursue it. Cheers, Nick. [1] http://docs.python.org/dev/library/functools.html#functools.lru_cache -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
when function body access function name, it requires "name lookup", that is a "runtime" behavior. i would like that python offer some "compile time" behavior just like this proposal - things(here, it's function) declare by the "as clause" is always "runtime independent" and only visible on it's own suite(which means you can not use the "as declared" NAME outside its indent block). 2011/8/8 Nick Coghlan <ncoghlan@gmail.com>:
On Sun, Aug 7, 2011 at 11:07 PM, Guido van Rossum <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
With a PEP 3135 closure style solution, the cell reference would be filled in at function definition time, so that part shouldn't be an issue.
Yes, I was thinking of something like that (though honestly I'd forgotten some of the details :-).
I'd forgotten many of the details as well, but was tracking down some super() strangeness recently (to answer a question Michael Foord asked, IIRC) and had to look it up.
IMO there is no doubt that if __function__ were to exist it should reference the innermost function, i.e. the thing that was created by the 'def' statement before any decorators were applied.
Yeah, I'd mostly realised that by the time I finished writing by last message, but figured I'd record the train of thought that got me there.
Reference by name lazily accesses the outermost one, but doesn't care how the decorators are applied (i.e. as part of the def statement or via post decoration).
What do you mean here by lazily?
Just the fact that the reference isn't resolved until the function executes rather than being resolved when it gets defined.
A __class__ style cell reference to the result of the 'def' statement would behave differently in the post decoration case.
Oh you were thinking of making it reference the result after decoration? Maybe I know too much about the implementation, but I would find that highly confusing. Do you even have a use case for that? If so, I think it should be a separate name, e.g. __decorated_function__.
The only reason I was thinking that way is that currently, if you do something like [1]:
@lru_cache() def fib(n): if n < 2: return n return fib(n-1) + fib(n-2)
then, at call time, 'fib' will resolve to the caching wrapper rather than to the undecorated function. Using a reference to the undecorated function instead (as would have to happen for a sane implementation of __func__) would be actively harmful since the recursive calls would bypass the cache unless the lru_cache decorator took steps to change the way the reference evolved:
@lru_cache() def fib(n): if n < 2: return n return __func__(n-1) + __func__(n-2) # Not the same, unless lru_cache adjusts the reference
This semantic mismatch has actually shifted my opinion from +0 to -1 on the idea. Relying on normal name lookup can be occasionally inconvenient, but it is at least clear what we're referring to. The existence of wrapper functions means that "this function" isn't as clear and unambiguous a phrase as it first seems.
(I think the reason we get away with it in the PEP 3135 case is that 'class wrappers' typically aren't handled via class decorators but via metaclasses, which do a better job of playing nicely with the implicit closure created to handle super() and __class__)
While referencing the innermost function would likely be wrong in any case involving function attributes, having the function in a valid state during decoration will likely mandate filling in the cell reference before invoking any decorators. Perhaps the best solution would be to syntactically reference the innermost function, but provide a clean way in functools to shift the cell reference to a different function (with functools.wraps doing that automatically).
Hm, making it dynamic sounds wrong. I think it makes more sense to just share the attribute dict (which is easily done through assignment to the wrapping function's __dict__).
Huh, I hadn't even thought of that as a potential alternative to the update() based approach currently used in functools.wraps (I had to jump into the interactive interpreter to confirm that functions really do let you swap out their instance dict).
It's interesting that, once again, the status quo deals with this according to ordinary name resolution rules: any wrapping of the function will be ignored, *unless* we store the wrapper back into the original location so the name resolution in the function body will see it.
Since the idea of implicitly sharing state between currently independent wrapper functions scares me, this strikes me as another reason to switch to '-1'.
This does seem like an area ripe for subtle decoration related bugs though, especially by contrast with lazy name based lookup.
TBH, personally I am in most cases unhappy with the aggressive copying of docstring and other metadata from the wrapped function to the wrapper function, and wish the idiom had never been invented.
IIRC, I was the one who actually committed the stdlib blessing of the idiom in the form of 'functools.wraps'. It was definitely a hack to deal with the increasing prevalence of wrapper functions as decorators became more popular - naive introspection was giving too many wrong answers and tweaking the recommended wrapping process so that 'f.__doc__' would work again seemed like a better option than defining a complex introspection protocol to handle wrapped functions.
I still think it was a reasonable way forward (and better than leaving things as they were), but it's definitely an approach with quite a few flaws.
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted:
import sys _this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a distraction and an unwarranted jump into the implementation. It may not even work: there are some recipes that replace sys.modules[__name__] with some wrapper object. If __this_module__ existed it would of course refer to the "real" module object involved.
Some invocations of runpy.run_module also cause the 'sys.modules' based idioms to fail, so there may be a case to be made for this one. I suspect some folks would use it to avoid global declarations as well (i.e. by just writing '__module__.x = y').
It might cause the cyclic GC some grief, though,so the implementation consequences would need to be investigated if someone wanted to pursue it.
Cheers, Nick.
[1] http://docs.python.org/dev/library/functools.html#functools.lru_cache
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Mon, Aug 8, 2011 at 3:27 PM, 海韵 <lyricconch@gmail.com> wrote:
when function body access function name, it requires "name lookup", that is a "runtime" behavior. i would like that python offer some "compile time" behavior just like this proposal - things(here, it's function) declare by the "as clause" is always "runtime independent" and only visible on it's own suite(which means you can not use the "as declared" NAME outside its indent block).
While a language could certainly be defined around such behaviour, that language isn't Python (Py3k except clauses notwithstanding). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Aug 7, 2011 at 7:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Aug 7, 2011 at 11:07 PM, Guido van Rossum <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
With a PEP 3135 closure style solution, the cell reference would be filled in at function definition time, so that part shouldn't be an issue.
Yes, I was thinking of something like that (though honestly I'd forgotten some of the details :-).
I'd forgotten many of the details as well, but was tracking down some super() strangeness recently (to answer a question Michael Foord asked, IIRC) and had to look it up.
IMO there is no doubt that if __function__ were to exist it should reference the innermost function, i.e. the thing that was created by the 'def' statement before any decorators were applied.
Yeah, I'd mostly realised that by the time I finished writing by last message, but figured I'd record the train of thought that got me there.
Reference by name lazily accesses the outermost one, but doesn't care how the decorators are applied (i.e. as part of the def statement or via post decoration).
What do you mean here by lazily?
Just the fact that the reference isn't resolved until the function executes rather than being resolved when it gets defined.
A __class__ style cell reference to the result of the 'def' statement would behave differently in the post decoration case.
Oh you were thinking of making it reference the result after decoration? Maybe I know too much about the implementation, but I would find that highly confusing. Do you even have a use case for that? If so, I think it should be a separate name, e.g. __decorated_function__.
The only reason I was thinking that way is that currently, if you do something like [1]:
@lru_cache() def fib(n): if n < 2: return n return fib(n-1) + fib(n-2)
then, at call time, 'fib' will resolve to the caching wrapper rather than to the undecorated function. Using a reference to the undecorated function instead (as would have to happen for a sane implementation of __func__) would be actively harmful since the recursive calls would bypass the cache unless the lru_cache decorator took steps to change the way the reference evolved:
@lru_cache() def fib(n): if n < 2: return n return __func__(n-1) + __func__(n-2) # Not the same, unless lru_cache adjusts the reference
How would the the reference be adjusted?
This semantic mismatch has actually shifted my opinion from +0 to -1 on the idea. Relying on normal name lookup can be occasionally inconvenient, but it is at least clear what we're referring to. The existence of wrapper functions means that "this function" isn't as clear and unambiguous a phrase as it first seems.
To me it just means that __func__ will remain esoteric, which is just fine with me. I wouldn't be surprised if there were use cases where it was *desirable* to have a way (from the inside) to access the undecorated function (somewhat similar to the thing with modules below). Also I really don't want the semantics of decorators to depart from the original "define the function, then apply this to it" thing. And I don't want to have to think about the possibility of __func__ being overridden by the wrapping decorator either (or by anything else).
(I think the reason we get away with it in the PEP 3135 case is that 'class wrappers' typically aren't handled via class decorators but via metaclasses, which do a better job of playing nicely with the implicit closure created to handle super() and __class__)
We didn't have class decorators then did we? Anyway I'm not sure what the semantics are, but I hope they will be such that __class__ references the undecorated, original class object used when the method was being defined. (If the class statement is executed repeatedly the __class__ should always refer to the "real" class actually involved in the method call.)
While referencing the innermost function would likely be wrong in any case involving function attributes, having the function in a valid state during decoration will likely mandate filling in the cell reference before invoking any decorators. Perhaps the best solution would be to syntactically reference the innermost function, but provide a clean way in functools to shift the cell reference to a different function (with functools.wraps doing that automatically).
Hm, making it dynamic sounds wrong. I think it makes more sense to just share the attribute dict (which is easily done through assignment to the wrapping function's __dict__).
Huh, I hadn't even thought of that as a potential alternative to the update() based approach currently used in functools.wraps (I had to jump into the interactive interpreter to confirm that functions really do let you swap out their instance dict).
Me too. :-) But I did remember that we might have made it that way, possibly for this very use case.
It's interesting that, once again, the status quo deals with this according to ordinary name resolution rules: any wrapping of the function will be ignored, *unless* we store the wrapper back into the original location so the name resolution in the function body will see it.
This makes sense because it builds complex functionality out of simpler building blocks. Combining two things together doesn't add any extra magic -- it's the building blocks themselves that add the magic.
Since the idea of implicitly sharing state between currently independent wrapper functions scares me, this strikes me as another reason to switch to '-1'.
I'm still wavering between -0 and +0; I see some merit but I think the high hopes of some folks for __func__ are unwarranted. Using the same cell-based mechanism as used for __class__ may or may not be the right implementation but I don't think that additional hacks based on mutating that cell should be considered. So it would really be a wash how it was done (at call time or at func def time). Are you aware of anything that mutates the __class__ cell? It would seem pretty tricky to do. FWIW I don't think I want __func__ to be available at all times, like someone (the OP?) mentioned. That seems an unnecessary slowdown of every call / increase of every frame.
This does seem like an area ripe for subtle decoration related bugs though, especially by contrast with lazy name based lookup.
TBH, personally I am in most cases unhappy with the aggressive copying of docstring and other metadata from the wrapped function to the wrapper function, and wish the idiom had never been invented.
IIRC, I was the one who actually committed the stdlib blessing of the idiom in the form of 'functools.wraps'. It was definitely a hack to deal with the increasing prevalence of wrapper functions as decorators became more popular - naive introspection was giving too many wrong answers and tweaking the recommended wrapping process so that 'f.__doc__' would work again seemed like a better option than defining a complex introspection protocol to handle wrapped functions.
I guess you rely more on interactive features like help() whereas I rely more on browsing the source code. :-)
I still think it was a reasonable way forward (and better than leaving things as they were), but it's definitely an approach with quite a few flaws.
You are forgiven. :-)
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted:
import sys _this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a distraction and an unwarranted jump into the implementation. It may not even work: there are some recipes that replace sys.modules[__name__] with some wrapper object. If __this_module__ existed it would of course refer to the "real" module object involved.
Some invocations of runpy.run_module also cause the 'sys.modules' based idioms to fail, so there may be a case to be made for this one. I suspect some folks would use it to avoid global declarations as well (i.e. by just writing '__module__.x = y').
+1. But what to call it? __module__ is a string in other places.
It might cause the cyclic GC some grief, though,so the implementation consequences would need to be investigated if someone wanted to pursue it.
Modules are already involved in much cyclical GC grief, and most have an infinite lifetime anyway (sys.modules keeps them alive). I doubt it will get any worse.
Cheers, Nick.
[1] http://docs.python.org/dev/library/functools.html#functools.lru_cache
-- --Guido van Rossum (python.org/~guido)
On Mon, Aug 8, 2011 at 11:07 PM, Guido van Rossum <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 7:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
then, at call time, 'fib' will resolve to the caching wrapper rather than to the undecorated function. Using a reference to the undecorated function instead (as would have to happen for a sane implementation of __func__) would be actively harmful since the recursive calls would bypass the cache unless the lru_cache decorator took steps to change the way the reference evolved:
@lru_cache() def fib(n): if n < 2: return n return __func__(n-1) + __func__(n-2) # Not the same, unless lru_cache adjusts the reference
How would the the reference be adjusted?
I was thinking of Michael's blog post about modifying cell contents [1], but I had forgotten that the write operation required mucking about with ctypes (since the cell_contents attribute of the cell is read only at the Python level). That's actually a good thing, since it means a cell based __func__ reference would consistently refer to the innermost function, ignoring any wrapper functions. [1] http://www.voidspace.org.uk/python/weblog/arch_d7_2011_05_28.shtml#e1214
This semantic mismatch has actually shifted my opinion from +0 to -1 on the idea. Relying on normal name lookup can be occasionally inconvenient, but it is at least clear what we're referring to. The existence of wrapper functions means that "this function" isn't as clear and unambiguous a phrase as it first seems.
To me it just means that __func__ will remain esoteric, which is just fine with me. I wouldn't be surprised if there were use cases where it was *desirable* to have a way (from the inside) to access the undecorated function (somewhat similar to the thing with modules below).
Yeah, now that I remember that you have to use the C API in order to monkey with the contents of a cell reference, I'm significantly happier with the idea that __func__ could be given solid 'always refers to the innermost unwrapped function definition' semantics. Referring to the function by name would remain the way to access the potentially wrapped version that is stored in the containing namespace. That's enough to get me back to -0. The reason I remain slightly negative is that I'd like to see some concrete use cases where ignoring wrapper functions is the right thing to do - every case that comes to mind for me is like the lru_cache() Fibonacci example, where bypassing the wrapper functions is precisely the *wrong* thing to do.
(I think the reason we get away with it in the PEP 3135 case is that 'class wrappers' typically aren't handled via class decorators but via metaclasses, which do a better job of playing nicely with the implicit closure created to handle super() and __class__)
We didn't have class decorators then did we? Anyway I'm not sure what the semantics are, but I hope they will be such that __class__ references the undecorated, original class object used when the method was being defined. (If the class statement is executed repeatedly the __class__ should always refer to the "real" class actually involved in the method call.)
Yeah, __class__ always refers to the original class object, as created by calling the metaclass. The idiom seems to be that people don't use class decorators to wrap classes anyway, as metaclasses are a better tool for that kind of thing - decorators are more used for things like registration or attribute modifications.
Since the idea of implicitly sharing state between currently independent wrapper functions scares me, this strikes me as another reason to switch to '-1'.
I'm still wavering between -0 and +0; I see some merit but I think the high hopes of some folks for __func__ are unwarranted. Using the same cell-based mechanism as used for __class__ may or may not be the right implementation but I don't think that additional hacks based on mutating that cell should be considered. So it would really be a wash how it was done (at call time or at func def time). Are you aware of anything that mutates the __class__ cell? It would seem pretty tricky to do.
No, I was misremembering how Michael's cell content modification trick worked, and that was throwing off my opinion of how easy it was to mess with the cell contents. Once people start using ctypes to access the C API all bets are off anyway.
FWIW I don't think I want __func__ to be available at all times, like someone (the OP?) mentioned. That seems an unnecessary slowdown of every call / increase of every frame.
Yeah, I think at least that part of the PEP 3135 approach should be copied, even if the implementation ended up being different. So the short version of my current opinion would be: __func__: -0 - this discussion has pretty much sorted out what the semantics of such a reference would be - we know at least one way to implement it that works (cell based, modelled on PEP 3135's __class__ reference) - lacking concrete use cases where it is demonstrably superior to reference by name lookup in the containing scope (given that many use cases are forced into the use of name lookup in order to refer to a wrapped version of the function) __this_module__: +0 - the sys.modules[__name__] approach is obscure and distracting when reading code - there are cases where that approach is unreliable (e.g. involving runpy.run_module with sys module alteration disabled) - obvious naming (i.e. __module__) is problematic, since class and function __module__ attributes are strings - will need to be careful to avoid creating uncollectable garbage due to the cyclic reference between the module and its global namespace (but shouldn't be any worse than the cycle created by any module that imports the sys module) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Guido van Rossum <guido@python.org> writes:
To me it just means that __func__ will remain esoteric, which is just fine with me. I wouldn't be surprised if there were use cases where it was *desirable* to have a way (from the inside) to access the undecorated function (somewhat similar to the thing with modules below).
Perhaps I misunderstand the functionality, but: The ability to access the undecorated function would be of great help in our Django-based project. Django is very decorator-happy, and we are trying to introspect code and, given a particular URL, determine the innermost function (more precisely, what location in our code) that actually handles that URL. Is that a use-case of interest? -- \ “Always code as if the guy who ends up maintaining your code | `\ will be a violent psychopath who knows where you live.” —John | _o__) F. Woods | Ben Finney
Ben Finney wrote:
The ability to access the undecorated function would be of great help in our Django-based project. Django is very decorator-happy, and we are trying to introspect code and, given a particular URL, determine the innermost function (more precisely, what location in our code) that actually handles that URL.
Is that a use-case of interest?
I shouldn't think the proposed __func__ local would be of any use here, because it is local to the function. As I understand your use-case, you need to inspect a function from the outside, where __func__ is not available, and peel back all the decorators to the innermost function. To do that, I would start by looking at: function.__closure__[0].cell_contents and recurse as necessary. -- Steven
On Mon, Aug 8, 2011 at 8:42 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Ben Finney wrote:
The ability to access the undecorated function would be of great help in our Django-based project. Django is very decorator-happy, and we are trying to introspect code and, given a particular URL, determine the innermost function (more precisely, what location in our code) that actually handles that URL.
Is that a use-case of interest?
I shouldn't think the proposed __func__ local would be of any use here, because it is local to the function. As I understand your use-case, you need to inspect a function from the outside, where __func__ is not available, and peel back all the decorators to the innermost function.
To do that, I would start by looking at:
function.__closure__[0].cell_contents
and recurse as necessary.
AFAIK there is no fool-proof way to peel back a decorator; decorators aren't required to have any specific implementation. If the use case was important it would reopen the question of always making this value available as a frame attribute. But I'm not sure why the line number indicated by the code object (which *is* accessible) is insufficient. -- --Guido van Rossum (python.org/~guido)
Guido van Rossum <guido@python.org> writes:
AFAIK there is no fool-proof way to peel back a decorator; decorators aren't required to have any specific implementation.
If the use case was important it would reopen the question of always making this value available as a frame attribute. But I'm not sure why the line number indicated by the code object (which *is* accessible) is insufficient.
Which code object, though? If we can't get at the innermost function (which is the code location we're interested in) handling the URL, how will we get its line number? -- \ “I bought some batteries, but they weren't included; so I had | `\ to buy them again.” —Steven Wright | _o__) | Ben Finney
On Tue, Aug 9, 2011 at 11:36 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
Which code object, though? If we can't get at the innermost function (which is the code location we're interested in) handling the URL, how will we get its line number?
In general, you can't, since Python can't readily tell the difference between decorators that wrap a function and those that replace it completely. Even the '__wrapped__' convention adopted by recent incarnations of functools.wraps (primarily to expose the function underlying lru_cache) only works for genuine wrapper functions that use that decorator.
From outside, a cell referencing the current function (if it was implemented that way) wouldn't tell you anything new, since the following identity would hold:
f is f.__closure__[f.__code__.co_freevars.index('__func__')].cell_contents On any given function, *if* the new cell existed at all, it would refer to that specific function, not an inner one. Cheers, Nick. P.S. An example with the existing PEP 3135 implicit cell creation:
class C: ... def f(self): ... print(__class__) ... f = C.f C is f.__closure__[f.__code__.co_freevars.index('__class__')].cell_contents True
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan@gmail.com> writes:
On Tue, Aug 9, 2011 at 11:36 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
Which code object, though? If we can't get at the innermost function (which is the code location we're interested in) handling the URL, how will we get its line number?
In general, you can't […]
Thanks. I don't know what Guido was referring to, then, with:
But I'm not sure why the line number indicated by the code object (which *is* accessible) is insufficient.
Since the inner function's code object isn't reliably available, it's insufficient for reliably getting to the undecorated function which handles a specific Django URL. -- \ “… whoever claims any right that he is unwilling to accord to | `\ his fellow-men is dishonest and infamous.” —Robert G. | _o__) Ingersoll, _The Liberty of Man, Woman and Child_, 1877 | Ben Finney
On 8/9/2011 12:01 AM, Ben Finney wrote:
Since the inner function's code object isn't reliably available, it's insufficient for reliably getting to the undecorated function which handles a specific Django URL.
I think you should try to keep an overt reference to the original function. -- Terry Jan Reedy
I was thinking from inside the executing code. It seems that your problem is how to get to the function object underneath the stack of decorators given an object that is the fully-decorated function. Never mind then. On Tue, Aug 9, 2011 at 12:01 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
Nick Coghlan <ncoghlan@gmail.com> writes:
On Tue, Aug 9, 2011 at 11:36 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
Which code object, though? If we can't get at the innermost function (which is the code location we're interested in) handling the URL, how will we get its line number?
In general, you can't […]
Thanks. I don't know what Guido was referring to, then, with:
But I'm not sure why the line number indicated by the code object (which *is* accessible) is insufficient.
Since the inner function's code object isn't reliably available, it's insufficient for reliably getting to the undecorated function which handles a specific Django URL.
-- \ “… whoever claims any right that he is unwilling to accord to | `\ his fellow-men is dishonest and infamous.” —Robert G. | _o__) Ingersoll, _The Liberty of Man, Woman and Child_, 1877 | Ben Finney
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
Guido van Rossum <guido@python.org> writes:
It seems that your problem is how to get to the function object underneath the stack of decorators given an object that is the fully-decorated function. Never mind then.
Yes. I'd still like a solution to that. -- \ “Broken promises don't upset me. I just think, why did they | `\ believe me?” —Jack Handey | _o__) | Ben Finney
On Wed, Aug 10, 2011 at 2:47 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Guido van Rossum <guido@python.org> writes:
It seems that your problem is how to get to the function object underneath the stack of decorators given an object that is the fully-decorated function. Never mind then.
Yes. I'd still like a solution to that.
The thing is that there isn't a general purpose way to resolve that question. Consider a wrapper like functools.lru_cache. It has 3 downcalls to other functions: tuple, sorted and the decorated user function. Without an explicit convention (like the __wrapped__ convention adopted by functools in 3.2), there's no way for external code to follow the chain downwards. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Aug 7, 2011 6:11 AM, "Guido van Rossum" <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 3:46 AM, Eric Snow <ericsnowcurrently@gmail.com>
On Sat, Aug 6, 2011 at 11:36 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Eric Snow wrote:
Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"?
I'm afraid I can't interpret this (which may be my ignorance rather
wrote: than
your fault).
No, the fault is likely mine. But it seems so clear to me. :)
The only guess I can make is based on what you say later:
"One new implicit name in locals().
so I presume you mean that the function should see a local variable (perhaps called "me", or "this"?) that is bound to itself.
One called __function__ (or the like). A "dunder" name is used to indicate its special nature and limit conflict with existing code.
Without thinking too much about this I like it.
The function that was called would be bound to that name at function execution time. Keep in mind that I am talking about the frame locals, not anything stored on the code object nor on the function object. Not to overdramatize it, but it would happen at the beginning of every call of every function. I don't know what that overhead would be.
It could be made into a "cell", which is the same way all locals are normally represented. This is very fast. Further the code for it could be triggered by the appearance of __function__ (if that's the keyword we choose) in the function body. I don't really care what happens if people use locals() -- that's inefficient and outmoded anyway. (Note that!)
Presumably if a function wants to use that same name as a local, nothing bad will happen, since the local assignment will just override the implicit assignment. But what about code that expects to see a nonlocal or global with the same name?
That's why a __dunder__ name is used.
What happens when two functions, sharing the same code object, get called from two threads at the same time? Are their locals independent?
I'm afraid I don't know. I expect that each would get executed in separate execution frames, and so have separate frame locals.
The frames are completely independent. They all point to the same code object and under the proposal they will all point to the same function object. I see no problems here except self-inflicted, like using __function__ to hold state that can't be accessed concurrently safely; note that recursive invocations have the same issue. I see it as a non-problem.
For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed.
Right. If it were expected that people would start writing recursive calls using __function__ routinely, in situations where a name reference works, I'd be very unhappy with the new feature. (And if someone wants to make the argument that recursive calls using __function__ are actually better in some way I am willing to filibuster.)
And I say that as somebody who does want a way for functions to know themselves. I don't think that use-case is so important that it should be implicitly added to every function, on the off-chance it is needed, rather than explicitly on demand.
For me the use case involves determining what function called my function. Currently you can tell in which execution frame a function was called, and thereby which code object, but reliably matching that to a function is not so simple. I recognize that my case is likely not a general one.
But it is a nice one. It solves some issues that pdb currently solves by just using a file/line reference.
To finish things off, bind to every new code object the function for which it was created, perhaps as "co_func". That way you will always know what function object was called and which one the code object came from originally.
What benefit will this give? Have you ever looked at a code object and said, "I need a way of knowing which function this is from?" If so, I'd love to know what problem you were trying to solve at the time!
You caught me! :) I don't already have a use case for this part. I had only considered that without this you could not determine where a code object came from, or if a function had borrowed another's code object. This is certainly only useful in the case that one function is using the code object of another, which we have all agreed is not that common. However, with a co_func I felt that all the bases would be covered.
Ah, but you can't do that! There are many situations where a single code object is used to create many different function objects. E.g. every time you have a nested function. Also the code object is immutable. This part is all carefully considered and should be left alone.
From all the responses it's apparent I have not communicated the idea well. The idea is for the *called* function object to be bound to __function__. Here's an example:
def g(): def f(): return return f f1 = g() f1() f2 = g() f2() (Thanks for pointing out that f1 and f2 share a code object.) In the call to g, g would be bound to __function__ in the function body. In the call to f1, __function__ would be f1 (in f_locals). And in the call to f2, __function__ would be f2. Also, while __function__ is not used in this example, it would still be there in the frame locals. Otherwise a function called by f would be unable to use it (via inspect.stack() and the like). Also, at definition time the original function would be bound as co_func. It would not be associated with __function__, except indirectly. This is so that you can tell the difference between the original function and one that is using the code object of the first. -eric
Code objects don't always get created as part of a function. They can
returned by compile. What should co_func be set to then?
None, since there was no function object created along with the code object. Same with generator expressions.
Just forget this part.
Finally, if the function has a reference to the code object, and the code object has a reference to the function, you have a reference cycle. That's not the end of the world now as it used to be, in early Python before
be the
garbage collector was added, but still, there better be a really good use-case to justify it.
(Perhaps a weak reference might be more appropriate?)
Good point.
Mostly I am trying to look for an angle that works without a lot of trouble. Can't fault me for trying in my own incoherent way. :)
On the rest of that rejected PEP:
- I'm not actually sure how easy it is to implement the setting of __function__ when the frame is created. IIRC the frame creation is rather far removed from the function object, as there are various cases where there is no function object (class bodies, module-level code) and in other cases the function is called via a bound method. Someone should write a working patch to figure out if this is a problem in practice.
- The primary use case for __function__ to me seems to access function attributes, but I'm not sure what's wrong with referencing these via the function name. Maybe it's when there's a method involved, since then you'd have to write <classname>.<methodname>.<attrname>.
- It seems that the "current class" question has already been solved for super(). If more is needed I'd be okay with extending the machinery used by super() so that you can access the magic "current class" variable explicitly too.
- For "current module" I've encountered a number of use cases, mostly having to do with wanting to define new names dynamically. Somehow I have found:
globals()[x] = y # Note that x is a variable, not a literal
cumbersome; I'd rather write:
setattr(__this_module__, x, y)
There are IIRC also some use cases where an API expects a module object (or at least something whose attributes it can set and/or get) and passing the current module is clumsy:
foo(sys.modules[__name__])
On the whole these use cases are all fairly weak though and I would give it a +0 at best. But rather than a prolonged discussion of the merits and use cases, I strongly recommend that somebody tries to come up with a working implementation and we'll strengthen the PEP from there.
-- --Guido van Rossum (python.org/~guido)
On Sun, Aug 7, 2011 at 8:10 AM, Guido van Rossum <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 3:46 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Sat, Aug 6, 2011 at 11:36 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Eric Snow wrote:
Right. If it were expected that people would start writing recursive calls using __function__ routinely, in situations where a name reference works, I'd be very unhappy with the new feature. (And if someone wants to make the argument that recursive calls using __function__ are actually better in some way I am willing to filibuster.)
They are better from a purity standpoint ... if your function is later renamed (or stored anonymously in an array of functions), and then the name is reused, then using the name allows silent bugs. As a style guide, I can certainly understand "Don't do that", but I prefer to program libraries defensively, because that sort of bug won't show up until the system is already pretty complex. Using __function__ is similar to using self instead of presuming that you know the class name, except that overridden class attributes (vs self) are far more common (and therefore also far easier to catch when debugging).
There are IIRC also some use cases where an API expects a module object (or at least something whose attributes it can set and/or get) and passing the current module is clumsy:
foo(sys.modules[__name__])
And it may not work, if the installation is complex enough that there are multiple modules with the same name. For me, this falls in the gray area of "Don't do that, but beware that others might." -jJ
On 8/7/2011 8:10 AM, Guido van Rossum wrote:
For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed.
Right. If it were expected that people would start writing recursive calls using __function__ routinely, in situations where a name reference works, I'd be very unhappy with the new feature.
I am willing to separate the recursion and attribute access use cases and not advocate that. I agree that def fact(n): # requires int n >= 0 return n*__function__(n-1) if n else 1 is less readable than fact(n-1)
(And if someone wants to make the argument that recursive calls using __function__ are actually better in some way I am willing to filibuster.)
Such calls would be slightly faster by avoiding a name lookup, but avoiding *any* function call by using iteration should be faster yet. For linear recursion (at most one call per call), this is usually trivial. In my book, I am stipulating that namespace manipulations that change the recursiveness of a function as written are 'forbidden' for the purpose of interpreting the code presented. This should be assumed or stated in other similar contexts. This restricts the recursion use case to multiple recursion (possibly multiple calls per call) in production library code that is not easily converted to iteration, is not intended to be read by anyone other than maintainers, and that might or even is expected to be used in an adverse environment. -- Terry Jan Reedy
On Sun, Aug 7, 2011 at 6:10 AM, Guido van Rossum <guido@python.org> wrote:
But rather than a prolonged discussion of the merits and use cases, I strongly recommend that somebody tries to come up with a working implementation and we'll strengthen the PEP from there.
-- --Guido van Rossum (python.org/~guido)
Here's a stab at a (relatively) simple approach to the function portion: http://bugs.python.org/issue12857 However, this does not expose __function__ in the locals. Instead it puts f_func in the frame object. Thus it provides the functionality, but doesn't invite abuse. And whether you like it or hate it, it's exactly what I've been looking for. -eric p.s. I tried doing it with a closure and, while it worked, I didn't like it. I will say that I came away with a _much_ better understanding of the CPython implementation (and gdb :). I feel much better about the f_func solution, particularly since we're really looking for the function associated with a frame. Trying to artificially force __function__ into the frame's locals just didn't feel right.
On Tue, Aug 30, 2011 at 4:10 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Here's a stab at a (relatively) simple approach to the function portion:
http://bugs.python.org/issue12857
However, this does not expose __function__ in the locals. Instead it puts f_func in the frame object. Thus it provides the functionality, but doesn't invite abuse. And whether you like it or hate it, it's exactly what I've been looking for.
That's actually quite an interesting idea, although I'm wondering if it could make some of our extant reference cycle issues associated with stack traces even worse. The fact that the patch is so simple is certainly rather appealing (although you have a few backwards compatibility issues to address, as I noted in my review). Also, I wouldn't be quite so quick to discard the function information in the class evaluation case. While the function involved there is a temporary one, it's still a real function. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 30, 2011 at 2:25 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's actually quite an interesting idea, although I'm wondering if it could make some of our extant reference cycle issues associated with stack traces even worse. The fact that the patch is so simple is certainly rather appealing (although you have a few backwards compatibility issues to address, as I noted in my review).
Thanks for taking a look, Nick. I'm addressing the backward incompatible stuff in a new patch a little later, and have updated the issue regarding the reference cycles.
Also, I wouldn't be quite so quick to discard the function information in the class evaluation case. While the function involved there is a temporary one, it's still a real function.
Yeah, I had considered that. However, isn't that just a CPython implementation detail? Does that matter? I don't have a problem with setting f_func for classes (and whatever else). Just wanted to clarify first. Thanks. -eric
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Aug 7, 2011 at 1:36 AM, Steven D'Aprano <steve@pearwood.info> wrote:
For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed.
Self-reflection is a purity issue. Are you worried about obscure bugs when something else hijacks the name of your function (or module or class), but your function is still somehow runnable? Practicality Beats Purity, but purity still has some value. I will also note that names are particularly likely to get reused in some contexts (GUIs, security proxies) where the writer of the original code can't rely on anything about the runtime environment.
I don't think that use-case is so important that it should be implicitly added to every function, on the off-chance it is needed, rather than explicitly on demand.
Agreed. -jJ
On Sun, Aug 7, 2011 at 2:00 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
On Sun, Aug 7, 2011 at 1:36 AM, Steven D'Aprano <steve@pearwood.info> wrote:
For most uses, standard recursion via the name is good enough, it's only a few corner cases where self-reflection (as I call it) is needed.
Self-reflection is a purity issue. Are you worried about obscure bugs when something else hijacks the name of your function (or module or class), but your function is still somehow runnable?
If you worry about that every time you write a recursive function you'll go insane.
Practicality Beats Purity, but purity still has some value. I will also note that names are particularly likely to get reused in some contexts (GUIs, security proxies) where the writer of the original code can't rely on anything about the runtime environment.
There are some standard situations where some standard recipes apply. But it's best to use those sparingly or your code will become unreadable.
I don't think that use-case is so important that it should be implicitly added to every function, on the off-chance it is needed, rather than explicitly on demand.
Right. -- --Guido van Rossum (python.org/~guido)
On 8/7/2011 12:32 AM, Eric Snow wrote:
On Sat, Aug 6, 2011 at 2:28 PM, Terry Reedy<tjreedy@udel.edu> wrote:
This proposal/desire comes up constantly on python-list, generally with support. I believe it was part of two recent threads. I would like to know if the rejection of the idea so far is a rejection in principle (and if so, why) or a rejection of specifics.
+1
Of the three code blocks, functions are the only ones for whom the resulting object and the execution of the code block are separate. So a code object could be executing for the original function or a different one that is sharing the code object.
Now I remember that the separation between code object and function object and the possibility of reusing code objects has been given as a reason to reject the idea. On the other hand, reusing code objects is so rare that I question the need to cater to it much.
Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"?
I do not understand this.
To finish things off, bind to every new code object the function for which it was created, perhaps as "co_func". That way you will always know what function object was called and which one the code object came from originally.
I don't see that solving the problem of how the function code accesses the object. Adding something to the default arg tuple might make more sense. -- Terry Jan Reedy
On Sun, Aug 7, 2011 at 4:17 PM, Terry Reedy <tjreedy@udel.edu> wrote:
On 8/7/2011 12:32 AM, Eric Snow wrote:
Of the three code blocks, functions are the only ones for whom the resulting object and the execution of the code block are separate. So a code object could be executing for the original function or a different one that is sharing the code object.
Now I remember that the separation between code object and function object and the possibility of reusing code objects has been given as a reason to reject the idea. On the other hand, reusing code objects is so rare that I question the need to cater to it much.
Nested function objects and class definitions say 'Hi!' - they reuse code blocks all the time. In the following code: def decorator(f): @wraps(f) def wrapper(*args, **kwds): return f(*args, **kwds) return wrapper All of the 'wrapper' instances share a single code object (stored as a constant in the code object for 'decorator'). If a proposal suggests storing mutable state on a code object it's time to stop and think of a new way (or drop the idea entirely).
Why not bind the called function-object to the frame locals, rather than the one for which the code object was created, perhaps as "__function__"?
I do not understand this.
I'm fairly sure I do understand it, and I don't think it could be made to work in a sensible way (either thread safety problems or else thoroughly confusing interactions with the descriptor machinery). I don't see any fundamental limitations preventing the use of a magic closure variable for __function__ along the lines of __class__ in PEP 3135 though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 2011-08-07 at 22:14 +1000, Nick Coghlan wrote:
On Sun, Aug 7, 2011 at 4:17 PM, Terry Reedy <tjreedy@udel.edu> wrote:
On 8/7/2011 12:32 AM, Eric Snow wrote:
Of the three code blocks, functions are the only ones for whom the resulting object and the execution of the code block are separate. So a code object could be executing for the original function or a different one that is sharing the code object.
Now I remember that the separation between code object and function object and the possibility of reusing code objects has been given as a reason to reject the idea. On the other hand, reusing code objects is so rare that I question the need to cater to it much.
Nested function objects and class definitions say 'Hi!' - they reuse code blocks all the time. In the following code:
def decorator(f): @wraps(f) def wrapper(*args, **kwds): return f(*args, **kwds) return wrapper
All of the 'wrapper' instances share a single code object (stored as a constant in the code object for 'decorator').
If a proposal suggests storing mutable state on a code object it's time to stop and think of a new way (or drop the idea entirely).
It seems to me, many of the attempts to improve functions involve making them more like objects. Since functions are a key building block of class's, I'd rather see functions simplified rather than made more complex. Which probably isn't possible at this time. As for a new way/direction ... Maybe a simplified function as a method may be possible? With that, it then becomes possible to have a Function class that people can alter by adding additional methods to. Then "def foo(): pass" could possibly be short for ... class f(Function): method __call__(self): #can this be more efficient over def? pass foo = f() Yes, functions are objects now, but modifying them is almost always hackish. Maybe moving in this direction can lead to some non-hackish alternatives for those use cases. Cheers, Ron
participants (13)
-
Antoine Pitrou -
Ben Finney -
Chris Rebert -
David Townshend -
Eric Snow -
Guido van Rossum -
Jim Jewett -
Nick Coghlan -
Rob Cliffe -
ron3200 -
Steven D'Aprano -
Terry Reedy -
海韵