Fwd: Re: Access to function objects
Sorry, this should have been a reply to all! ---------- Forwarded message ---------- From: "David Townshend" <aquavitae69@gmail.com> Date: Aug 8, 2011 9:24 PM Subject: Re: [Python-ideas] Access to function objects To: "Guido van Rossum" <guido@python.org> On Aug 8, 2011 3:08 PM, "Guido van Rossum" <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 7:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Aug 7, 2011 at 11:07 PM, Guido van Rossum <guido@python.org>
wrote:
On Sun, Aug 7, 2011 at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
With a PEP 3135 closure style solution, the cell reference would be filled in at function definition time, so that part shouldn't be an issue.
Yes, I was thinking of something like that (though honestly I'd forgotten some of the details :-).
I'd forgotten many of the details as well, but was tracking down some super() strangeness recently (to answer a question Michael Foord asked, IIRC) and had to look it up.
IMO there is no doubt that if __function__ were to exist it should reference the innermost function, i.e. the thing that was created by the 'def' statement before any decorators were applied.
Yeah, I'd mostly realised that by the time I finished writing by last message, but figured I'd record the train of thought that got me there.
Reference by name lazily accesses the outermost one, but doesn't care how the decorators are applied (i.e. as part of the def statement or via post decoration).
What do you mean here by lazily?
Just the fact that the reference isn't resolved until the function executes rather than being resolved when it gets defined.
A __class__ style cell reference to the result of the 'def' statement would behave differently in the post decoration case.
Oh you were thinking of making it reference the result after decoration? Maybe I know too much about the implementation, but I would find that highly confusing. Do you even have a use case for that? If so, I think it should be a separate name, e.g. __decorated_function__.
The only reason I was thinking that way is that currently, if you do something like [1]:
@lru_cache() def fib(n): if n < 2: return n return fib(n-1) + fib(n-2)
then, at call time, 'fib' will resolve to the caching wrapper rather than to the undecorated function. Using a reference to the undecorated function instead (as would have to happen for a sane implementation of __func__) would be actively harmful since the recursive calls would bypass the cache unless the lru_cache decorator took steps to change the way the reference evolved:
@lru_cache() def fib(n): if n < 2: return n return __func__(n-1) + __func__(n-2) # Not the same, unless lru_cache adjusts the reference
How would the the reference be adjusted?
This semantic mismatch has actually shifted my opinion from +0 to -1 on the idea. Relying on normal name lookup can be occasionally inconvenient, but it is at least clear what we're referring to. The existence of wrapper functions means that "this function" isn't as clear and unambiguous a phrase as it first seems.
To me it just means that __func__ will remain esoteric, which is just fine with me. I wouldn't be surprised if there were use cases where it was *desirable* to have a way (from the inside) to access the undecorated function (somewhat similar to the thing with modules below).
Also I really don't want the semantics of decorators to depart from the original "define the function, then apply this to it" thing. And I don't want to have to think about the possibility of __func__ being overridden by the wrapping decorator either (or by anything else).
(I think the reason we get away with it in the PEP 3135 case is that 'class wrappers' typically aren't handled via class decorators but via metaclasses, which do a better job of playing nicely with the implicit closure created to handle super() and __class__)
We didn't have class decorators then did we? Anyway I'm not sure what the semantics are, but I hope they will be such that __class__ references the undecorated, original class object used when the method was being defined. (If the class statement is executed repeatedly the __class__ should always refer to the "real" class actually involved in the method call.)
While referencing the innermost function would likely be wrong in any case involving function attributes, having the function in a valid state during decoration will likely mandate filling in the cell reference before invoking any decorators. Perhaps the best solution would be to syntactically reference the innermost function, but provide a clean way in functools to shift the cell reference to a different function (with functools.wraps doing that automatically).
Hm, making it dynamic sounds wrong. I think it makes more sense to just share the attribute dict (which is easily done through assignment to the wrapping function's __dict__).
Huh, I hadn't even thought of that as a potential alternative to the update() based approach currently used in functools.wraps (I had to jump into the interactive interpreter to confirm that functions really do let you swap out their instance dict).
Me too. :-)
But I did remember that we might have made it that way, possibly for this very use case.
It's interesting that, once again, the status quo deals with this according to ordinary name resolution rules: any wrapping of the function will be ignored, *unless* we store the wrapper back into the original location so the name resolution in the function body will see it.
This makes sense because it builds complex functionality out of simpler building blocks. Combining two things together doesn't add any extra magic -- it's the building blocks themselves that add the magic.
Since the idea of implicitly sharing state between currently independent wrapper functions scares me, this strikes me as another reason to switch to '-1'.
I'm still wavering between -0 and +0; I see some merit but I think the high hopes of some folks for __func__ are unwarranted. Using the same cell-based mechanism as used for __class__ may or may not be the right implementation but I don't think that additional hacks based on mutating that cell should be considered. So it would really be a wash how it was done (at call time or at func def time). Are you aware of anything that mutates the __class__ cell? It would seem pretty tricky to do.
FWIW I don't think I want __func__ to be available at all times, like someone (the OP?) mentioned. That seems an unnecessary slowdown of every call / increase of every frame.
Yes, that was my idea, (hence the "as" syntax). However, this discussion is getting a bit out of my depth now and I don't really know the implications of my suggestion!
This does seem like an area ripe for subtle decoration related bugs though, especially by contrast with lazy name based lookup.
TBH, personally I am in most cases unhappy with the aggressive copying of docstring and other metadata from the wrapped function to the wrapper function, and wish the idiom had never been invented.
IIRC, I was the one who actually committed the stdlib blessing of the idiom in the form of 'functools.wraps'. It was definitely a hack to deal with the increasing prevalence of wrapper functions as decorators became more popular - naive introspection was giving too many wrong answers and tweaking the recommended wrapping process so that 'f.__doc__' would work again seemed like a better option than defining a complex introspection protocol to handle wrapped functions.
I guess you rely more on interactive features like help() whereas I rely more on browsing the source code. :-)
I still think it was a reasonable way forward (and better than leaving things as they were), but it's definitely an approach with quite a few flaws.
You are forgiven. :-)
While this may sound a little hypocritical coming from the author of PEPs 366 and 395, I'm wary of adding new implicit module globals for problems with relatively simple and robust alternatives. In this case, it's fairly easy to get access to the current module using the idiom Guido quoted:
import sys _this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a distraction and an unwarranted jump into the implementation. It may not even work: there are some recipes that replace sys.modules[__name__] with some wrapper object. If __this_module__ existed it would of course refer to the "real" module object involved.
Some invocations of runpy.run_module also cause the 'sys.modules' based idioms to fail, so there may be a case to be made for this one. I suspect some folks would use it to avoid global declarations as well (i.e. by just writing '__module__.x = y').
+1. But what to call it? __module__ is a string in other places.
It might cause the cyclic GC some grief, though,so the implementation consequences would need to be investigated if someone wanted to pursue it.
Modules are already involved in much cyclical GC grief, and most have an infinite lifetime anyway (sys.modules keeps them alive). I doubt it will get any worse.
Cheers, Nick.
[1] http://docs.python.org/dev/library/functools.html#functools.lru_cache
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
participants (1)
-
David Townshend