Sorry, this should have been a reply to all!
---------- Forwarded message ----------
From: "David Townshend" <aquavitae69@gmail.com>
Date: Aug 8, 2011 9:24 PM
Subject: Re: [Python-ideas] Access to function objects
To: "Guido van Rossum" <guido@python.org>
On Aug 8, 2011 3:08 PM, "Guido van Rossum" <guido@python.org> wrote:
On Sun, Aug 7, 2011 at 7:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Aug 7, 2011 at 11:07 PM, Guido van Rossum <guido@python.org>
On Sun, Aug 7, 2011 at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com>
wrote:
With a PEP 3135 closure style solution, the cell reference would be
filled in at function definition time, so that part shouldn't be an
issue.
Yes, I was thinking of something like that (though honestly I'd
forgotten some of the details :-).
I'd forgotten many of the details as well, but was tracking down some
super() strangeness recently (to answer a question Michael Foord
asked, IIRC) and had to look it up.
IMO there is no doubt that if __function__ were to exist it should
reference the innermost function, i.e. the thing that was created by
the 'def' statement before any decorators were applied.
Yeah, I'd mostly realised that by the time I finished writing by last
message, but figured I'd record the train of thought that got me
there.
Reference by name lazily accesses the outermost one, but doesn't care
how the decorators are applied (i.e. as part of the def statement or
via post decoration).
What do you mean here by lazily?
Just the fact that the reference isn't resolved until the function
executes rather than being resolved when it gets defined.
A __class__ style cell reference to the result
of the 'def' statement would behave differently in the post decoration
case.
Oh you were thinking of making it reference the result after
decoration? Maybe I know too much about the implementation, but I
would find that highly confusing. Do you even have a use case for
that? If so, I think it should be a separate name, e.g.
__decorated_function__.
The only reason I was thinking that way is that currently, if you do
something like [1]:
@lru_cache()
def fib(n):
if n < 2:
return n
return fib(n-1) + fib(n-2)
then, at call time, 'fib' will resolve to the caching wrapper rather
than to the undecorated function. Using a reference to the undecorated
function instead (as would have to happen for a sane implementation of
__func__) would be actively harmful since the recursive calls would
bypass the cache unless the lru_cache decorator took steps to change
the way the reference evolved:
@lru_cache()
def fib(n):
if n < 2:
return n
return __func__(n-1) + __func__(n-2) # Not the same, unless lru_cache
adjusts the reference
How would the the reference be adjusted?
This semantic mismatch has actually shifted my opinion from +0 to -1
on the idea. Relying on normal name lookup can be occasionally
inconvenient, but it is at least clear what we're referring to. The
existence of wrapper functions means that "this function" isn't as
clear and unambiguous a phrase as it first seems.
To me it just means that __func__ will remain esoteric, which is just
fine with me. I wouldn't be surprised if there were use cases where it
was *desirable* to have a way (from the inside) to access the
undecorated function (somewhat similar to the thing with modules
below).
Also I really don't want the semantics of decorators to depart from
the original "define the function, then apply this to it" thing. And I
don't want to have to think about the possibility of __func__ being
overridden by the wrapping decorator either (or by anything else).
(I think the reason we get away with it in the PEP 3135 case is that
'class wrappers' typically aren't handled via class decorators but via
metaclasses, which do a better job of playing nicely with the implicit
closure created to handle super() and __class__)
We didn't have class decorators then did we? Anyway I'm not sure what
the semantics are, but I hope they will be such that __class__
references the undecorated, original class object used when the method
was being defined. (If the class statement is executed repeatedly the
__class__ should always refer to the "real" class actually involved in
the method call.)
While referencing the innermost function would likely be wrong in any
case involving function attributes, having the function in a valid
state during decoration will likely mandate filling in the cell
reference before invoking any decorators. Perhaps the best solution
would be to syntactically reference the innermost function, but
provide a clean way in functools to shift the cell reference to a
different function (with functools.wraps doing that automatically).
Hm, making it dynamic sounds wrong. I think it makes more sense to
just share the attribute dict (which is easily done through assignment
to the wrapping function's __dict__).
Huh, I hadn't even thought of that as a potential alternative to the
update() based approach currently used in functools.wraps (I had to
jump into the interactive interpreter to confirm that functions really
do let you swap out their instance dict).
Me too. :-)
But I did remember that we might have made it that way, possibly for
this very use case.
It's interesting that, once again, the status quo deals with this
according to ordinary name resolution rules: any wrapping of the
function will be ignored, *unless* we store the wrapper back into the
original location so the name resolution in the function body will see
it.
This makes sense because it builds complex functionality out of
simpler building blocks. Combining two things together doesn't add any
extra magic -- it's the building blocks themselves that add the magic.
Since the idea of implicitly sharing state between currently
independent wrapper functions scares me, this strikes me as another
reason to switch to '-1'.
I'm still wavering between -0 and +0; I see some merit but I think the
high hopes of some folks for __func__ are unwarranted. Using the same
cell-based mechanism as used for __class__ may or may not be the right
implementation but I don't think that additional hacks based on
mutating that cell should be considered. So it would really be a wash
how it was done (at call time or at func def time). Are you aware of
anything that mutates the __class__ cell? It would seem pretty tricky
to do.
FWIW I don't think I want __func__ to be available at all times, like
someone (the OP?) mentioned. That seems an unnecessary slowdown of
every call / increase of every frame.
Yes, that was my idea, (hence the "as" syntax). However, this discussion is
getting a bit out of my depth now and I don't really know the implications
of my suggestion!
This does seem like an area ripe for subtle decoration related bugs
though, especially by contrast with lazy name based lookup.
TBH, personally I am in most cases unhappy with the aggressive copying
of docstring and other metadata from the wrapped function to the
wrapper function, and wish the idiom had never been invented.
IIRC, I was the one who actually committed the stdlib blessing of the
idiom in the form of 'functools.wraps'. It was definitely a hack to
deal with the increasing prevalence of wrapper functions as decorators
became more popular - naive introspection was giving too many wrong
answers and tweaking the recommended wrapping process so that
'f.__doc__' would work again seemed like a better option than defining
a complex introspection protocol to handle wrapped functions.
I guess you rely more on interactive features like help() whereas I
rely more on browsing the source code. :-)
I still think it was a reasonable way forward (and better than leaving
things as they were), but it's definitely an approach with quite a few
flaws.
You are forgiven. :-)
While this may sound a little hypocritical coming from the author of
PEPs 366 and 395, I'm wary of adding new implicit module globals for
problems with relatively simple and robust alternatives. In this case,
it's fairly easy to get access to the current module using the idiom
Guido quoted:
import sys
_this = sys.modules[__name__]
(or using dict-style access on globals())
Yeah, well, in most cases I find having to reference sys.modules a
distraction and an unwarranted jump into the implementation. It may
not even work: there are some recipes that replace
sys.modules[__name__] with some wrapper object. If __this_module__
existed it would of course refer to the "real" module object involved.
Some invocations of runpy.run_module also cause the 'sys.modules'
based idioms to fail, so there may be a case to be made for this one.
I suspect some folks would use it to avoid global declarations as well
(i.e. by just writing '__module__.x = y').
+1. But what to call it? __module__ is a string in other places.
It might cause the cyclic GC some grief, though,so the implementation
consequences would need to be investigated if someone wanted to pursue
it.
Modules are already involved in much cyclical GC grief, and most have
an infinite lifetime anyway (sys.modules keeps them alive). I doubt it
will get any worse.
Cheers,
Nick.
[1]
http://docs.python.org/dev/library/functools.html#functools.lru_cache
--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
http://mail.python.org/mailman/listinfo/python-ideas
David Townshend