[Python-ideas] make __closure__ writable

[Eric Snow]

On Tue, Mar 20, 2012 at 6:06 AM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> I did provide such example earlier in this thread.  I'm copying and
> pasting it to this mail.  Please read the example carefully, as it
> explains why returning new types.FunctionType() is not enough.
>
> ----
>
> Yes, your approach will work if your decorator is the only one applied.
> But, as I said, if you have many of them (see below), you can't just
> return a new function out of your decorator, you need to change the
> underlying "in-place".  Consider the following:
>
> def modifier(func):
>  orig_func = func
>
>  while func.__wrapped__:
>   func = func.__wrapped__
>
>  # patch func.__code__ and func.__closure__
>  return orig_func # no need to wrap anything
>
> def some_decorator(func):
>  def wrapper(*args, **kwargs):
>     # some code
>     return func(*args, **kwargs)
>  functools.wraps(wrapper, func)
>  return wrapper
>
> @modifier
> @some_decorator
> def foo():
>  # this code needs to be verified/augmented/etc

Couldn't something like the following work?

def modifier(func):
    """Traverse the decorator "stack" and patch the bottom-most
wrapped function."""
    # relies on __wrapped__ being set at each level of the decorator
stack and on the
    # wrapped function being bound in func.__closure__.

    if not hasattr(func, "__wrapped__"):
        # patch func.__code__ and func.__closure__
        code = ...
        closure = ...
    else:
        code = func.__code__
        closure = list(func.__closure__)
        closure[closure.index(func.__wrapped__)] = modifier(func.__wrapped__)

    return type(func)(code, func.__globals__, func.__name__,
func.__defaults__, tuple(closure))


Also, I'm guessing that your actual use-case looks more like the following:

    from some_third_party_module import foo
    #assert foo.__wrapped__ == foo.__closure__[0]
    foo = modifier(foo)  # hacks up foo.__wrapped__

Hacking the innards of an existing function object is touchy stuff,
probably the riskiest kind of monkey-patching.  You're basically
taking the chance of breaking (in ugly, unexpected ways) other code
that uses that function you just hacked.  Still, there are certainly
valid use-cases (and we're all consenting adults here).

However, I'd hate for projects to start getting blamed for
difficult-to-debug problems that are the result of some other project
that did this sort of hack.  It's nice when your expectations for a
function's behavior (or any code for that matter) can remain stable,
regardless of what libraries are installed along-side.

-eric

p.s. I want to reiterate my understanding that nearly everything
involving the internals of functions is pretty delicate (read:
fragile), in part due to being the focal point for optimization.
Hacking it like this is therefore a delicate undertaking and
definitely skirts the edges of creating implementation-specific code.
Don't shy away.  Just be extra cautious.

>
> So, in the above snippet, if you don't want to discard the
> @some_decorator by returning a new function object, you need to modify
> the 'foo' from the @modifier.
>
> In a complex framework, where you can't guarantee that your magic
> decorator will always be called first, rewriting the __closure__
> attribute is the only way.
>
> Again, since the __code__ attribute is modifiable, and __closure__
> works in tight conjunction with it, I see no point in protecting it.
>
> On 2012-03-20, at 5:34 AM, Mark Shannon wrote:
>
>> Yury Selivanov wrote:
>>> I've created an issue: http://bugs.python.org/issue14369
>>
>> I think that creating an issue may be premature, given that you have had
>> no positive feedback on the idea.
>>
>> I still think making __closure__ mutable is unnecessary.
>> If you insist that it is it, then please provide an example which would
>> work with your proposed change, but cannot be made to work using
>> types.FunctionType() to create a new closure.
>>
>> Cheers,
>> Mark.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas