Re: [Python-ideas] Problems (and solutions?) in writing decorators

March 12, 2019

      Thanks David,
...
I did write the book _Functional Programming in Python_, so I'm not entirely unfamiliar with function wrappers.
Nice ! I did not realize ; good job here, congrats! ;)

--
I carefully read the documentation you pointed at, https://wrapt.readthedocs.io/en/latest/decorators.html#decorators-with-optio...
This is the example shown by the author:

def with_optional_arguments(wrapped=None, myarg1=1, myarg2=2):
    if wrapped is None:
        return functools.partial(with_optional_arguments,
                myarg1=myarg1, myarg2=myarg2)

    @wrapt.decorator
    def wrapper(wrapped, instance, args, kwargs):
        return wrapped(*args, **kwargs)

    return wrapper(wrapped)

As you can see:

  *   the developer has to explicitly handle the no-parenthesis case (the first two lines of code).
  *   And in the next lines of the doc you see his recommendations “For this to be used in this way, it is a requirement that the decorator arguments be supplied as keyword arguments. If using Python 3, the requirement to use keyword only arguments can again be enforced using the keyword only argument syntax.”
  *   Finally, but this is just a comment: this is not “flat” mode but nested mode (the code returns a decorator that returns a function wrapper)

So if I’m not misleading, the problem is not really solved. Or at least, not the way I would like the problem to be solved : it is solved here (a) only if the developer takes extra care and (b) reduces the way the decorator can be used (no positional args). This is precisely because I was frustrated by all these limitations that depend on the desired signature that I wrote decopatch. As a developer I do not want to care about which trick to use in which situation (mandatory args, optional args, var-positional args..). My decorators may change signature during the development cycle, and if I frequently had to change trick during development as I changed the signature - that is a bit tiring.

--
Concerning creation of signature-preserving wrappers: @wrapt.decorator is not signature preserving, I just checked it. You can check it with the following experiment:

def dummy(wrapped):
    @wrapt.decorator
    def wrapper(wrapped, instance, args, kwargs):
        print("wrapper called")
        return wrapped(*args, **kwargs)
    return wrapper(wrapped)

@dummy
def function(a, b):
    pass

If you call

function(1)

you will see that “wrapper called” is displayed before the TypeError is raised…

The signature-preserving equivalent of @wrapt.decorator, @decorator.decorator, is the source of inspiration for makefun. You can see `makefun` as a generalization of the core of `decorator`.

--
...
I'm not sure I ever want them (decopatch and makefun) independently in practice
I totally understand.
But some projects actually need makefun and not decopatch because their need is different: they just want to create a function dynamically. This is low-level tooling, really.
So at least now there is a clear separation of concerns (and dedicated issues management/roadmap, which is also quite convenient. Not to mention readability !).
To cover your concern: decopatch depends on makefun, so both come at the same time when you install decopatch, and decopatch by default relies on makefun when you use it in “double-flat” mode to create wrappers as explained here https://smarie.github.io/python-decopatch/#even-simpler
--

Thanks again for this discussion! It is challenging but it is necessary, to make sure I did not answer a non-existent need ;)
Kind regards

--
Sylvain

De : David Mertz <mertz@gnosis.cx>
Envoyé : mardi 12 mars 2019 15:30
À : Sylvain MARIE <sylvain.marie@se.com>
Cc : Steven D'Aprano <steve@pearwood.info>; python-ideas <python-ideas@python.org>
Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators

[External email: Use caution with links and attachments]

________________________________

The documentation for wrapt mentions:

Decorators With Optional Arguments

Although opinion can be mixed about whether the pattern is a good one, if the decorator arguments all have default values, it is also possible to implement decorators which have optional arguments.
As Graham hints in his docs, I think repurposing decorator factories as decorators is an antipattern. Explicit is better than implicit.

While I *do* understands that what decotools and makefun do are technically independent, I'm not sure I ever want them independently in practice. I did write the book _Functional Programming in Python_, so I'm not entirely unfamiliar with function wrappers.
On Tue, Mar 12, 2019, 10:18 AM David Mertz <mertz@gnosis.cx<mailto:mertz@gnosis.cx>> wrote:
The wrapt module I linked to (not funtools.wraps) provides all the capabilities you mention since 2013. It allows mixed use of decorators as decorator factories. It has a flat style.

There are some minor API difference between your libraries and wrapt, but the concept is very similar. Since yours is something new, I imagine you perceive some win over what wrapt does.
On Tue, Mar 12, 2019, 9:52 AM Sylvain MARIE <sylvain.marie@se.com<mailto:sylvain.marie@se.com>> wrote:
David, Steven,

Thanks for your interest !

As you probably know, decorators and function wrappers are *completely different concepts*. A decorator can directly return the decorated function (or class), it does not have to return a wrapper. Even more, it can entirely replace the decorated item with something else (not even a function or class!). Try it: it is possible to write a decorator to replace a function with an integer, even though it is probably not quite useful :)

`decopatch` helps you write decorators, whatever they are. It "just" solves the annoying issue of having to handle the no-parenthesis and with-parenthesis calls. In addition as a 'goodie', it proposes two development styles: *nested* (you have to return a function) and *flat* (you directly write what will happen when the decorator is applied to something).
--
Now about creating signature-preserving function wrappers (in a decorator, or outside a decorator - again, that's not related). That use case is supposed to be covered by functools.wrapt. Unfortunately as explained here https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F308999%2Fwhat-does-functools-wraps-do%2F55102697%2355102697&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186393926&sdata=Dmkd0Ld1HsuCJtJDCS6GDUgwH3GO66zwYMpZ9Ow%2B18k%3D&reserved=0> this is not the case because with functools.wrapt:
 - the wrapper code will execute even when the provided arguments are invalid.
 - the wrapper code cannot easily access an argument using its name, from the received *args, **kwargs. Indeed one would have to handle all cases (positional, keyword, default) and therefore to use something like Signature.bind().

For this reason I proposed a replacement in `makefun`: https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-makefun%2F%23signature-preserving-function-wrappers&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186403936&sdata=SO319cmnKXiUK6Zek%2FSioHIhFfnQCZRpAF3kim84mv8%3D&reserved=0>
--
Now bridging the gap. Of course a very interesting use cases for decorators is to create decorators that create a signature-preserving wrapper. It is possible to combine decopatch and makefun for this: https://smarie.github.io/python-decopatch/#3-creating-function-wrappers<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-decopatch%2F%233-creating-function-wrappers&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186403936&sdata=FpCGLYwJHFJMWmStJhI%2F7NzXxmJPwC3hf9afg2zfXik%3D&reserved=0> .
Decopatch even proposes a "double-flat" development style where you directly write the wrapper body, as explained in the doc.

Did I answer your questions ?
Thanks again for the quick feedback !
Best,

Sylvain

-----Message d'origine-----
De : Python-ideas <python-ideas-bounces+sylvain.marie=se.com@python.org<mailto:se.com@python.org>> De la part de Steven D'Aprano
Envoyé : mardi 12 mars 2019 12:30
À : python-ideas@python.org<mailto:python-ideas@python.org>
Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators

[External email: Use caution with links and attachments]

________________________________

On Tue, Mar 12, 2019 at 09:36:41AM +0000, Sylvain MARIE via Python-ideas wrote:
...
I therefore proposed
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma
rie.github.io<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frie.github.io&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186413941&sdata=8QNKZapq0Z9VQtREcqt4f8EcKbTteHWqiAhD1kVnDyc%3D&reserved=0>%2Fpython-makefun%2F&data=02%7C01%7Csylvain.marie%40s
e.com<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fe.com&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186413941&sdata=73ekIlXJ7nwxcsHYz91JotQ%2F0eGH7h6YC2e3U5pqXow%3D&reserved=0>%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae
68fef%7C0%7C0%7C636879872385158085&sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2
Fekk35bnYGvmEFJyCXaLDyLm9I%3D&reserved=0 . In particular it
provides an equivalent of `@functools.wraps` that is truly
signature-preserving
Tell us more about that please. I'm very interested in getting decorators preserve the original signature.

--
Steven
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org<mailto:Python-ideas@python.org>
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas&data=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085&sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3D&reserved=0<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186423950&sdata=X1o6EwD3M3jw6n%2Fqrsx0IFoBK8N2nsoSq4nKDgcmAZQ%3D&reserved=0>
Code of Conduct: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F&data=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085&sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3D&reserved=0<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186423950&sdata=gatcCdLn8s9Zhh26txhKRsqHIYkKpEghOTNmVJ%2BDZJw%3D&reserved=0>

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________