[Python-ideas] Problems (and solutions?) in writing decorators

David Mertz mertz at gnosis.cx
Tue Mar 12 14:15:04 EDT 2019


One of the nice things in wrapt is that Dumpleton lets you use the same
decorator for functions, regular methods, static methods, and class
methods.  Does yours handle that sort of "polymorphism"?

FWIW, thanks for the cool work with your libraries!

I don't think I will want the specific with-or-without parens feature,
since it feels too implicit.  Typing `@deco_factory()` really isn't too
much work for me to use the two characters extra.  But given that I feel
the option is an antipattern, I don't want to add core language features to
make the pattern easier.  Both you and Graham Dumpleton have found
workarounds to get that behavior when it is wanted, but I don't want it to
be "too easy."

FWIW... I think I'd be tempted to use a metaclass approach so that both the
class and instance are callable.  The class would be called with a single
function argument (i.e. a decorator), but if called with any other
signature it would manufacture a callable instance that was parameterized
by the initialization arguments (i.e. a decorator factory).  Actually, I
haven't looked at your actual code, maybe that's what you do.

Best, David...

On Tue, Mar 12, 2019 at 12:44 PM Sylvain MARIE <sylvain.marie at se.com> wrote:

> Thanks David,
>
>
>
> > I did write the book _Functional Programming in Python_, so I'm not
> entirely unfamiliar with function wrappers.
>
>
>
> Nice ! I did not realize ; good job here, congrats! ;)
>
>
>
> --
>
> I carefully read the documentation you pointed at,
> https://wrapt.readthedocs.io/en/latest/decorators.html#decorators-with-optional-arguments
>
> This is the example shown by the author:
>
>
>
> def with_optional_arguments(wrapped=None, myarg1=1, myarg2=2):
>
>     if wrapped is None:
>
>         return functools.partial(with_optional_arguments,
>
>                 myarg1=myarg1, myarg2=myarg2)
>
>
>
>     @wrapt.decorator
>
>     def wrapper(wrapped, instance, args, kwargs):
>
>         return wrapped(*args, **kwargs)
>
>
>
>     return wrapper(wrapped)
>
>
>
> As you can see:
>
>    - the developer has to explicitly handle the no-parenthesis case (the
>    first two lines of code).
>    - And in the next lines of the doc you see his recommendations “For
>    this to be used in this way, it is a requirement that the decorator
>    arguments be supplied as keyword arguments. If using Python 3, the
>    requirement to use keyword only arguments can again be enforced using the
>    keyword only argument syntax.”
>    - Finally, but this is just a comment: this is not “flat” mode but
>    nested mode (the code returns a decorator that returns a function wrapper)
>
>
>
> So if I’m not misleading, the problem is not really solved. Or at least,
> not the way I would like the problem to be solved : it is solved here (a)
> only if the developer takes extra care and (b) reduces the way the
> decorator can be used (no positional args). This is precisely because I was
> frustrated by all these limitations that depend on the desired signature
> that I wrote decopatch. As a developer I do not want to care about which
> trick to use in which situation (mandatory args, optional args,
> var-positional args..). My decorators may change signature during the
> development cycle, and if I frequently had to change trick during
> development as I changed the signature - that is a bit tiring.
>
>
>
> --
>
> Concerning creation of signature-preserving wrappers: @wrapt.decorator is
> not signature preserving, I just checked it. You can check it with the
> following experiment:
>
>
>
> *def *dummy(wrapped):
>     @wrapt.decorator
>     *def *wrapper(wrapped, instance, args, kwargs):
>         print(*"wrapper called"*)
>         *return *wrapped(*args, **kwargs)
>     *return *wrapper(wrapped)
>
>
>
> @dummy
> *def *function(a, b):
>     *pass*
>
>
>
> If you call
>
>
>
> function(1)
>
>
>
> you will see that “wrapper called” is displayed before the TypeError is
> raised…
>
>
>
> The signature-preserving equivalent of @wrapt.decorator,
> @decorator.decorator, is the source of inspiration for makefun. You can
> see `makefun` as a generalization of the core of `decorator`.
>
>
>
> --
>
> > I'm not sure I ever want them (decopatch and makefun) independently in
> practice
>
>
>
> I totally understand.
>
> But some projects actually need makefun and not decopatch because their
> need is different: they just want to create a function dynamically. This is
> low-level tooling, really.
>
> So at least now there is a clear separation of concerns (and dedicated
> issues management/roadmap, which is also quite convenient. Not to mention
> readability !).
>
> To cover your concern: decopatch depends on makefun, so both come at the
> same time when you install decopatch, and decopatch by default relies on
> makefun when you use it in “double-flat” mode to create wrappers as
> explained here https://smarie.github.io/python-decopatch/#even-simpler
>
> --
>
>
>
> Thanks again for this discussion! It is challenging but it is necessary,
> to make sure I did not answer a non-existent need ;)
>
> Kind regards
>
>
>
> --
>
> Sylvain
>
>
>
> *De :* David Mertz <mertz at gnosis.cx>
> *Envoyé :* mardi 12 mars 2019 15:30
> *À :* Sylvain MARIE <sylvain.marie at se.com>
> *Cc :* Steven D'Aprano <steve at pearwood.info>; python-ideas <
> python-ideas at python.org>
> *Objet :* Re: [Python-ideas] Problems (and solutions?) in writing
> decorators
>
>
>
> [External email: Use caution with links and attachments]
> ------------------------------
>
>
>
> The documentation for wrapt mentions:
>
>
> Decorators With Optional Arguments
>
> Although opinion can be mixed about whether the pattern is a good one, if
> the decorator arguments all have default values, it is also possible to
> implement decorators which have optional arguments.
>
> As Graham hints in his docs, I think repurposing decorator factories as
> decorators is an antipattern. Explicit is better than implicit.
>
>
>
> While I *do* understands that what decotools and makefun do are
> technically independent, I'm not sure I ever want them independently in
> practice. I did write the book _Functional Programming in Python_, so I'm
> not entirely unfamiliar with function wrappers.
>
> On Tue, Mar 12, 2019, 10:18 AM David Mertz <mertz at gnosis.cx> wrote:
>
> The wrapt module I linked to (not funtools.wraps) provides all the
> capabilities you mention since 2013. It allows mixed use of decorators as
> decorator factories. It has a flat style.
>
>
>
> There are some minor API difference between your libraries and wrapt, but
> the concept is very similar. Since yours is something new, I imagine you
> perceive some win over what wrapt does.
>
> On Tue, Mar 12, 2019, 9:52 AM Sylvain MARIE <sylvain.marie at se.com> wrote:
>
> David, Steven,
>
> Thanks for your interest !
>
> As you probably know, decorators and function wrappers are *completely
> different concepts*. A decorator can directly return the decorated function
> (or class), it does not have to return a wrapper. Even more, it can
> entirely replace the decorated item with something else (not even a
> function or class!). Try it: it is possible to write a decorator to replace
> a function with an integer, even though it is probably not quite useful :)
>
> `decopatch` helps you write decorators, whatever they are. It "just"
> solves the annoying issue of having to handle the no-parenthesis and
> with-parenthesis calls. In addition as a 'goodie', it proposes two
> development styles: *nested* (you have to return a function) and *flat*
> (you directly write what will happen when the decorator is applied to
> something).
> --
> Now about creating signature-preserving function wrappers (in a decorator,
> or outside a decorator - again, that's not related). That use case is
> supposed to be covered by functools.wrapt. Unfortunately as explained here
> https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F308999%2Fwhat-does-functools-wraps-do%2F55102697%2355102697&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186393926&sdata=Dmkd0Ld1HsuCJtJDCS6GDUgwH3GO66zwYMpZ9Ow%2B18k%3D&reserved=0>
> this is not the case because with functools.wrapt:
>  - the wrapper code will execute even when the provided arguments are
> invalid.
>  - the wrapper code cannot easily access an argument using its name, from
> the received *args, **kwargs. Indeed one would have to handle all cases
> (positional, keyword, default) and therefore to use something like
> Signature.bind().
>
> For this reason I proposed a replacement in `makefun`:
> https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-makefun%2F%23signature-preserving-function-wrappers&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186403936&sdata=SO319cmnKXiUK6Zek%2FSioHIhFfnQCZRpAF3kim84mv8%3D&reserved=0>
> --
> Now bridging the gap. Of course a very interesting use cases for
> decorators is to create decorators that create a signature-preserving
> wrapper. It is possible to combine decopatch and makefun for this:
> https://smarie.github.io/python-decopatch/#3-creating-function-wrappers
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.github.io%2Fpython-decopatch%2F%233-creating-function-wrappers&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186403936&sdata=FpCGLYwJHFJMWmStJhI%2F7NzXxmJPwC3hf9afg2zfXik%3D&reserved=0>
> .
> Decopatch even proposes a "double-flat" development style where you
> directly write the wrapper body, as explained in the doc.
>
> Did I answer your questions ?
> Thanks again for the quick feedback !
> Best,
>
> Sylvain
>
> -----Message d'origine-----
> De : Python-ideas <python-ideas-bounces+sylvain.marie=se.com at python.org>
> De la part de Steven D'Aprano
> Envoyé : mardi 12 mars 2019 12:30
> À : python-ideas at python.org
> Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators
>
> [External email: Use caution with links and attachments]
>
> ________________________________
>
>
>
> On Tue, Mar 12, 2019 at 09:36:41AM +0000, Sylvain MARIE via Python-ideas
> wrote:
>
> > I therefore proposed
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma
> > rie.github.io
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frie.github.io&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186413941&sdata=8QNKZapq0Z9VQtREcqt4f8EcKbTteHWqiAhD1kVnDyc%3D&reserved=0>
> %2Fpython-makefun%2F&data=02%7C01%7Csylvain.marie%40s
> > e.com
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fe.com&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186413941&sdata=73ekIlXJ7nwxcsHYz91JotQ%2F0eGH7h6YC2e3U5pqXow%3D&reserved=0>
> %7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae
> > 68fef%7C0%7C0%7C636879872385158085&sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2
> > Fekk35bnYGvmEFJyCXaLDyLm9I%3D&reserved=0 . In particular it
> > provides an equivalent of `@functools.wraps` that is truly
> > signature-preserving
>
> Tell us more about that please. I'm very interested in getting decorators
> preserve the original signature.
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
>
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas&data=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085&sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3D&reserved=0
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186423950&sdata=X1o6EwD3M3jw6n%2Fqrsx0IFoBK8N2nsoSq4nKDgcmAZQ%3D&reserved=0>
> Code of Conduct:
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F&data=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085&sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3D&reserved=0
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F&data=02%7C01%7Csylvain.marie%40se.com%7Ca7dee43a3bdf494e37c508d6a6f73f7d%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879978186423950&sdata=gatcCdLn8s9Zhh26txhKRsqHIYkKpEghOTNmVJ%2BDZJw%3D&reserved=0>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> ______________________________________________________________________
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> ______________________________________________________________________
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20190312/76b14031/attachment-0001.html>


More information about the Python-ideas mailing list