Add decorator_with_params function to functools module
In the current python codebases, there are a lot of decorators with parameters, and their code in most cases contains a lot of code boilerplates. The purpose of this feature is to add `decorator_with_params` (I think the name can be changed to smth better) function to `functools` module. This function will allow using a decorated function as a simple decorator and decorator with parameter. I believe the real example will bring more context, for instance, `typing._tp_cache` function can be rewritten from: ``` def _tp_cache(func=None, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ def decorator(func): cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass return func(*args, **kwds) return inner if func is not None: return decorator(func) return decorator ``` To ``` @decorator_with_params def _tp_cache(func, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner ``` And `_tp_cache` can be used as: ``` @_tp_cache(typed=True) def __getitem__(self, parameters): return self._getitem(self, parameters) ... @_tp_cache def __getitem__(self, parameters): return self._getitem(self, parameters) ``` Proposed implementation is quite simple (I used it in a lot of projects): ``` def decorator_with_params(deco): @wraps(deco) def decorator(func=None, /, *args, **kwargs): if func is not None: return deco(func, *args, **kwargs) def wrapper(f): return deco(f, *args, **kwargs) return wrapper return decorator ``` I believe it will be a great enhancement for the Python standard library and will save such important indentations. I have already opened an issue https://bugs.python.org/issue42455 (Sorry about that I didn't know that smth like python-ideas exists).
You can use classes as decorators, it's quite more simple: https://towardsdatascience.com/using-class-decorators-in-python-2807ef52d273...
The idea of `decorator_factory` is to eliminate boilerplate code that used to create a decorator with parameters. A perfect example is how `dataclass` decorator can be simplified from this: ``` def dataclass(cls=None, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): def wrap(cls): return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) # See if we're being called as @dataclass or @dataclass(). if cls is None: # We're called with parens. return wrap # We're called as @dataclass without parens. return wrap(cls) ``` To this: @functools.decorator_factory def dataclass(cls, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) ```
Is it really worth it? Fact is, while it can shave off some lines of code, I think it is interesting to know _which_ lines of code - Usually when one writes a decorator, it is expected that they will know what they are writing, and will want to be in control of their code. Delegating this to a decorator-decorator that is to be copied and pasted, and will definitely change call-order, and when your decorator is called, is something that I, at least, would be wary to use. Meanwhile, when I want this pattern, it really takes me 2 LoC inside the decorator to have the same functionality, and still be 100% in control of when my function is called: ``` from functools import partial def mydecorator(func=None, /, *, param1=None, **kwargs): if func is None: return partial(mydcorator, param1=None, **kwargs) # decorator code goes here ... ``` So, yes, your proposal has some utility - but I consider it to be marginal - it is the kind of stuff that I'd rather see on a 3rdy party package with extra-stuff to help building decorators than on stdlib. On Mon, 30 Nov 2020 at 17:04, Yurii Karabas <1998uriyyo@gmail.com> wrote:
Sorry for the long response. After investigation, I have found that there is a third-party library called `decorator` which do smth similar to what I propose (I didn't know about that library when I created this thread). As I said, the idea of `decorator_factory` is to create a unified way of how to deal with simple decorator with parameters. For me, this is smth similar to `functools.lru_cache`, before it was introduced I written such functionality by myself many times. Thank you for your opinion, and sorry about the long response.
After discussion at the python issue tracker, the better name `decorator_factory` was proposed. I have just gone through the standard library code to find places where `decorator_factory` can be used. 1. `dataclasses.dataclass` ``` def dataclass(cls=None, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): """Returns the same class as was passed in, with dunder methods added based on the fields defined in the class. Examines PEP 526 __annotations__ to determine fields. If init is true, an __init__() method is added to the class. If repr is true, a __repr__() method is added. If order is true, rich comparison dunder methods are added. If unsafe_hash is true, a __hash__() method function is added. If frozen is true, fields may not be assigned to after instance creation. """ def wrap(cls): return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) # See if we're being called as @dataclass or @dataclass(). if cls is None: # We're called with parens. return wrap # We're called as @dataclass without parens. return wrap(cls) ``` ``` @functools.decorator_factory def dataclass(cls, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): """Returns the same class as was passed in, with dunder methods added based on the fields defined in the class. Examines PEP 526 __annotations__ to determine fields. If init is true, an __init__() method is added to the class. If repr is true, a __repr__() method is added. If order is true, rich comparison dunder methods are added. If unsafe_hash is true, a __hash__() method function is added. If frozen is true, fields may not be assigned to after instance creation. """ return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) ``` 2. `functools.lru_cache` ``` def lru_cache(maxsize=128, typed=False): """Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bound. If *typed* is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results. Arguments to the cached function must be hashable. View the cache statistics named tuple (hits, misses, maxsize, currsize) with f.cache_info(). Clear the cache and statistics with f.cache_clear(). Access the underlying function with f.__wrapped__. See: http://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_...) """ # Users should only access the lru_cache through its public API: # cache_info, cache_clear, and f.__wrapped__ # The internals of the lru_cache are encapsulated for thread safety and # to allow the implementation to change (including a possible C version). if isinstance(maxsize, int): # Negative maxsize is treated as 0 if maxsize < 0: maxsize = 0 elif callable(maxsize) and isinstance(typed, bool): # The user_function was passed in directly via the maxsize argument user_function, maxsize = maxsize, 128 wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) wrapper.cache_parameters = lambda : {'maxsize': maxsize, 'typed': typed} return update_wrapper(wrapper, user_function) elif maxsize is not None: raise TypeError( 'Expected first argument to be an integer, a callable, or None') def decorating_function(user_function): wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) wrapper.cache_parameters = lambda : {'maxsize': maxsize, 'typed': typed} return update_wrapper(wrapper, user_function) return decorating_function ``` ``` @decorator_factory def lru_cache(user_function, /, maxsize=128, typed=False): """Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bound. If *typed* is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results. Arguments to the cached function must be hashable. View the cache statistics named tuple (hits, misses, maxsize, currsize) with f.cache_info(). Clear the cache and statistics with f.cache_clear(). Access the underlying function with f.__wrapped__. See: http://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_...) """ # Users should only access the lru_cache through its public API: # cache_info, cache_clear, and f.__wrapped__ # The internals of the lru_cache are encapsulated for thread safety and # to allow the implementation to change (including a possible C version). if isinstance(maxsize, int): # Negative maxsize is treated as 0 if maxsize < 0: maxsize = 0 elif maxsize is not None: raise TypeError( 'Expected first argument to be an integer, a callable, or None') wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) wrapper.cache_parameters = lambda : {'maxsize': maxsize, 'typed': typed} return update_wrapper(wrapper, user_function) ``` 3. `reprlib.recursive_repr` ``` def recursive_repr(fillvalue='...'): 'Decorator to make a repr function return fillvalue for a recursive call' def decorating_function(user_function): repr_running = set() def wrapper(self): key = id(self), get_ident() if key in repr_running: return fillvalue repr_running.add(key) try: result = user_function(self) finally: repr_running.discard(key) return result # Can't use functools.wraps() here because of bootstrap issues wrapper.__module__ = getattr(user_function, '__module__') wrapper.__doc__ = getattr(user_function, '__doc__') wrapper.__name__ = getattr(user_function, '__name__') wrapper.__qualname__ = getattr(user_function, '__qualname__') wrapper.__annotations__ = getattr(user_function, '__annotations__', {}) return wrapper return decorating_function ``` ``` @decorator_factory def recursive_repr(user_function, fillvalue='...'): 'Decorator to make a repr function return fillvalue for a recursive call' repr_running = set() def wrapper(self): key = id(self), get_ident() if key in repr_running: return fillvalue repr_running.add(key) try: result = user_function(self) finally: repr_running.discard(key) return result # Can't use functools.wraps() here because of bootstrap issues wrapper.__module__ = getattr(user_function, '__module__') wrapper.__doc__ = getattr(user_function, '__doc__') wrapper.__name__ = getattr(user_function, '__name__') wrapper.__qualname__ = getattr(user_function, '__qualname__') wrapper.__annotations__ = getattr(user_function, '__annotations__', {}) return wrapper ``` 4. `typing._tp_cache` ``` def _tp_cache(func=None, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ def decorator(func): cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner if func is not None: return decorator(func) return decorator ``` ``` @functools.decorator_factory def _tp_cache(func, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner ```
You can use classes as decorators, it's quite more simple: https://towardsdatascience.com/using-class-decorators-in-python-2807ef52d273...
The idea of `decorator_factory` is to eliminate boilerplate code that used to create a decorator with parameters. A perfect example is how `dataclass` decorator can be simplified from this: ``` def dataclass(cls=None, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): def wrap(cls): return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) # See if we're being called as @dataclass or @dataclass(). if cls is None: # We're called with parens. return wrap # We're called as @dataclass without parens. return wrap(cls) ``` To this: @functools.decorator_factory def dataclass(cls, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) ```
Is it really worth it? Fact is, while it can shave off some lines of code, I think it is interesting to know _which_ lines of code - Usually when one writes a decorator, it is expected that they will know what they are writing, and will want to be in control of their code. Delegating this to a decorator-decorator that is to be copied and pasted, and will definitely change call-order, and when your decorator is called, is something that I, at least, would be wary to use. Meanwhile, when I want this pattern, it really takes me 2 LoC inside the decorator to have the same functionality, and still be 100% in control of when my function is called: ``` from functools import partial def mydecorator(func=None, /, *, param1=None, **kwargs): if func is None: return partial(mydcorator, param1=None, **kwargs) # decorator code goes here ... ``` So, yes, your proposal has some utility - but I consider it to be marginal - it is the kind of stuff that I'd rather see on a 3rdy party package with extra-stuff to help building decorators than on stdlib. On Mon, 30 Nov 2020 at 17:04, Yurii Karabas <1998uriyyo@gmail.com> wrote:
Sorry for the long response. After investigation, I have found that there is a third-party library called `decorator` which do smth similar to what I propose (I didn't know about that library when I created this thread). As I said, the idea of `decorator_factory` is to create a unified way of how to deal with simple decorator with parameters. For me, this is smth similar to `functools.lru_cache`, before it was introduced I written such functionality by myself many times. Thank you for your opinion, and sorry about the long response.
After discussion at the python issue tracker, the better name `decorator_factory` was proposed. I have just gone through the standard library code to find places where `decorator_factory` can be used. 1. `dataclasses.dataclass` ``` def dataclass(cls=None, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): """Returns the same class as was passed in, with dunder methods added based on the fields defined in the class. Examines PEP 526 __annotations__ to determine fields. If init is true, an __init__() method is added to the class. If repr is true, a __repr__() method is added. If order is true, rich comparison dunder methods are added. If unsafe_hash is true, a __hash__() method function is added. If frozen is true, fields may not be assigned to after instance creation. """ def wrap(cls): return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) # See if we're being called as @dataclass or @dataclass(). if cls is None: # We're called with parens. return wrap # We're called as @dataclass without parens. return wrap(cls) ``` ``` @functools.decorator_factory def dataclass(cls, /, *, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False): """Returns the same class as was passed in, with dunder methods added based on the fields defined in the class. Examines PEP 526 __annotations__ to determine fields. If init is true, an __init__() method is added to the class. If repr is true, a __repr__() method is added. If order is true, rich comparison dunder methods are added. If unsafe_hash is true, a __hash__() method function is added. If frozen is true, fields may not be assigned to after instance creation. """ return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen) ``` 2. `functools.lru_cache` ``` def lru_cache(maxsize=128, typed=False): """Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bound. If *typed* is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results. Arguments to the cached function must be hashable. View the cache statistics named tuple (hits, misses, maxsize, currsize) with f.cache_info(). Clear the cache and statistics with f.cache_clear(). Access the underlying function with f.__wrapped__. See: http://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_...) """ # Users should only access the lru_cache through its public API: # cache_info, cache_clear, and f.__wrapped__ # The internals of the lru_cache are encapsulated for thread safety and # to allow the implementation to change (including a possible C version). if isinstance(maxsize, int): # Negative maxsize is treated as 0 if maxsize < 0: maxsize = 0 elif callable(maxsize) and isinstance(typed, bool): # The user_function was passed in directly via the maxsize argument user_function, maxsize = maxsize, 128 wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) wrapper.cache_parameters = lambda : {'maxsize': maxsize, 'typed': typed} return update_wrapper(wrapper, user_function) elif maxsize is not None: raise TypeError( 'Expected first argument to be an integer, a callable, or None') def decorating_function(user_function): wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) wrapper.cache_parameters = lambda : {'maxsize': maxsize, 'typed': typed} return update_wrapper(wrapper, user_function) return decorating_function ``` ``` @decorator_factory def lru_cache(user_function, /, maxsize=128, typed=False): """Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache can grow without bound. If *typed* is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results. Arguments to the cached function must be hashable. View the cache statistics named tuple (hits, misses, maxsize, currsize) with f.cache_info(). Clear the cache and statistics with f.cache_clear(). Access the underlying function with f.__wrapped__. See: http://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_...) """ # Users should only access the lru_cache through its public API: # cache_info, cache_clear, and f.__wrapped__ # The internals of the lru_cache are encapsulated for thread safety and # to allow the implementation to change (including a possible C version). if isinstance(maxsize, int): # Negative maxsize is treated as 0 if maxsize < 0: maxsize = 0 elif maxsize is not None: raise TypeError( 'Expected first argument to be an integer, a callable, or None') wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) wrapper.cache_parameters = lambda : {'maxsize': maxsize, 'typed': typed} return update_wrapper(wrapper, user_function) ``` 3. `reprlib.recursive_repr` ``` def recursive_repr(fillvalue='...'): 'Decorator to make a repr function return fillvalue for a recursive call' def decorating_function(user_function): repr_running = set() def wrapper(self): key = id(self), get_ident() if key in repr_running: return fillvalue repr_running.add(key) try: result = user_function(self) finally: repr_running.discard(key) return result # Can't use functools.wraps() here because of bootstrap issues wrapper.__module__ = getattr(user_function, '__module__') wrapper.__doc__ = getattr(user_function, '__doc__') wrapper.__name__ = getattr(user_function, '__name__') wrapper.__qualname__ = getattr(user_function, '__qualname__') wrapper.__annotations__ = getattr(user_function, '__annotations__', {}) return wrapper return decorating_function ``` ``` @decorator_factory def recursive_repr(user_function, fillvalue='...'): 'Decorator to make a repr function return fillvalue for a recursive call' repr_running = set() def wrapper(self): key = id(self), get_ident() if key in repr_running: return fillvalue repr_running.add(key) try: result = user_function(self) finally: repr_running.discard(key) return result # Can't use functools.wraps() here because of bootstrap issues wrapper.__module__ = getattr(user_function, '__module__') wrapper.__doc__ = getattr(user_function, '__doc__') wrapper.__name__ = getattr(user_function, '__name__') wrapper.__qualname__ = getattr(user_function, '__qualname__') wrapper.__annotations__ = getattr(user_function, '__annotations__', {}) return wrapper ``` 4. `typing._tp_cache` ``` def _tp_cache(func=None, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ def decorator(func): cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner if func is not None: return decorator(func) return decorator ``` ``` @functools.decorator_factory def _tp_cache(func, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner ```
participants (3)
-
Joao S. O. Bueno -
Marco Sulla -
Yurii Karabas