Deprecation utilities for the warnings module
(Apologies if it's a duplicate. I originally posted to python-ideas@googlegroups.com) I've been recently working on an internal library written entirely in Python. The library while being under development was as actively used by other teams. The product was under pressure and hasty design decisions were made. Certain interfaces were exposed before scope of the problem was entirely clear. Didn't take long before the code started to beg for refactoring. But compatibility needed to be preserved. As a result a set of utilities, which I'd like to share, came to life. --- The warnings module is a robust foundation. Authors can warn what should be avoided and users can choose how to act. However in its current form it still requires utilities to be useful. E.g. it's non-trivial to properly replace a deprecated class. I'd like to propose an extension for the warnings module to address this problem. The extensions consists of 4 decorators: - @deprecated - @obsolete - @deprecated_arg - @obsolete_arg The @deprecated decorator marks an object to issue a warning if it's used: - Callable objects issue a warning upon a call - Property attributes issue a warning upon an access - Classes issue a warning upon instantiation and subclassing (directly or via a metaclass) The @obsolete decorator marks an object in the same way as @deprecated does but forwards usage to the replacement: - Callable object redirect the call - Property attribute redirect the access (get / set / del) - Class is replaced in a way that during both instantiation and subclassing replacement is used In case of classes extra care is taken to preserve validity of existing isinstance and issubclass checks. The @deprecated_arg and @obsolete_arg work with signatures of callable objects. Upon a call either a warning is issued or an argument mapping is performed. Please take a look at the implementation and a few examples: https://gist.github.com/Kentzo/53df97c7a54609d3febf5f8eb6b67118 Why I think it should be a part of stdlib: - Library authors are reluctant to add dependencies especially when it's for internal usage - Ease of use will reduce compatibility problems and ease migration burden since the soltion will be readily available - IDEs and static analyzers are more likely to support it --- Joshua Harlow shared a link to OpenStack's debtcollector: https://docs.openstack.org/debtcollector/latest/reference/index.html Certain aspects of my implementation are inspired by it. Please let me know what you think about the idea in general and implementation in particular. If that's something the community is interested in, I'm willing to work on it.
I have to say, this would be amazing! I've basically had to create many of these by hand over time, and I doubt I'm the only person who's wondered how this isn't in the stdlib! On Thu, Sep 13, 2018, 7:18 PM Ilya Kulakov <kulakov.ilya@gmail.com> wrote:
-- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/
I'd like to propose an extension for the warnings module to address this problem.
I like all of that. The only issue I have with it is that the warnings module is designed to namespace depredations so you can turn them on per library and this code doesn’t seem to handle that. We really want to avoid libraries using these convenience functions instead of creating their own warning that can be properly filtered. I’m not sure what the solution to this would be. Maybe just accessing <yourmodule>.DeprecationWarning dynamically? Seems a bit magical though. / Anders
Op vr 14 sep. 2018 om 08:07 schreef Anders Hovmöller <boxed@killingar.net>:
I feel there could be solutions. Either module.__getattribute__ (which, IIRC, was implemented recently), or just using a proxy/wrapper class around non-Union[class, function] objects. Those are rare, though, in imported modules. And I don't think this library will see much use without the subjects being imported first - at least I know I do my refactors on a per-file basis. And if you can't, you might want to split your files first.
Hi Anders, If correctly understood your concern, it's about usage of stdlib's *Warning classes directly that makes all warnings coming from different libraries indistinguishable. I think that's not the case, since warnings.filterwarnings allows to specify custom filter using a regular expression to match module names. Therefore it's not redundant to subclass *Warning for namespacing alone.
If correctly understood your concern, it's about usage of stdlib's *Warning classes directly that makes all warnings coming from different libraries indistinguishable.
That was my concern yes.
I think that's not the case, since warnings.filterwarnings allows to specify custom filter using a regular expression to match module names.
And what does that match against? The module name of the exception type right?
Therefore it's not redundant to subclass *Warning for namespacing alone.
Not redundant? You mean you must subclass? In that case my concern stands. / Anders
Therefore it's not redundant to subclass *Warning for namespacing alone.
Not redundant? You mean you must subclass? In that case my concern stands.
An unfortunate typo, meant "it's redundant".
And what does that match against? The module name of the exception type right?
It matches agains a location where warn is called after taking stacklevel into account. Consider the following example: test.py: import warnings warnings.warn("test") warnings.warn("__main__ from test", stacklevel=2) $ python -c "import warnings, test; warnings.warn('__main__')" test.py:2: UserWarning: test warnings.warn("test") -c:1: UserWarning: __main__ from test -c:1: UserWarning: __main__ $ python -W "ignore:::test:" -c "import warnings, test; warnings.warn('__main__')" -c:1: UserWarning: __main__ from test -c:1: UserWarning: __main__ $ python -W "ignore:::__main__:" -c "import warnings, test; warnings.warn('__main__')" test.py:2: UserWarning: test warnings.warn("test") End-user can distinguish warnings of the same category by specifying their origin (where warning is issued in runtime).
After spending more time thinking about the implementation I came to a conclusion that it's not easy to generalize replacement of classes. Yes, with some work it's possible to ensure that old name references a new one. But that's not sufficient. If new class has different interface then user's code will fail. Library's author could provide a mapping via a dict or by manually implementing methods but that'd be only half of the problem. User's code would still pass "old" objects back to the library. So effectively the library would have to support both old and new making all work meaningless. I think it's possible to design a smart wrapper that behaves as an old class in user's code, but as a new class inside library. E.g. by analyzing traceback.extract_stack. But that is too clever and almost certainly comes with its own bag of flaws. I'm confident that this problem leaves the "obsolete" decorators out of scope of this enhancement. Would anyone be interested in using plain deprecation decorators? They are still quite useful: - Can warn when class is subclassed or class-level attributes are accessed (vs warn inside __init__/__new__) - Much easier to be seen by static analyzers Best Regards, Ilya Kulakov
I have to say, this would be amazing! I've basically had to create many of these by hand over time, and I doubt I'm the only person who's wondered how this isn't in the stdlib! On Thu, Sep 13, 2018, 7:18 PM Ilya Kulakov <kulakov.ilya@gmail.com> wrote:
-- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/
I'd like to propose an extension for the warnings module to address this problem.
I like all of that. The only issue I have with it is that the warnings module is designed to namespace depredations so you can turn them on per library and this code doesn’t seem to handle that. We really want to avoid libraries using these convenience functions instead of creating their own warning that can be properly filtered. I’m not sure what the solution to this would be. Maybe just accessing <yourmodule>.DeprecationWarning dynamically? Seems a bit magical though. / Anders
Op vr 14 sep. 2018 om 08:07 schreef Anders Hovmöller <boxed@killingar.net>:
I feel there could be solutions. Either module.__getattribute__ (which, IIRC, was implemented recently), or just using a proxy/wrapper class around non-Union[class, function] objects. Those are rare, though, in imported modules. And I don't think this library will see much use without the subjects being imported first - at least I know I do my refactors on a per-file basis. And if you can't, you might want to split your files first.
Hi Anders, If correctly understood your concern, it's about usage of stdlib's *Warning classes directly that makes all warnings coming from different libraries indistinguishable. I think that's not the case, since warnings.filterwarnings allows to specify custom filter using a regular expression to match module names. Therefore it's not redundant to subclass *Warning for namespacing alone.
If correctly understood your concern, it's about usage of stdlib's *Warning classes directly that makes all warnings coming from different libraries indistinguishable.
That was my concern yes.
I think that's not the case, since warnings.filterwarnings allows to specify custom filter using a regular expression to match module names.
And what does that match against? The module name of the exception type right?
Therefore it's not redundant to subclass *Warning for namespacing alone.
Not redundant? You mean you must subclass? In that case my concern stands. / Anders
Therefore it's not redundant to subclass *Warning for namespacing alone.
Not redundant? You mean you must subclass? In that case my concern stands.
An unfortunate typo, meant "it's redundant".
And what does that match against? The module name of the exception type right?
It matches agains a location where warn is called after taking stacklevel into account. Consider the following example: test.py: import warnings warnings.warn("test") warnings.warn("__main__ from test", stacklevel=2) $ python -c "import warnings, test; warnings.warn('__main__')" test.py:2: UserWarning: test warnings.warn("test") -c:1: UserWarning: __main__ from test -c:1: UserWarning: __main__ $ python -W "ignore:::test:" -c "import warnings, test; warnings.warn('__main__')" -c:1: UserWarning: __main__ from test -c:1: UserWarning: __main__ $ python -W "ignore:::__main__:" -c "import warnings, test; warnings.warn('__main__')" test.py:2: UserWarning: test warnings.warn("test") End-user can distinguish warnings of the same category by specifying their origin (where warning is issued in runtime).
After spending more time thinking about the implementation I came to a conclusion that it's not easy to generalize replacement of classes. Yes, with some work it's possible to ensure that old name references a new one. But that's not sufficient. If new class has different interface then user's code will fail. Library's author could provide a mapping via a dict or by manually implementing methods but that'd be only half of the problem. User's code would still pass "old" objects back to the library. So effectively the library would have to support both old and new making all work meaningless. I think it's possible to design a smart wrapper that behaves as an old class in user's code, but as a new class inside library. E.g. by analyzing traceback.extract_stack. But that is too clever and almost certainly comes with its own bag of flaws. I'm confident that this problem leaves the "obsolete" decorators out of scope of this enhancement. Would anyone be interested in using plain deprecation decorators? They are still quite useful: - Can warn when class is subclassed or class-level attributes are accessed (vs warn inside __init__/__new__) - Much easier to be seen by static analyzers Best Regards, Ilya Kulakov
participants (4)
-
Anders Hovmöller -
Ilya Kulakov -
Jacco van Dorp -
Ryan Gonzalez