seeking feedback on a proposed deprecation library to be used at type checking time (in pytype)
Hi everyone, I'm working on a proposal to introduce a deprecation library. It is intended to be used at type checking time, and will be supported by pytype. The primary benefit (over DeprecationWarning at runtime) is to provide deprecation signals at static analysis time (think type checking, linting, IDE support). **The gist of the deprecation library** ============================ The gist of the deprecation library is to use a data class Deprecated wrapped in a typing.Annotated to mark variables/methods/functions as deprecated. Here are a couple of examples: # API: @dataclasses.dataclass class Deprecated: reason: str # Usages: my_constant: Annotated[int, Deprecated("my_constant is deprecated.")] = 123 def do_something(name: str, arg: Annotated[Any, Deprecated("`arg` is deprecated.")] = None) -> None: # The parameter `arg` is deprecated def my_function(name: str) -> Annotated[str, Deprecated("my_function is deprecated.")]: # Putting Deprecated in the return value type marks the method/function as deprecated. class MyClass: def __init__(self) -> Annotated[None, Deprecated("MyClass is deprecated.")]: # Marking __init__ as deprecated makes the class deprecated It is a no-op at runtime, and per PEP 593 it should be ignored by type checkers that don't support it. We also intend to include a higher-level decorator that will be turned into the lower-level annotations at type checking time. Here is an example: # API: def deprecated(message: str, stacklevel: int = 2) -> Callable[[_T], _T]: def decorator(wrapped): @functools.wraps(wrapped) def wrapper(*args, **kwargs): warnings.warn(message, DeprecationWarning, stacklevel=stacklevel) return wrapped(*args, **kwargs) return wrapper return decorator # Usage: @deprecated("my_func is deprecated") def my_func(name): print(f"Called my_func with {name}") At type checking time, it turns the function to def my_func(name) -> Annotated[?, Deprecated("my_func is deprecated")]. ? is either a concrete type if the type checker can infer its return type, or Any if not. At runtime, it raises a DeprecationWarning whenever my_func is called. **Feedback I'm looking for** ====================== Above isn't very detailed nor a finalized design. But I hope the idea is just clear enough to seek some initial feedback. The things I'm specifically looking for are: * Would the use of these APIs cause issues for other type checkers? Per my interpretation of PEP 593, it shouldn't. I tested with pytype, mypy, pyright, and pyre, one hiccup I found is pyre thinks annotating __init__ with Annotated[None, Deprecated("")] is an error (Incompatible constructor annotation [17]: __init__ is annotated as returning typing.Annotated[None], but it should return None.) * Is there interest from other type checkers to support such a deprecation library, so that static analysis tooling can provide better support detecting the use of deprecated APIs? Thanks, Yilei
Eric Traut and I recently discussed this topic for pyright: https://github.com/microsoft/pyright/discussions/2300. I'm definitely on board with the idea of marking deprecations in a way that is visible to type checkers. El jue, 16 sept 2021 a las 11:22, Yilei Yang (<yileiyang9@gmail.com>) escribió:
Hi everyone,
I'm working on a proposal to introduce a deprecation library. It is intended to be used at type checking time, and will be supported by pytype. The primary benefit (over DeprecationWarning at runtime) is to provide deprecation signals at static analysis time (think type checking, linting, IDE support).
**The gist of the deprecation library** ============================
The gist of the deprecation library is to use a data class Deprecated wrapped in a typing.Annotated to mark variables/methods/functions as deprecated. Here are a couple of examples:
# API: @dataclasses.dataclass class Deprecated: reason: str
# Usages: my_constant: Annotated[int, Deprecated("my_constant is deprecated.")] = 123 def do_something(name: str, arg: Annotated[Any, Deprecated("`arg` is deprecated.")] = None) -> None: # The parameter `arg` is deprecated def my_function(name: str) -> Annotated[str, Deprecated("my_function is deprecated.")]: # Putting Deprecated in the return value type marks the method/function as deprecated. class MyClass: def __init__(self) -> Annotated[None, Deprecated("MyClass is deprecated.")]: # Marking __init__ as deprecated makes the class deprecated
It is a no-op at runtime, and per PEP 593 it should be ignored by type checkers that don't support it.
We also intend to include a higher-level decorator that will be turned into the lower-level annotations at type checking time. Here is an example:
# API: def deprecated(message: str, stacklevel: int = 2) -> Callable[[_T], _T]: def decorator(wrapped): @functools.wraps(wrapped) def wrapper(*args, **kwargs): warnings.warn(message, DeprecationWarning, stacklevel=stacklevel) return wrapped(*args, **kwargs) return wrapper return decorator
# Usage: @deprecated("my_func is deprecated") def my_func(name): print(f"Called my_func with {name}")
At type checking time, it turns the function to def my_func(name) -> Annotated[?, Deprecated("my_func is deprecated")]. ? is either a concrete type if the type checker can infer its return type, or Any if not. At runtime, it raises a DeprecationWarning whenever my_func is called.
**Feedback I'm looking for** ======================
Above isn't very detailed nor a finalized design. But I hope the idea is just clear enough to seek some initial feedback. The things I'm specifically looking for are:
* Would the use of these APIs cause issues for other type checkers? Per my interpretation of PEP 593, it shouldn't. I tested with pytype, mypy, pyright, and pyre, one hiccup I found is pyre thinks annotating __init__ with Annotated[None, Deprecated("")] is an error (Incompatible constructor annotation [17]: __init__ is annotated as returning typing.Annotated[None], but it should return None.) * Is there interest from other type checkers to support such a deprecation library, so that static analysis tooling can provide better support detecting the use of deprecated APIs?
Thanks, Yilei _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Hi Yilei, Thanks for the proposal! Detecting deprecated functions/classes etc. is indeed a very useful and important feature. However, I am generally lukewarm about doing this within a standardized type system. Here are my thoughts: 1) To me, "the function I am calling is deprecated" does not seem to violate any typing-related constraints, at least not in a traditional sense. We are not doing stuffs like passing an int to a function that expects a string or having an argument mismatch issues, which all lead to runtime errors. No soundness issues are involved here, and I'm not sure deprecation detection belongs to that camp. Of course, it does not mean that type systems cannot be extended to catch more issues, but for extensions like this I would generally expect more precise descriptions of the proposed system -- e.g. what are the additional typing constraints you want to introduce (like when and where should a type checker reports this "deprecation" error?), how does the new type interacts with pre-existing types (like how it affects subtyping and what are the meanings/behaviors of `Union[int, Annotated[str, Deprecated(...)]` or `Dict[str, Iterable[Annotated[int, Deprecated(...)]]]`?), etc. 2) Let's assume that we end up deciding we want to support deprecation detection in the type system. In that case, I would still argue against relying on `Annotated` for the job. My reading of PEP 593 is that the interpretation of `Annotated` is meant to be tooling-specific -- type checkers themselves may not want to assign specific meanings to those annotations as those meanings may create conflicts with downstream toolings. If you want to standardize your extensions to the type system, I personally think it would be much cleaner to just add new classes to stdlib (e.g. something like `typing.Deprecated[T]`?), which can do whatever `Annotated` can do but without the issues. 3) Let's assume that we end up adopting your approach. In that case, I would propose 2 small revisions to your `deprecated` API. The first one is that Pyre does not really want to support unbounded `Callable[[T], T]` in return type. See https://github.com/facebook/pyre-check/issues/329 . It would be nice if it can be rewritten into a callback protocol. The second one is that the `deprecated` function can be more fully typed with ParamSpec introduced in PEP 612. There's also this issue you mentioned in your message, but that's something to be fixed on the Pyre side. 4) Finally, let me also propose an alternative approach where the check gets implemented not inside the type system but as a (tool-specific) downstream linter instead. The idea is to have type checker expose an interface that returns all of the callsites it knows, and then let the linter go over them: for each of them see if the possible callees resides in a pre-determined deprecation list (which can be obtained by looking for syntactical hints around functions/classes/globals such as decorators, or even just a pre-defined list of names). If we do it like this, no standardized type system change is needed. The only thing that may need to be standardized here is the form of syntactical hints, which would be much lighter-weight than a standardized type system extension. - Jia
Some quick notes: On Thu, Sep 16, 2021 at 2:53 PM Jia Chen via Typing-sig < typing-sig@python.org> wrote:
1) To me, "the function I am calling is deprecated" does not seem to violate any typing-related constraints, at least not in a traditional sense.
Well, deprecation in Python usually implies that at some future point the function (etc.) won't exist at all, so we can see this as type-checking for multiple versions at once.
2) Let's assume that we end up deciding we want to support deprecation detection in the type system. In that case, I would still argue against relying on `Annotated` for the job.
Agreed. `Annotated` is a measure of last resort. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Thanks for your thoughts so far everyone!
Eric Traut and I recently discussed this topic for pyright: https://github.com/microsoft/pyright/discussions/2300. I'm definitely on board with the idea of marking deprecations in a way that is visible to type checkers.
Aha, the ideas there are quite similar. A new type typing.Deprecated would enable deprecating parameters/constants/etc. A decorator, in addition to the type, would also be nice. (And if we use typing.Annotated, the type alone would force the users to include the original type since Annotated doesn't support inference. So a decorator would also be more friendly to use.) But if we can just use typing.Deprecated and make it accept two optional parameters: 1) the original type; 2) the deprecation reason, then users don't need to include the original type: ```python def foo(x, y: typing.Deprecated): ... def foo2(x: int, y: typing.Deprecated[int]): ... def foo3(x: int, y: typing.Deprecated[int, "deprecation reason"]): ... ``` And also in my proposal, I'm using the return value of the method/function to mark as deprecated: # The following means the function `foo4` is deprecated: def foo4(x: int) -> typing.Deprecated[str]: ... I'm not quite happy about this, since unfamiliar readers could read it as `foo4` returns something deprecated (as opposed to `foo4` itself is deprecated). So the use of decorator can be preferred for methods/functions.
1) To me, "the function I am calling is deprecated" does not seem to violate any typing-related constraints, at least not in a traditional sense. We are not doing stuffs like passing an int to a function that expects a string or having an argument mismatch issues, which all lead to runtime errors. No soundness issues are involved here, and I'm not sure deprecation detection belongs to that camp. Of course, it does not mean that type systems cannot be extended to catch more issues, but for extensions like this I would generally expect more precise descriptions of the proposed system -- e.g. what are the additional typing constraints you want to introduce (like when and where should a type checker reports this "deprecation" error?), how does the new type interacts with pre-existing types (like how it affects subtyping and what are the meanings/behaviors of `Union[int, Annotated[str, Deprecated(...)]` or `Dict[str, Iterable[Annotated[int, Deprecated(...)]]]`?), etc.
I'm agreeing on the point that we should not require type checkers to do the _actual reporting_ to users. Type checkers can choose to do so, but shouldn't be required to report. The main requirement is for the type checkers to understand the proposed new addition to the type system. re subtyping: (assuming we end up using typing.Deprecated), we could require that typing.Deprecated must be used at the outer most place. And typing.Deprecated isn't transferrable, meaning: ```python # foo.py: value: typing.Deprecated[int] = 1 # bar.py: import foo value = foo.value # The type of bar should be int, not typing.Deprecated[int] items = {'foo': foo.value} # The type of items should be Dict[str, int], not Dict[str, typing.Deprecated[int]] # user.py: import foo import bar _ = foo.value # This should be reported as use of deprecated API _ = bar.value # This should not be reported as use of deprecated API _ = bar.items # This should not be reported as use of deprecated API ```
2) Let's assume that we end up deciding we want to support deprecation detection in the type system. In that case, I would still argue against relying on `Annotated` for the job. My reading of PEP 593 is that the interpretation of `Annotated` is meant to be tooling-specific -- type checkers themselves may not want to assign specific meanings to those annotations as those meanings may create conflicts with downstream toolings. If you want to standardize your extensions to the type system, I personally think it would be much cleaner to just add new classes to stdlib (e.g. something like `typing.Deprecated[T]`?), which can do whatever `Annotated` can do but without the issues.
Agreed too. If we end up a proposal for everyone to adopt, it shouldn't use Annotated. I'm coming from an angle that we want to first experiment with this in pytype, and for other type checkers, at least the use of such annotations shouldn't cause issues if they don't want to adopt.
3) Let's assume that we end up adopting your approach. In that case, I would propose 2 small revisions to your `deprecated` API. The first one is that Pyre does not really want to support unbounded `Callable[[T], T]` in return type. See https://github.com/facebook/pyre-check/issues/329 . It would be nice if it can be rewritten into a callback protocol. The second one is that the `deprecated` function can be more fully typed with ParamSpec introduced in PEP 612. There's also this issue you mentioned in your message, but that's something to be fixed on the Pyre side.
That's a good suggestion, thanks!
4) Finally, let me also propose an alternative approach where the check gets implemented not inside the type system but as a (tool-specific) downstream linter instead. The idea is to have type checker expose an interface that returns all of the callsites it knows, and then let the linter go over them: for each of them see if the possible callees resides in a pre-determined deprecation list (which can be obtained by looking for syntactical hints around functions/classes/globals such as decorators, or even just a pre-defined list of names). If we do it like this, no standardized type system change is needed. The only thing that may need to be standardized here is the form of syntactical hints, which would be much lighter-weight than a standardized type system extension.
What would this form of syntactical hints be though? And how it would be lighter-weight than say, a typing.Deprecated type?
Well, deprecation in Python usually implies that at some future point the function (etc.) won't exist at all, so we can see this as type-checking for multiple versions at once.
Yeah this is fair. I'm a bit skeptical about the feasibility of cleanly formalizing the notion of "multi-version checking" with types and constraint systems though.
I'm not quite happy about this, since unfamiliar readers could read it as `foo4` returns something deprecated (as opposed to `foo4` itself is deprecated).
Yes this is one aspect I dislike about the proposal: types go with the return values, not the function. Interpreting the type as if it were applying to the entire function does not seem to be compatible with how the rest of the type system works.
What would this form of syntactical hints be though? And how it would be lighter-weight than say, a typing.Deprecated type?
When I used the word "lightweight", I did not mean the syntactical verbosity of your construct. Instead, I was referring to how much more complexity the construct is going to introduce to the type system. Ideally I would like to see this feature adding zero complexity: no subtyping rule change or constraint system tweaking is needed. To put it in another way, type checkers could simply throw all the wrapping `typing.Deprecated` away, and their core behaviors wouldn't be affected. The pyright proposal would be a good example of this. There's no need to specially attach or assign new types to the decorator. For parameters and global attributes, I'm OK with your `typing.Deprecated` proposal as long as it's only used for syntactical marking purpose -- i.e. they can only appear on param and global annotations, must be used as outermost location (how does this work with `typing.Final` which has the same requirement?), and cannot appear as annotations for local variables / return types / class attributes, etc. What I would suggest against is to assign special type-level semantics to this `typing.Deprecated` construct. - Jia
Following up this thread. Looks like we share a common interest in a decorator approach (though I haven't heard from mypy). And if we allow deprecating overloads (to deprecate parameters), I believe this covers the majority of the use cases and is a real improvement even without a `typing.Deprecated` type. If I want to formally propose this shared decorator, where should I start? Does it make sense for this decorator to live in the typing module? If not, where then? - Yilei
participants (5)
-
Guido van Rossum -
Jelle Zijlstra -
Jia Chen -
Yilei Yang -
Yilei Yang