Typing of sentinel objects
Hi, I'm working on PEP 661: Sentinel Values [1]. One thing I think it should get right is that it should be possible to have strict type annotations for sentinel values. Many sentinel implementations don't support this. I see that this has come up before on this mailing list [2] and as an issue in the GitHub repo [3]. Using Literal and Final, as suggested in the discussions of the PEP [4] and as suggested by Guido on the aforementioned GitHub issue [5], seems to be a very nice approach. 1. What would be needed to make that work with mypy? 2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools? Please note that I'm far from well-versed on typing in Python, so assume I know only the very basics. - Tal Einat [1]: https://www.python.org/dev/peps/pep-0661/ [2]: https://mail.python.org/archives/list/typing-sig@python.org/thread/TI5Y2HTVT... [3]: https://github.com/python/typing/issues/689 [4]: https://discuss.python.org/t/pep-661-sentinel-values/9126/2 [5]: https://github.com/python/typing/issues/689#issuecomment-561425237
El mar, 8 jun 2021 a las 8:28, Tal Einat (<taleinat@gmail.com>) escribió:
Hi,
I'm working on PEP 661: Sentinel Values [1].
One thing I think it should get right is that it should be possible to have strict type annotations for sentinel values. Many sentinel implementations don't support this.
I see that this has come up before on this mailing list [2] and as an issue in the GitHub repo [3].
Using Literal and Final, as suggested in the discussions of the PEP [4] and as suggested by Guido on the aforementioned GitHub issue [5], seems to be a very nice approach.
Looking at your PEP, I think it would be more user-friendly if the sentinel can be used directly as a type annotation, something like this: SENTINEL = sentinel("SENTINEL") def print_exception(*, value=Exception | SENTINEL) -> None: ... This would be similar to how None works, and it would make the type annotation more concise than when we use Literal.
1. What would be needed to make that work with mypy?
Somebody would need to contribute an implementation. I don't think it would be very difficult since this isn't introducing any deep new concept to the type system. This doesn't necessarily need to be done right now, though; the implementation can come after the PEP is accepted. Is there going to be a backport for the sentinels module on PyPI? That would make it easier for type checkers to already implement support.
2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools?
My preference would be that the PEP specifies how this new feature interacts with typing, either by allowing sentinels as types directly (my proposal above) or by saying that PEP 586 is amended to allow sentinels in Literal.
Please note that I'm far from well-versed on typing in Python, so assume I know only the very basics.
- Tal Einat
[1]: https://www.python.org/dev/peps/pep-0661/ [2]: https://mail.python.org/archives/list/typing-sig@python.org/thread/TI5Y2HTVT... [3]: https://github.com/python/typing/issues/689 [4]: https://discuss.python.org/t/pep-661-sentinel-values/9126/2 [5]: https://github.com/python/typing/issues/689#issuecomment-561425237 _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Am 08.06.21 um 17:44 schrieb Jelle Zijlstra:
Looking at your PEP, I think it would be more user-friendly if the sentinel can be used directly as a type annotation, something like this:
SENTINEL = sentinel("SENTINEL")
def print_exception(*, value=Exception | SENTINEL) -> None: ...
This would be similar to how None works, and it would make the type annotation more concise than when we use Literal. +1 2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools? My preference would be that the PEP specifies how this new feature interacts with typing, either by allowing sentinels as types directly (my proposal above) or by saying that PEP 586 is amended to allow sentinels in Literal.
+1 Sorry for just +1'ing, but Jelle formulated the thoughts I had. Type checkers will need to special case sentinels anyway, so using Literal[] seems to be just unnecessary noise. - Sebastian
+1 Are there any existing provisions in type checking tools to denote that a type hint of n should be interpreted as type(n) — basically the case of a singleton? The obvious case is None → type(None). I wonder if there would a more general case beyond sentinel to be supported here. Paul On Tue, 2021-06-08 at 17:54 +0200, Sebastian Rittau wrote:
Am 08.06.21 um 17:44 schrieb Jelle Zijlstra:
Looking at your PEP, I think it would be more user-friendly if the sentinel can be used directly as a type annotation, something like this:
SENTINEL = sentinel("SENTINEL")
def print_exception(*, value=Exception | SENTINEL) -> None: ...
This would be similar to how None works, and it would make the type annotation more concise than when we use Literal.
+1
2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools? My preference would be that the PEP specifies how this new feature interacts with typing, either by allowing sentinels as types directly (my proposal above) or by saying that PEP 586 is amended to allow sentinels in Literal. +1 Sorry for just +1'ing, but Jelle formulated the thoughts I had. Type checkers will need to special case sentinels anyway, so using Literal[] seems to be just unnecessary noise. - Sebastian
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
El mar, 8 jun 2021 a las 9:06, Paul Bryan (<pbryan@anode.ca>) escribió:
+1
Are there any existing provisions in type checking tools to denote that a type hint of n should be interpreted as type(n) — basically the case of a singleton?
The obvious case is None → type(None). I wonder if there would a more general case beyond sentinel to be supported here.
No others are supported right now. Ellipsis and NotImplemented have come up as other use cases.
Paul
On Tue, 2021-06-08 at 17:54 +0200, Sebastian Rittau wrote:
Am 08.06.21 um 17:44 schrieb Jelle Zijlstra:
Looking at your PEP, I think it would be more user-friendly if the sentinel can be used directly as a type annotation, something like this:
SENTINEL = sentinel("SENTINEL")
def print_exception(*, value=Exception | SENTINEL) -> None: ...
This would be similar to how None works, and it would make the type annotation more concise than when we use Literal.
+1
2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools? My preference would be that the PEP specifies how this new feature interacts with typing, either by allowing sentinels as types directly (my proposal above) or by saying that PEP 586 is amended to allow sentinels in Literal.
+1
Sorry for just +1'ing, but Jelle formulated the thoughts I had. Type checkers will need to special case sentinels anyway, so using Literal[] seems to be just unnecessary noise. - Sebastian _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
On Tue, Jun 8, 2021 at 8:54 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 08.06.21 um 17:44 schrieb Jelle Zijlstra:
Looking at your PEP, I think it would be more user-friendly if the sentinel can be used directly as a type annotation, something like this:
SENTINEL = sentinel("SENTINEL")
def print_exception(*, value=Exception | SENTINEL) -> None: ...
This would be similar to how None works, and it would make the type annotation more concise than when we use Literal.
+1
2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools? My preference would be that the PEP specifies how this new feature interacts with typing, either by allowing sentinels as types directly (my proposal above) or by saying that PEP 586 is amended to allow sentinels in Literal.
+1
Sorry for just +1'ing, but Jelle formulated the thoughts I had. Type checkers will need to special case sentinels anyway, so using Literal[] seems to be just unnecessary noise.
Why do type checkers have to special-case sentinels? Until this idiom has become popular, using `Literal[SENTINEL]` seems the most logical approach. The only change to type checkers in that case would be to treat `sentinel` as one of the types whose values can be used in `Literal[]`. (More generally speaking, allowing all values to be used directly instead of literals would of course be nice, but the current use of strings for forward references pretty much blocks that; I'm not sure we should start making individual exceptions for sentinels.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Am 08.06.21 um 18:51 schrieb Guido van Rossum:
Why do type checkers have to special-case sentinels? Until this idiom has become popular, using `Literal[SENTINEL]` seems the most logical approach. The only change to type checkers in that case would be to treat `sentinel` as one of the types whose values can be used in `Literal[]`.
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work: X = foo() def bar(x: Literal[X]) -> None: ... This means that "X = sentinel()" would need to be special-cased. - Sebastian
On Tue, 2021-06-08 at 20:40 +0200, Sebastian Rittau wrote:
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work: X = foo() def bar(x: Literal[X]) -> None: ... This means that "X = sentinel()" would need to be special-cased.
Why can't a sentinel class itself be used? Why would it need to be instantiated? Paul
Am 08.06.21 um 20:54 schrieb Paul Bryan:
On Tue, 2021-06-08 at 20:40 +0200, Sebastian Rittau wrote:
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work:
X = foo() def bar(x: Literal[X]) -> None: ...
This means that "X = sentinel()" would need to be special-cased.
Why can't a sentinel class itself be used? Why would it need to be instantiated?
Do you mean something like this? # sentinels.pyi class Sentinel: ... # this is currently not part of the PEP and wouldn't exist at runtime def sentinel(...) -> Sentinel: ... # foo.py X = sentinel() def bar(x: X) -> None: ... Apart from the part that a "Sentinel" class currently isn't part of the PEP (although I'd prefer if it was), this means that all sentinels would be type compatible with each other and using the wrong sentinel wouldn't be caught at type checking time. If this is not what you meant, could you give an example? - Sebastian
I suppose I'm hoping for some variant of the Use class objects rejected idea from the PEP: define your own specific sentinel class, which subclasses a Sentinel base class, or has a Sentinel metaclass. The base class would be responsible for providing the __repr__ and __str__ implementations. Then, one uses the specific subclass as the sentinel, and it can be easily type checked. So, ideally, something like: class MySentinel(Sentinel): pass Paul On Tue, 2021-06-08 at 21:02 +0200, Sebastian Rittau wrote:
Am 08.06.21 um 20:54 schrieb Paul Bryan:
On Tue, 2021-06-08 at 20:40 +0200, Sebastian Rittau wrote:
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work: X = foo() def bar(x: Literal[X]) -> None: ... This means that "X = sentinel()" would need to be special-cased.
Why can't a sentinel class itself be used? Why would it need to be instantiated?
Do you mean something like this? # sentinels.pyi
class Sentinel: ... # this is currently not part of the PEP and wouldn't exist at runtime def sentinel(...) -> Sentinel: ... # foo.py
X = sentinel() def bar(x: X) -> None: ... Apart from the part that a "Sentinel" class currently isn't part of the PEP (although I'd prefer if it was), this means that all sentinels would be type compatible with each other and using the wrong sentinel wouldn't be caught at type checking time. If this is not what you meant, could you give an example? - Sebastian _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
Am 08.06.21 um 21:32 schrieb Paul Bryan:
I suppose I'm hoping for some variant of the /Use class objects/ rejected idea from the PEP: define your own specific sentinel class, which subclasses a Sentinel base class, or has a Sentinel metaclass. The base class would be responsible for providing the __repr__ and __str__ implementations. Then, one uses the specific subclass as the sentinel, and it can be easily type checked. So, ideally, something like:
class MySentinel(Sentinel): pass
While I like the sentinel() function syntax better as it's shorter and matches more what is currently done (i.e. Sentinel = object()), using a subclass would have the advantage of immediately working with all type checkers. - Sebastian
On Tue, Jun 8, 2021 at 11:40 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 08.06.21 um 18:51 schrieb Guido van Rossum:
Why do type checkers have to special-case sentinels? Until this idiom has become popular, using `Literal[SENTINEL]` seems the most logical approach. The only change to type checkers in that case would be to treat `sentinel` as one of the types whose values can be used in `Literal[]`.
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work:
X = foo() def bar(x: Literal[X]) -> None: ...
This means that "X = sentinel()" would need to be special-cased.
Hm, I think you're right. I was thinking of X = 42 def bar(x: Literal[X]): ... but I don't even know if that works, and even if it did, it wouldn't work if you replaced 42 with a call to int(...). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Tue, Jun 8, 2021 at 11:09 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Jun 8, 2021 at 11:40 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 08.06.21 um 18:51 schrieb Guido van Rossum:
Why do type checkers have to special-case sentinels? Until this idiom has become popular, using `Literal[SENTINEL]` seems the most logical approach. The only change to type checkers in that case would be to treat `sentinel` as one of the types whose values can be used in `Literal[]`.
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work:
X = foo() def bar(x: Literal[X]) -> None: ...
This means that "X = sentinel()" would need to be special-cased.
Hm, I think you're right. I was thinking of
X = 42
def bar(x: Literal[X]): ...
but I don't even know if that works, and even if it did, it wouldn't work if you replaced 42 with a call to int(...).
The precedent here is Enum values. How does Literal work with Enum values? Could sentinels use the same mechanism? - Tal
On Tue, Jun 8, 2021 at 11:19 PM Tal Einat <taleinat@gmail.com> wrote:
On Tue, Jun 8, 2021 at 11:09 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Jun 8, 2021 at 11:40 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 08.06.21 um 18:51 schrieb Guido van Rossum:
Why do type checkers have to special-case sentinels? Until this idiom has become popular, using `Literal[SENTINEL]` seems the most logical approach. The only change to type checkers in that case would be to treat `sentinel` as one of the types whose values can be used in `Literal[]`.
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work:
X = foo() def bar(x: Literal[X]) -> None: ...
This means that "X = sentinel()" would need to be special-cased.
Hm, I think you're right. I was thinking of
X = 42
def bar(x: Literal[X]): ...
but I don't even know if that works, and even if it did, it wouldn't work if you replaced 42 with a call to int(...).
The precedent here is Enum values. How does Literal work with Enum values? Could sentinels use the same mechanism?
Enums have a considerable amount of special-casing, both at runtime and in static checkers. Basically, when comparing Color.RED and Color.BLUE, the static checker knows that these are not the same value. (With the exception of aliases, I'm not sure how those work.) But when you compare Color(x) and Color.RED, the checker doesn't have enough information to tell whether they are the same, and will always give you an error. (The same as when you take a value of type int and try to match it against Literal[42].) Without special-casing the static checker wouldn't know whether X and Y could be the same or not if you write X = sentinel() Y = sentinel() and it also wouldn't know that X cannot be changed to a different value. Maybe it could be made to work if you write X: Final = sentinel() but this would require an extension of the concept of Literal[], which, as currently defined, only supports a few types (IIRC bool, int, str, and enums). I think it doesn't even support aliases, so X: Final = 42 does not make X compatible with Literal[42] (though I'm not sure of that). (Can you tell I'm not set up to try anything out? Nor do I feel like re-reading the PEP... :-) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Wed, Jun 9, 2021 at 9:53 AM Guido van Rossum <guido@python.org> wrote:
On Tue, Jun 8, 2021 at 11:19 PM Tal Einat <taleinat@gmail.com> wrote:
On Tue, Jun 8, 2021 at 11:09 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Jun 8, 2021 at 11:40 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 08.06.21 um 18:51 schrieb Guido van Rossum:
Why do type checkers have to special-case sentinels? Until this idiom has become popular, using `Literal[SENTINEL]` seems the most logical approach. The only change to type checkers in that case would be to treat `sentinel` as one of the types whose values can be used in `Literal[]`.
Unless I misunderstand something (quite possible!), the following idiom does not currently work with type checkers and is not supposed to work:
X = foo() def bar(x: Literal[X]) -> None: ...
This means that "X = sentinel()" would need to be special-cased.
Hm, I think you're right. I was thinking of
X = 42
def bar(x: Literal[X]): ...
but I don't even know if that works, and even if it did, it wouldn't work if you replaced 42 with a call to int(...).
The precedent here is Enum values. How does Literal work with Enum values? Could sentinels use the same mechanism?
Enums have a considerable amount of special-casing, both at runtime and in static checkers. Basically, when comparing Color.RED and Color.BLUE, the static checker knows that these are not the same value. (With the exception of aliases, I'm not sure how those work.) But when you compare Color(x) and Color.RED, the checker doesn't have enough information to tell whether they are the same, and will always give you an error. (The same as when you take a value of type int and try to match it against Literal[42].)
Without special-casing the static checker wouldn't know whether X and Y could be the same or not if you write
X = sentinel() Y = sentinel()
and it also wouldn't know that X cannot be changed to a different value.
Maybe it could be made to work if you write
X: Final = sentinel()
but this would require an extension of the concept of Literal[], which, as currently defined, only supports a few types (IIRC bool, int, str, and enums).
Thanks for the clarification, Guido! The more I think about it, the more I'm inclined to find an implementation that would require no special support from type checkers. That is, an implementation which would allow existing type analysis to "just work". But it seems that would be impossible with an interface of the form currently suggested in the PEP, `NotGiven = sentinel('NotGiven').` :( May I ask, how do Python static type analyzers currently handle namedtuples? Those use a similar interface... - Tal
Am 09.06.21 um 09:04 schrieb Tal Einat:
The more I think about it, the more I'm inclined to find an implementation that would require no special support from type checkers. That is, an implementation which would allow existing type analysis to "just work". But it seems that would be impossible with an interface of the form currently suggested in the PEP, `NotGiven = sentinel('NotGiven').`
I honestly think that requiring type checkers to special case this is fine. All implementations that would work with the existing type system are a bit awkward to use. In my opinion the type system should serve the language, but the language shouldn't be constrained by it. (Although often when something doesn't work within the type system, it's an indication that it might not be the best/clearest solution. Sentinels are special enough to earn special treatment.) The only thing that I think would help the typing case is having a Sentinel base class as I mentioned in the discuss thread. This allows us to write functions that accept arbitrary sentinels (for whatever reason) in a type-safe manner. - Sebastian
I'm quite surprised that PEP 661 and the following discussion doesn't mentioned PEP 484, which already define a sentinel pattern (which are named singleton types): https://www.python.org/dev/peps/pep-0484/#support-for-singleton-types-in-uni... In fact, the following example works fine in mypy: ```python from typing import Union from enum import Enum class Empty(Enum): token = 0 _empty = Empty.token def func(x: Union[int, Empty]) -> int: return 0 if x is _empty else x # check must be done with `is` ``` Yes, it's not a one-liner declaration, but it has existed for 6 years, and it's a quite logical pattern which don't need additional stuff (and it also allows more customization, like overriding `__bool__`). Personally, because sentinel is not something I do often, I don't mind having more than 1 lines to declare it. However, I admit that using directly the sentinel as a type annotation instead of its type would be very nice, and that's unfortunately not possible with the existing enum pattern (and will never be because of PEP 604). P.S. I don't see PEP 604 compatibility in the current reference implementation.
If the singleton pattern of PEP 484 is too cumbersome, why not update [PEP 586](https://www.python.org/dev/peps/pep-0586/#legal-and-illegal-parameterization...) (which defines `Literal` type) to extend the legal parameterization by adding `object` variable declared as `Final` ? (Because `object` instances are immutable, so they meet the `Literal` criteria when they are also declared as `Final`) My previous example could be written: ```python from typing import Final, Literal, Union empty: Final = object() def func(x: Union[int, Literal[empty]]) -> int: return 0 if x is empty else x ``` This would be a quite simple and elegant solution, which doesn't require any additional stuff too. The enum singleton pattern could still be used when more customization (magic method overriding) is needed.
On Wed, Jun 9, 2021 at 3:33 PM Joseph Perez <joperez@hotmail.fr> wrote:
If the singleton pattern of PEP 484 is too cumbersome, why not update [PEP 586](https://www.python.org/dev/peps/pep-0586/#legal-and-illegal-parameterization...) (which defines `Literal` type) to extend the legal parameterization by adding `object` variable declared as `Final` ? (Because `object` instances are immutable, so they meet the `Literal` criteria when they are also declared as `Final`)
My previous example could be written: ```python from typing import Final, Literal, Union
empty: Final = object()
def func(x: Union[int, Literal[empty]]) -> int: return 0 if x is empty else x ```
This would be a quite simple and elegant solution, which doesn't require any additional stuff too.
The PEP details why I consider the empty = object() pattern to be problematic in several senses: Besides the strict typing issue (which could perhaps be addressed using Literal), it also suffers from having a very poor repr and from not behaving correctly when copied or unpickled. https://www.python.org/dev/peps/pep-0661/#use-notgiven-object That being said, all of these could be addressed by using a dedicated class rather than object(). If we could allow for good type annotations without needing to generate a class for each sentinel, that seems like it would be ideal. - Tal
participants (6)
-
Guido van Rossum
-
Jelle Zijlstra
-
Joseph Perez
-
Paul Bryan
-
Sebastian Rittau
-
Tal Einat