Mailman 3 AnyOf - Union for return types - Typing-sig

AnyOf - Union for return types

Sebastian Rittau

21 Sep 2020 21 Sep '20

10:15 p.m.

This has previously been discussed here: https://github.com/python/typing/issues/566. In typeshed we have found that there a quite a few cases where functions can return two or more different, incompatible types, that can't be distinguished by the input types alone. A few examples: * ip_address() can return either an IPv4Address or an IPv6Address, depending on the input string. * float ** float returns either a float or a complex. Currently, typeshed marks this as returning just float. * json.loads() et al. can return a selection of types, depending on the input. * urlopen() can return either an HTTPResponse or an addinfourl. There are more examples in the linked issue. In these cases, returning a Union would technically be correct, but would also be inconvenient for callers that know what type to expect. We usually give up and just return Any. My suggestion would be to add a type tentatively called "AnyOf", that acts as an unsafe union: ip_address(address: str) -> AnyOf[IPv4Address, IPv6Address] I believe this could be useful for type checkers. While the use of AnyOf is unsafe, it is safer than just treating these return values as Any. It's also theoretically possible to use some clever narrowing. That said, there has been some pushback from the mypy core team in the linked issue, especially since this is most likely complex to implement. An easy way for type checkers to implement this, without sacrificing any already existing type safety is to just treating AnyOf exactly like Any. At least as a first step. Should type checkers implement full support for AnyOf later, we could already have better types in typeshed ready to use. But in the last years, typeshed has also gained users outside of type checkers. For example, PyCharm and jedi use it for autocompletion. I believe that AnyOf could be useful for those projects, as well as others. Is there interest in a feature like this? I would be willing to write a PEP and contribute the AnyOf = Any solution to mypy. Unfortunately I don't have the bandwidth to learn how to implement this "properly" in mypy. - Sebastian

Show replies by thread

Guido van Rossum

21 Sep 21 Sep

11:14 p.m.

If we implement this, are we sure that IDEs like PyCharm and/or vscode (e.g. pylance) will use and implement this? Or is that part just speculation? On Mon, Sep 21, 2020 at 06:45 Sebastian Rittau wrote:

...

This has previously been discussed here:

https://github.com/python/typing/issues/566.

In typeshed we have found that there a quite a few cases where functions

can return two or more different, incompatible types, that can't be

distinguished by the input types alone. A few examples:

* ip_address() can return either an IPv4Address or an IPv6Address,

depending on the input string.

* float ** float returns either a float or a complex. Currently,

typeshed marks this as returning just float.

* json.loads() et al. can return a selection of types, depending on

the input.

* urlopen() can return either an HTTPResponse or an addinfourl.

There are more examples in the linked issue.

In these cases, returning a Union would technically be correct, but

would also be inconvenient for callers that know what type to expect.

We usually give up and just return Any.

My suggestion would be to add a type tentatively called "AnyOf",

that acts as an unsafe union:

ip_address(address: str) -> AnyOf[IPv4Address, IPv6Address]

I believe this could be useful for type checkers. While the use of

AnyOf is unsafe, it is safer than just treating these return values

as Any. It's also theoretically possible to use some clever

narrowing. That said, there has been some pushback from the mypy

core team in the linked issue, especially since this is most likely

complex to implement. An easy way for type checkers to implement

this, without sacrificing any already existing type safety is to

just treating AnyOf exactly like Any. At least as a first step.

Should type checkers implement full support for AnyOf later, we

could already have better types in typeshed ready to use.

But in the last years, typeshed has also gained users outside of

type checkers. For example, PyCharm and jedi use it for

autocompletion. I believe that AnyOf could be useful for those

projects, as well as others.

Is there interest in a feature like this? I would be willing to

write a PEP and contribute the AnyOf = Any solution to mypy.

Unfortunately I don't have the bandwidth to learn how to implement

this "properly" in mypy.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

Zac Hatfield Dodds

11:28 p.m.

Rather than defining a new Any type, this seems to me rather like a TypeVar with what the Mypy docs call value restriction (not a bound). For example:

...

* ip_address() can return either an IPv4Address or an IPv6Address, depending on the input string. * urlopen() can return either an HTTPResponse or an addinfourl.

def ip_address(s: str) -> TypeVar("address", IPv4Address, IPv6Address): ... # and similarly for urlopen() I think this would also require new logic from typecheckers because I don't believe you can currently have a return typevar without an argument typevar, but the spelling makes sense to me and narrowing should be as obvious as it ever is between typecheckers.

...

* json.loads() et al. can return a selection of types, depending on the input.

JSON is really a case for recursive types - "we usually omit the base case" isn't much worse to me than "one or more layers in it's just Any". --Zac On Tue, 22 Sep 2020 at 00:15, Guido van Rossum wrote:

...

If we implement this, are we sure that IDEs like PyCharm and/or vscode (e.g. pylance) will use and implement this? Or is that part just speculation?

On Mon, Sep 21, 2020 at 06:45 Sebastian Rittau wrote:

...
This has previously been discussed here:

https://github.com/python/typing/issues/566.

In typeshed we have found that there a quite a few cases where functions

can return two or more different, incompatible types, that can't be

distinguished by the input types alone. A few examples:

* ip_address() can return either an IPv4Address or an IPv6Address,

depending on the input string.

* float ** float returns either a float or a complex. Currently,

typeshed marks this as returning just float.

* json.loads() et al. can return a selection of types, depending on

the input.

* urlopen() can return either an HTTPResponse or an addinfourl.

There are more examples in the linked issue.

In these cases, returning a Union would technically be correct, but

would also be inconvenient for callers that know what type to expect.

We usually give up and just return Any.

My suggestion would be to add a type tentatively called "AnyOf",

that acts as an unsafe union:

ip_address(address: str) -> AnyOf[IPv4Address, IPv6Address]

I believe this could be useful for type checkers. While the use of

AnyOf is unsafe, it is safer than just treating these return values

as Any. It's also theoretically possible to use some clever

narrowing. That said, there has been some pushback from the mypy

core team in the linked issue, especially since this is most likely

complex to implement. An easy way for type checkers to implement

this, without sacrificing any already existing type safety is to

just treating AnyOf exactly like Any. At least as a first step.

Should type checkers implement full support for AnyOf later, we

could already have better types in typeshed ready to use.

But in the last years, typeshed has also gained users outside of

type checkers. For example, PyCharm and jedi use it for

autocompletion. I believe that AnyOf could be useful for those

projects, as well as others.

Is there interest in a feature like this? I would be willing to

write a PEP and contribute the AnyOf = Any solution to mypy.

Unfortunately I don't have the bandwidth to learn how to implement

this "properly" in mypy.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: zac.hatfield.dodds@gmail.com

Sebastian Rittau

22 Sep 22 Sep

1:11 a.m.

Am 21.09.20 um 16:14 schrieb Guido van Rossum:

...

If we implement this, are we sure that IDEs like PyCharm and/or vscode (e.g. pylance) will use and implement this? Or is that part just speculation?

At this point it is speculation, but I'd be very interested what maintainers of those projects think about this proposal. - Sebastian

Sebastian Rittau

2:24 a.m.

Am 21.09.20 um 16:28 schrieb Zac Hatfield Dodds:

...

Rather than defining a new Any type, this seems to me rather like a TypeVar with what the Mypy docs call value restriction (not a bound). For example:

...
* ip_address() can return either an IPv4Address or an IPv6Address, depending on the input string. * urlopen() can return either an HTTPResponse or an addinfourl.

def ip_address(s: str) -> TypeVar("address", IPv4Address, IPv6Address): ... # and similarly for urlopen()

I think this would also require new logic from typecheckers because I don't believe you can currently have a return typevar without an argument typevar, but the spelling makes sense to me and narrowing should be as obvious as it ever is between typecheckers.

I don't think that TypeVars are a good fit. As you point out, TypeVars only make sense if there are at least two instances of it in a definition. They have a completely different purpose (being variables) than what AnyOf is supposed to do.

...

...
* json.loads() et al. can return a selection of types, depending on the input.

JSON is really a case for recursive types - "we usually omit the base case" isn't much worse to me than "one or more layers in it's just Any".

It would definitely benefit greatly from recursive types, although I think this is a bit of an orthogonal issue. - Sebastian

Elazar

4:05 a.m.

Isn't that an (unsafe) intersection type? Elazar On Mon, 21 Sep 2020 at 20:27, Sebastian Rittau wrote:

...

Am 21.09.20 um 16:28 schrieb Zac Hatfield Dodds:

...
Rather than defining a new Any type, this seems to me rather like a TypeVar with what the Mypy docs call value restriction (not a bound). For example:

...
* ip_address() can return either an IPv4Address or an IPv6Address, depending on the input string. * urlopen() can return either an HTTPResponse or an addinfourl.

def ip_address(s: str) -> TypeVar("address", IPv4Address, IPv6Address): ... # and similarly for urlopen()

I think this would also require new logic from typecheckers because I don't believe you can currently have a return typevar without an argument typevar, but the spelling makes sense to me and narrowing should be as obvious as it ever is between typecheckers.

I don't think that TypeVars are a good fit. As you point out, TypeVars only make sense if there are at least two instances of it in a definition. They have a completely different purpose (being variables) than what AnyOf is supposed to do.

...
...
* json.loads() et al. can return a selection of types, depending on the input.

JSON is really a case for recursive types - "we usually omit the base case" isn't much worse to me than "one or more layers in it's just Any".

It would definitely benefit greatly from recursive types, although I think this is a bit of an orthogonal issue.

- Sebastian _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: elazarg@gmail.com

Richard Levasseur

4:22 a.m.

On Mon, Sep 21, 2020 at 6:45 AM Sebastian Rittau wrote:

...

This has previously been discussed here: https://github.com/python/typing/issues/566.

In typeshed we have found that there a quite a few cases where functions can return two or more different, incompatible types, that can't be distinguished by the input types alone. A few examples:

* ip_address() can return either an IPv4Address or an IPv6Address, depending on the input string. * float ** float returns either a float or a complex. Currently, typeshed marks this as returning just float. * json.loads() et al. can return a selection of types, depending on the input. * urlopen() can return either an HTTPResponse or an addinfourl.

There are more examples in the linked issue.

It looks like there's two different categories of inputs: a serialized format (json, ip address, arg parsing, struct unpacking) and float pow operator? TBH, I don't see how a function accepting a serialized input that decodes to different output types could safely use this AnyOf concept. The example that springs to mind is JSON decoding, which IME frequently has unexpected values (untrusted user input; old/inconsistent data format (e.g. "date" was encoding as int timestamp, then later switched to ISO timestamp); etc), which causes problems much later on (call stack or calendar wise) in some unrelated code. But, fundamentally, the same is true for the other examples. In particular, the problematic case that pops into mind is how the type annotations of a helper function become untrustworthy. e.g. given this: def get_name(json_str) -> str: """Gets the username from the bla bla bla lots of lines""" return json.loads(json_str) Then all I see -- because all I see is usages of get_name(), or am skimming the code because I don't have time/desire to read the full implementation -- is "a function that'll give me the name from the json str", not "a function that *might* give me the name, but really just whatever object was in the json and I need to be aware of that". Because I generally expect functions to be well behaved and reject invalid input, so when it says it returns a str, I trust that it'll do that or an exception is going to happen. IME, manually narrowing a union is easy: assert with isinstance(). It's cheap, easy, and detects the failure at the point of origin. FWIW, I think Pytype has (or had? pytype devs can correct me/clarify ) some functionality that, in specific cases, usage of a Union value was considered valid as long as any of the contained types was satisfied. This was a behavior that I grew to dislike because it'd bite much later. I would think my code was type-safe, but then, months later, find some edge case months later where I forgot to do a pre-check and ended up treating an int as a str or some such. I *think* this behavior was introduced long ago for pragmatic reasons (made it easier to introduce type checking), but I also think it's been cleaned up / restricted over time as typing has grown. For float pow -- and I don't recall the exact API definitions -- isn't it well defined (I couldn't find the language def from a quick search)? i.e if any input arg is complex, then the output is complex.

...

In these cases, returning a Union would technically be correct, but would also be inconvenient for callers that know what type to expect. We usually give up and just return Any.

My suggestion would be to add a type tentatively called "AnyOf", that acts as an unsafe union:

ip_address(address: str) -> AnyOf[IPv4Address, IPv6Address]

I believe this could be useful for type checkers. While the use of AnyOf is unsafe, it is safer than just treating these return values as Any. It's also theoretically possible to use some clever narrowing. That said, there has been some pushback from the mypy core team in the linked issue, especially since this is most likely complex to implement. An easy way for type checkers to implement this, without sacrificing any already existing type safety is to just treating AnyOf exactly like Any. At least as a first step.

Should type checkers implement full support for AnyOf later, we could already have better types in typeshed ready to use.

But in the last years, typeshed has also gained users outside of type checkers. For example, PyCharm and jedi use it for autocompletion. I believe that AnyOf could be useful for those projects, as well as others.

Is there interest in a feature like this? I would be willing to write a PEP and contribute the AnyOf = Any solution to mypy. Unfortunately I don't have the bandwidth to learn how to implement this "properly" in mypy.

- Sebastian _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: richardlev@gmail.com

Eric Traut

4:30 p.m.

I see some value in adding an "AnyOf" construct for return types, but a type checker won't be able to do much with this information because (as others have pointed out), it would be unsafe to make many assumptions about the type. The most significant benefit is that language servers could provide completion suggestions — something developers would definitely appreciate. That said, I don't think we need to introduce a new "AnyOf" type. As Zac pointed out, there's already a construct in the Python type system that has the desired properties of "AnyOf". Rather than invent a new construct, I'd prefer to leverage the constrained TypeVar. Adding support for a new "AnyOf" type would likely be a lot of work in Pyright, but leveraging constrained TypeVars would be nearly free. In fact, it mostly works already, and I was able to implement completion suggestions and type narrowing with just a few additional lines of code. I'm guessing this it would be easy for most existing type checkers to add this support. Mypy might require some small changes because it appears to use the first type in a constrained TypeVar rather than retaining the full list of constraints. I'm assuming that's not the intended behavior and is just a bug. Zac said:

...

I don't believe you can currently have a return typevar without an argument typevar...

I don't see anything in PEP 484 that states that a TypeVar used within a return type annotation must also appear in a parameter type annotation. Mypy doesn't appear to enforce this either, nor does the "stubtest" CI test for typeshed. There are legitimate uses for this. Here's an example: ```python def my_decorator(timeout: float) -> Callable[[_T], _T]: ... ``` Building on Zac's example, here's how this could work: ```python from typing import TypeVar from ipaddress import IPv4Address, IPv6Address IPAddress = TypeVar("IPAddress", IPv4Address, IPv6Address) def ip_address(s: str) -> IPAddress: if len(s.split('.')) == 4: return IPv4Address(s) else: return IPv6Address(s) a = ip_address("1.2.3.4") reveal_type(a) if isinstance(a, IPv4Address): print("IPv4!") reveal_type(a) else: print("IPv6!") reveal_type(a) ``` Pyright's output is: ``` Found 1 source file 15:13 - info: Type of "a" is "IPAddress" 19:17 - info: Type of "a" is "IPv4Address" 22:17 - info: Type of "a" is "IPv6Address" 0 errors, 0 warnings, 3 infos ``` Mypy's output is: ``` test.py:9: error: Incompatible return value type (got "IPv4Address", expected "IPv6Address") test.py:11: error: Incompatible return value type (got "IPv6Address", expected "IPv4Address") test.py:15: note: Revealed type is 'ipaddress.IPv4Address*' test.py:19: note: Revealed type is 'ipaddress.IPv4Address*' Found 2 errors in 1 file (checked 1 source file) ``` (This looks like a bug. Perhaps someone from the mypy team could speak to how difficult a fix would be.) (@Zac — You might be interested in the fact that I recently added support for recursive types in Pyright. https://devblogs.microsoft.com/python/pylance-introduces-five-new-features-t...) -Eric -- Eric Traut Microsoft

Rebecca Chen

5:13 p.m.

From my recent efforts to clean up incorrect TypeVar usages in the Google codebase, I've discovered that users already find many creative ways to misunderstand and misuse TypeVar. Echoing Sebastian's earlier point, I think it would be a bad idea to reuse TypeVar to also mean AnyOf, since that would contradict the current state of a TypeVar being meaningful only when it appears multiple times in a definition and likely cause even more confusion. Re: Richard's observation about pytype and unsafe unions: yes, we had them for practical reasons, and yes, we've been making things stricter ever since because of reports of missed bugs and surprising behavior. Best, Rebecca On Tue, Sep 22, 2020 at 12:30 AM Eric Traut wrote:

...

I see some value in adding an "AnyOf" construct for return types, but a type checker won't be able to do much with this information because (as others have pointed out), it would be unsafe to make many assumptions about the type. The most significant benefit is that language servers could provide completion suggestions — something developers would definitely appreciate.

That said, I don't think we need to introduce a new "AnyOf" type. As Zac pointed out, there's already a construct in the Python type system that has the desired properties of "AnyOf". Rather than invent a new construct, I'd prefer to leverage the constrained TypeVar.

Adding support for a new "AnyOf" type would likely be a lot of work in Pyright, but leveraging constrained TypeVars would be nearly free. In fact, it mostly works already, and I was able to implement completion suggestions and type narrowing with just a few additional lines of code. I'm guessing this it would be easy for most existing type checkers to add this support.

Mypy might require some small changes because it appears to use the first type in a constrained TypeVar rather than retaining the full list of constraints. I'm assuming that's not the intended behavior and is just a bug.

Zac said:

...
I don't believe you can currently have a return typevar without an argument typevar...

I don't see anything in PEP 484 that states that a TypeVar used within a return type annotation must also appear in a parameter type annotation. Mypy doesn't appear to enforce this either, nor does the "stubtest" CI test for typeshed. There are legitimate uses for this. Here's an example:

```python def my_decorator(timeout: float) -> Callable[[_T], _T]: ... ```

Building on Zac's example, here's how this could work:

```python from typing import TypeVar from ipaddress import IPv4Address, IPv6Address

IPAddress = TypeVar("IPAddress", IPv4Address, IPv6Address)

def ip_address(s: str) -> IPAddress: if len(s.split('.')) == 4: return IPv4Address(s) else: return IPv6Address(s)

a = ip_address("1.2.3.4") reveal_type(a)

if isinstance(a, IPv4Address): print("IPv4!") reveal_type(a) else: print("IPv6!") reveal_type(a) ```

Pyright's output is: ``` Found 1 source file 15:13 - info: Type of "a" is "IPAddress" 19:17 - info: Type of "a" is "IPv4Address" 22:17 - info: Type of "a" is "IPv6Address" 0 errors, 0 warnings, 3 infos ```

Mypy's output is: ``` test.py:9: error: Incompatible return value type (got "IPv4Address", expected "IPv6Address") test.py:11: error: Incompatible return value type (got "IPv6Address", expected "IPv4Address") test.py:15: note: Revealed type is 'ipaddress.IPv4Address*' test.py:19: note: Revealed type is 'ipaddress.IPv4Address*' Found 2 errors in 1 file (checked 1 source file) ``` (This looks like a bug. Perhaps someone from the mypy team could speak to how difficult a fix would be.)

(@Zac — You might be interested in the fact that I recently added support for recursive types in Pyright. https://devblogs.microsoft.com/python/pylance-introduces-five-new-features-t... )

-Eric

-- Eric Traut Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: rechen@google.com

Sebastian Rittau

6:09 p.m.

Am 21.09.20 um 21:22 schrieb Richard Levasseur:

...

It looks like there's two different categories of inputs: a serialized format (json, ip address, arg parsing, struct unpacking) and float pow operator? In the examples above, yes. There could be other categories in the other examples. TBH, I don't see how a function accepting a serialized input that decodes to different output types could safely use this AnyOf concept. The example that springs to mind is JSON decoding, which IME frequently has unexpected values (untrusted user input; old/inconsistent data format (e.g. "date" was encoding as int timestamp, then later switched to ISO timestamp); etc), which causes problems much later on (call stack or calendar wise) in some unrelated code. But, fundamentally, the same is true for the other examples.

AnyOf is not safe, but it is safer than the status quo. For example, the following type errors are currently accepted, but could be caught with AnyOf: j = json.loads(...) j.get(12) j.gets("foo") def foo(x: Mapping[int, Any]) -> None: ... foo(j)

...

IME, manually narrowing a union is easy: assert with isinstance(). It's cheap, easy, and detects the failure at the point of origin. This has been debated before and the current consensus for typeshed is not to use safe unions in return types (with exceptions). This proposal is supposed to improve on that status quo. Generally, typeshed prefers false negatives over false positives, which is especially necessary to ease the transition burden. For float pow -- and I don't recall the exact API definitions -- isn't it well defined (I couldn't find the language def from a quick search)? i.e if any input arg is complex, then the output is complex.

Yes, but e.g. (-2) ** -0.5 also returns a complex number. - Sebastian

Guido van Rossum

11:36 p.m.

I wonder if we need something similar to Annotated[type, arg, ...]. Here for the static type checker ‘type’ is used, but the other arguments can be types that the IDE uses for suggestions. Then you can write ** as -> Something[float, complex], and Jason as -> Something[Any, dict, list, float, int, bool, None]. (Int is not really needed.) I don’t have a good suggestion for a name. —Guido -- --Guido (mobile)

dimvar＠google.com

23 Sep 23 Sep

1:39 a.m.

If the main motivation for AnyOf is IDEs and not typechecking, maybe we can define the annotation for AnyOf in a comment. There is precedent for this kind of thing (e.g., javadoc, Closure Compiler). Re: the use of AnyOf for typing. I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Sebastian Rittau

6:13 p.m.

Am 22.09.20 um 18:39 schrieb dimvar--- via Typing-sig:

...

Re: the use of AnyOf for typing. I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Unfortunately that is not true as the many examples in the originally mentioned ticket prove. Experience has shown that safe unions are inconvenient as return types in many cases, especially when typing previously untyped code. Unsafe unions provide a substantial type safety benefit over the only alternative of using Any. - Sebastian

Guido van Rossum

11:47 p.m.

So for a checker that interprets AnyOf fully, there would be no difference with Union, right? It’s just a spelling of Union with a default behavior of Any. Because clearly it can’t be meant to mean Intersection, right? Still, the example of float**float concerns me. It returns AnyOf[float, complex] but the default should be float. Your proposed notation doesn’t have a way to specify that. (Another fallback could be the join of all the types, but that would be complex, which is undesirable for float**float for pragmatic reasons.) —Guido On Wed, Sep 23, 2020 at 02:13 Sebastian Rittau wrote:

...

Am 22.09.20 um 18:39 schrieb dimvar--- via Typing-sig:

...
Re: the use of AnyOf for typing.

...
I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Unfortunately that is not true as the many examples in the originally

mentioned ticket prove. Experience has shown that safe unions are

inconvenient as return types in many cases, especially when typing

previously untyped code. Unsafe unions provide a substantial type safety

benefit over the only alternative of using Any.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

Jelle Zijlstra

24 Sep 24 Sep

12:06 a.m.

El mié., 23 sept. 2020 a las 16:48, Guido van Rossum () escribió:

...

So for a checker that interprets AnyOf fully, there would be no difference with Union, right? It’s just a spelling of Union with a default behavior of Any.

My understanding is that a type checker should accept an operation on a value of type `AnyOf[A, B]` if it is valid on either A or B, but an operation on a value of type `Union[A, B]` is acceptable only if it is valid on both A and B. So `len(x)` where `x: AnyOf[str, int]` would be valid.

That does sound similar to `Intersection`, now that I think of it.

...

Because clearly it can’t be meant to mean Intersection, right?

Still, the example of float**float concerns me. It returns AnyOf[float, complex] but the default should be float. Your proposed notation doesn’t have a way to specify that. (Another fallback could be the join of all the types, but that would be complex, which is undesirable for float**float for pragmatic reasons.)

—Guido

On Wed, Sep 23, 2020 at 02:13 Sebastian Rittau wrote:

...
Am 22.09.20 um 18:39 schrieb dimvar--- via Typing-sig:

...
Re: the use of AnyOf for typing.

...
I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Unfortunately that is not true as the many examples in the originally

mentioned ticket prove. Experience has shown that safe unions are

inconvenient as return types in many cases, especially when typing

previously untyped code. Unsafe unions provide a substantial type safety

benefit over the only alternative of using Any.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com

Elazar

12:11 a.m.

It is casting a union into an intersection. * Any of the types can be assigned into it. * It behaves like all of the types combined. Elazar בתאריך יום ד׳, 23 בספט׳ 2020, 18:07, מאת Jelle Zijlstra ‏< jelle.zijlstra@gmail.com>:

...

El mié., 23 sept. 2020 a las 16:48, Guido van Rossum () escribió:

...
So for a checker that interprets AnyOf fully, there would be no difference with Union, right? It’s just a spelling of Union with a default behavior of Any.

My understanding is that a type checker should accept an operation on a value of type `AnyOf[A, B]` if it is valid on either A or B, but an operation on a value of type `Union[A, B]` is acceptable only if it is valid on both A and B. So `len(x)` where `x: AnyOf[str, int]` would be valid.

That does sound similar to `Intersection`, now that I think of it.

...
Because clearly it can’t be meant to mean Intersection, right?

Still, the example of float**float concerns me. It returns AnyOf[float, complex] but the default should be float. Your proposed notation doesn’t have a way to specify that. (Another fallback could be the join of all the types, but that would be complex, which is undesirable for float**float for pragmatic reasons.)

—Guido

On Wed, Sep 23, 2020 at 02:13 Sebastian Rittau wrote:

...
Am 22.09.20 um 18:39 schrieb dimvar--- via Typing-sig:

...
Re: the use of AnyOf for typing.

...
I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Unfortunately that is not true as the many examples in the originally

mentioned ticket prove. Experience has shown that safe unions are

inconvenient as return types in many cases, especially when typing

previously untyped code. Unsafe unions provide a substantial type safety

benefit over the only alternative of using Any.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com

_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: elazarg@gmail.com

Guido van Rossum

12:12 a.m.

Hm, I hadn’t thought of it that way, but you're right. But that would mean that using an Intersection type is usually a lie, right? (I.e. unsound, I believe is the term?) On Wed, Sep 23, 2020 at 08:07 Jelle Zijlstra wrote:

...

El mié., 23 sept. 2020 a las 16:48, Guido van Rossum () escribió:

...
So for a checker that interprets AnyOf fully, there would be no difference with Union, right? It’s just a spelling of Union with a default behavior of Any.

My understanding is that a type checker should accept an operation on a value of type `AnyOf[A, B]` if it is valid on either A or B, but an operation on a value of type `Union[A, B]` is acceptable only if it is valid on both A and B. So `len(x)` where `x: AnyOf[str, int]` would be valid.

That does sound similar to `Intersection`, now that I think of it.

...
Because clearly it can’t be meant to mean Intersection, right?

Still, the example of float**float concerns me. It returns AnyOf[float, complex] but the default should be float. Your proposed notation doesn’t have a way to specify that. (Another fallback could be the join of all the types, but that would be complex, which is undesirable for float**float for pragmatic reasons.)

—Guido

On Wed, Sep 23, 2020 at 02:13 Sebastian Rittau wrote:

...
Am 22.09.20 um 18:39 schrieb dimvar--- via Typing-sig:

...
Re: the use of AnyOf for typing.

...
I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Unfortunately that is not true as the many examples in the originally

mentioned ticket prove. Experience has shown that safe unions are

inconvenient as return types in many cases, especially when typing

previously untyped code. Unsafe unions provide a substantial type safety

benefit over the only alternative of using Any.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: jelle.zijlstra@gmail.com

--

--Guido (mobile)

Anthony Sottile

1:19 a.m.

I think this would still be useful for replacing functions which return `-> Any` currently (because of the burden of returning a Union on callers - essentially a slightly-safer-but-still-unsafe-Any). for example, I think it would be useful for a type checker / IDE to produce an error on something like this: resp = json.load(f) print(resp.definitely_not_a_thing) but allow any of the attributes / methods of the union of all of the attributes / methods of the `AnyOf` type: resp = json.load(f) print(resp['this_is_a_thing']) It's still unsound, but it's a strict improvement over Any I guess for clarity in definition: - Union[T1, T2] only allows attributes which are the _intersection_ of those provided by T1 & T2 - AnyOf[T1, T2] allows attributes which are the _union_ of those provided by T1 | T2 Anthony On Wed, Sep 23, 2020 at 8:15 AM Guido van Rossum wrote:

...

Hm, I hadn’t thought of it that way, but you're right. But that would mean that using an Intersection type is usually a lie, right? (I.e. unsound, I believe is the term?)

On Wed, Sep 23, 2020 at 08:07 Jelle Zijlstra wrote:

...
El mié., 23 sept. 2020 a las 16:48, Guido van Rossum () escribió:

...
So for a checker that interprets AnyOf fully, there would be no difference with Union, right? It’s just a spelling of Union with a default behavior of Any.

My understanding is that a type checker should accept an operation on a value of type `AnyOf[A, B]` if it is valid on either A or B, but an operation on a value of type `Union[A, B]` is acceptable only if it is valid on both A and B. So `len(x)` where `x: AnyOf[str, int]` would be valid.

That does sound similar to `Intersection`, now that I think of it.

...
Because clearly it can’t be meant to mean Intersection, right?

Still, the example of float**float concerns me. It returns AnyOf[float, complex] but the default should be float. Your proposed notation doesn’t have a way to specify that. (Another fallback could be the join of all the types, but that would be complex, which is undesirable for float**float for pragmatic reasons.)

—Guido

On Wed, Sep 23, 2020 at 02:13 Sebastian Rittau wrote:

...
Am 22.09.20 um 18:39 schrieb dimvar--- via Typing-sig:

...
Re: the use of AnyOf for typing.

...
I agree with the folks saying that it's probably not a good idea to introduce an unsafe union. In the cases where more type safety than just using Any is desired, one can use an ordinary union, and the callers of the function can cast the result to their desired type.

Unfortunately that is not true as the many examples in the originally

mentioned ticket prove. Experience has shown that safe unions are

inconvenient as return types in many cases, especially when typing

previously untyped code. Unsafe unions provide a substantial type safety

benefit over the only alternative of using Any.

- Sebastian

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: guido@python.org

-- --Guido (mobile)

_______________________________________________

Typing-sig mailing list -- typing-sig@python.org

To unsubscribe send an email to typing-sig-leave@python.org

https://mail.python.org/mailman3/lists/typing-sig.python.org/

Member address: jelle.zijlstra@gmail.com

-- --Guido (mobile) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: asottile@umich.edu

Eric Traut

1:28 a.m.

That depends on how you define "intersection type". If you are using the classic definition from type theory, then yes, use of an intersection type is unsound. TypeScript, Scala and other newer languages define "intersection type" a bit differently. https://www.typescriptlang.org/docs/handbook/unions-and-intersections.html#i.... They treat an intersection as a new type that combines the subtypes into one — i.e. it has all the members of all of the subtypes — to support ad hoc polymorphism. That's safe, but it is meaningful only for structural types. And it doesn't really apply to the AnyOf discussion.

Guido van Rossum

2:24 a.m.

TL;DR: AnyOf us neither Union nor Intersection -- it behaves like one or the other depending on whether it's in a "giving" or "receiving" position. This makes it convenient (but unsound). On Wed, Sep 23, 2020 at 9:19 AM Anthony Sottile wrote:

...

[...] It's still unsound, but it's a strict improvement over Any

I guess for clarity in definition: - Union[T1, T2] only allows attributes which are the _intersection_ of those provided by T1 & T2 - AnyOf[T1, T2] allows attributes which are the _union_ of those provided by T1 | T2

Thinking about it more, maybe AnyOf should not be the same as Intersection. A key idea is that types have two roles: "receiving" and "giving". E.g. consider ``` def fun(arg: A) -> R: return ... ``` 1. Union From the caller's perspective, `arg: A` is a receiving position; if A is X|Y (using the new PEP 604 notation for unions) then we can pass either an X or a Y (or something if type X|Y), which is "easy" (by which I mean it's easy to satisfy the type checker). And the caller is receiving R. If R is a union, the caller has to discriminate the union somehow (typically using isinstance(); or using `if r is None` if R has the form X|None). I will call the requirement to discriminate "hard" (because it requires extra code that for various reasons often isn't there in code that was originally untyped). But from the perspective of the callee the roles are reversed: if A is X|Y then it has to discriminate arg ("hard"), whereas if R is X|Y, foo can return either an X or a Y indiscriminately ("easy"). Note that hard and easy are swapped compared to the caller's perspective (I'm sure this relates to contravariance somehow :-). None of that is news. 2. Intersection If we had classic intersection types spelled as X&Y, from the caller's perspective, if A is X&Y, we have to provide something that is both an X and a Y (perhaps involving multiple inheritance), which I consider "hard", whereas if R is X&Y, we can just use it as if it is both an X and a Y ("easy"). (From the docs Eric pointed to, it looks like JavaScript's Intersection type can be seen as multiple inheritance from two TypedDicts.) For the callee, the roles of intersection types are again reversed -- `arg: X&Y` is "easy", while `-> X&Y` means we have to work "hard" at constructing something that's both X and Y. Again, hard and easy are swapped compared to the caller -- and they are also swapped compared to Union types (a Union arg is hard for the function body, while an Intersection return is hard for it). That should also not be news, if you see Intersection as the "opposite" of Union. (It's less useful mostly because constructing something that's both an X and a Y is even more of a pain than discriminating a union, special cases for TypedDict notwithstanding.) 3. AnyOf Now we get to the kicker. A function that returns AnyOf[X, Y] makes it "easy" for the caller -- at least, easy to satisfy the type checker. So it behaves like an Intersection in a "giving" position. But it should be easy for the callee too! You can return either an X or a Y to make the checker happy, so here it behaves like a union. And from this we derive that AnyOf types are easy in all positions. 4. Afterthoughts Why is AnyOf so convenient? It gives up soundness for convenience, without degenerating entirely to Any. The behavior is similar to Any, because it is both a local top type and a local bottom type, if those concepts make sense. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

Sebastian Rittau

2:28 a.m.

Am 23.09.20 um 18:19 schrieb Anthony Sottile:

...

I think this would still be useful for replacing functions which return `-> Any` currently (because of the burden of returning a Union on callers - essentially a slightly-safer-but-still-unsafe-Any).

for example, I think it would be useful for a type checker / IDE to produce an error on something like this:

resp = json.load(f) print(resp.definitely_not_a_thing)

but allow any of the attributes / methods of the union of all of the attributes / methods of the `AnyOf` type:

resp = json.load(f) print(resp['this_is_a_thing'])

It's still unsound, but it's a strict improvement over Any

I guess for clarity in definition: - Union[T1, T2] only allows attributes which are the _intersection_ of those provided by T1 & T2 - AnyOf[T1, T2] allows attributes which are the _union_ of those provided by T1 | T2

Thank you for explaining it more eloquently than I could. "Intersection" was probably not the right to use, as Eric Traut pointed out. - Sebastian

1308

Age (days ago)

1310

Last active (days ago)

List overview

Download

20 comments

10 participants

participants (10)

Anthony Sottile
dimvar＠google.com
Elazar
Eric Traut
Guido van Rossum
Jelle Zijlstra
Rebecca Chen
Richard Levasseur
Sebastian Rittau
Zac Hatfield Dodds

AnyOf - Union for return types

tags

participants (10)