Just two random thoughts, not intended as an in-depth analysis of this proposal. Am 16.04.21 um 18:59 schrieb Jukka Lehtosalo:

However, what about type aliases:

  A = () => int  # Hmm?

An alternative for aliases is to rethink the rejected idea from PEP 613 and allow type aliases to be defined like this: A: TypeAlias[() => int] (I really wish we could use <> instead of [], though.)

  type A = list[int]

As introducing "type" as a keyword could be problematic (although I might be mistaken), we could reuse "def":     def A = list[int]  - Sebastian

This proposal also naturally extends to: type import mod and from mod type import X, Y These use cases don't need the custom syntax support, but with the same lazy evaluation they would form a natural replacement for guarding imports with `if TYPE_CHECKING:`, which would also be compatible with PEP 649 and runtime evaluation of annotations. Carl On Fri, Apr 16, 2021 at 11:00 AM Jukka Lehtosalo wrote:

Many people seem to agree that Python typing still has some rough edges. Various PEPs have fixed warts (or proposed fixes), including PEPs 585, 563, and 649. Still, several issues remain:

1. Type annotation syntax is not always elegant (consider callable types). There's no good way to add custom type syntax without "polluting" non-typing use cases.

2. String literal escaping is ugly, and it's still needed from time to time.

3. Runtime introspection of types feels like a second-class citizen, even though it's pretty popular (pydantic, etc.).

Instead of trying to tackle these issues piecemeal, I've tried to come up with ideas to solve all of the above issues. They don't form a complete proposal, and I haven't worked out many details yet, but I'm hoping that this could be a discussion starter, and perhaps it can help us eventually design something that is workable.

Here I explain my first idea -- type assignment. Right now, we can't easily add dedicated syntax for callable types, for example, without either making this syntax available everywhere, or making it *un*available in type aliases and other common use cases, which don't use the annotation syntax.

Supporting new syntax in annotations seems manageable:

def doit(f: () => int) -> None: ... # Could be supported

However, what about type aliases:

A = () => int # Hmm?

If this works, nothing stops people from writing weird code like this:

d = {'x': (1 + y) => foo(1, 2)}

I'd rather have dedicated typing syntax reserved for type-related uses, mainly to avoid confusing Python users who don't use typing. If typing syntax is only available via a small number of special syntactic forms, it's easier to teach users to ignore it, if they don't care about it.

I'm proposing a new construct called "type assignment" where the RHS is lazily evaluated, and in the future it might support typing-only syntax. There would be two variants. First, we can define a type alias like this:

type A = list[int]

If we'd use the new syntax for callable types, we could write an alias like this:

type C = (int) => str # No problem

However, the following would be a syntax error, since it's not in a "type context":

C = (int) => str # Error, as it probably should be

Since the right hand side of type assignment contains types, we can avoid evaluating them during module import time, using either PEP 563 semantics (stringification) or something resembling PEP 649 (lazy evaluation). This means that we won't need to use explicit string literal escaping:

type A = B # OK! class B: ...

If the RHS is a function call, we'd call the function at runtime, but arguments would be (partially) stringified/lazy and they can contain type syntax (I'm glossing over some details):

type N = NewType(int) type T = TypeVar(bound=(int) => str) # Type-specific syntax is ok type D = typing.TypedDict({"x": C}) # C can be a forward reference! class C: ...

Note that the results here are usable in runtime contexts, since we call the function:

x = N(2) # Ok

The general syntax would be "type <id> = <name>(...)". We'd pass the stringified name as an implicit first argument for a cleaner syntax.

We can write the earlier examples today like this:

T = TypeVar("T", bound=Callable[[int], str]) N = NewType("N", int) D = typing.TypedDict("D", {"x": "C"}) # Need escaping here class C: ...

The new syntax uses 'type', which is widely used as a name. The new meaning would only apply in this assignment context, and all existing uses are still fine. 'type' wouldn't be a keyword.

To support the function call syntax lazily, we want to be able to distinguish between types and non-types in arguments.

First, any literals would be eagerly calculated and not escaped:

type T = f(1, "y", kw=False) # evaluated as T = f(1, "y", kw=False)

Name expressions are lazy, to allow forward references:

type T = f(C, kw=D) # T = f("C", kw="D") [if using PEP 563]

If using delayed evaluation instead of stringification, we can create some "lazy type reference" objects instead of strings (this is kind of hand-wavy).

Similarly, tuples, lists and dictionaries would be eager (but items can be lazy):

type T = f(C, [("x", list[C])]) # T = f("C", [("x", "list[C]")])

This would resemble PEP 563 stringification, but it needs to be a little more flexible.

The type statement syntax is general enough to support various existing use cases, including NewType, TypeVar, TypedDict, NamedTuple, and others, and it's also extensible to new forms.

I'll write separate posts about my other ideas I mentioned earlier.

Jukka _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: carl@oddbird.net

Hi Julia, On Fri, Apr 16, 2021 at 11:00 AM Jukka Lehtosalo wrote:

If the RHS is a function call, we'd call the function at runtime, but arguments would be (partially) stringified/lazy and they can contain type syntax (I'm glossing over some details):

type N = NewType(int) type T = TypeVar(bound=(int) => str) # Type-specific syntax is ok type D = typing.TypedDict({"x": C}) # C can be a forward reference! class C: ...

Note that the results here are usable in runtime contexts, since we call the function:

x = N(2) # Ok

My main question about the details of your proposal is this piece. Partial laziness adds a lot of complexity both to implementation and simply to understanding the semantics; is it really necessary? Might it not be acceptable to say that `x = N(2)` can only occur after any forward references involved in the definition of `N` have been defined? (In the common cases I think it would occur inside functions; the ordering is only really an issue if it occurs at top level.) And if so, could we avoid the complexities of partial laziness and instead just always wrap the entire RHS in lazy evaluation? The other issue is implicitly passing the stringified name as first arg. Personally I would rather handle the lazy evaluation in a generalized way as something like “module level descriptors”, in which case the name issue could be handled via the existing `__set_name__` protocol instead. Overall I like the direction of the proposal, and agree that a more holistic solution to handling of type syntax at runtime is needed. Carl

On Fri, Apr 16, 2021 at 5:14 PM Guido van Rossum wrote:

Or perhaps

type: import a from b import c def f() -> int: pass

Sure, that works too!

(You could even add an 'else:' clause. :-)

Since unlike `if TYPE_CHECKING:`, the semantics of `type:` would be that names defined in it are available at runtime (just lazily), I guess the semantics of the `else` would be that it runs eagerly and any names defined in it take precedence over those defined in `type:` at runtime? Carl

I wonder if it would be useful to let this syntax be used for overloads as well. The current overload syntax is quite verbose, but could perhaps look like this: type def a(arg: int) -> int: ... type def a(arg: str) -> str: ... type def a(arg: object) -> NoReturn: ... def a(arg: Union[int, str, object]) -> Union[int, str]: if not isinstance(arg, (int, str)): raise TypeError return arg

On Sat, Apr 17, 2021 at 12:29 PM Carl Meyer wrote:

On Fri, Apr 16, 2021 at 11:00 AM Jukka Lehtosalo wrote:

...

Note that the results here are usable in runtime contexts, since we call the function:

x = N(2) # Ok

My main question about the details of your proposal is this piece. Partial laziness adds a lot of complexity both to implementation and simply to understanding the semantics; is it really necessary? Might it not be acceptable to say that `x = N(2)` can only occur after any forward references involved in the definition of `N` have been defined? (In the common cases I think it would occur inside functions; the ordering is only really an issue if it occurs at top level.) And if so, could we avoid the complexities of partial laziness and instead just always wrap the entire RHS in lazy evaluation?

The partial laziness seems helpful to avoid explicit escaping in use cases like this: type T = TypeVar(bound=D) class C(Generic[T]): ... # If we evaluate T fully, we get NameError about D class D: ... Code like this seems somewhat common. I could find over 100 forward references in type variable bounds in a large internal codebase. Not having partial laziness could result in getting a NameError potentially far away from the source, which can be confusing. Changing the order of definitions will sometimes help, but it's not always practical. It can create a lot of needless code churn, especially if we are talking about changing the order of big classes. Some forward references are caused by import cycles, and avoiding import cycles is often not possible without major compromises. If we'd limit the use of forward references, users might prefer the old way as more flexible. There are other possible tricky situations that involve base classes. Example: type NT = NamedTuple(x=D) class class C(NT): ... class D: ... Occasionally type objects are also stored in some module-level data structures. Hypothetical example: type X = NamedTuple(...) type Y = NamedTuple(...) dd = { 'key1': X, 'key2': Y, ... } The other issue is implicitly passing the stringified name as first arg.

Personally I would rather handle the lazy evaluation in a generalized way as something like “module level descriptors”, in which case the name issue could be handled via the existing `__set_name__` protocol instead.

I don't much like the magic first argument, to be honest. Something like an implicit call to '__set_name__' sounds better to me. I'd just like to be able to drop the explicit string literal name argument somehow, since it looks quite awkward -- especially if the type name is long. Jukka

On Mon, Apr 19, 2021 at 5:49 AM Jukka Lehtosalo wrote:

The partial laziness seems helpful to avoid explicit escaping in use cases like this:

type T = TypeVar(bound=D) class C(Generic[T]): ... # If we evaluate T fully, we get NameError about D class D: ...

Code like this seems somewhat common. I could find over 100 forward references in type variable bounds in a large internal codebase. Not having partial laziness could result in getting a NameError potentially far away from the source, which can be confusing.

Changing the order of definitions will sometimes help, but it's not always practical. It can create a lot of needless code churn, especially if we are talking about changing the order of big classes. Some forward references are caused by import cycles, and avoiding import cycles is often not possible without major compromises. If we'd limit the use of forward references, users might prefer the old way as more flexible.

I think you're right. I've not been able to think of a better option than some form of partial laziness. I hope it could be kept as simple and consistent as possible.

I'd just like to be able to drop the explicit string literal name argument somehow, since it looks quite awkward -- especially if the type name is long.

Agreed. Carl

I like the idea of having special syntax for types definitions. However, I'm curious how this would interact with the existing (type-free) way of defining types? Do you see type assignment eventually becoming the one and only way? Sergei On Fri, Apr 16, 2021 at 6:00 PM Jukka Lehtosalo wrote:

Many people seem to agree that Python typing still has some rough edges. Various PEPs have fixed warts (or proposed fixes), including PEPs 585, 563, and 649. Still, several issues remain:

1. Type annotation syntax is not always elegant (consider callable types). There's no good way to add custom type syntax without "polluting" non-typing use cases.

2. String literal escaping is ugly, and it's still needed from time to time.

3. Runtime introspection of types feels like a second-class citizen, even though it's pretty popular (pydantic, etc.).

Instead of trying to tackle these issues piecemeal, I've tried to come up with ideas to solve all of the above issues. They don't form a complete proposal, and I haven't worked out many details yet, but I'm hoping that this could be a discussion starter, and perhaps it can help us eventually design something that is workable.

Here I explain my first idea -- type assignment. Right now, we can't easily add dedicated syntax for callable types, for example, without either making this syntax available everywhere, or making it *un*available in type aliases and other common use cases, which don't use the annotation syntax.

Supporting new syntax in annotations seems manageable:

def doit(f: () => int) -> None: ... # Could be supported

However, what about type aliases:

A = () => int # Hmm?

If this works, nothing stops people from writing weird code like this:

d = {'x': (1 + y) => foo(1, 2)}

I'd rather have dedicated typing syntax reserved for type-related uses, mainly to avoid confusing Python users who don't use typing. If typing syntax is only available via a small number of special syntactic forms, it's easier to teach users to ignore it, if they don't care about it.

I'm proposing a new construct called "type assignment" where the RHS is lazily evaluated, and in the future it might support typing-only syntax. There would be two variants. First, we can define a type alias like this:

type A = list[int]

If we'd use the new syntax for callable types, we could write an alias like this:

type C = (int) => str # No problem

However, the following would be a syntax error, since it's not in a "type context":

C = (int) => str # Error, as it probably should be

Since the right hand side of type assignment contains types, we can avoid evaluating them during module import time, using either PEP 563 semantics (stringification) or something resembling PEP 649 (lazy evaluation). This means that we won't need to use explicit string literal escaping:

type A = B # OK! class B: ...

If the RHS is a function call, we'd call the function at runtime, but arguments would be (partially) stringified/lazy and they can contain type syntax (I'm glossing over some details):

type N = NewType(int) type T = TypeVar(bound=(int) => str) # Type-specific syntax is ok type D = typing.TypedDict({"x": C}) # C can be a forward reference! class C: ...

Note that the results here are usable in runtime contexts, since we call the function:

x = N(2) # Ok

The general syntax would be "type <id> = <name>(...)". We'd pass the stringified name as an implicit first argument for a cleaner syntax.

We can write the earlier examples today like this:

T = TypeVar("T", bound=Callable[[int], str]) N = NewType("N", int) D = typing.TypedDict("D", {"x": "C"}) # Need escaping here class C: ...

The new syntax uses 'type', which is widely used as a name. The new meaning would only apply in this assignment context, and all existing uses are still fine. 'type' wouldn't be a keyword.

To support the function call syntax lazily, we want to be able to distinguish between types and non-types in arguments.

First, any literals would be eagerly calculated and not escaped:

type T = f(1, "y", kw=False) # evaluated as T = f(1, "y", kw=False)

Name expressions are lazy, to allow forward references:

type T = f(C, kw=D) # T = f("C", kw="D") [if using PEP 563]

If using delayed evaluation instead of stringification, we can create some "lazy type reference" objects instead of strings (this is kind of hand-wavy).

Similarly, tuples, lists and dictionaries would be eager (but items can be lazy):

type T = f(C, [("x", list[C])]) # T = f("C", [("x", "list[C]")])

This would resemble PEP 563 stringification, but it needs to be a little more flexible.

The type statement syntax is general enough to support various existing use cases, including NewType, TypeVar, TypedDict, NamedTuple, and others, and it's also extensible to new forms.

I'll write separate posts about my other ideas I mentioned earlier.

Jukka _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: sergei.a.lebedev@gmail.com