Right, with the final couple of updates to the PEP (culminating in https://github.com/python/peps/pull/1859) I think we're done. (Eric, heads up: we made a couple of final semantic changes, detailed in https://github.com/python/peps/pull/1856).

Guido, I think we're ready for PEP 646 to be sent to the steering council! What are the next steps?

On Tue, 23 Feb 2021 at 20:36, Matthew Rahtz <mrahtz@google.com> wrote:

I think I'm basically with Pradeep here - the question how to represent arrays with fully or partially arbitrary shapes seems like an issue with a lot of subtleties that we should leave for a future PEP. I feel very strongly about not committing to anything here just yet; for shape typing to be useful in some important examples I have in mind at DeepMind, we're going to need to get this just right.

> It means that there's no way for a type checker to distinguish between "I forgot to add (or haven't gotten around to adding) type arguments" and "I explicitly and intentionally mean `Tensor[Any, ...]`.

This therefore unfortunately has to necessarily be the case as this PEP, I reckon. Rectifying this will definitely be a priority for our future PEPs.

> As a compromise, how about if we allow the syntax Tensor[T, ...] only if T is `Any`?

Hmm. This is interesting, but I still hesitate - if we do decide we want `...` to mean 'This part is arbitrary', then this would prevent us from being able to specify 'An array with an artbirary data type and an arbitrary shape' as `Tensor[Any, ...]`. (To emphasise, I'm not saying that we necessarily should use '...' to mean 'This part is arbitrary', but only that it seems like a plausible enough option, and a) we shouldn't complicate this PEP by talking about that, and b) we should make this PEP flexible enough to accomodate that potential option.)

My opinions about `*Tuple[Any, ...]` have shifted in the past few days, though. It is very verbose. Too verbose for my liking. My 'emulating a newcomer' narrative is still like "Why do we put the tuple with known contents there and then unpack it? Why not just put the contents there directly? Rrrrrr"

So the solution I'm currently leaning towards is the "No explicit syntax" option. We would state that an unparameterized alias `Float32Array` should be considered compatible in both directions with an arbitrary number of type parameters, without stating explicitly how an unparameterized `Float32Array` should be represented, giving us some wiggle room for later.

On Tue, 23 Feb 2021 at 02:18, S Pradeep Kumar <gohanpra@gmail.com> wrote:

Eric, responding to your original email:

> I noticed that the updated PEP includes a proposal for unparameterized generic type aliases. It currently indicates that a missing type argument for a variadic type var should be interpreted as a zero-length tuple. This interpretation would be inconsistent with every other case where type arguments are omitted, so I don't think that's the right answer.

> PEP:

> IntTuple = Tuple[int, *Ts]

> As this example shows, all type parameters passed to the alias are bound to the type variable tuple. If no type parameters are given, or if an explicitly empty list of type parameters are given, type variable tuple in the alias is simply ignored:

>

> # Both equivalent to Tuple[int]

> IntTuple

> IntTuple[()]

Sorry, I had missed this example in the latest PEP update. I agree that x: IntTuple should not be treated as IntTuple[()].

To be consistent with unparameterized Tensor we should treat `x: IntTuple` as an arbitrary-shaped Tuple, not as `IntTuple[()]`. A generic alias with a free `*Ts` should be treated just as a `class IntTupleClass(BaseClass[int, *Ts], Generic[*Ts])`. In the latter case, x: IntTupleClass would be treated as an arbitrary-shaped variadic. The unparameterized alias should behave the same way.

> In my opinion, omitting a type argument for a variadic type parameter should imply `Tuple[Any, ...]`. That's consistent with how `Tuple` works.
> If the type alias is concatenating other types with the variadic type var into a tuple, then the entire tuple should become a `Tuple[Any ...]`.

I think we agree about this. If we had the *Tuple[Any, ...] syntax we would replace Ts to get Tuple[int, *Tuple[Any, ...]], but since we're not adding this syntax in this PEP, we can keep it simple and just fall back to Tuple[Any, ...] as a whole.

> The proposal is effectively saying "the type system supports open-ended variadic generics, but the only way to specify them is to use a syntax that we want to discourage".

That's partially true. The type system supports arbitrary-shaped variadics to the very limited extent that it won't complain about passing Tensor[int, str] to Tensor or vice versa. Even this is only in the PEP to support pre-existing code. A typechecker could comply with the PEP by special-casing unparameterized variadic classes and aliases, without implementing open-ended variadic generics.

> The first option (no explicit syntax) doesn't address my primary concern. It means that there's no way for a type checker to distinguish between "I forgot to add (or haven't gotten around to adding) type arguments" and "I explicitly and intentionally mean `Tensor[Any, ...]`.

> In its strictest mode, pyright complains about both `Ndim` and `Shape` missing type arguments. It's a common mistake for developers to forget type arguments on generic types, so this is an important error. You can fix the problem with `Ndim` by changing it to `Ndim[Any]`, but you can't do the same with `Shape` because `Shape[Any]` would imply a single dimension for the variadic.

Ah, I see. In your strictest mode, you want to (a) warn users when they inadvertently use an unparameterized Tensor but (b) not warn them when they explicitly denote something as having arbitrary shape.

I can't think of a good alternative solution. The problem is that `Tensor[Any, ...]` doesn't seem to be on the road to precise future syntax like `Tensor[np.float64, *Tuple[Any, ...]]`. And we'd have to forbid `Tensor[int, ...]`, `Tensor[T, ...]`, etc. because of the problems mentioned earlier.

How about making `Tensor[Any, ...]` a separate PEP? We could bikeshed other possibilities like `Tensor[...]` that disallow using `int`. We could also judge whether the new syntax is worth the niche but important use case. (And we wouldn't have to add typing.py support for it as part of PEP 646)

On Mon, Feb 22, 2021 at 12:46 PM Eric Traut <eric@traut.com> wrote:

The first option (no explicit syntax) doesn't address my primary concern. It means that there's no way for a type checker to distinguish between "I forgot to add (or haven't gotten around to adding) type arguments" and "I explicitly and intentionally mean `Tensor[Any, ...]`.

The second option requires us to expand the syntax and increase the complexity of this PEP. I don't think we should do that either.

I understand your point that `[T, ...]` may not make sense for some variadic generic types (except the case where T is `Any`). I guess that doesn't concern me as much as it concerns you. It simply won't be used in ways that don't make sense. I also don't worry so much about users getting confused because they're already familiar with the semantics of "..." when used with tuples. But I understand your points here.

As a compromise, how about if we allow the syntax Tensor[T, ...] only if T is `Any`? In other words, `Tensor[np.float64, ...]` would be flagged as an error but `Tensor[Any, ...]` would be accepted.
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
https://mail.python.org/mailman3/lists/typing-sig.python.org/
Member address: gohanpra@gmail.com

--

S Pradeep Kumar

_______________________________________________
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
https://mail.python.org/mailman3/lists/typing-sig.python.org/
Member address: mrahtz@google.com