PEP 646 (Variadic Generics): final call for comments
Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council. If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.) Thanks! Matthew and Pradeep
I just wanted to thank Matthew & Pradeep for writing this PEP and for clarifications to the broader context of PEP 646 for array typing in https://github.com/python/peps/pull/1904. As someone who is heavily involved in the Python numerical computing community (e.g., NumPy, JAX, Xarray), but who is not so familiar with the details of Python's type system, it is reassuring to see that a broad range of use-cases related to type checking of named axes & shapes have been considered, and could build upon the infrastructure in this PEP. Type checking for shapes is something the NumPy community is very interested in -- there are more thumbs up on the relevant issue on NumPy's GitHub than any others (https://github.com/numpy/numpy/issues/7370) and we recently added a "typing" module that is under active development. It will certainly require experimentation to figure out the best ways to use type checking for ndarrays, but this PEP looks like an excellent foundation for such work. On Sat, Mar 20, 2021 at 9:52 AM Matthew Rahtz via Python-Dev < python-dev@python.org> wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5GWS2GGV... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, 2021-03-20 at 14:08 +0000, Matthew Rahtz via Python-Dev wrote:
Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hey, Any updates on this? I am very excited about this PEP :) Cheers, Filipe Laíns
Still working on it :) Specifically the grammar changes in cpython, though I haven't found much time to work on it recently. Tune into the tensor typing meeting on Monday for a more detailed update. On Thu, 8 Jul 2021 at 19:49, Filipe Laíns <lains@archlinux.org> wrote:
Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to
On Sat, 2021-03-20 at 14:08 +0000, Matthew Rahtz via Python-Dev wrote: the
steering council.
If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hey,
Any updates on this? I am very excited about this PEP :)
Cheers, Filipe Laíns
I'd be interested in using this the mechanisms defined in this PEP to define rank-generic Tensor types in TensorFlow, which are important in specifying `tf.function` signatures in a Pythonic way, using type annotations (rather than the custom `input_signature` mechanism we have today - see this issue: https://github.com/tensorflow/tensorflow/issues/31579). Variadic generics are among the last few missing pieces to create an elegant set of type definitions for tensors and shapes. Best, Dan Moldovan Senior Software Engineer, TensorFlow Dev Team
Matthew Rahtz wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's. I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.:: idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1] However, the PEP is silent about indexing assignment, e.g.:: array[0, *idxs_to_select, -1] = 1 IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan? For a second point, the PEP says:
this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous): def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ... If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view? And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me? SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to? TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin <pviktori@redhat.com> wrote:
Matthew Rahtz wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's.
I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.::
idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]
However, the PEP is silent about indexing assignment, e.g.::
array[0, *idxs_to_select, -1] = 1
IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan?
The previous SC approved the PEP despite this. If you want to convince the SC to request this feature parity in the PEP, I won't stop you. But unless that happens I would rather not update the PEP again (it's been tough to get to this point). Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.
For a second point, the PEP says:
this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous):
def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
I don't think that sentence is trying to forbid this. The problem appears in things like def foo(*args: tuple[*Ts1, *Ts2]) -> ... Maybe the wording in the PEP can be imrpoved?
If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view?
Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)
And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me?
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
I'll leave this to the PEP authors to address. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 12. 01. 22 17:58, Guido van Rossum wrote:
On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin <pviktori@redhat.com <mailto:pviktori@redhat.com>> wrote:
Matthew Rahtz wrote: > Hi everyone, > > We've got to the stage now with PEP 646 that we're feeling pretty happy > with it. So far though we've mainly been workshopping it in typing-sig, so > as PEP 1 requires we're asking for some feedback here too before submitting > it to the steering council. > > If you have time over the next couple of weeks, please take a look at the > current draft and let us know your thoughts: > https://www.python.org/dev/peps/pep-0646/ <https://www.python.org/dev/peps/pep-0646/> (Note that the final couple of > sections are out of date; https://github.com/python/peps/pull/1880 <https://github.com/python/peps/pull/1880> > clarifies which grammar changes would be required, now that PEP 637 has > been rejected. We also have a second PR in progress at > https://github.com/python/peps/pull/1881 <https://github.com/python/peps/pull/1881> clarifying some of the motivation.) > > Thanks! > Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's.
I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.::
idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]
However, the PEP is silent about indexing assignment, e.g.::
array[0, *idxs_to_select, -1] = 1
IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan?
The previous SC approved the PEP despite this.
If you want to convince the SC to request this feature parity in the PEP, I won't stop you.
But unless that happens I would rather not update the PEP again (it's been tough to get to this point).
Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.
As I learned after drafting that question, at least some members of the previous SC feel like this went without saying. So I'm a bit surprised by this reply. Guess it goes to show that PEPs need to be precise :) Looking at the current grammar, I see that the proposed change to `slices` will affect indexed assignment. And the reference implementation also allows it. With that in mind, can we treat this as this as a clarification, rather than a substantial change? A simple note that indexed assignment is also affected should be enough. (Tests/docs in the eventual implementation won't be so simple, but they'll be needed anyway.) Of course, if __setitem__ isn't meant to be affected, the proposed grammar change needs to be revisited.
For a second point, the PEP says:
> this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous):
def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
I don't think that sentence is trying to forbid this. The problem appears in things like
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
Maybe the wording in the PEP can be imrpoved?
Looks like it can -- the current wording does forbid it:
As of this PEP, only a single type variable tuple may appear in a type parameter list
But improving that smells like a rabbit hole. It's probably better to keep it disallowed (as a static typing expression) in the current PEP.
If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view?
Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)
+1, glad we're on the same page here.
And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me?
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
I'll leave this to the PEP authors to address.
Sounds good. Thank you!
The wording there probably should be improved. I had a different interpretation when I read that, so that suggests it needs to be clarified. We should ensure to draw a clear distinction between type parameters and type arguments. (Generic classes and functions are parameterized over type parameters and they have a type parameter list that is implicit in the syntax. Generic classes can be explicitly instantiated by giving them type arguments, and an instantiation has a (explicit or implicit) type argument list.) So when I read: """ As of this PEP, only a single type variable tuple may appear in a type parameter list: class Array(Generic[*Ts1, *Ts2]): ... # Error\ """ I interpreted it to mean that the error is that the type _parameters_ of the generic class Array include *Ts1 and *Ts2 (not that they were used as type arguments to Generic). Similarly, this should be an error: class Array(dict[*Ts1], Generator[*Ts2]): ... even though there is only a single type variable tuple appearing in a type _argument_ list. The reason for the restriction is that the tupling of Array's type arguments is not explicit in an instantiation of Array, so we rely on this restriction so that they can be unambiguously tupled. I don't think there is necessarily a similar restriction on a generic function's type parameters, because we don't have the ability to explicitly instantiate generic functions anyway. An alternative wording is along the lines of: "As of this PEP, only a single type variable tuple may appear among a generic class's type parameters." def foo(*args: tuple[*Ts1, *Ts2]) -> ... is already prohibited by "Multiple Unpackings in a Tuple: Not Allowed". There are three other occurrences of "type parameter list" in the PEP. Two of them talk about instantiating generic type aliases and should be changed to "type argument list". The last one is less clear, I can't quite parse out what it's trying to say. On Wed, Jan 12, 2022 at 5:04 PM Guido van Rossum <guido@python.org> wrote:
On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin <pviktori@redhat.com> wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at
Matthew Rahtz wrote: the
current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's.
I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.::
idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]
However, the PEP is silent about indexing assignment, e.g.::
array[0, *idxs_to_select, -1] = 1
IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan?
The previous SC approved the PEP despite this.
If you want to convince the SC to request this feature parity in the PEP, I won't stop you.
But unless that happens I would rather not update the PEP again (it's been tough to get to this point).
Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.
For a second point, the PEP says:
this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous):
def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
I don't think that sentence is trying to forbid this. The problem appears in things like
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
Maybe the wording in the PEP can be imrpoved?
If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view?
Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)
And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me?
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
I'll leave this to the PEP authors to address.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZHBIGZQP... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks also Kevin for this feedback! Good point about being careful to distinguish type parameters vs type arguments. If I understand correctly, you're making two points: 1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it *is* allowed. 2. Use of wrong terminology elsewhere in the PEP. Agreed. I'll draft a PR tweaking the wording to fix both these points. On Thu, 13 Jan 2022 at 11:28, Kevin Millikin <kmillikin@deepmind.com> wrote:
The wording there probably should be improved. I had a different interpretation when I read that, so that suggests it needs to be clarified.
We should ensure to draw a clear distinction between type parameters and type arguments. (Generic classes and functions are parameterized over type parameters and they have a type parameter list that is implicit in the syntax. Generic classes can be explicitly instantiated by giving them type arguments, and an instantiation has a (explicit or implicit) type argument list.)
So when I read:
""" As of this PEP, only a single type variable tuple may appear in a type parameter list:
class Array(Generic[*Ts1, *Ts2]): ... # Error\ """
I interpreted it to mean that the error is that the type _parameters_ of the generic class Array include *Ts1 and *Ts2 (not that they were used as type arguments to Generic). Similarly, this should be an error:
class Array(dict[*Ts1], Generator[*Ts2]): ...
even though there is only a single type variable tuple appearing in a type _argument_ list.
The reason for the restriction is that the tupling of Array's type arguments is not explicit in an instantiation of Array, so we rely on this restriction so that they can be unambiguously tupled.
I don't think there is necessarily a similar restriction on a generic function's type parameters, because we don't have the ability to explicitly instantiate generic functions anyway.
An alternative wording is along the lines of: "As of this PEP, only a single type variable tuple may appear among a generic class's type parameters."
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
is already prohibited by "Multiple Unpackings in a Tuple: Not Allowed".
There are three other occurrences of "type parameter list" in the PEP. Two of them talk about instantiating generic type aliases and should be changed to "type argument list". The last one is less clear, I can't quite parse out what it's trying to say.
On Wed, Jan 12, 2022 at 5:04 PM Guido van Rossum <guido@python.org> wrote:
On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin <pviktori@redhat.com> wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at
Matthew Rahtz wrote: the
current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's.
I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.::
idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]
However, the PEP is silent about indexing assignment, e.g.::
array[0, *idxs_to_select, -1] = 1
IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan?
The previous SC approved the PEP despite this.
If you want to convince the SC to request this feature parity in the PEP, I won't stop you.
But unless that happens I would rather not update the PEP again (it's been tough to get to this point).
Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.
For a second point, the PEP says:
this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous):
def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
I don't think that sentence is trying to forbid this. The problem appears in things like
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
Maybe the wording in the PEP can be imrpoved?
If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view?
Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)
And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me?
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
I'll leave this to the PEP authors to address.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZHBIGZQP... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Jan 13, 2022 at 8:18 AM Matthew Rahtz <mrahtz@google.com> wrote:
Thanks also Kevin for this feedback!
Good point about being careful to distinguish type parameters vs type arguments. If I understand correctly, you're making two points:
1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it *is* allowed.
That looks like a syntax error. We only allow *Ts if the argument is of the form *a. `def f(x: *Ts1)` is not allowed. Maybe you were thinking of `def f(x: Array[*Ts1], y: Array[*Ts2]) as the example? That seems unambiguous since the two positional arguments are given separately. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
1. The wording of the 'Multiple Type Variable Tuples: Not Allowed'
That looks like a syntax error. We only allow *Ts if the argument is of
[Matthew] section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it is allowed. [Guido] the form *a. `def f(x: *Ts1)` is not allowed.
Maybe you were thinking of `def f(x: Array[*Ts1], y: Array[*Ts2]) as the example? That seems unambiguous since the two positional arguments are given separately.
Sorry, yes!
Yes, exactly. Specifically, the "wrong" example in section 'Multiple Type Variable Tuples: Not Allowed' suggests that maybe what is wrong is that `Generic` was given more than one unpacked type variable tuple. The actual problem is a consequence of that: `class Array` has more than one type variable tuple as type parameters. (But there are other ways that could happen and all of them should be wrong.) I think it might be good to say that explicitly in that section. On Thu, Jan 13, 2022 at 4:18 PM Matthew Rahtz <mrahtz@google.com> wrote:
Thanks also Kevin for this feedback!
Good point about being careful to distinguish type parameters vs type arguments. If I understand correctly, you're making two points:
1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it *is* allowed.
2. Use of wrong terminology elsewhere in the PEP. Agreed.
I'll draft a PR tweaking the wording to fix both these points.
On Thu, 13 Jan 2022 at 11:28, Kevin Millikin <kmillikin@deepmind.com> wrote:
The wording there probably should be improved. I had a different interpretation when I read that, so that suggests it needs to be clarified.
We should ensure to draw a clear distinction between type parameters and type arguments. (Generic classes and functions are parameterized over type parameters and they have a type parameter list that is implicit in the syntax. Generic classes can be explicitly instantiated by giving them type arguments, and an instantiation has a (explicit or implicit) type argument list.)
So when I read:
""" As of this PEP, only a single type variable tuple may appear in a type parameter list:
class Array(Generic[*Ts1, *Ts2]): ... # Error\ """
I interpreted it to mean that the error is that the type _parameters_ of the generic class Array include *Ts1 and *Ts2 (not that they were used as type arguments to Generic). Similarly, this should be an error:
class Array(dict[*Ts1], Generator[*Ts2]): ...
even though there is only a single type variable tuple appearing in a type _argument_ list.
The reason for the restriction is that the tupling of Array's type arguments is not explicit in an instantiation of Array, so we rely on this restriction so that they can be unambiguously tupled.
I don't think there is necessarily a similar restriction on a generic function's type parameters, because we don't have the ability to explicitly instantiate generic functions anyway.
An alternative wording is along the lines of: "As of this PEP, only a single type variable tuple may appear among a generic class's type parameters."
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
is already prohibited by "Multiple Unpackings in a Tuple: Not Allowed".
There are three other occurrences of "type parameter list" in the PEP. Two of them talk about instantiating generic type aliases and should be changed to "type argument list". The last one is less clear, I can't quite parse out what it's trying to say.
On Wed, Jan 12, 2022 at 5:04 PM Guido van Rossum <guido@python.org> wrote:
On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin <pviktori@redhat.com> wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at
Matthew Rahtz wrote: the
current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's.
I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.::
idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]
However, the PEP is silent about indexing assignment, e.g.::
array[0, *idxs_to_select, -1] = 1
IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan?
The previous SC approved the PEP despite this.
If you want to convince the SC to request this feature parity in the PEP, I won't stop you.
But unless that happens I would rather not update the PEP again (it's been tough to get to this point).
Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.
For a second point, the PEP says:
this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous):
def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
I don't think that sentence is trying to forbid this. The problem appears in things like
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
Maybe the wording in the PEP can be imrpoved?
If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view?
Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)
And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me?
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
I'll leave this to the PEP authors to address.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZHBIGZQP... Code of Conduct: http://python.org/psf/codeofconduct/
*First point (indexing assignment)* [Guido]
Agreed. I just misremembered this, my bad! Please do the clarification etc.
Will do. *Second point (multiple TypeVarTuples)* [Guido]
I would love it for the cases where it's *not* ambiguous to just work (once type checkers support it). I'd like the PEP to be written so as not to disallow those. Type checkers that only support a single star even when it's unambiguous should be considered out of compliance. (But could still be fully compliant otherwise.) User pressure will then sort things out. That seems better than requiring a follow-up PEP just to allow non-ambiguous cases that follow the PEP's semantics.
I agree that would be nice, but I feel a bit worried this could result in additional churn. I was going to propose adding just a single sentence to the PEP along the lines of "Multiple unpacking are ok when the result would be unambiguous", but as you point out, there's no simple rule for deciding what would and wouldn't be ambiguous - so we'd probably have to spell out it out with examples in the PEP, which would a) make the PEP even longer, and b) maybe put us at risk of the SC wanting to re-re-review. So overall I'd strongly prefer to just add a sentence clarifying support at the grammar level to satisfy Petr, but leave it at that. Wdyt? To be clear, though:
This should clearly be disallowed: def f(*args: *tuple[*Ts1, *Ts2])... This (and similar variants) should also be disallowed: def f(*args: *tuple[*Ts, *Ts])... This should be allowed: def f(*args: *tuple[int, *Ts, str, T])... But not this: def f(*args: *tuple[*Ts1, str, *Ts2])...
It seems you are saying that this would be disallowed, even though
Right, I think the current wording in the PEP ("Multiple Unpackings in a Tuple: Not Allowed") is sufficient to make it clear these are disallowed. there's no ambiguity?
def f(a: Array[*Ts1], b: Array[*Ts2])...
This should definitely be fine, and I don't think there's any wording in the PEP which would suggest they be disallowed. *Third point (aliases)* [Guido]
As a corollary, this would also be correct, right? SplitDataSet[Height, Width] and it would make Ts be (Height, Width). Right?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] Given the previous example and my corollary, this seems ambiguous, just
Right, exactly. like `def f(*tuple[*Ts1, *Ts2])`. Oh, right, good point. So it sounds like the algorithm I described for aliases isn't *quite* accurate - it's not that we assign types to type variables by going from left to right and assigning greedily, it's that we assign types to type variables using the same mechanisms as in other contexts. So I guess the result is that multiple TypeVarTuples in aliases should be disallowed, for the same reasons as disallowing multiple TypeVarTuples in parameter lists elsewhere? (I'll wait for confirmation on this from you and Pradeep, then draft a PR clarifying this.) On Thu, 13 Jan 2022 at 16:44, Kevin Millikin <kmillikin@deepmind.com> wrote:
Yes, exactly.
Specifically, the "wrong" example in section 'Multiple Type Variable Tuples: Not Allowed' suggests that maybe what is wrong is that `Generic` was given more than one unpacked type variable tuple. The actual problem is a consequence of that: `class Array` has more than one type variable tuple as type parameters. (But there are other ways that could happen and all of them should be wrong.)
I think it might be good to say that explicitly in that section.
On Thu, Jan 13, 2022 at 4:18 PM Matthew Rahtz <mrahtz@google.com> wrote:
Thanks also Kevin for this feedback!
Good point about being careful to distinguish type parameters vs type arguments. If I understand correctly, you're making two points:
1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it *is* allowed.
2. Use of wrong terminology elsewhere in the PEP. Agreed.
I'll draft a PR tweaking the wording to fix both these points.
On Thu, 13 Jan 2022 at 11:28, Kevin Millikin <kmillikin@deepmind.com> wrote:
The wording there probably should be improved. I had a different interpretation when I read that, so that suggests it needs to be clarified.
We should ensure to draw a clear distinction between type parameters and type arguments. (Generic classes and functions are parameterized over type parameters and they have a type parameter list that is implicit in the syntax. Generic classes can be explicitly instantiated by giving them type arguments, and an instantiation has a (explicit or implicit) type argument list.)
So when I read:
""" As of this PEP, only a single type variable tuple may appear in a type parameter list:
class Array(Generic[*Ts1, *Ts2]): ... # Error\ """
I interpreted it to mean that the error is that the type _parameters_ of the generic class Array include *Ts1 and *Ts2 (not that they were used as type arguments to Generic). Similarly, this should be an error:
class Array(dict[*Ts1], Generator[*Ts2]): ...
even though there is only a single type variable tuple appearing in a type _argument_ list.
The reason for the restriction is that the tupling of Array's type arguments is not explicit in an instantiation of Array, so we rely on this restriction so that they can be unambiguously tupled.
I don't think there is necessarily a similar restriction on a generic function's type parameters, because we don't have the ability to explicitly instantiate generic functions anyway.
An alternative wording is along the lines of: "As of this PEP, only a single type variable tuple may appear among a generic class's type parameters."
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
is already prohibited by "Multiple Unpackings in a Tuple: Not Allowed".
There are three other occurrences of "type parameter list" in the PEP. Two of them talk about instantiating generic type aliases and should be changed to "type argument list". The last one is less clear, I can't quite parse out what it's trying to say.
On Wed, Jan 12, 2022 at 5:04 PM Guido van Rossum <guido@python.org> wrote:
On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin <pviktori@redhat.com> wrote:
Matthew Rahtz wrote:
Hi everyone,
We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.
If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)
Thanks! Matthew and Pradeep
Hi, I'm very late to the discussion -- I relied on the typing-sig and SC to handle this, but now that I'm on the SC, I no longer have that luxury :) This mail has my own opinions, not necessarily the SC's.
I've read the PEP, and I quite like it! It's clear that typing-sig thought this through very well. The thing that surprised me is the proposed changes that affect more than typing annotations. Quite deep in the PEP, the "Grammar Changes" section explains the (quite exciting) change to make star-unpacking possible in normal indexing operations, e.g.::
idxs_to_select = (1, 2) array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]
However, the PEP is silent about indexing assignment, e.g.::
array[0, *idxs_to_select, -1] = 1
IMO, it would be very confusing to not keep these in sync. If they are, the assignment change should be documented and tested appropriately. Is that the plan?
The previous SC approved the PEP despite this.
If you want to convince the SC to request this feature parity in the PEP, I won't stop you.
But unless that happens I would rather not update the PEP again (it's been tough to get to this point).
Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.
For a second point, the PEP says:
this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.
Typing annotations are sometimes used for other things than *static* typing, and I wouldn't be surprised if type checkers themselves started allowing this (as a non-standard extension in cases where things aren't ambiguous):
def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
I don't think that sentence is trying to forbid this. The problem appears in things like
def foo(*args: tuple[*Ts1, *Ts2]) -> ...
Maybe the wording in the PEP can be imrpoved?
If multiple unpackings in a tuple aren't blocked by the compiler, they should be tested and documented as syntactically valid annotations -- just not valid static typing annotations (even though other uses are currently deprecated). In particular, once the compiler allows multiple unpackings, disallowing them at the syntax level later would mean breaking backwards compatibility. Do we share that view?
Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)
And after reading the PEP again, I'm unclear on some details in the Aliases section. Could you please clarify these examples for me?
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid? What would this be equivalent to?
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?
I'll leave this to the PEP authors to address.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZHBIGZQP... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks for this feedback, Petr! *First point (indexing assignment)* Great catch; we hadn't thought about this. I agree it would be better to keep these in sync. I just tested this in our current CPython implementation, and can confirm it looks like this already works fine. So as much as I agree with Guido in preferring not to make too many more updates to the PEP, I guess we can indeed just fix this with a small clarification. I'll also add some tests for this to our CPython implementation. *Second point (multiple TypeVarTuples)* In terms of the wording in the PEP - Guido, our intention actually *was* to prohibit even the straightforward cases for now. Iirc, our reasoning was that we thought the decision boundary between "straightforward to infer type assignment" and "nontrivial to infer type assignment" (and of course "no unique type assignment") was tricky enough that we shouldn't complicate the already-long PEP by trying to describe all the cases where it was and wasn't ok. Petr, do I understand that the crux for you is basically that we should commit to multiple-stars at the syntax level - the main implication being to make sure we've properly documented and tested it? I'm happy with this; we plan to follow up with another PEP that *does* talk about when multiple unpackings are ok anyway. I guess we should just a) clarify in the PEP that allowing multiple unpackings in the grammar isn't accidental, and b) test this in our CPython implementation? *Third point (aliases)* This is actually a great point - I had to think about this myself. Since PEP 484 doesn't explicitly spell out how assignment of type to type variables in generic aliases works either, I had to try some things with mypy playground to figure out what the current rules are. I think the logic is, if we have an alias like Foo = tuple[T1, T1, T2] Foo[int, str] then we construct a list of unique type variables in the alias (here [T1, T2]), then assign types to those variables by going left to right through the type argument list when the alias is instantiated. Guido, can you remember from your time with mypy whether this is correct? Pradeep, I guess you'll also know about this? Based on that precedent, I believe that: SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid; equivalent Tuple[Array[Height], Array[Height]] (indeed, I just tested this with Pyre, and that matches our implementation there) For this one: TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height] Since we we allow a TypeVarTuple to be bound to an *empty* list of types, when we try to assign type arguments [Height] to the unique list of type variables [Ts1, Ts2], I think we should end up with Ts1 containing Height and Ts2 being empty; thus: TwoArrays[Height] # Valid; equivalent to Tuple[Array[Height], Array[()]] But I actually can't get this to type check correctly in Pyre. Pradeep, this is the test I was using: https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a Am I doing something wrong? Also, by that logic, in this situation *all* the type arguments would be assigned to Ts1, and Ts2 would always end up being empty. That seems a bit weird but...*shrug* it also seems correct. In any case, this is definitely something we should explain better in the PEP. I'll make a TODO for myself to write something on this once Pradeep and Guido have confirmed whether my understanding is correct.
On Thu, Jan 13, 2022 at 7:23 AM Matthew Rahtz <mrahtz@google.com> wrote:
Thanks for this feedback, Petr!
*First point (indexing assignment)*
Great catch; we hadn't thought about this. I agree it would be better to keep these in sync.
I just tested this in our current CPython implementation, and can confirm it looks like this already works fine. So as much as I agree with Guido in preferring not to make too many more updates to the PEP, I guess we can indeed just fix this with a small clarification. I'll also add some tests for this to our CPython implementation.
Agreed. I just misremembered this, my bad! Please do the clarification etc.
*Second point (multiple TypeVarTuples)*
In terms of the wording in the PEP - Guido, our intention actually *was* to prohibit even the straightforward cases for now. Iirc, our reasoning was that we thought the decision boundary between "straightforward to infer type assignment" and "nontrivial to infer type assignment" (and of course "no unique type assignment") was tricky enough that we shouldn't complicate the already-long PEP by trying to describe all the cases where it was and wasn't ok.
Petr, do I understand that the crux for you is basically that we should commit to multiple-stars at the syntax level - the main implication being to make sure we've properly documented and tested it? I'm happy with this; we plan to follow up with another PEP that *does* talk about when multiple unpackings are ok anyway. I guess we should just a) clarify in the PEP that allowing multiple unpackings in the grammar isn't accidental, and b) test this in our CPython implementation?
I would love it for the cases where it's *not* ambiguous to just work (once type checkers support it). I'd like the PEP to be written so as not to disallow those. Type checkers that only support a single star even when it's unambiguous should be considered out of compliance. (But could still be fully compliant otherwise.) User pressure will then sort things out. That seems better than requiring a follow-up PEP just to allow non-ambiguous cases that follow the PEP's semantics. FWIW I still feel we don't have a simple enough example to distinguish the situations. This should clearly be disallowed: def f(*args: *tuple[*Ts1, *Ts2])... This (and similar variants) should also be disallowed: def f(*args: *tuple[*Ts, *Ts])... This should be allowed: def f(*args: *tuple[int, *Ts, str, T])... But not this: def f(*args: *tuple[*Ts1, str, *Ts2])... (Even though there might be cases where it's unambiguous, like f(1, 2, 3, "", 4, 5).) It seems you are saying that this would be disallowed, even though there's no ambiguity? def f(a: Array[*Ts1], b: Array[*Ts2])... I'm failing to see the point. (Other than that perhaps Pyre doesn't support it? :-) The two positional (non-variadic!) arguments have different types, each of which is an Array with variadic length. I can see nothing ambiguous about this. If you indeed want to disallow this, can you show a scenario where there's an ambiguity? Also, in call sites I feel that multiple stars shouldn't be an issue. Example: def f(*args: *Ts)... f(*Array[int, int], *Array[str, str]) # Ts becomes [int, int, str, str]
*Third point (aliases)*
This is actually a great point - I had to think about this myself.
Since PEP 484 doesn't explicitly spell out how assignment of type to type variables in generic aliases works either, I had to try some things with mypy playground to figure out what the current rules are. I think the logic is, if we have an alias like
Foo = tuple[T1, T1, T2] Foo[int, str]
then we construct a list of unique type variables in the alias (here [T1, T2]), then assign types to those variables by going left to right through the type argument list when the alias is instantiated. Guido, can you remember from your time with mypy whether this is correct? Pradeep, I guess you'll also know about this?
Yes, that's how it works. Also, Foo[int] would be an error in that case.
Based on that precedent, I believe that:
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid; equivalent Tuple[Array[Height], Array[Height]]
(indeed, I just tested this with Pyre, and that matches our implementation there)
Yeah, looks fine. As a corollary, this would also be correct, right? SplitDataSet[Height, Width] and it would make Ts be (Height, Width). Right?
For this one:
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height]
Given the previous example and my corollary, this seems ambiguous, just like `def f(*tuple[*Ts1, *Ts2])`.
Since we we allow a TypeVarTuple to be bound to an *empty* list of types, when we try to assign type arguments [Height] to the unique list of type variables [Ts1, Ts2], I think we should end up with Ts1 containing Height and Ts2 being empty; thus:
TwoArrays[Height] # Valid; equivalent to Tuple[Array[Height], Array[()]]
I hope not.
But I actually can't get this to type check correctly in Pyre. Pradeep, this is the test I was using: https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a Am I doing something wrong?
Also, by that logic, in this situation *all* the type arguments would be assigned to Ts1, and Ts2 would always end up being empty. That seems a bit weird but...*shrug* it also seems correct.
No?
In any case, this is definitely something we should explain better in the PEP. I'll make a TODO for myself to write something on this once Pradeep and Guido have confirmed whether my understanding is correct.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
I'm seeing enough different interpretations to think things aren't quite specified -- but I'm not sure if it matters. (1) Is any of this something that should affect computation, or is it really just a question of how to interpret possibly ambiguous documentation? (2) Are any of these troubling cases something that a person should actually write for a normal situation? Or are they just arguments about which abbreviations are acceptable? Or about how automatically-generated (inferred) type descriptions should be written? (3) Are the slice-expansion questions all assumed to be indexing an n-dimensional array, as opposed to [start, stop, step]? Is that explicit in the PEP, and just not in the extracts here? (4) Expanding multiple * shouldn't be ambiguous; the problem is figuring out what to condense into which if two are adjacent. So s1, s2 =[a,b], (1,2,3) [*s1, *s2] should turn into [a, b, 1, 2, 3] The problem is that [*s3, *s4] = (a, b, 1, 2, 3) is ambiguous ... and I didn't really get that distinction from Petr's question or the answers. I can't tell whether I've missed something crucial, or others are arguing over angels on a pinhead ... so whatever the PEP ends up deciding, it should be explicit. (I *think* the earlier parts of this thread are consistent with this, and discussing whether to say explicitly that certain formats are forbidden (but maybe not enforced by the grammar), meaningless, or valid but currently meaningless outside of typing.) -jJ
On 13. 01. 22 16:23, Matthew Rahtz wrote:
Thanks for this feedback, Petr!
*First point (indexing assignment)*
Great catch; we hadn't thought about this. I agree it would be better to keep these in sync.
I just tested this in our current CPython implementation, and can confirm it looks like this already works fine. So as much as I agree with Guido in preferring not to make too many more updates to the PEP, I guess we can indeed just fix this with a small clarification. I'll also add some tests for this to our CPython implementation.
Thanks.
*Second point (multiple TypeVarTuples)*
In terms of the wording in the PEP - Guido, our intention actually /was/ to prohibit even the straightforward cases for now. Iirc, our reasoning was that we thought the decision boundary between "straightforward to infer type assignment" and "nontrivial to infer type assignment" (and of course "no unique type assignment") was tricky enough that we shouldn't complicate the already-long PEP by trying to describe all the cases where it was and wasn't ok.
Petr, do I understand that the crux for you is basically that we should commit to multiple-stars at the syntax level - the main implication being to make sure we've properly documented and tested it? I'm happy with this; we plan to follow up with another PEP that /does/ talk about when multiple unpackings are ok anyway. I guess we should just a) clarify in the PEP that allowing multiple unpackings in the grammar isn't accidental, and b) test this in our CPython implementation?
Even less, actually. The PEP doesn't make a very clear distinction between invalid Python syntax vs. invalid type annotation, so I wanted to check if we're on the same page here: the newly valid syntax will be subject to PEP 387. We clearly are on the same page, and I don't think you need to update the PEP.
*Third point (aliases)* * * This is actually a great point - I had to think about this myself.
Since PEP 484 doesn't explicitly spell out how assignment of type to type variables in generic aliases works either, I had to try some things with mypy playground to figure out what the current rules are. I think the logic is, if we have an alias like
Foo = tuple[T1, T1, T2] Foo[int, str]
then we construct a list of unique type variables in the alias (here [T1, T2]), then assign types to those variables by going left to right through the type argument list when the alias is instantiated. Guido, can you remember from your time with mypy whether this is correct? Pradeep, I guess you'll also know about this?
Based on that precedent, I believe that:
SplitDataset = Tuple[Array[*Ts], Array[*Ts]] SplitDataset[Height] # Valid; equivalent Tuple[Array[Height], Array[Height]]
(indeed, I just tested this with Pyre, and that matches our implementation there)
For this one:
TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]] TwoArrays[Height]
Since we we allow a TypeVarTuple to be bound to an /empty/ list of types, when we try to assign type arguments [Height] to the unique list of type variables [Ts1, Ts2], I think we should end up with Ts1 containing Height and Ts2 being empty; thus:
TwoArrays[Height] # Valid; equivalent to Tuple[Array[Height], Array[()]]
But I actually can't get this to type check correctly in Pyre. Pradeep, this is the test I was using: https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a <https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a> Am I doing something wrong?
Also, by that logic, in this situation /all/ the type arguments would be assigned to Ts1, and Ts2 would always end up being empty. That seems a bit weird but.../shrug/ it also seems correct.
In any case, this is definitely something we should explain better in the PEP. I'll make a TODO for myself to write something on this once Pradeep and Guido have confirmed whether my understanding is correct.
When I asked my curious question, I thought I misread a piece of text, not that it's a detail that went unnoticed, and could delay the PEP. I can't speak for the whole SC, but on the Monday meeting I'll suggest accepting the PEP with a note that - index assignment is also affected, and - the details around multiple unpackings in a type expression aren't specified precisely. This gives individual type checkers some leeway, but can be tightened in future PEPs.
Even less, actually. The PEP doesn't make a very clear distinction between invalid Python syntax vs. invalid type annotation, so I wanted to check if we're on the same page here: the newly valid syntax will be subject to PEP 387. We clearly are on the same page, and I don't think you need to update the PEP.
Ok, fair enough.
When I asked my curious question, I thought I misread a piece of text, not that it's a detail that went unnoticed, and could delay the PEP. I can't speak for the whole SC, but on the Monday meeting I'll suggest accepting the PEP with a note that - index assignment is also affected, and - the details around multiple unpackings in a type expression aren't specified precisely. This gives individual type checkers some leeway, but can be tightened in future PEPs.
Cool. Thanks, Petr!
participants (9)
-
Dan Moldovan -
Filipe Laíns -
Glenn Linderman -
Guido van Rossum -
Jim J. Jewett -
Kevin Millikin -
Matthew Rahtz -
Petr Viktorin -
Stephan Hoyer