Debrief for Callable syntax (PEP 677)
Hello all, I know many of us have been privately discussing the PEP 677 (Callable syntax) rejection and what it might mean for future typing efforts. I wanted to start a thread so that we can pool our thoughts and converge on some lessons for the future. ("We" below means various people from typing-sig - Steven, Eric, Guido, me, and everybody else who participated.) Timeline: + May 2021: We started discussing callable syntax, among other syntax proposals, at the Typing Summit + June 2021: Steven kicked off a typing-sig thread about callable syntax [2] + September 2021: We all had a Zoom meetup about Callable syntax [3]. The main outcome was that, as a group, we dropped the def-style syntax proposal because it was too verbose. We didn't reach consensus about shorthand vs extended style. + early October 2021: We mailed python-dev to get preliminary feedback about new callable syntax [4]. There was no strong pushback at that point, as far as I can see. People pointed out edge cases such as trailing commas and proposed alternatives such as function-name-as-a-type. The Steering Council asked us to explicitly discuss the syntax changes. + November 2021: We had another heated Zoom discussion about shorthand vs extended syntax. [5] We still didn't reach consensus. Given that shorthand syntax was forward-compatible with extended syntax, Steven and I drafted a PEP with shorthand syntax, hoping to get it in before the deadline for 3.11. + December 2021: We put up the draft PEP in python-dev [6]. There was some strong pushback, especially in this thread [7] - "tl;dr: I find it very troubling that we are going on a path where we need to increase the language complexity (syntax) only in the cause 'easier' typing. So I am opposed to this change". + December 2021: We discovered some wrinkles in the proposal: readers could find it hard to distinguish the multiple `->`s in a function signature (this was point (4) in the SC rejection notice); readers could get confusing errors because of the relative binding of `->` and `|` [8]. + February 2022: The SC rejected PEP 677 [1]. The reasons were: (1) "A feature for use only in a fraction of type annotations ... did not feel like a strong enough reason to justify the complexity needed to parse this new syntax and provide meaningful error messages" (2) "While the current Callable[x, y] syntax is not loved, it does work." (3) "Each time we add syntax solely for typing it shifts us further in the direction of typing being its own mini-language so we aim to tread lightly in what gets added here." (4) "We did not like the visual and cognitive consequence of multiple `->` tokens in a def." + February 2022: SC clarification in that thread based on Shantanu's question: "If the 677 proposed syntax had been expanded to include more def features instead of being conservative I think it would've been an even easier rejection for many of us". My takeaways: 1. We could have tested the syntax in real-world code a lot earlier. That would have caught the readability issue with multiple arrows and the relative binding of `->` and `|`. This can be a lesson for future syntax changes. 2. It would have been good to hear the December python-dev pushback earlier. If I'd heard that in the early-October python-dev thread, I would have been much more pessimistic about the PEP's chances. Maybe instead of the informal mail in October, we could have put up a draft PEP right away. That might have gotten more eyeballs in python-dev. 3. The def vs shorthand discussion was useful and helped us decisively eliminate a proposal. 4. In retrospect, the shorthand vs extended discussion seems less useful. From the SC clarification above, it looks like it was always going to favor a simpler proposal over a more complicated proposal. Once we realized that we weren't going to reach consensus anytime soon, maybe we could have avoided the extended discussions (but maybe not). 5. It was a good call to defer the full CPython implementation until the SC verdict. That would have been a month or two of work that went nowhere. 6. Finally, on a positive note, this syntax rejection doesn't mean we can't have any typing-related Python language changes. For example, I'd be interested in seeing builtins for `mapping[str, int]` and `sequence[int]`, similar to `dict` and `list`. We could even consider `callable[[int, str], bool]`. If there's enough interest in such ideas, we could have a discussion session for this at the PyCon Typing Summit. Eric Traut had mentioned that he'd provisionally implemented callable syntax in Pyright's parser. With this rejection, I'm guessing he'd need to roll that back. Let me know if anyone has any objection. Overall, I think this PEP was still worth submitting. It got us valuable feedback from python-dev and the SC about future syntax changes. Special thanks to Steven Troxler, who put in a *lot* of work writing the PEP, understanding the CPython code, and implementing the syntax changes. Thanks also to everyone who participated in the vigorous debates, particularly Eric Traut, who presented alternative proposals, and Guido van Rossum, who provided much wisdom and guidance behind the scenes. What are your takeaways? What could we (typing-sig as a whole or individual people) have done differently? Also feel free to vent your disappointment, surprise, or worry :) -- S Pradeep Kumar References: [1]: https://mail.python.org/archives/list/python-dev@python.org/thread/NHCLHCU2X... [2]: https://mail.python.org/archives/list/typing-sig@python.org/thread/3JNXLYH5V... [3]: https://docs.google.com/document/d/17iqV7WWvB0IwA43EPlIqlUS6Xuvk08X3sEudAA-g... [4]: https://mail.python.org/archives/list/python-dev@python.org/thread/VBHJOS3LO... [5]: https://docs.google.com/document/d/17iqV7WWvB0IwA43EPlIqlUS6Xuvk08X3sEudAA-g... [6]: https://mail.python.org/archives/list/python-dev@python.org/thread/OGACYN2X7... [7]: https://mail.python.org/archives/list/python-dev@python.org/thread/FI4AFU3I2... [8]: https://mail.python.org/archives/list/typing-sig@python.org/thread/DWIKV5XSI...
Thanks for kicking this thread off Pradeep! My main takaway is that syntax changes are hard. This doesn't surprise me, my impression going into this was that the Steering Council was likely to lean against syntax changes for typing, but the consensus at PyCon was that we should put a PEP forward anyway so that we could learn from the response. I am glad we went with the simpler option in a stand-alone PEP, since I think drafting an extended syntax PEP would have required even more work. In retrospect we might have been able to cut some of the discussion short if we had decided on this sooner, I do appreciate that in the rejection thread the SC clarified that a one-shot extended syntax PEP would not have been better. I do wish that there were an easier way to engage with python-dev for more feedback prior to a full-blown PEP - I do agree that the level of opposition once the PEP came out was a little surprising given that in October the thread was pretty encouraging. That said, one goal of drafting a full PEP is to clarify all aspects of a proposal so that python developers can form coherent opinions in the discussion phase. So I don't blame python-dev for not pushing back until the draft was out, and I'm not sure there's a better way. The truth is that some PEPs are going to be controversial, and a lot of that feedback will come after the draft is finished. I think one challenge is the split discussion across typing-sig and python-dev, that made managing communications quite a bit harder - particularly because the culture of typing-sig is very pro-typing, as in "let's make this feature as useful for typing as possible", whereas in python-dev the developers are rightly concerned about how a syntax change will fit into the language overall. Perhaps in the future we can identify when a proposal affects the core language heavily, and find a way to move discussion to python-dev early in the process. It's always disappointing to get a rejection, but I do think this PEP helped us get clear guidance from the SC about future directions. Many thanks to Eric, Jelle, and especially Guido and Pradeep for their help getting the PEP finished. Cheers, Steven
Great summary Pradeep. I had some discussions with SC members when I said we would be proposing a PEP for Dataclasses and I very much got the message "if your PEP does not involve any changes to the language we'll largely defer to the Typing SIG but otherwise your chances are really slim", which merely confirms what has been said above. I'm assuming this is because at this point typing is still seen as something that is of interest to a relatively small subset of the total Python user base and secondly it has largely (entirely?) been able to be bolted on without language changes, and so that's a tough line for the SC to cross. Perhaps as more and more package authors adopt type annotations and bump up against their limitations, attitudes will soften somewhat. But I did wonder about the statement "If the 677 proposed syntax had been expanded to include more def features instead of being conservative I think it would've been an even easier rejection for many of us". My understanding is the PEP was essentially syntactic sugar to provide a more elegant way of writing Callables, but did not address the current *expressive* limitations of Callables. Perhaps if there had been more emphasis on addressing those shortcomings, instead of just providing cleaner syntax, this might have carried more weight even if it was more ambitious in scope, as it would be solving not just a convenience problem but also a type checking problem. But TBH I did not read the PEP, just saw some of the discussion here, so I may be completely wrong here. On Tue, Feb 15, 2022 at 7:34 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Thanks for kicking this thread off Pradeep!
My main takaway is that syntax changes are hard. This doesn't surprise me, my impression going into this was that the Steering Council was likely to lean against syntax changes for typing, but the consensus at PyCon was that we should put a PEP forward anyway so that we could learn from the response.
I am glad we went with the simpler option in a stand-alone PEP, since I think drafting an extended syntax PEP would have required even more work. In retrospect we might have been able to cut some of the discussion short if we had decided on this sooner, I do appreciate that in the rejection thread the SC clarified that a one-shot extended syntax PEP would not have been better.
I do wish that there were an easier way to engage with python-dev for more feedback prior to a full-blown PEP - I do agree that the level of opposition once the PEP came out was a little surprising given that in October the thread was pretty encouraging.
That said, one goal of drafting a full PEP is to clarify all aspects of a proposal so that python developers can form coherent opinions in the discussion phase. So I don't blame python-dev for not pushing back until the draft was out, and I'm not sure there's a better way. The truth is that some PEPs are going to be controversial, and a lot of that feedback will come after the draft is finished.
I think one challenge is the split discussion across typing-sig and python-dev, that made managing communications quite a bit harder - particularly because the culture of typing-sig is very pro-typing, as in "let's make this feature as useful for typing as possible", whereas in python-dev the developers are rightly concerned about how a syntax change will fit into the language overall.
Perhaps in the future we can identify when a proposal affects the core language heavily, and find a way to move discussion to python-dev early in the process.
It's always disappointing to get a rejection, but I do think this PEP helped us get clear guidance from the SC about future directions.
Many thanks to Eric, Jelle, and especially Guido and Pradeep for their help getting the PEP finished.
Cheers, Steven _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gram@geekraver.com
Thank you, Pradeep. With this feature rejected, I think we can get back to "functions as types" discussion. Because it kinda does what we wanted to achieve with this PEP, but with no syntax changes. ср, 16 февр. 2022 г. в 07:47, Graham Wheeler <graham@grahamwheeler.com>:
Great summary Pradeep. I had some discussions with SC members when I said we would be proposing a PEP for Dataclasses and I very much got the message "if your PEP does not involve any changes to the language we'll largely defer to the Typing SIG but otherwise your chances are really slim", which merely confirms what has been said above. I'm assuming this is because at this point typing is still seen as something that is of interest to a relatively small subset of the total Python user base and secondly it has largely (entirely?) been able to be bolted on without language changes, and so that's a tough line for the SC to cross. Perhaps as more and more package authors adopt type annotations and bump up against their limitations, attitudes will soften somewhat. But I did wonder about the statement "If the 677 proposed syntax had been expanded to include more def features instead of being conservative I think it would've been an even easier rejection for many of us". My understanding is the PEP was essentially syntactic sugar to provide a more elegant way of writing Callables, but did not address the current *expressive* limitations of Callables. Perhaps if there had been more emphasis on addressing those shortcomings, instead of just providing cleaner syntax, this might have carried more weight even if it was more ambitious in scope, as it would be solving not just a convenience problem but also a type checking problem. But TBH I did not read the PEP, just saw some of the discussion here, so I may be completely wrong here.
On Tue, Feb 15, 2022 at 7:34 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Thanks for kicking this thread off Pradeep!
My main takaway is that syntax changes are hard. This doesn't surprise
me, my impression going into this was that the Steering Council was likely to lean against syntax changes for typing, but the consensus at PyCon was that we should put a PEP forward anyway so that we could learn from the response.
I am glad we went with the simpler option in a stand-alone PEP, since I
think drafting an extended syntax PEP would have required even more work. In retrospect we might have been able to cut some of the discussion short if we had decided on this sooner, I do appreciate that in the rejection thread the SC clarified that a one-shot extended syntax PEP would not have been better.
I do wish that there were an easier way to engage with python-dev for
more feedback prior to a full-blown PEP - I do agree that the level of opposition once the PEP came out was a little surprising given that in October the thread was pretty encouraging.
That said, one goal of drafting a full PEP is to clarify all aspects of
a proposal so that python developers can form coherent opinions in the discussion phase. So I don't blame python-dev for not pushing back until the draft was out, and I'm not sure there's a better way. The truth is that some PEPs are going to be controversial, and a lot of that feedback will come after the draft is finished.
I think one challenge is the split discussion across typing-sig and
python-dev, that made managing communications quite a bit harder - particularly because the culture of typing-sig is very pro-typing, as in "let's make this feature as useful for typing as possible", whereas in python-dev the developers are rightly concerned about how a syntax change will fit into the language overall.
Perhaps in the future we can identify when a proposal affects the core
language heavily, and find a way to move discussion to python-dev early in the process.
It's always disappointing to get a rejection, but I do think this PEP
helped us get clear guidance from the SC about future directions.
Many thanks to Eric, Jelle, and especially Guido and Pradeep for their
help getting the PEP finished.
Cheers, Steven _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gram@geekraver.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: n.a.sobolev@gmail.com
Thanks for the thorough post-mortem. I'm learning stuff too here. (Yesterday, Brett pointed out to me that seeming '->' *inside* an argument list is quite confusing when you only expect it to mark the end of the parameter list.) I somehow feel the need to explain what "python-dev" is. It's not "the Python core devs". It's "a mailing list discussing Python development" (i.e. developing the language, not using it). It doesn't reach all core devs, and many members are not core devs, they just like to talk about Python's design and implementation and how to evolve it. It's quite flawed, but it's still the best forum we have for PEPs that touch the language design. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
participants (5)
-
Graham Wheeler -
Guido van Rossum -
S Pradeep Kumar -
Steven Troxler -
Никита Соболев