Enhancing generic type documentation in the standard library
Hello, everyone! I believe our documentation about types needs another overhaul. We now have lots of generic types in the standard library, but their formal type parameters are poorly documented—or not documented at all—in the standard library documentation. More importantly: the documentation we have about specific covariant/contravariant types is now in entries in the `typing` module that are all deprecated since PEP 585 was implemented in Python 3.9. Below I present two of many examples where the documentation of a generic type is not great. However, if people agree this is a problem, we need to discuss where and how to put the documentation in a way that is not too disruptive to users of Python who don't know or don't care about type hints, for many reasons that we should not judge. For example, where do we document the fact that `dict` accepts two invariant formal type parameters, and that `frozenset` accepts one contravariant type parameter? What do you think? Cheers, Luciano _________________________________________ EXAMPLE 1: `Callable` variance is not documented For example, in the `Callable` type, the `ReturnType` parameter is covariant, and the `ParameterType` parameters are all contravariant. But there is no mention of variance in this entry: https://docs.python.org/3/library/typing.html?highlight=typing#typing.Callab... Also, no mention of the fact that `collections.abc.Callable` is generic here: https://docs.python.org/3/library/collections.abc.html#collections.abc.Calla... PEP 483—The Theory of Type Hints—is the only official Python doc where I found the information about the variance of the formal type parameters of `Callable`. The explanation there is brilliant [0]. [0] https://peps.python.org/pep-0483/#covariance-and-contravariance Regardless, the intended audience of PEPs is "core developers"—which is neither a superset nor a subset of "Python devs now using type hints". We should not rely on PEPs to document features for Python users in general. _________________________________________ EXAMPLE 2: `Generator` variance could be better documented The entry for `typing.Generator` [1] has this heading: class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) Answer quickly: how many formal type parameters does `Generator` require? Which are covariant? Which are contravariant? [1] https://docs.python.org/3/library/typing.html?highlight=typing#typing.Genera... Nowhere in that page [1] there's an explanation of the `*_co` and `*_contra` naming convention, much less their semantics. Fortunately, the text of the `typing.Generator` entry says: "A generator can be annotated by the generic type `Generator[YieldType, SendType, ReturnType]". Unfortunately, `typing.Generator` is deprecated and will be gone in 5 years or so... The same issue applies to all the other generic types: builtins (`dict`, `frozenset`), ABCs, etc. Their formal type parameters they accept as generics are either undocumented, or documented in parts of the `typing` module that are already deprecated. Thoughts? -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Wed, Mar 23, 2022 at 11:03 AM Luciano Ramalho <luciano@ramalho.org> wrote:
Hello, everyone!
I believe our documentation about types needs another overhaul.
The SC somewhat agrees! 😉 See https://mail.python.org/archives/list/typing-sig@python.org/thread/TVMQJXOJF... where we have asked for at least the specs to get consolidated into proper documentation instead of spread out across various PEPs. My personal hope is this will also lead to better docs overall in a central location instead of spread out among modules, tools, etc. -Brett
On Wed, Mar 23, 2022 at 5:01 PM Brett Cannon <brett@python.org> wrote:
The SC somewhat agrees! 😉 See https://mail.python.org/archives/list/typing-sig@python.org/thread/TVMQJXOJF... where we have asked for at least the specs to get consolidated into proper documentation instead of spread out across various PEPs.
That makes me somewhat happy. Just kidding. I totally agree that the typing PEPs need to be consolidated into specs, which then need to be kept up-to-date. The issue I raised is related but different: we now have dozens of generic types in the standard library that are not fully documented in PEPs or hardly anywhere else. For example, PEP 585–Type Hinting Generics In Standard Collections merely lists the types which, back then, were changed to accept type parameters within [ ]. That PEP does not document the number, meaning, or variance of the accepted type parameters. When Guido and I refactored the `typing` module docs for Python 3.9, we marked dozens of types as deprecated. The entries for those deprecated types, such as `typing.Generator`, are the only places where the parameters accepted by the generic type are documented—albeit in a way that's not user-friendly: class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co])
My personal hope is this will also lead to better docs overall in a central location instead of spread out among modules, tools, etc.
We share that hope and I am willing to help. Documenting the generic types in the standard library is a much smaller task than turning the typing PEPs into specs. It seems like a good next step on the way to better typing docs. Cheers, Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Mon, Mar 28, 2022 at 3:58 PM Luciano Ramalho <luciano@ramalho.org> wrote:
Since the typing-sig will likely be doing the work and driving such an effort I would ask over there. I personally would love for a typing.python.org or equivalent subsection of docs.python.org to exist. -Brett
On Wed, 30 Mar 2022, 3:49 am Brett Cannon, <brett@python.org> wrote:
The import system gained its own subsection of the language reference to get info out of the historical PEPs. Perhaps gradual typing could go the same way, with the initial content being: * the type hinting docs for the builtin generics * pointers to the PEPs that provide the interim docs until the reference section is fleshed out (we used the latter trick to bootstrap the PyPA specs page without having to rewrite all the content: some of the pages are just pointers to the relevant PEP) Cheers, Nick.
I personally would love for a typing.python.org or equivalent subsection of docs.python.org to exist.
+1 There’s a wrinkle here, however. The type specs are Python, but how they are used/interpreted is up to third party packages. So it’s a bit tricky to know exactly what to document where. I don’t think that’s insurmountable, but something to keep in mind. For example, while the clear specs are the first step, what the community really could use is a good “recommended practices for static typing” doc — and that’s harder to do without reference to particular tools. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Not defining the semantics of annotations was a brave move, but inevitably led to the situation where different use cases, all quite reasonable, would spring into being. Now they have, the development team has to decide a) which ones to sanction and b) which will be left out in the cold. I wish them well. Kind regards, Steve On Wed, Mar 30, 2022 at 5:24 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Wed, Mar 30, 2022 at 12:42 PM Steve Holden <steve@holdenweb.com> wrote:
Interesting, in the JavaScript (excuse me, ECMAScript) world, there's currently a proposal on the table to do just that, on steroids: https://github.com/giltayar/proposal-types-as-comments (full disclosure: some of the proposal authors are colleagues of mine). We'll have to see how they fare.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Mar 23, 2022 at 11:03 AM Luciano Ramalho <luciano@ramalho.org> wrote:
Hello, everyone!
I believe our documentation about types needs another overhaul.
The SC somewhat agrees! 😉 See https://mail.python.org/archives/list/typing-sig@python.org/thread/TVMQJXOJF... where we have asked for at least the specs to get consolidated into proper documentation instead of spread out across various PEPs. My personal hope is this will also lead to better docs overall in a central location instead of spread out among modules, tools, etc. -Brett
On Wed, Mar 23, 2022 at 5:01 PM Brett Cannon <brett@python.org> wrote:
The SC somewhat agrees! 😉 See https://mail.python.org/archives/list/typing-sig@python.org/thread/TVMQJXOJF... where we have asked for at least the specs to get consolidated into proper documentation instead of spread out across various PEPs.
That makes me somewhat happy. Just kidding. I totally agree that the typing PEPs need to be consolidated into specs, which then need to be kept up-to-date. The issue I raised is related but different: we now have dozens of generic types in the standard library that are not fully documented in PEPs or hardly anywhere else. For example, PEP 585–Type Hinting Generics In Standard Collections merely lists the types which, back then, were changed to accept type parameters within [ ]. That PEP does not document the number, meaning, or variance of the accepted type parameters. When Guido and I refactored the `typing` module docs for Python 3.9, we marked dozens of types as deprecated. The entries for those deprecated types, such as `typing.Generator`, are the only places where the parameters accepted by the generic type are documented—albeit in a way that's not user-friendly: class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co])
My personal hope is this will also lead to better docs overall in a central location instead of spread out among modules, tools, etc.
We share that hope and I am willing to help. Documenting the generic types in the standard library is a much smaller task than turning the typing PEPs into specs. It seems like a good next step on the way to better typing docs. Cheers, Luciano
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Mon, Mar 28, 2022 at 3:58 PM Luciano Ramalho <luciano@ramalho.org> wrote:
Since the typing-sig will likely be doing the work and driving such an effort I would ask over there. I personally would love for a typing.python.org or equivalent subsection of docs.python.org to exist. -Brett
On Wed, 30 Mar 2022, 3:49 am Brett Cannon, <brett@python.org> wrote:
The import system gained its own subsection of the language reference to get info out of the historical PEPs. Perhaps gradual typing could go the same way, with the initial content being: * the type hinting docs for the builtin generics * pointers to the PEPs that provide the interim docs until the reference section is fleshed out (we used the latter trick to bootstrap the PyPA specs page without having to rewrite all the content: some of the pages are just pointers to the relevant PEP) Cheers, Nick.
I personally would love for a typing.python.org or equivalent subsection of docs.python.org to exist.
+1 There’s a wrinkle here, however. The type specs are Python, but how they are used/interpreted is up to third party packages. So it’s a bit tricky to know exactly what to document where. I don’t think that’s insurmountable, but something to keep in mind. For example, while the clear specs are the first step, what the community really could use is a good “recommended practices for static typing” doc — and that’s harder to do without reference to particular tools. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Not defining the semantics of annotations was a brave move, but inevitably led to the situation where different use cases, all quite reasonable, would spring into being. Now they have, the development team has to decide a) which ones to sanction and b) which will be left out in the cold. I wish them well. Kind regards, Steve On Wed, Mar 30, 2022 at 5:24 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Wed, Mar 30, 2022 at 12:42 PM Steve Holden <steve@holdenweb.com> wrote:
Interesting, in the JavaScript (excuse me, ECMAScript) world, there's currently a proposal on the table to do just that, on steroids: https://github.com/giltayar/proposal-types-as-comments (full disclosure: some of the proposal authors are colleagues of mine). We'll have to see how they fare.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (6)
-
Brett Cannon -
Christopher Barker -
Guido van Rossum -
Luciano Ramalho -
Nick Coghlan -
Steve Holden