Mailman 3 May 2022 - Typing-sig

Re: Typing Meetup
by S Pradeep Kumar 19 Sep '23

19 Sep '23

Updating the Typing Meetup date to Wednesday as there were some conflicts. Let me know if there are any major time conflicts. Date: *Wednesday, November 03, 2021.*Time: 10 am San Francisco time (UTC-7) / 5 pm London time (UTC) Duration: 1 hour Potential discussion topics: + Typing **kwargs using a TypedDict (Franek Magiera?, Pradeep) [1] + Remaining questions about Callable syntax (Steven) + Non-required Protocol fields (Sebastian Rittau?) [2] + Type support for sealed classes (Adrian Freund?) [3] + Safe, typed JSON validation + Type system documentation (Shannon?) If you're working on these or any other topics and would like to start a discussion at the meetup, reach out to me. If you have interesting topics you'd like to see discussed, share them too. Once we have topics, we can always start a discussion even if there's no proposer yet. We will probably have time for 3-4 topics in this session. On Fri, Oct 15, 2021 at 12:55 PM S Pradeep Kumar <gohanpra(a)gmail.com> wrote: >>> >>>> Hello all, >>>> >>>> We plan to hold a virtual Typing Meetup to discuss key typing-related >>>> issues in real-time. The idea is that we can make progress on important or >>>> controversial discussions, get early feedback on in-progress PEPs, and >>>> gauge interest in new proposals. >>>> >>>> A virtual meetup will be more accessible to people in other timezones >>>> than Bay Area in-person meetups. It should also help consolidate >>>> discussions that are often spread out across mailing lists or Github repos. >>>> The tentative plan is to have a monthly meetup, but we can change that as >>>> needed. >>>> >>>> Date: Monday, November 01, 2021. >>>> Time: 10 am San Francisco time (UTC-7) / 5 pm London time (UTC) >>>> Duration: 1 hour >>>> >>>> Time and date are flexible. I'll share the zoom link and agenda the >>>> week before. I'll also record the meetup and share afterwards. >>>> >>>> If you're interested in presenting a new proposal or your work on some >>>> existing proposal, please respond here. We can shoot for 15 minutes per >>>> presentation. I know there are quite a few interesting threads on >>>> typing-sig that haven't seen much activity in a while :) >>>> -- >>>> S Pradeep Kumar >>>> _______________________________________________ >>>> Typing-sig mailing list -- typing-sig(a)python.org >>>> To unsubscribe send an email to typing-sig-leave(a)python.org >>>> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >>>> Member address: guido(a)python.org >>> >>> Links: [1] Typing **kwargs using TypedDict: https://mail.python.org/archives/list/typing-sig@python.org/thread/IUCXT2IQ… [2] Non-required Protocol fields: https://mail.python.org/archives/list/typing-sig@python.org/thread/P6FCGRJU… [3] Sealed classes: https://mail.python.org/archives/list/typing-sig@python.org/thread/QSCT2N4R… -- S Pradeep Kumar

10 60

Type narrowing for TypeGuard in the negative case
by Eric Traut 04 Aug '23

04 Aug '23

Since the introduction of PEP 647 (User-defined Type Guards), we've received a steady stream of input from users saying that they don't like the limitation that type narrowing is applied only in the positive case and is not applied in the negative case. In general, type narrowing is not safe to perform if a user-defined type guard returns False, so I've stood by the original decision not to provide type narrowing in the negative case, but there are cases where such narrowing is safe and desirable. Here's a recent thread where this was discussed in some detail: https://github.com/python/typing/issues/996#issuecomment-1002716830 In this discussion, Ilya Kamen proposed a solution. The proposal is to extend the existing `TypeGuard` to support an optional second type argument. If present, the second argument indicates the narrowed type in the negative type narrowing situation. Here's a simple (admittedly contrived) example: ```python def is_str(val: str | int) -> TypeGuard[str, int]: return isinstance(val, str) def func(val: str | int): if is_str(val): reveal_type(val) # str else: reveal_type(val) # int ``` I've implemented this proposal in the latest published version of pyright (1.1.202) so folks can experiment with it and see if they like it. If there's general consensus that it's the right approach, I can file an amendment for the existing PEP 647 to include this functionality. It was trivial to add to pyright, so I'm optimistic that it would likewise be easy to add to the other type checkers. Another common request for PEP 647 is the desire for an "assert" form of TypeGuard — a way to indicate that a function performs runtime validation of a type, raising an exception if the type is incorrect. It occurred to me that this two-argument form of `TypeGuard` could also be used to handle this use case. The second argument would be specified as a `NoReturn`. I've provisionally implemented this in pyright as well (although it's not in the currently-published version — you'll need to wait until 1.1.203 is published). Here's what this would look like: ```python def validate_sequence(val: Sequence[_T | None]) -> TypeGuard[Sequence[_T], NoReturn]: """Verifies that the sequence contains no None values""" if len([x for x in val if x is not None]) > 0: raise Exception() return True def func(val: Sequence[int | None]): validate_sequence(val) reveal_type(val) # Sequence[int] ``` I'm interested in input on these proposals. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft

7 8

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 20 Jun '23

20 Jun '23

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

17 35

Runtime Access to Type Parameters
by Saul Shanabrook 07 Jun '23

07 Jun '23

I would like to propose being able to access, at runtime, the type parameters in classmethods and in constructors. I wrote up a draft PEP on this to capture the thinking here: https://github.com/saulshanabrook/peps/blob/master/pep-9999.md It seems easier to add them to classmethods, so even that addition alone would be very welcome! It could be added to Guido's existing draft PR for pep 585, by making a small change in the getattr function (https://github.com/python/cpython/pull/18239/files#diff-a34ae826f869897e56e…), to first do a `__getattribute__` and check if the result is a descriptor (has a `__get__` attribute), and if so passing the generic class itself in as the class, instead of the origin class. I wrote up a little example of that in the PEP. To make it available in the constructor, I proposed adding a global function `get_parameters` to `typing` which returns the parameters in the constructor. I also provide a little mock implementation of that. Note that work is only for custom classes which inherit from `typing.Generic` not for any built in collections.

5 6

Support Algebraic Data Types with @sealed
by John Hagen 20 Mar '23

20 Mar '23

The original (superseded) PEP 622 proposed adding a @sealed decorator so that users could define algebraic data types (ADTs) that static type checkers could verify are `match`ed exhaustively: - https://peps.python.org/pep-0622/#sealed-classes-as-algebraic-data-types There is a somewhat fairly-upvoted question about ADTs in Python, indicating interest in this area: - https://stackoverflow.com/questions/16258553 @sealed would be very valuable towards making ADT usage within Python more type-safe. Is there any historical discussion about why @sealed was dropped from the final pattern matching PEP? What would be the next steps needed to push this forward? Would writing a new PEP be the next step?

15 32

Defaults for TypeVar-likes
by James H-B 03 Nov '22

03 Nov '22

I've been toying with the idea of writting a PEP for this long requested feature (https://github.com/python/typing/issues/307). I've got a preliminary draft of it with the motivation and a specification for the feature. In short features like ```py T = TypeVar("T", default=int) # This means that if no type is specified T = int @dataclass class Box(Generic[T]): value: T | None = None reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] ``` would be supported as would ```py YieldT = TypeVar("YieldT") SendT = TypeVar("SendT", default=None) ReturnT = TypeVar("ReturnT", default=None) class Generator(Generic[YieldT, SendT, ReturnT]): ... Generator[int] == Generator[int, None] == Generator[int, None, None] ``` Feedback and any questions on this would be very appreciated. https://gist.github.com/Gobot1234/8c9bfe8eb88f5ad42bf69b6f118033a7 Thanks, James H-B.

8 20

Re: Support for import hooks for static, type checkers and IDEs
by Mikhail Golubev 06 Oct '22

06 Oct '22

Aha, thanks for your reply, Bernat! It seems like it hasn't been sent to typing-sig, though. Hopefully, the automatic quotation will keep everyone on the same page. Your presentation on the topic is just what I needed. (and the terriers are adorable!) > Can you detail how you achieve this? As far as I'm aware pylance doesn't do this, am I right Eric? Effectively, we scan through names in sys.builtin_module_names and binary modules found in sys.path, import them and try reconstructing a stub definition for each of them using the introspection facilities of the language. It's the second kind of stubs we operate with internally, in addition to .pyi stubs. These don't have type annotations, but e.g. preserve docstrings of the definitions. We also use them for navigation e.g. when one invokes "Go to definition" on a built-in function, such as "print". In theory, the same mechanism could be reused for modules that become available dynamically through import hooks, if it's what you're implying. On Mon, May 9, 2022 at 9:26 PM Bernat Gabor <gaborjbernat(a)gmail.com> wrote: > generate stubs for binary modules >> > > Can you detail how you achieve this? As far as I'm aware pylance doesn't > do this, am I right Eric? > >> >> When should this process be repeated? For .pth files, we rely on regular >> filesystem events from under site-packages. Am I right that >> STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" >> directory, so we can expect that if it ever changes it will trigger another >> filesystem event in a known location? > > > Yes, in my vision, the file would be part of the dist-info, and as such > trackable as any other file for changes. > > Aren't editable installs, as they are, a mechanism too complicated just >> for exposing the content of "src" directory in sys.path? >> > > Exposing the src directory is what we historically did, and gets you 80% > there, but is not actually a true representation of what the user would > want in the other 20%. Editable installations at their core are a method to > develop a library without needing to reinstall it after every change. In > the trivial case, yes it can be achieved by just injecting the src folder > onto the sys.path via some mechanism (PYTHONPATH, cwd change, PTH file, > sitecustomize.py, etc). However, there are more complicated cases when this > approach will not work: > > - selective inclusion of what gets packaged (only package these > business logic files) > - selective exclusion of what gets packaged (don't package test files) > - any kind of code generation during the package build (either > C-extension or generate python code from a JSON schema) > > The goal we want to achieve in the new editable installations is to allow > supporting these additional use cases, even if with some limitations (e.g. > updates to files get picked up automatically but adding/removing new files > would still require a reinstallation - through tooling could hide this > away). My presentation from PyCon covered some of these, see > https://gaborbernat.github.io/pep-660-662-2022 (recording should also be > available in a few weeks). > > > On Mon, May 9, 2022 at 9:45 AM Mikhail Golubev < > mikhail.golubev(a)jetbrains.com> wrote: > >> Hi everyone! Sorry for joining the conversation *that* late. >> >> In short, PyCharm does support the current means of editable installs >> (.pth files) and does it very similar to what Eric described for Pyright. >> Namely, we have an interpreter "introspection" phase when we dynamically >> evaluate the content of sys.path with a helper script, as well as generate >> stubs for binary modules, retrieve the list of installed packages, etc. >> Because extra roots registered with .pth files automatically become visible >> in sys.path, we recognize them without any special handling. Although I'm >> not a huge fan of the idea, it is possible to add an extra step in this >> phase for checking either some physical file with metadata, such >> as STATIC_EXTRA.json, suggested by Bernat, or even another runtime value in >> addition to sys.path. >> >> Two questions, though. >> >> When should this process be repeated? For .pth files, we rely on regular >> filesystem events from under site-packages. Am I right that >> STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" >> directory, so we can expect that if it ever changes it will trigger another >> filesystem event in a known location? >> >> who prefer to put the library into a src folder and use editable >>> installations to manifest the content of the src folder to the import >>> system (either via pth files, symlinks or import hooks). >> >> >> This is a bit confusing to me. Aren't editable installs, as they are, a >> mechanism too complicated just for exposing the content of "src" directory >> in sys.path? I believe that existing tooling offers plenty of options for >> altering the content of sys.path before running an executable script or >> tests, e.g. in PyCharm we automatically add the directories marked as >> "source roots" in PYTHONPATH before launching project run configurations, >> in tox.ini it's possible to prepend PYTHONPATH with an extra directory >> before launching tests, etc. There must be something else editable installs >> are necessary for and what "flat" projects, without the "src" subfolder, >> cannot achieve automatically. Is it extra steps in a package's setup.py, or >> working with multi-package projects where a few libraries are being >> developed simultaneously and share the same virtual environment? Could >> someone give me more context, please? >> >> On Sun, May 8, 2022 at 4:42 AM Bernat Gabor <gaborjbernat(a)gmail.com> >> wrote: >> >>> If there are serious discussions about deprecating ".pth" files, have >>>> there been any proposals for a replacement mechanism? Bernat, do you think >>>> that the "STATIC_EXTRA.json" mechanism you have in mind could be that >>>> replacement? >>> >>> >>> No that's not my goal here. I think the primary reason people stay away >>> from import hooks is that IDEs don't support them. I know many people in my >>> work team spoke up against adopting PEP-660 when they found out it would >>> lose auto-imports. Very-very few people take the effort to work on >>> improving the existing systems. The majority just work around the problem >>> by using the old solutions (e.g. PTH) file with all their problems. Every >>> so often they ran into an issue caused by it, but then just blame wasting a >>> few hours for not having better solutions available. I think the same is >>> true with loading python modules from a database. Bigger companies might >>> have a proprietary custom plugin that works around the issue, and the rest >>> just live with what they have, they are well aware that if company Y wants >>> to load code from DB X pylance adding support for it is likely to close to >>> zero. After all, it's a custom DB and custom checker. >>> >>> Finally, for me personally as a packaging tools maintainer, it would be >>> very important that what we come up here does not impact the interpreter at >>> runtime (that should keep using the dynamic import hook). However, for ease >>> of development of libraries, I'd like libraries to have a mechanism to feed >>> their snapshot of dynamic code to the static checker. Otherwise extending >>> and writing a library would become even harder, and we're already >>> struggling with finding maintainers for projects. I'm OK with this better >>> solution taking a longer time than our current workaround ones (after all >>> until we get there we can keep using the current workarounds). >>> >>> Bernat >>> >>> On Sat, May 7, 2022 at 7:26 PM Bernat Gabor <gaborjbernat(a)gmail.com> >>> wrote: >>> >>>> Sadly, while the db/dynamic code stub files can be used to work around >>>> the problem; for files that live on the disk just not under the >>>> site-packages folder, they're less useful. >>>> >>>> What I mean that if I have a SRC folder on the disk and I'm using >>>> import hooks to expose them to the python interpreter as they would be >>>> installed into the purelib/platlib locations I cannot use the stub >>>> mechanism to expose these files to the static checkers. At best I could >>>> take all python files and copy them via a pyi files to a stub package. I'd >>>> like to get to a world where editable installations (even when implemented >>>> via import hooks) mostly work in IDEs. This will make significantly easier >>>> the life of these library developers. And while I agree users of the >>>> library are likely two order of magnitude bigger than maintainers, I think >>>> we can also agree that for a sustainable ecosystem the life of maintainers >>>> should also be easy as possible. Maintaining a library is already hard >>>> let's not make it harder by taking away the convenience of autocomplete in >>>> IDEs. >>>> >>>> On Sat, 7 May 2022, 19:02 Guido van Rossum, <guido(a)python.org> wrote: >>>> >>>>> On Sat, May 7, 2022 at 17:30 Bernat Gabor <gaborjbernat(a)gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> TBH, if all we're trying to solve is editable installs, I am assuming >>>>>>> that those would be used by a vanishingly small proportion of users >>>>>>> (otherwise we're doing something else wrong with package installs), so I >>>>>>> would personally be okay if an editable install wasn't visible to my IDE or >>>>>>> type checker, or if those tools were to fall back to typeshed or a PEP 561 >>>>>>> stub package. >>>>>> >>>>>> >>>>>> Generally, they are two types of people when developing libraries: >>>>>> >>>>>> - who put everything in the current working directory and use the >>>>>> fact that the current working directory is added to the sys.path >>>>>> automatically - these people do not need editable installations as the lib >>>>>> appears to be installed without requiring installation, >>>>>> - who prefer to put the library into a src folder and use >>>>>> editable installations to manifest the content of the src folder to the >>>>>> import system (either via pth files, symlinks or import hooks). >>>>>> >>>>>> In my experience, there's roughly a 60 to 40 divide in favour of the >>>>>> first. Especially more experienced/advanced users tend to prefer the src >>>>>> layout over the inline approach. Based on this I don't think it's accurate >>>>>> to say that it's used by a vanishingly small proportion of users. I >>>>>> personally use it for all my projects, and for why src layout can be better >>>>>> than the inline source see Paul Ganssle blog post here >>>>>> https://blog.ganssle.io/tag/tox.html explaining some of the problems >>>>>> (what's in your source directory is not necessarily what'll be put in your >>>>>> purelib/platlib during installation). >>>>>> >>>>> >>>>> >>>>> Ah, that’s not what I meant. I meant very few users are developing >>>>> packages. E.g. there are millions of pandas users but few of them are >>>>> modifying pandas. (I hope. :-) >>>>> >>>>> Also, I do not think that what we want to solve is editable >>>>>> installations per se. What I'd like to solve here is supporting import >>>>>> hooks for static checkers. Editable installations are just a good example >>>>>> of where import hooks can be used. >>>>>> >>>>> >>>>> Agreed. >>>>> >>>>> Sadly, while the db/dynamic code stub files can be used to work around >>>>>> the problem; for disks that live on the disk just not under the >>>>>> site-packages folder, they're less useful. >>>>>> >>>>> >>>>> Not sure I follow. >>>>> >>>>> —Guido >>>>> >>>>> All the best, >>>>>> >>>>>> Bernat >>>>>> >>>>>> On Sat, May 7, 2022 at 6:11 PM Guido van Rossum <guido(a)python.org> >>>>>> wrote: >>>>>> >>>>>>> On Sat, May 7, 2022 at 4:14 PM Eric Traut <eric(a)traut.com> wrote: >>>>>>> >>>>>>>> Most static analysis tools already support “.pth” files. Jia >>>>>>>> mentioned that pyre doesn’t yet, but it should be straightforward for them >>>>>>>> to add this. My recommendation is that we rally behind “.pth” files. I >>>>>>>> think this approach is more pragmatic than introducing yet another >>>>>>>> mechanism that needs to be implemented by all of the tools in the ecosystem. >>>>>>>> >>>>>>> >>>>>>> Hm. You're the first person I've met in a long time who regards PTH >>>>>>> files as something we should do more of -- IIUC they are almost universally >>>>>>> considered a necessary evil. Especially the fact that they can be used to >>>>>>> do other stuff than add entries to sys.path. But also that -- a long >>>>>>> sys.path tends to slow down module search, and it seems just wrong that >>>>>>> adding N packages would append N sys.path entries: if we wanted each >>>>>>> package to have its own entry we should have designed it as a mapping, not >>>>>>> a search path. >>>>>>> >>>>>>> TBH, if all we're trying to solve is editable installs, I am >>>>>>> assuming that those would be used by a vanishingly small proportion of >>>>>>> users (otherwise we're doing something else wrong with package installs), >>>>>>> so I would personally be okay if an editable install wasn't visible to my >>>>>>> IDE or type checker, or if those tools were to fall back to typeshed or a >>>>>>> PEP 561 stub package. >>>>>>> >>>>>>> Regarding your claim that almost nobody loads code from a database, >>>>>>> apparently that's actually very popular in the banking sector ( >>>>>>> https://calpaterson.com/bank-python.html) -- and those people have >>>>>>> a *lot* of Python code. >>>>>>> >>>>>>> I believe that VS Code is using (or going to use, or may use?) a >>>>>>> Virtual Filesystem to access user code. Would it really be so terrible if >>>>>>> there were some corner of the VFS namespace to map to the Python module >>>>>>> (spec) namespace? It could be served from a Python process that can give >>>>>>> you the source of a module given its full name (without executing the >>>>>>> module, of course -- the import loader machinery should be able to >>>>>>> guarantee that already). >>>>>>> >>>>>>> -- >>>>>>> --Guido van Rossum (python.org/~guido) >>>>>>> *Pronouns: he/him **(why is my pronoun here?)* >>>>>>> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…> >>>>>>> >>>>>> _______________________________________________ >>>>>>> Typing-sig mailing list -- typing-sig(a)python.org >>>>>>> To unsubscribe send an email to typing-sig-leave(a)python.org >>>>>>> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >>>>>>> Member address: gaborjbernat(a)gmail.com >>>>>>> >>>>>> -- >>>>> --Guido (mobile) >>>>> >>>> _______________________________________________ >>> Typing-sig mailing list -- typing-sig(a)python.org >>> To unsubscribe send an email to typing-sig-leave(a)python.org >>> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >>> Member address: mikhail.golubev(a)jetbrains.com >>> >> >> >> -- >> Mikhail Golubev >> Software Developer >> JetBrains >> http://www.jetbrains.com >> The drive to develop >> > -- Mikhail Golubev Software Developer JetBrains http://www.jetbrains.com The drive to develop

8 7

Explicit Type Aliases
by Shannon Zhu 17 Aug '22

17 Aug '22

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

PEP 692: Using TypedDict for more precise **kwargs typing
by Franek Magiera 16 Aug '22

16 Aug '22

Adding support for more precise **kwargs (consisting of different types) typing using TypedDict has been a very desired feature in python community [1]. The feature proposal has been discussed both on the mailing list and at a typing meetup and received positive feedback. I've submitted PEP 692 that aims to add support for more precise **kwargs typing. The PR is available at https://github.com/python/peps/pull/2620 And the preview at https://pep-previews--2620.org.readthedocs.build/pep-0692/ - **kwargs can consist of different types - A new syntax `def foo(**kwargs: **Movie): ...` that enables the feature is introduced Please let me know what you think about it. Franek [1] https://github.com/python/mypy/issues/4441

10 16

Support for import hooks for static, type checkers and IDEs
by Bernat Gabor 16 Aug '22

16 Aug '22

As some of you might be aware the python packaging ecosystem has adopted a standardized solution of editable installations in form of https://peps.python.org/pep-0660. Historically editable installations worked via `pth` files that inject the projects working directory onto the sys.path. This solution, while it mostly works, is fairly inflexible when it comes to more advanced functionality (such as excluding files from some modules, loading the code from in-memory or the database). A more powerful solution from the runtimes point of view is to use import hooks to dynamically allow/exclude module imports. The problem with this solution today is that static (and type) checkers are not capable of knowing/indexing the project layout, and as such most IDEs e.g. underline the imports with a red line as cannot be resolved from their POV. A possible suggestion was to allow the py.typed (or another file) to contain a rough layout of the project, e.g. a list of {module x is importable from file at path y}. This file then could be loaded and indexed by both type and static checkers, and periodically be refreshed by tooling for discovering new files/modules. See discussion at https://discuss.python.org/t/pep-660-support-and-ides/13878/13. Opinions or better options to tackle this problem? Thanks, Bernat

8 22

PyCon 2022 Typing Summit Recap
by Shannon Zhu 10 Jul '22

10 Jul '22

Hello everyone, I’m excited to share some resources from the 2022 Typing Summit! The summit was held last week, on 4/28 at PyCon in Salt Lake City. It was in-person this year without support for live streaming, so we know a lot of you couldn’t fly out but are still interested in catching up with the topics discussed. We took some best-effort video and notes for each of the presentations and discussions, and some of the speakers have also uploaded their slides and speaker notes to share. You can find links to video and slides per topic (where available), as well as paraphrased audience-member notes and action-items here: https://docs.google.com/document/d/1_CBM9SeBrFTFULtwAzGmxlBRhqzpIArQ65cWa2W… Please thank Matthew Rahtz for personally taking and cutting video of every topic. They are linked individually in the doc above, and you can find them all in one place here: https://drive.google.com/drive/folders/1IVV0URv7_CCxiOvY6GC6adB72d5LKcvI<https://drive.google.com/drive/folders/1IVV0URv7_CCxiOvY6GC6adB72d5LKcvI> We will be uploading a full-length video of the typing summit to PyCon’s YouTube channel in the coming days. Note: Unfortunately due to battery life and other technical limitations, the video slides are hard to read, and we missed the last bit of the panel discussion. If anyone else at the summit had taken notes or wanted to share takeaways or further questions, please feel free to respond to this thread. 😊 Thanks again to all of the presenters for the work you’ve put in, and thanks to everyone who made it out in person to listen and discuss! It’s exciting to see so many typing related features already going into 3.11 and all the potential for future changes to improve the typing experience in Python. Best, Shannon & Alfonso

6 5

Almost accepting PEP 681 – Data Class Transforms
by Petr Viktorin 06 Jun '22

06 Jun '22

Hello, As an initial implementation that will be improved in the future, the specification in PEP 681 is fine. Feel free to add the decorator to Python 3.11 at your convenience. However, the PEP includes several worrying recommendations like: - we recommend that the maintainers of attrs move away from the legacy semantics and adopt auto_attribs behaviors by default. - We chose not to support this feature and recommend that attrs users avoid converters. - Attrs users should use the dataclass-standard eq and order parameter names instead. These are probably meant as recommendations from typing-sig, but an accepted PEP represents consensus of the entire Python community. A typing PEP is not an appropriate place to make recommendations like this, especially without reaching out to the maintainer of attrs. As far as I know,the attrs and pydantic libraries are using the reference implementation, but their authors weren't consulted on the PEP itself. Could you either change the wording (e.g. say that the unsupported features need bespoke type-checker functionality for proper type checking), or work with attrs to make the same recommendations in its documentation? Happy typing, — Petr, on behalf of the Steering Council

9 11

Abstract attributes for Mixins
by Thomas Kehrenberg 02 Jun '22

02 Jun '22

When working with mixins, I often wish I could specify abstract attributes – to have the contract “if the class provides these attributes, then this mixin provides this functionality.” Currently, it’s possible to write the following: class ProvidesLength(ABC): # my mixin data: list[float] def __len__(self) -> int: return len(self.data) @dataclass class A(ProvidesLength): data: list[float] = field(default_factory=list) class B(ProvidesLength): def __init__(self, data: list[float] | None = None): self.data = [] if data is None else data class C(ProvidesLength): @property def data(self) -> list[float]: # incompatible with supertype return [0.0, 1.0, 2.0] However, type checkers will not warn me if the concrete class doesn’t specify the `data` attribute in the above, because type checkers treat the type declaration `data: list[float]` like a variable definition. Ideally, there’d be some way to say that an attribute is abstract, like we can do with methods. It might look like this: from typing import ABSTRACT class ProvidesLength(ABC): data: list[float] = ABSTRACT def __len__(self) -> int: return len(self.data) but I’m not attached to any syntax. I know that I can use abstract properties to get something similar: class LengthMixin(ABC): @property @abstractmethod def data(self) -> list[float]: ... def __len__(self) -> int: return len(self.data) @dataclass class A(LengthMixin): data_: list[float] = field(default_factory=list) @property def data(self) -> list[float]: return self.data_ class B(LengthMixin): def __init__(self, data: list[float] | None = None): self._data = [] if data is None else data @property def data(self) -> list[float]: return self._data But this is just very verbose and doesn’t play that well with dataclasses (I can’t use a dataclass field to implement the abstract property). Another thing that actually works with pyright is to use a Protocol instead of an ABC: class ProvidesLength(Protocol): data: list[float] def __len__(self) -> int: return len(self.data) class A(ProvidesLength): # pyright complains about missing `data` data2: list[float] But Protocols do not feel like the appropriate concept to use for mixins. Besides, it doesn’t work in other type checkers I’ve tried (mypy and pycharm). Other possible syntaxes for abstract attributes: class Mixin(ABC): a: Abstract[list[float]] # wrapping in Abstract[] b: list[float] = ... # ellipses as value Best, Thomas

3 2

Generic type alias documentation?
by Paul Bryan 01 Jun '22

01 Jun '22

My Google-fu skills may be waning. I can't seem to find documentation for the new style generic type aliases. Example: tuple[int, ...] I can find the older-style typing.Tuple documentation. Are the new generic type aliases documented somewhere in the Python docs site?

4 5

typing.override decorator
by Shannon Zhu 31 May '22

31 May '22

Hi everyone! Pyre has received several user requests over the last few months to support @override functionality in the Python type system. I’ve compiled some of details here to see what you all think (I hope the formatting works over the mailing list): Motivation: One core functionality of the type checker is to flag when refactors or changes break pre-existing semantic structures in the code, so users can identify and make fixes across their project without doing a manual audit of their code. Unlike most other language type systems, Python does not provide a way to identify call sites that need to be changed to stay consistent when an overridden function API changes. This makes refactoring code dangerous and increases the cost of re-architectures and codemods. Take for example, a simple inheritance structure: class Base: def foo(self, input: int) -> int: return input class Child(Base): def foo(self, input: int) -> int: return input def callsite(input: Child) -> None: input.foo(1) If an overridden method on the superclass is renamed or deleted, the type checker will only identify and flag call sites that deal with objects of the base class directly, but still happily accept any call sites dealing with subclasses. class Base: def new_foo(self, input: int, flag: bool) -> int: # new semantics / side effects return input class Child(Base): # this method now silently does something fundamentally different - defines a new function unrelated to what it used to override def foo(self, input: int) -> int: return input def callsite(input: Child) -> None: input.foo(1) # no indication this API changed While this will run when executed, the type checker omits to mention the fact that a core piece of semantics have changed: the subclass method that was formerly overriding a superclass method is now, silently, defining a brand new method on the class and fooling call-sites into thinking nothing has changed with the old API. Explicitly marking a method as intending to override something in the parent would help us flag these problems in a complicated inheritance system. We’ve seen runtime bug reports usually caused by an engineer renaming a parent class method and forgetting to rename that method in all child classes. This blows up at runtime and causes some user frustration because this pattern is caught in type systems of many other languages (see below). Syntax & Semantics: A method decorated with @typing.override will throw a type error if it is not actively overriding a compatible method in a (non-Protocol) parent class. from typing import override class Base: def foo(self) -> int: return 1 def bar(self, x: str) -> str: return x class Child(Base): @override def foo(self) -> int: return 2 @override def bar(self, y: int) -> int: # type error: signature mismatch return y @override def baz() -> int: # type error: no overridden method return 1 Note: Method decorators @staticmethod and @classmethod should always be applied last - in other words, @override should wrap the method more closely (should only matter for runtime). from typing import override class Base: @staticmethod def foo() -> int: return 1 @classmethod def bar(cls, x: str) -> str: return x class Child(Base): @staticmethod @override def foo() -> int: return 2 @override def bar(self, x: str) -> str: # type error: signature mismatch return x Adoption: The proposed syntax and semantics above imply that the use of the @override decorator will be purely opt-in. Alternatively, type checkers may want to offer an option for users to flag methods that are actively overriding a method in the parent class but are missing the decorator. There are a couple ways this may be possible: 1. Class Inheritance: Denote a class as “enforcing explicit overrides” (syntax suggestions welcome) and error on any subclass methods that override parent methods but are not decorated with @override. This is what the PyPi runtime library (see below) provides via inheritance of a helper EnforceOverrides class. It would affect downstream dependencies that may subclass your logic but not be type checked together, and may be confusing 1. Everywhere: Make explicit annotation of @override a core expectation of well-annotated code, enforcing its usage in strictly typed modules across the board. This is probably too invasive. Precedent in other languages: A non-exhaustive list of other languages that provide the same feature: * Hack has <<__Override>><https://docs.hhvm.com/hack/attributes/predefined-attributes#__override> * Java has @Override<https://docs.oracle.com/javase/tutorial/java/IandI/override.html> * Scala has override<https://www.javatpoint.com/scala-method-overriding> * Kotlin has override<https://kotlinlang.org/docs/inheritance.html#overriding-methods> * Swift has override<https://docs.swift.org/swift-book/LanguageGuide/Inheritance.html#ID198> * C++ has override<https://en.cppreference.com/w/cpp/language/override> * JS/Flow discourage classes (in favor of functions) Comparison to third-party runtime enforcement library: There already exists a library<https://pypi.org/project/overrides/> that enforces @overrides (sic) and @final at runtime. PEP 591<https://peps.python.org/pep-0591/> added a final decorator to the typing module. The override component of the runtime library is not supported statically at all, which has added some confusion around the mix/matched support where the runtime library is imported and used. The ways in which the current static typing.override proposal differs from the runtime library are: 1. Proposed spelling is override rather than overrides 2. May not enforce an option where methods in classes inheriting from EnforceOverrides that are missing this decorator will throw type errors* *The runtime library provides two motivation examples for EnforceOverrides unrelated to feature adoption that I do not find motivating for the type checking use case: 1. “A method that is added to a superclass is shadowed by an existing method with the same name in a subclass.” The type checker will throw errors already here, unless the name, all parameters types and names (and arity), and return type are all perfectly compatible. That seems extremely unlikely unless the methods are in fact intended to do the same thing, which suggests this is not flagging a real error. 1. “A method of a superclass that is overridden by a subclass but the signature of the overridden method is incompatible with that of the inherited one.” Similarly, the type checker will already throw incompatible override errors to flag this. Looking for feedback on whether this would make a good addition to the typing module, as well as thoughts on some of the open questions regarding an “EnforceOverrides” equivalent or any other concerns. Thanks! Shannon

8 8

Opened and changed typing issues week 2022-W21
by Typing Bot 30 May '22

30 May '22

No issues or pull requests with the label 'topic: feature' were opened or updated last week in the typing repository on GitHub. All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Unbounded TypeVar on variable's type info
by Nicolas Haller 28 May '22

28 May '22

Hello, I have a use case that I'm not sure if this is something the current typing system could handle (and if it should). Basically, I'd like to use generics on a variable's type info to constrain the variable a bit more. My prime example would be to constrain a dict value based on the dict key, something like that: U = TypeVar("U", bound=Item) _CHECKS_TO_APPLY: dict[Type[U], list[Type[Check[U]]]] In that example, _CHECKS_TO_APPLY stores what kind of "Check" is applicable on a type "Item". The "Check" type is a hierarchy of specialized generic classes taking the type "Item" they can check as a generic parameter. I put a longer example there: https://gist.github.com/Gaasmann/beeebf526c163280cff796b842ee800a Right now, I'm not sure if what I'm doing is valid. In any case, mypy isn't happy: generic_typing.py:61: error: Type variable "generic_typing.U" is unbound generic_typing.py:61: note: (Hint: Use "Generic[U]" or "Protocol[U]" base class to bind "U" inside a class) generic_typing.py:61: note: (Hint: Use "U" in function signature to bind "U" inside a function) Do you think this use case makes sense, and if this is something the system should be able to handle? Thanks, -- Nicolas

2 2

Opened and changed typing issues week 2022-W20
by Typing Bot 23 May '22

23 May '22

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were updated last week: #182 Define a JSON type opened by @brettcannon https://github.com/python/typing/issues/182 #548 Higher-Kinded TypeVars opened by @tek https://github.com/python/typing/issues/548 #1183 Annotate sync/async code via Generic | generic TypeVar / Type aliases in Generic opened by @Bobronium https://github.com/python/typing/issues/1183 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

typing_extensions is now a separate repo
by Jelle Zijlstra 19 May '22

19 May '22

The typing_extensions module is now developed in its own repo, https://github.com/python/typing_extensions. Previously, it was part of https://github.com/python/typing, but that repo is used for several different purposes and typing_extensions stood out as unlike the others. Please watch the new repo if you're interested in typing_extensions development. The typing repo still hosts the https://github.com/python/typing/discussions forum, the https://typing.readthedocs.io/en/latest/, and issues proposing bigger changes to the type system.

1 0

Stubs: Supporting __all__ += [...]
by Sebastian Rittau 17 May '22

17 May '22

It would be useful if type checkers would support the `+=` operator for `__all__`, at least in stubs. For example, have a look at this abomination from typing.pyi: if sys.version_info >= (3, 11): __all__ = [ "Annotated", # snipped about 60 items "TypeGuard", ] elif sys.version_info >= (3, 10): __all__ = [ "Annotated", # snipped about 60 items again "TypeGuard", ] elif sys.version_info >= (3, 9): __all__ = [ # mostly the same items again ] elif sys.version_info >= (3, 8): __all__ = [ # mostly the same items again ] elif sys.version_info >= (3, 7): __all__ = [ # mostly the same items again ] else: __all__ = [ # mostly the same items again ] While this is an extreme case, adding items to modules is of course a common occurrence, and the current duplication is hard to read and error-prone. This would much better be written as: __all__ = [ # lots of items ] if sys.version_info >= (3, 7): __all__ += [SomeItem, AnotherItem] # etc. for all other versions This makes is immediately clear which items were added in which Python version. This leads to a few questions: * Do type checkers maybe already support this? * Would this be hard to implement? * How could removals be handled? - Sebastian

3 2

Re: Stubs: Supporting __all__ += [...]
by Joren Hammudoglu 17 May '22

17 May '22

Yes I agree. Another use-case of this is this little toy project of mine: https://github.com/jorenham/exports. Its @exports decorator simply adds the decorated function or class to __all__. But as it currently stands, type checkers do not pick up on this.

2 1

Inheriting from Union and Literal
by j.scholbach＠posteo.de 17 May '22

17 May '22

Currently, it does not seem to be possible to inherit from Union or Literal. Would it make sense to provide this functionality? Use case: I would like to narrow down a type. I want to use the typing system as a safeguard that the restricted child type is actually smaller than the parent type. Here is an example: ```Python Currency: Literal['CHF', 'EUR', 'USD',] EuropeanCurrency: Literal['CHF', 'EUR'] FooCurrency: Literal['FOO', 'EUR'] # Should result in an error, as 'FOO' is not a Currency. ``` Has this been discussed before? --Jonathan

3 3

Opened and changed typing issues week 2022-W19
by Typing Bot 16 May '22

16 May '22

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were opened last week: #1183 Annotate sync/async code via Generic | generic TypeVar / Type aliases in Generic opened by @Bobronium https://github.com/python/typing/issues/1183 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Discussion about a CST parser for 3.10+ (and a separate error-recovering parser)
by S Pradeep Kumar 10 May '22

10 May '22

TL;DR: A bunch of us met informally at PyCon to discuss adding a common CST parsing library for 3.10+. We also talked about an error-recovering parser. Pablo gave us a lot of feedback and advice about a path forward. I'm sharing the meeting notes for posterity. *Attendees*: Jelle Zijlstra, Zsolt Dollenstein (libcst, etc.), Kevin Millikin (DeepMind), Matthew Rahtz (DeepMind), Shantanu Jain (mypy), Pyre (Shannon, Jia, Pradeep), Pablo Galindo Salgado (Steering Council), and any others I didn’t note down! *Notes* (May 01, 2022): - Requirements - All attendees cared about having the tool support 3.10+. - CST parsing was important for Libcst, Black, linter frameworks such as Fixit, and DeepMind. - Error recovery was important for Pyre (so it can show results in the IDE as the user types code) and for Fixit. - DeepMind: we have a custom language server that uses a parser generated by Textmapper <https://textmapper.org/>, implemented in Go. Conceivably, we should be able to get Textmapper to generate Python code as well. But we'd also like a Python parser that supported (batch) static analysis and (batch) source-to-source transformations. Their existing solution doesn’t support 3.10. - Problems mentioned by Pablo - Difficulties with PEG: Soft keywords are the most challenging part for table-based parsers. Even with an LL(k) parser, some constructs need more than k=2 lookup. - Questions for Pablo: What does it take to add a CST parser? - Extending CPython’s PEG to CST is possible, but not free. - Need to balance features and speed. - CST is more expensive than AST: The tree is bigger and needs more memory. The tokenizer needs to chunk more. The parser needs to match more. - What would it take to maintain a CST parser in the standard library? - This is quite tricky. Many people will resist it. - Maintaining it in the standard library is challenging. However, this is more doable than any other change. - Need to create another version of the parser. pegen has a Python backend. Doing this in C will be extremely costly. The Python version will be easier. - Cython could be 2x the speed. C parser would be ~100x faster. - Error-prone to change the C parser: There was a recent change that needed to throw a syntax error at the start of an unmatched paren. There were ~63 bug fixes after implementing the change. - Pablo: What about Parso? - Zsolt: The Parso developer is apparently either not going to make it work on 3.10 or is going to make it closed-source. - Pablo: Issues in building an error-recovering PEG parser - PEG is tricky. It doesn’t have a concept of error. If one rule fails, some other may succeed. - Python was built on an LL(1) parser, so the deepest-rule heuristic works well. - One interesting technique for PEG error recovery is to add one of the tokens that you expected and reparse to see if it advances. This can be extremely fast. The challenge is to mutate the token without messing with the caching. - If you use technology other than PEG - you will need to find a way to support newer features, need to handle soft keywords, need and the equivalent of infinite backtracking and infinite lookahead. - Pablo: If you build a PEG CST parser (not in Rust), I can champion adding it to the standard library. - Pablo: For Black, could adapt pegen to produce a lib2to3-compatible CST. Would be slow. - Zsolt: Would rather rewrite on top of LibCST. - Takeaways - Zsolt: I’m committing to adding an error-recovering CST parser that supports PEG. It won’t be in pure Python, but we could create a wheel for it. - Kevin: Deepmind language server is currently a non-PEG parser supporting error recoverability; unclear how to migrate. We're a language server that needs to be in Go. Would like to do more source-to-source transformation, static analysis. Don't care about error recovery. - General: Look into Pyright’s error-recovering parser to understand its strengths and limitations. - General: Also look into tree-sitter-python <https://github.com/tree-sitter/tree-sitter-python>, which seems to already support 3.10’s match statement. Calls to action: 1. If you are interested in a common CST parser, please reply on this thread so that the other interested folks are aware of you. 2. If you are interested in an error-recovering parser, please reply below. e.g., you may want to show errors or auto-complete data as people are typing code in the IDE. 3. If you already use an error-recovering parser that supports 3.10+ features, please share your solution below. (I believe Pyright rolls its own parser in TypeScript. I’m curious about other Python IDE-like tools.) -- S Pradeep Kumar

3 2

Opened and changed typing issues week 2022-W18
by Typing Bot 09 May '22

09 May '22

No issues or pull requests with the label 'topic: feature' were opened or updated last week in the typing repository on GitHub. All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0

Generic NamedTuples
by Jelle Zijlstra 07 May '22

07 May '22

At the moment, this will throw a runtime error: from typing import Generic, NamedTuple, TypeVar T = TypeVar("T") class NT(NamedTuple, Generic[T]): foo: int bar: T There is an open CPython PR to add runtime support for this pattern: https://github.com/python/cpython/pull/92027. I'm inclined to accept this change, because it seems like a natural extension and would work well with the rest of the type system. Mypy doesn't currently support this for internal reasons, but I'm told Pyre already supports it. Let me know if there's some problem I missed with this idea. We just discussed this at the typing summit at PyCon, and one concern was brought up about variance. I think variance should work the same as with non-NamedTuple generic classes and no special treatment is required. Relatedly, there is another open PR that allows generalized multiple inheritance on NamedTuple (https://github.com/python/cpython/pull/31781). This would allow code like this: class A: @property def prop(self) -> int: ... class B(NamedTuple, A): foo: int assert_type(B(1).prop, int) I think this is too much to ask static type checkers to support: regular classes are too different from namedtuples. But I'd be happy to hear other opinions, especially from type checker maintainers.

6 11

Opened and changed typing issues week 2022-W17
by Typing Bot 02 May '22

02 May '22

The following is an overview of all issues and pull requests in the typing repository on GitHub with the label 'topic: feature' that were opened or updated last week, excluding closed issues. --------------------------------------------------- The following issues and pull requests were updated last week: #593 Buffer protocol types opened by @srittau https://github.com/python/typing/issues/593 --------------------------------------------------- All issues and pull requests with the label 'topic: feature' can be viewed under the following URL: https://github.com/python/typing/issues?q=label%3A%22topic%3A+feature%22

1 0