Mailman 3 September 2022 - Typing-sig

Support Algebraic Data Types with @sealed
by John Hagen 20 Mar '23

20 Mar '23

The original (superseded) PEP 622 proposed adding a @sealed decorator so that users could define algebraic data types (ADTs) that static type checkers could verify are `match`ed exhaustively: - https://peps.python.org/pep-0622/#sealed-classes-as-algebraic-data-types There is a somewhat fairly-upvoted question about ADTs in Python, indicating interest in this area: - https://stackoverflow.com/questions/16258553 @sealed would be very valuable towards making ADT usage within Python more type-safe. Is there any historical discussion about why @sealed was dropped from the final pattern matching PEP? What would be the next steps needed to push this forward? Would writing a new PEP be the next step?

15 32

Extending published stubs
by Irv Lustig 20 Mar '23

20 Mar '23

I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `(a)pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register…). So for a `DataFrame`, if you do `(a)pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated. Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it. Has this idea been discussed anywhere?

5 10

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 23 Dec '22

23 Dec '22

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

16 34

Re: Typing Meetup
by S Pradeep Kumar 15 Nov '22

15 Nov '22

Updating the Typing Meetup date to Wednesday as there were some conflicts. Let me know if there are any major time conflicts. Date: *Wednesday, November 03, 2021.*Time: 10 am San Francisco time (UTC-7) / 5 pm London time (UTC) Duration: 1 hour Potential discussion topics: + Typing **kwargs using a TypedDict (Franek Magiera?, Pradeep) [1] + Remaining questions about Callable syntax (Steven) + Non-required Protocol fields (Sebastian Rittau?) [2] + Type support for sealed classes (Adrian Freund?) [3] + Safe, typed JSON validation + Type system documentation (Shannon?) If you're working on these or any other topics and would like to start a discussion at the meetup, reach out to me. If you have interesting topics you'd like to see discussed, share them too. Once we have topics, we can always start a discussion even if there's no proposer yet. We will probably have time for 3-4 topics in this session. On Fri, Oct 15, 2021 at 12:55 PM S Pradeep Kumar <gohanpra(a)gmail.com> wrote: >>> >>>> Hello all, >>>> >>>> We plan to hold a virtual Typing Meetup to discuss key typing-related >>>> issues in real-time. The idea is that we can make progress on important or >>>> controversial discussions, get early feedback on in-progress PEPs, and >>>> gauge interest in new proposals. >>>> >>>> A virtual meetup will be more accessible to people in other timezones >>>> than Bay Area in-person meetups. It should also help consolidate >>>> discussions that are often spread out across mailing lists or Github repos. >>>> The tentative plan is to have a monthly meetup, but we can change that as >>>> needed. >>>> >>>> Date: Monday, November 01, 2021. >>>> Time: 10 am San Francisco time (UTC-7) / 5 pm London time (UTC) >>>> Duration: 1 hour >>>> >>>> Time and date are flexible. I'll share the zoom link and agenda the >>>> week before. I'll also record the meetup and share afterwards. >>>> >>>> If you're interested in presenting a new proposal or your work on some >>>> existing proposal, please respond here. We can shoot for 15 minutes per >>>> presentation. I know there are quite a few interesting threads on >>>> typing-sig that haven't seen much activity in a while :) >>>> -- >>>> S Pradeep Kumar >>>> _______________________________________________ >>>> Typing-sig mailing list -- typing-sig(a)python.org >>>> To unsubscribe send an email to typing-sig-leave(a)python.org >>>> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >>>> Member address: guido(a)python.org >>> >>> Links: [1] Typing **kwargs using TypedDict: https://mail.python.org/archives/list/typing-sig@python.org/thread/IUCXT2IQ… [2] Non-required Protocol fields: https://mail.python.org/archives/list/typing-sig@python.org/thread/P6FCGRJU… [3] Sealed classes: https://mail.python.org/archives/list/typing-sig@python.org/thread/QSCT2N4R… -- S Pradeep Kumar

8 51

Draft PEP for Improved Type Parameter Syntax
by Eric Traut 08 Nov '22

08 Nov '22

In last month's typing meetup, Sebastian presented some options for improving the syntax for type parameters. I subsequently posted some slides that explored a bunch of options. For reference, here's a link to those slides: https://docs.google.com/presentation/d/1aVHTaj8zGYAvM27uft1ktdthwEKO4KH4qaR…. In the interest of continuing to make progress on this topic, I've drafted a PEP that places some stakes in the ground. Here's a link to the PEP: https://github.com/erictraut/peps/blob/typeparams/pep-9999.rst. I'm interested in feedback. Since there appears to be broad interest in addressing this problem, perhaps we can use the next typing meetup to continue this discussion. -- Eric Traut Contributor to Pyright & Pylance Microsoft

16 51

Defaults for TypeVar-likes
by James H-B 03 Nov '22

03 Nov '22

I've been toying with the idea of writting a PEP for this long requested feature (https://github.com/python/typing/issues/307). I've got a preliminary draft of it with the motivation and a specification for the feature. In short features like ```py T = TypeVar("T", default=int) # This means that if no type is specified T = int @dataclass class Box(Generic[T]): value: T | None = None reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] ``` would be supported as would ```py YieldT = TypeVar("YieldT") SendT = TypeVar("SendT", default=None) ReturnT = TypeVar("ReturnT", default=None) class Generator(Generic[YieldT, SendT, ReturnT]): ... Generator[int] == Generator[int, None] == Generator[int, None, None] ``` Feedback and any questions on this would be very appreciated. https://gist.github.com/Gobot1234/8c9bfe8eb88f5ad42bf69b6f118033a7 Thanks, James H-B.

8 20

Readonly TypedDicts
by Alice Purcell 02 Nov '22

02 Nov '22

I would like to propose adding a boolean `readonly` parameter to TypedDict, which would remove all mutate operations from the resulting type. # Specification ```python class Request(TypedDict, total=False, readonly=True): some_field: string | None request: Request = { "some_field": "foo" } request["some_field"] = None # error del request["some_field"] # error ``` I suggest making it illegal to have a subclass that does not have the same readonly value as its superclass(es), as on the one hand, it would be natural to expect the following to be equivalent: ```python # Declare some_field explicitly class MutableRequest(TypedDict, total=False): some_field: string | None # Inherit some_field from Request class MutableRequest(Request, readonly=False): pass ```python But on the other hand, by analogy with total we would expect only the fields explicitly declared on the class to be affected by the readonly flag. I'm not aware of a usecase for the latter, and the ambiguity between the two possible interpretations would probably be a source of issues. # Use case We are making increasing use of TypedDict to type hint code that uses dictionaries to hold requests to and responses from services (both client and server side). However, we run into problems whenever any constraint is loosened, e.g. passing data from a response where a field is guaranteed to a request where it is optional: ```python class Response(TypedDict): some_field: string class Request(TypedDict, total=False): some_field: string | None class Server1: def fetch_data() -> Response: ... class Server2: def accept_data(request: Request): ... def my_code(server1: Server1, server2: Server2): data: Response = server1.fetch_data() server2.accept_data(data) # error: Response and Request are incompatible ```python This last line is not type-safe, as there are mutation operations on Request that aren't valid on Response. If we could remove those mutation operations from the Request type, then it would be a valid subtype of Response. This problem seems quite common, e.g. https://github.com/python/mypy/pull/12142. I am thinking of making a PEP (Pablo Galindo has agreed to sponsor). What does everyone think? Is the proposal sound and implementable? --- Alice Bloomberg

4 5

Re: Support for import hooks for static, type checkers and IDEs
by Mikhail Golubev 05 Oct '22

05 Oct '22

Aha, thanks for your reply, Bernat! It seems like it hasn't been sent to typing-sig, though. Hopefully, the automatic quotation will keep everyone on the same page. Your presentation on the topic is just what I needed. (and the terriers are adorable!) > Can you detail how you achieve this? As far as I'm aware pylance doesn't do this, am I right Eric? Effectively, we scan through names in sys.builtin_module_names and binary modules found in sys.path, import them and try reconstructing a stub definition for each of them using the introspection facilities of the language. It's the second kind of stubs we operate with internally, in addition to .pyi stubs. These don't have type annotations, but e.g. preserve docstrings of the definitions. We also use them for navigation e.g. when one invokes "Go to definition" on a built-in function, such as "print". In theory, the same mechanism could be reused for modules that become available dynamically through import hooks, if it's what you're implying. On Mon, May 9, 2022 at 9:26 PM Bernat Gabor <gaborjbernat(a)gmail.com> wrote: > generate stubs for binary modules >> > > Can you detail how you achieve this? As far as I'm aware pylance doesn't > do this, am I right Eric? > >> >> When should this process be repeated? For .pth files, we rely on regular >> filesystem events from under site-packages. Am I right that >> STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" >> directory, so we can expect that if it ever changes it will trigger another >> filesystem event in a known location? > > > Yes, in my vision, the file would be part of the dist-info, and as such > trackable as any other file for changes. > > Aren't editable installs, as they are, a mechanism too complicated just >> for exposing the content of "src" directory in sys.path? >> > > Exposing the src directory is what we historically did, and gets you 80% > there, but is not actually a true representation of what the user would > want in the other 20%. Editable installations at their core are a method to > develop a library without needing to reinstall it after every change. In > the trivial case, yes it can be achieved by just injecting the src folder > onto the sys.path via some mechanism (PYTHONPATH, cwd change, PTH file, > sitecustomize.py, etc). However, there are more complicated cases when this > approach will not work: > > - selective inclusion of what gets packaged (only package these > business logic files) > - selective exclusion of what gets packaged (don't package test files) > - any kind of code generation during the package build (either > C-extension or generate python code from a JSON schema) > > The goal we want to achieve in the new editable installations is to allow > supporting these additional use cases, even if with some limitations (e.g. > updates to files get picked up automatically but adding/removing new files > would still require a reinstallation - through tooling could hide this > away). My presentation from PyCon covered some of these, see > https://gaborbernat.github.io/pep-660-662-2022 (recording should also be > available in a few weeks). > > > On Mon, May 9, 2022 at 9:45 AM Mikhail Golubev < > mikhail.golubev(a)jetbrains.com> wrote: > >> Hi everyone! Sorry for joining the conversation *that* late. >> >> In short, PyCharm does support the current means of editable installs >> (.pth files) and does it very similar to what Eric described for Pyright. >> Namely, we have an interpreter "introspection" phase when we dynamically >> evaluate the content of sys.path with a helper script, as well as generate >> stubs for binary modules, retrieve the list of installed packages, etc. >> Because extra roots registered with .pth files automatically become visible >> in sys.path, we recognize them without any special handling. Although I'm >> not a huge fan of the idea, it is possible to add an extra step in this >> phase for checking either some physical file with metadata, such >> as STATIC_EXTRA.json, suggested by Bernat, or even another runtime value in >> addition to sys.path. >> >> Two questions, though. >> >> When should this process be repeated? For .pth files, we rely on regular >> filesystem events from under site-packages. Am I right that >> STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" >> directory, so we can expect that if it ever changes it will trigger another >> filesystem event in a known location? >> >> who prefer to put the library into a src folder and use editable >>> installations to manifest the content of the src folder to the import >>> system (either via pth files, symlinks or import hooks). >> >> >> This is a bit confusing to me. Aren't editable installs, as they are, a >> mechanism too complicated just for exposing the content of "src" directory >> in sys.path? I believe that existing tooling offers plenty of options for >> altering the content of sys.path before running an executable script or >> tests, e.g. in PyCharm we automatically add the directories marked as >> "source roots" in PYTHONPATH before launching project run configurations, >> in tox.ini it's possible to prepend PYTHONPATH with an extra directory >> before launching tests, etc. There must be something else editable installs >> are necessary for and what "flat" projects, without the "src" subfolder, >> cannot achieve automatically. Is it extra steps in a package's setup.py, or >> working with multi-package projects where a few libraries are being >> developed simultaneously and share the same virtual environment? Could >> someone give me more context, please? >> >> On Sun, May 8, 2022 at 4:42 AM Bernat Gabor <gaborjbernat(a)gmail.com> >> wrote: >> >>> If there are serious discussions about deprecating ".pth" files, have >>>> there been any proposals for a replacement mechanism? Bernat, do you think >>>> that the "STATIC_EXTRA.json" mechanism you have in mind could be that >>>> replacement? >>> >>> >>> No that's not my goal here. I think the primary reason people stay away >>> from import hooks is that IDEs don't support them. I know many people in my >>> work team spoke up against adopting PEP-660 when they found out it would >>> lose auto-imports. Very-very few people take the effort to work on >>> improving the existing systems. The majority just work around the problem >>> by using the old solutions (e.g. PTH) file with all their problems. Every >>> so often they ran into an issue caused by it, but then just blame wasting a >>> few hours for not having better solutions available. I think the same is >>> true with loading python modules from a database. Bigger companies might >>> have a proprietary custom plugin that works around the issue, and the rest >>> just live with what they have, they are well aware that if company Y wants >>> to load code from DB X pylance adding support for it is likely to close to >>> zero. After all, it's a custom DB and custom checker. >>> >>> Finally, for me personally as a packaging tools maintainer, it would be >>> very important that what we come up here does not impact the interpreter at >>> runtime (that should keep using the dynamic import hook). However, for ease >>> of development of libraries, I'd like libraries to have a mechanism to feed >>> their snapshot of dynamic code to the static checker. Otherwise extending >>> and writing a library would become even harder, and we're already >>> struggling with finding maintainers for projects. I'm OK with this better >>> solution taking a longer time than our current workaround ones (after all >>> until we get there we can keep using the current workarounds). >>> >>> Bernat >>> >>> On Sat, May 7, 2022 at 7:26 PM Bernat Gabor <gaborjbernat(a)gmail.com> >>> wrote: >>> >>>> Sadly, while the db/dynamic code stub files can be used to work around >>>> the problem; for files that live on the disk just not under the >>>> site-packages folder, they're less useful. >>>> >>>> What I mean that if I have a SRC folder on the disk and I'm using >>>> import hooks to expose them to the python interpreter as they would be >>>> installed into the purelib/platlib locations I cannot use the stub >>>> mechanism to expose these files to the static checkers. At best I could >>>> take all python files and copy them via a pyi files to a stub package. I'd >>>> like to get to a world where editable installations (even when implemented >>>> via import hooks) mostly work in IDEs. This will make significantly easier >>>> the life of these library developers. And while I agree users of the >>>> library are likely two order of magnitude bigger than maintainers, I think >>>> we can also agree that for a sustainable ecosystem the life of maintainers >>>> should also be easy as possible. Maintaining a library is already hard >>>> let's not make it harder by taking away the convenience of autocomplete in >>>> IDEs. >>>> >>>> On Sat, 7 May 2022, 19:02 Guido van Rossum, <guido(a)python.org> wrote: >>>> >>>>> On Sat, May 7, 2022 at 17:30 Bernat Gabor <gaborjbernat(a)gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> TBH, if all we're trying to solve is editable installs, I am assuming >>>>>>> that those would be used by a vanishingly small proportion of users >>>>>>> (otherwise we're doing something else wrong with package installs), so I >>>>>>> would personally be okay if an editable install wasn't visible to my IDE or >>>>>>> type checker, or if those tools were to fall back to typeshed or a PEP 561 >>>>>>> stub package. >>>>>> >>>>>> >>>>>> Generally, they are two types of people when developing libraries: >>>>>> >>>>>> - who put everything in the current working directory and use the >>>>>> fact that the current working directory is added to the sys.path >>>>>> automatically - these people do not need editable installations as the lib >>>>>> appears to be installed without requiring installation, >>>>>> - who prefer to put the library into a src folder and use >>>>>> editable installations to manifest the content of the src folder to the >>>>>> import system (either via pth files, symlinks or import hooks). >>>>>> >>>>>> In my experience, there's roughly a 60 to 40 divide in favour of the >>>>>> first. Especially more experienced/advanced users tend to prefer the src >>>>>> layout over the inline approach. Based on this I don't think it's accurate >>>>>> to say that it's used by a vanishingly small proportion of users. I >>>>>> personally use it for all my projects, and for why src layout can be better >>>>>> than the inline source see Paul Ganssle blog post here >>>>>> https://blog.ganssle.io/tag/tox.html explaining some of the problems >>>>>> (what's in your source directory is not necessarily what'll be put in your >>>>>> purelib/platlib during installation). >>>>>> >>>>> >>>>> >>>>> Ah, that’s not what I meant. I meant very few users are developing >>>>> packages. E.g. there are millions of pandas users but few of them are >>>>> modifying pandas. (I hope. :-) >>>>> >>>>> Also, I do not think that what we want to solve is editable >>>>>> installations per se. What I'd like to solve here is supporting import >>>>>> hooks for static checkers. Editable installations are just a good example >>>>>> of where import hooks can be used. >>>>>> >>>>> >>>>> Agreed. >>>>> >>>>> Sadly, while the db/dynamic code stub files can be used to work around >>>>>> the problem; for disks that live on the disk just not under the >>>>>> site-packages folder, they're less useful. >>>>>> >>>>> >>>>> Not sure I follow. >>>>> >>>>> —Guido >>>>> >>>>> All the best, >>>>>> >>>>>> Bernat >>>>>> >>>>>> On Sat, May 7, 2022 at 6:11 PM Guido van Rossum <guido(a)python.org> >>>>>> wrote: >>>>>> >>>>>>> On Sat, May 7, 2022 at 4:14 PM Eric Traut <eric(a)traut.com> wrote: >>>>>>> >>>>>>>> Most static analysis tools already support “.pth” files. Jia >>>>>>>> mentioned that pyre doesn’t yet, but it should be straightforward for them >>>>>>>> to add this. My recommendation is that we rally behind “.pth” files. I >>>>>>>> think this approach is more pragmatic than introducing yet another >>>>>>>> mechanism that needs to be implemented by all of the tools in the ecosystem. >>>>>>>> >>>>>>> >>>>>>> Hm. You're the first person I've met in a long time who regards PTH >>>>>>> files as something we should do more of -- IIUC they are almost universally >>>>>>> considered a necessary evil. Especially the fact that they can be used to >>>>>>> do other stuff than add entries to sys.path. But also that -- a long >>>>>>> sys.path tends to slow down module search, and it seems just wrong that >>>>>>> adding N packages would append N sys.path entries: if we wanted each >>>>>>> package to have its own entry we should have designed it as a mapping, not >>>>>>> a search path. >>>>>>> >>>>>>> TBH, if all we're trying to solve is editable installs, I am >>>>>>> assuming that those would be used by a vanishingly small proportion of >>>>>>> users (otherwise we're doing something else wrong with package installs), >>>>>>> so I would personally be okay if an editable install wasn't visible to my >>>>>>> IDE or type checker, or if those tools were to fall back to typeshed or a >>>>>>> PEP 561 stub package. >>>>>>> >>>>>>> Regarding your claim that almost nobody loads code from a database, >>>>>>> apparently that's actually very popular in the banking sector ( >>>>>>> https://calpaterson.com/bank-python.html) -- and those people have >>>>>>> a *lot* of Python code. >>>>>>> >>>>>>> I believe that VS Code is using (or going to use, or may use?) a >>>>>>> Virtual Filesystem to access user code. Would it really be so terrible if >>>>>>> there were some corner of the VFS namespace to map to the Python module >>>>>>> (spec) namespace? It could be served from a Python process that can give >>>>>>> you the source of a module given its full name (without executing the >>>>>>> module, of course -- the import loader machinery should be able to >>>>>>> guarantee that already). >>>>>>> >>>>>>> -- >>>>>>> --Guido van Rossum (python.org/~guido) >>>>>>> *Pronouns: he/him **(why is my pronoun here?)* >>>>>>> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…> >>>>>>> >>>>>> _______________________________________________ >>>>>>> Typing-sig mailing list -- typing-sig(a)python.org >>>>>>> To unsubscribe send an email to typing-sig-leave(a)python.org >>>>>>> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >>>>>>> Member address: gaborjbernat(a)gmail.com >>>>>>> >>>>>> -- >>>>> --Guido (mobile) >>>>> >>>> _______________________________________________ >>> Typing-sig mailing list -- typing-sig(a)python.org >>> To unsubscribe send an email to typing-sig-leave(a)python.org >>> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >>> Member address: mikhail.golubev(a)jetbrains.com >>> >> >> >> -- >> Mikhail Golubev >> Software Developer >> JetBrains >> http://www.jetbrains.com >> The drive to develop >> > -- Mikhail Golubev Software Developer JetBrains http://www.jetbrains.com The drive to develop

8 7

Type narrowing for TypeGuard in the negative case
by Eric Traut 29 Sep '22

29 Sep '22

Since the introduction of PEP 647 (User-defined Type Guards), we've received a steady stream of input from users saying that they don't like the limitation that type narrowing is applied only in the positive case and is not applied in the negative case. In general, type narrowing is not safe to perform if a user-defined type guard returns False, so I've stood by the original decision not to provide type narrowing in the negative case, but there are cases where such narrowing is safe and desirable. Here's a recent thread where this was discussed in some detail: https://github.com/python/typing/issues/996#issuecomment-1002716830 In this discussion, Ilya Kamen proposed a solution. The proposal is to extend the existing `TypeGuard` to support an optional second type argument. If present, the second argument indicates the narrowed type in the negative type narrowing situation. Here's a simple (admittedly contrived) example: ```python def is_str(val: str | int) -> TypeGuard[str, int]: return isinstance(val, str) def func(val: str | int): if is_str(val): reveal_type(val) # str else: reveal_type(val) # int ``` I've implemented this proposal in the latest published version of pyright (1.1.202) so folks can experiment with it and see if they like it. If there's general consensus that it's the right approach, I can file an amendment for the existing PEP 647 to include this functionality. It was trivial to add to pyright, so I'm optimistic that it would likewise be easy to add to the other type checkers. Another common request for PEP 647 is the desire for an "assert" form of TypeGuard — a way to indicate that a function performs runtime validation of a type, raising an exception if the type is incorrect. It occurred to me that this two-argument form of `TypeGuard` could also be used to handle this use case. The second argument would be specified as a `NoReturn`. I've provisionally implemented this in pyright as well (although it's not in the currently-published version — you'll need to wait until 1.1.203 is published). Here's what this would look like: ```python def validate_sequence(val: Sequence[_T | None]) -> TypeGuard[Sequence[_T], NoReturn]: """Verifies that the sequence contains no None values""" if len([x for x in val if x is not None]) > 0: raise Exception() return True def func(val: Sequence[int | None]): validate_sequence(val) reveal_type(val) # Sequence[int] ``` I'm interested in input on these proposals. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft

6 7

Literal -> _LiteralGenericAlias?
by Paul Bryan 29 Sep '22

29 Sep '22

I'm trying to understand why Literal["a", "b", "c"] results in an object derived from _GenericAlias. It's not using TypeVar substitutions, so it's not generic per se. Is it just to provide the origin and arguments semantics? That said, would would be the best way to distinguish between a "real" generic alias from literals and unions (both of which derive from _GenericAlias)?

2 2

Typing-sig September 2022