Extending published stubs
I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `@pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register_...). So for a `DataFrame`, if you do `@pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated. Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it. Has this idea been discussed anywhere?
TypeScript has a similar concept: https://www.typescriptlang.org/docs/handbook/declaration-merging.html#module... Probably, this is the only similar reference I know. ср, 14 сент. 2022 г. в 17:49, Irv Lustig <irv@princeton.com>:
I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `@pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register_...). So for a `DataFrame`, if you do `@pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated.
Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it.
Has this idea been discussed anywhere? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: n.a.sobolev@gmail.com
The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported. martin On Wed, Sep 14, 2022 at 7:49 AM Irv Lustig <irv@princeton.com> wrote:
I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `@pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register_...). So for a `DataFrame`, if you do `@pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated.
Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it.
Has this idea been discussed anywhere? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com
I like the overlay idea. Is it worth it to put together a PEP proposal for this? -Irv On Wed, Sep 14, 2022 at 6:49 PM Martin DeMello <mdemello@google.com> wrote:
The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported.
martin
Hi, I like this overlay idea and I think I have a good use case for it. It is not possible to represent all the things that Django can do in the type system at present. Therefore, to support type-checking those things, people have developed a mypy plug-in which handles them. But that is of course limited to mypy, which limits its utility. An example of what it does is adding an extra attribute to a class with a name derived from another attribute whenever it is of a certain type. I’ve been prototyping a tool (not published yet) that can automatically generate these extra annotations (and sometimes modifications to existing annotations). It currently uses libcst’s codemod facility to modify the .py files in place, but it would be much better if it could just generate overlays alongside the existing files. This would avoid cluttering the original source files. It would also be cool to allow for optional comments in the overlay file which would indicate which line in the original source file the generated code line came from (kind of like a source map in TypeScript) so that any error messages would point to the original source file. I then imagine a watcher which looks for changes in the .py files and generates new overlay files whenever they are changed. I would be super excited to see overlay files become a reality, to support these kinds of situations. Best regards, Seth Yastrov
On 15 Sep 2022, at 15.21, Irv Lustig <irv@princeton.com> wrote:
I like the overlay idea. Is it worth it to put together a PEP proposal for this?
-Irv
On Wed, Sep 14, 2022 at 6:49 PM Martin DeMello <mdemello@google.com> wrote: The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported.
martin
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: syastrov@gmail.com
I'll put together a PEP exploring the idea more fully. martin On Sun, Sep 18, 2022 at 11:22 PM <syastrov@gmail.com> wrote:
Hi,
I like this overlay idea and I think I have a good use case for it.
It is not possible to represent all the things that Django can do in the type system at present. Therefore, to support type-checking those things, people have developed a mypy plug-in which handles them. But that is of course limited to mypy, which limits its utility.
An example of what it does is adding an extra attribute to a class with a name derived from another attribute whenever it is of a certain type.
I’ve been prototyping a tool (not published yet) that can automatically generate these extra annotations (and sometimes modifications to existing annotations). It currently uses libcst’s codemod facility to modify the .py files in place, but it would be much better if it could just generate overlays alongside the existing files. This would avoid cluttering the original source files. It would also be cool to allow for optional comments in the overlay file which would indicate which line in the original source file the generated code line came from (kind of like a source map in TypeScript) so that any error messages would point to the original source file. I then imagine a watcher which looks for changes in the .py files and generates new overlay files whenever they are changed.
I would be super excited to see overlay files become a reality, to support these kinds of situations.
Best regards, Seth Yastrov
On 15 Sep 2022, at 15.21, Irv Lustig <irv@princeton.com> wrote:
I like the overlay idea. Is it worth it to put together a PEP proposal for this?
-Irv
On Wed, Sep 14, 2022 at 6:49 PM Martin DeMello <mdemello@google.com> wrote:
The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported.
martin
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: syastrov@gmail.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com
I haven't had the time to do any work on the PEP yet :( I'll move it up the priority list, since we would like to support it more fully in pytype. martin On Tue, Mar 7, 2023 at 9:30 AM Irv Lustig <irv@princeton.com> wrote:
Just following up here - are you planning on creating a PEP for this? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com
I'd be happy to review what you put together, whenever you get around to it.
A couple use cases for this I’ve encountered: - As an application developer, wanting to provide type stubs for a small subset of Luigi’s (the popular data pipeline library) public API. In particular I wanted to fully type the https://github.com/spotify/luigi/blob/master/luigi/parameter.py module since these parameters are descriptor-like in behavior and thus without any stubs make a mess of using any Task attribute. For this use case a module level overlay would work fine. But I did also think about typing single functions within classes that would particularly benefit from overloads and such, for that to work more granular overlay levels would have to be supported or else I would be forced to type the entire class even if I’m fine with basic type inference for the rest of the methods. - Generating partial stubs to replace manual overload boilerplate or things the type system won’t let you do, similar to discussions in https://github.com/python/typing/issues/1371. I think supporting more granular overlays for this would be ideal.
One more use-case: When using django-stubs people complain that installing some runtime apps does not affect typing. One example: adding new django middleware handlers, it can add new attributes to a request object, right now there's no way to handle this without rewriting all HttpRequest objects manually. пн, 20 мар. 2023 г. в 09:55, Adrian Garcia Badaracco <adrian@adriangb.com>:
A couple use cases for this I’ve encountered: - As an application developer, wanting to provide type stubs for a small subset of Luigi’s (the popular data pipeline library) public API. In particular I wanted to fully type the https://github.com/spotify/luigi/blob/master/luigi/parameter.py module since these parameters are descriptor-like in behavior and thus without any stubs make a mess of using any Task attribute. For this use case a module level overlay would work fine. But I did also think about typing single functions within classes that would particularly benefit from overloads and such, for that to work more granular overlay levels would have to be supported or else I would be forced to type the entire class even if I’m fine with basic type inference for the rest of the methods. - Generating partial stubs to replace manual overload boilerplate or things the type system won’t let you do, similar to discussions in https://github.com/python/typing/issues/1371. I think supporting more granular overlays for this would be ideal. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: n.a.sobolev@gmail.com
any update on this? I have this issue - https://github.com/microsoft/pylance-release/issues/4008 - and I think it could be used for tools to support some dynamic behavior cases.
participants (6)
-
Adrian Garcia Badaracco -
Heejae Chang -
Irv Lustig -
Martin DeMello -
syastrov＠gmail.com -
Никита Соболев