Mailman 3 Extending published stubs - Typing-sig

newer
Opened and changed typing issues...

Extending published stubs

Irv Lustig

14 Sep 2022 14 Sep '22

2:39 p.m.

I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `@pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register_...). So for a `DataFrame`, if you do `@pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated. Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it. Has this idea been discussed anywhere?

Show replies by date

Никита Соболев

14 Sep 14 Sep

10:25 p.m.

TypeScript has a similar concept: https://www.typescriptlang.org/docs/handbook/declaration-merging.html#module... Probably, this is the only similar reference I know. ср, 14 сент. 2022 г. в 17:49, Irv Lustig :

...

I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `@pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register_...). So for a `DataFrame`, if you do `@pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated.

Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it.

Has this idea been discussed anywhere? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: n.a.sobolev@gmail.com

Martin DeMello

10:48 p.m.

The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported. martin On Wed, Sep 14, 2022 at 7:49 AM Irv Lustig wrote:

...

I'm the leader of the pandas-stubs project. In pandas, users can extend the library by creating their own accessor via `@pd.api.extensions.register_dataframe_accessor` (see https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.register_...). So for a `DataFrame`, if you do `@pd.api.extensions.register_dataframe_accessor("xyz")`, you now have `DataFrame.xyz` returning the class you annotated.

Now suppose you want to create a stub for your new accessor, that would extend the existing pandas-stubs. AFAIK, there isn't a way to do something like this in a stub: ``` from pandas import DataFrame @typing.extend class DataFrame: def xyz(self) -> MyXyzClass: ... ``` The idea here is that there is a class (in this case, `DataFrame`) that already has stubs, and you want to add a method to it.

Has this idea been discussed anywhere? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com

Irv Lustig

15 Sep 15 Sep

1:20 p.m.

I like the overlay idea. Is it worth it to put together a PEP proposal for this? -Irv On Wed, Sep 14, 2022 at 6:49 PM Martin DeMello wrote:

...

The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported.

martin

...

syastrov＠gmail.com

19 Sep 19 Sep

6:22 a.m.

Hi, I like this overlay idea and I think I have a good use case for it. It is not possible to represent all the things that Django can do in the type system at present. Therefore, to support type-checking those things, people have developed a mypy plug-in which handles them. But that is of course limited to mypy, which limits its utility. An example of what it does is adding an extra attribute to a class with a name derived from another attribute whenever it is of a certain type. I’ve been prototyping a tool (not published yet) that can automatically generate these extra annotations (and sometimes modifications to existing annotations). It currently uses libcst’s codemod facility to modify the .py files in place, but it would be much better if it could just generate overlays alongside the existing files. This would avoid cluttering the original source files. It would also be cool to allow for optional comments in the overlay file which would indicate which line in the original source file the generated code line came from (kind of like a source map in TypeScript) so that any error messages would point to the original source file. I then imagine a watcher which looks for changes in the .py files and generates new overlay files whenever they are changed. I would be super excited to see overlay files become a reality, to support these kinds of situations. Best regards, Seth Yastrov

...

On 15 Sep 2022, at 15.21, Irv Lustig wrote:

I like the overlay idea. Is it worth it to put together a PEP proposal for this?

-Irv

...
On Wed, Sep 14, 2022 at 6:49 PM Martin DeMello wrote: The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported.

martin

...

Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: syastrov@gmail.com

Martin DeMello

11:19 p.m.

I'll put together a PEP exploring the idea more fully. martin On Sun, Sep 18, 2022 at 11:22 PM wrote:

...

Hi,

I like this overlay idea and I think I have a good use case for it.

It is not possible to represent all the things that Django can do in the type system at present. Therefore, to support type-checking those things, people have developed a mypy plug-in which handles them. But that is of course limited to mypy, which limits its utility.

An example of what it does is adding an extra attribute to a class with a name derived from another attribute whenever it is of a certain type.

I’ve been prototyping a tool (not published yet) that can automatically generate these extra annotations (and sometimes modifications to existing annotations). It currently uses libcst’s codemod facility to modify the .py files in place, but it would be much better if it could just generate overlays alongside the existing files. This would avoid cluttering the original source files. It would also be cool to allow for optional comments in the overlay file which would indicate which line in the original source file the generated code line came from (kind of like a source map in TypeScript) so that any error messages would point to the original source file. I then imagine a watcher which looks for changes in the .py files and generates new overlay files whenever they are changed.

I would be super excited to see overlay files become a reality, to support these kinds of situations.

Best regards, Seth Yastrov

On 15 Sep 2022, at 15.21, Irv Lustig wrote:

I like the overlay idea. Is it worth it to put together a PEP proposal for this?

-Irv

On Wed, Sep 14, 2022 at 6:49 PM Martin DeMello wrote:

...
The pytype team has internally discussed the idea of supporting "stub overlays". Our idea was to do this at a module level, so that definitions would be recursively merged without any explicit annotations; in this instance you would have a file named `pandas_overlay.pyi` and define `class DataFrame` in it, which would modify `pandas.DataFrame` when imported.

martin

...
_______________________________________________

Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: syastrov@gmail.com

_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com

Irv Lustig

7 Mar 7 Mar

5:29 p.m.

Just following up here - are you planning on creating a PEP for this?

Martin DeMello

10:24 p.m.

I haven't had the time to do any work on the PEP yet :( I'll move it up the priority list, since we would like to support it more fully in pytype. martin On Tue, Mar 7, 2023 at 9:30 AM Irv Lustig wrote:

...

Just following up here - are you planning on creating a PEP for this? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mdemello@google.com

Irv Lustig

8 Mar 8 Mar

3:25 p.m.

I'd be happy to review what you put together, whenever you get around to it.

Adrian Garcia Badaracco

20 Mar 20 Mar

6:54 a.m.

A couple use cases for this I’ve encountered: - As an application developer, wanting to provide type stubs for a small subset of Luigi’s (the popular data pipeline library) public API. In particular I wanted to fully type the https://github.com/spotify/luigi/blob/master/luigi/parameter.py module since these parameters are descriptor-like in behavior and thus without any stubs make a mess of using any Task attribute. For this use case a module level overlay would work fine. But I did also think about typing single functions within classes that would particularly benefit from overloads and such, for that to work more granular overlay levels would have to be supported or else I would be forced to type the entire class even if I’m fine with basic type inference for the rest of the methods. - Generating partial stubs to replace manual overload boilerplate or things the type system won’t let you do, similar to discussions in https://github.com/python/typing/issues/1371. I think supporting more granular overlays for this would be ideal.

Никита Соболев

10:38 a.m.

One more use-case: When using django-stubs people complain that installing some runtime apps does not affect typing. One example: adding new django middleware handlers, it can add new attributes to a request object, right now there's no way to handle this without rewriting all HttpRequest objects manually. пн, 20 мар. 2023 г. в 09:55, Adrian Garcia Badaracco :

...

A couple use cases for this I’ve encountered: - As an application developer, wanting to provide type stubs for a small subset of Luigi’s (the popular data pipeline library) public API. In particular I wanted to fully type the https://github.com/spotify/luigi/blob/master/luigi/parameter.py module since these parameters are descriptor-like in behavior and thus without any stubs make a mess of using any Task attribute. For this use case a module level overlay would work fine. But I did also think about typing single functions within classes that would particularly benefit from overloads and such, for that to work more granular overlay levels would have to be supported or else I would be forced to type the entire class even if I’m fine with basic type inference for the rest of the methods. - Generating partial stubs to replace manual overload boilerplate or things the type system won’t let you do, similar to discussions in https://github.com/python/typing/issues/1371. I think supporting more granular overlays for this would be ideal. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: n.a.sobolev@gmail.com

Heejae Chang

26 Apr 26 Apr

9:44 p.m.

any update on this? I have this issue - https://github.com/microsoft/pylance-release/issues/4008 - and I think it could be used for tools to support some dynamic behavior cases.

364

Age (days ago)

588

Last active (days ago)

List overview

Download

11 comments

6 participants

participants (6)

Adrian Garcia Badaracco
Heejae Chang
Irv Lustig
Martin DeMello
syastrov＠gmail.com
Никита Соболев

Extending published stubs

tags

participants (6)