Questions about overload usage
We're working on a way to access overloaded functions at runtime in https://github.com/python/cpython/pull/31716. This will create a typing.get_overloads() function that returns the overloads for a function. A few questions came up that I'd like to get feedback on from people who maintain typed codebases: - The current implementation will make @overload about 6x slower (46 ns to 286 ns). Will this be a problem for users? - Performance will be worse for functions with a large number of overloads (more than 20 right now). Is this common? (I think the highest overload count for typeshed is builtins.pow(), which has 17.) - get_overloads() will behave oddly for overloads on functions nested within other functions. Is that something that happens in practice?
Speaking on a ~100k loc internal mostly typed codebase, searching codebase I see about ~50 overloads. The highest number of distinct overloads I see for 1 function is 4. Admittingly, that specific function should have more overloads, but maintaining 15 overloads for a function any time it gets a new argument is tedious enough that I only included the most likely overloads to be accessed instead of having most precise types for all paths. Given the rarity of them I'd be very surprised if I noticed the performance difference. My guess is numpy would have a high amount of overloads for data type rules if/when they fully support data type conversion rules. Precise type for np.add/np.multiply will need a lot of overloads. Not sure if they have those overloads yet. My gut most of impact would be startup time for cli/short used scripts. As for nested function overloads, I see 0. I also wouldn't mind minor refactor if needed for that case. I remember hitting rule of TypeAlias not supported in function scope once and just did small refactor for it.
My guess is numpy would have a high amount of overloads for data type rules if/when they fully support data type conversion rules. Precise type for np.add/np.multiply will need a lot of overloads. Not sure if they have those overloads yet.
This is correct, and those overloads are already largely in place (and even ignoring dtypes there are many places where, unfortunately, overloads are necessary). The type hints themselves are confined to .pyi stub files though, with no plans for moving them to inline annotations.
I have seen overloads with much higher numerosity, e.g. https://github.com/henribru/google-api-python-client-stubs/blob/master/googl... That said, I don’t think such cases are too common. mypy’s overlap check is quadratic in number of overloads, so mypy users probably don’t have many hundred count overloads. On the other hand, I believe pyright skips its corresponding check for high overload count, so seems likely at least one pyright user had this issue. On Mon, Apr 4, 2022 at 12:16 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
We're working on a way to access overloaded functions at runtime in https://github.com/python/cpython/pull/31716. This will create a typing.get_overloads() function that returns the overloads for a function.
A few questions came up that I'd like to get feedback on from people who maintain typed codebases: - The current implementation will make @overload about 6x slower (46 ns to 286 ns). Will this be a problem for users? - Performance will be worse for functions with a large number of overloads (more than 20 right now). Is this common? (I think the highest overload count for typeshed is builtins.pow(), which has 17.) - get_overloads() will behave oddly for overloads on functions nested within other functions. Is that something that happens in practice? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
El lun, 4 abr 2022 a las 6:19, Shantanu Jain (<hauntsaninja@gmail.com>) escribió:
I have seen overloads with much higher numerosity, e.g. https://github.com/henribru/google-api-python-client-stubs/blob/master/googl...
Thanks for that example, but it won't be affected by this change because it's in a stub file and my change only affects the runtime implementation of `@overload`. That said, this suggests an obvious workaround if someone else has a monstrosity like that in a .py file: just move it to a stub.
That said, I don’t think such cases are too common. mypy’s overlap check is quadratic in number of overloads, so mypy users probably don’t have many hundred count overloads. On the other hand, I believe pyright skips its corresponding check for high overload count, so seems likely at least one pyright user had this issue.
On Mon, Apr 4, 2022 at 12:16 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
We're working on a way to access overloaded functions at runtime in https://github.com/python/cpython/pull/31716. This will create a typing.get_overloads() function that returns the overloads for a function.
A few questions came up that I'd like to get feedback on from people who maintain typed codebases: - The current implementation will make @overload about 6x slower (46 ns to 286 ns). Will this be a problem for users? - Performance will be worse for functions with a large number of overloads (more than 20 right now). Is this common? (I think the highest overload count for typeshed is builtins.pow(), which has 17.) - get_overloads() will behave oddly for overloads on functions nested within other functions. Is that something that happens in practice? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
participants (4)
-
Bas van Beek -
Jelle Zijlstra -
Mehdi2277 -
Shantanu Jain