PEP 612: Parameter Specification Variables
Hi All, I want to announce the submission of a new typing PEP, for ParameterSpecifications (https://github.com/python/peps/pull/1259). We discussed this concept at the last typing summit (presentation can be found here https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variab...), and there seemed to be general consensus on the utility of this idea, and the appropriateness of the particular solution. One note that I know there was some disagreement on was the name for the feature itself, with some folks preferring ArgSpec. I wrote up a brief note in the PEP itself why we're not as big fans of that name, but I'd love to hear all of your opinions on this issue, as well as on anything else in the PEP. Best, Mark Mendoza
Hi all I'm one of the Hypothesis core developers, for which this will be very useful indeed - we have several decorators which modify the signature of the wrapped function, and currently we just check the return types. I am *also* responsible for the `from_type()` strategy, which does runtime inspection of types (or type-annotated functions, classes, methods, ...) and works out how to call them. So I would really appreciate it if the implementation of ParameterSpecifications and the various type operators preserved enough information at runtime to reconstruct the semantics of that use! (Contrast TypedDict with optional keys: https://bugs.python.org/issue38834, fixed for 3.9). Happy to review things, draft our implementation against a prototype, etc. - just let me know. Best wishes from a continent literally on fire, Zac On Fri, 20 Dec 2019 at 09:56, Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi All, I want to announce the submission of a new typing PEP, for ParameterSpecifications (https://github.com/python/peps/pull/1259). We discussed this concept at the last typing summit (presentation can be found here https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variab...), and there seemed to be general consensus on the utility of this idea, and the appropriateness of the particular solution.
One note that I know there was some disagreement on was the name for the feature itself, with some folks preferring ArgSpec. I wrote up a brief note in the PEP itself why we're not as big fans of that name, but I'd love to hear all of your opinions on this issue, as well as on anything else in the PEP.
Best, Mark Mendoza _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
Hi Zac, I'm not super familiar with the details of the runtime support for type variables etc., but in principle this definitely seems doable! I will definitely reach out when we're working on that side of the implementation for test cases/prototype testing etc. Glad to hear that you're interested! Best, Mark
Hi All, I'm transferring a response over from https://github.com/python/peps/pull/1259 to a comment from Andrew Svetlov <andrew.svetlov@gmail.com> to keep our discussion consolidated.
I still think that modifying the parameters list signature is important. One very common example is `unittest.mock`:
``` @patch('__main__.SomeClass') def function(normal_argument, mock_class): print(mock_class is SomeClass) ```
Another one is `click`:
``` @cli.command() @click.pass_context def sync(ctx): click.echo('Debug is %s' % (ctx.obj['DEBUG'] and 'on' or 'off')) ```
Note, `pass_context` decorator adds a parameter to the function signature. I can imagine the reverse operation easily.
Signature modifying decorators are definitely on our radar, but I'm not sure that this PEP is the appropriate moment to try to address them. For altering the arguments, we have another PEP coming down the pipe that may fit your needs, ListVariadics. For addition: ``` from typing import Callable, TypeVar from pyre_extensions.type_variable_operators import Concatenate Ts = pyre_extensions.ListVariadic("Ts") def prepend_addition_argument(f: Callable[[Ts], int]) -> Callable[[Concatenate[int, Ts]], str]: def inner(x: int, *args: Ts) -> str: return str(x + f( *args)) return inner @prepend_addition_argument def foo(x: int, y: int) -> int: return x + y reveal_type(foo) # typing.Callable(foo)[[int, int, int], str] ``` For removal: ``` from typing import Callable, TypeVar, List from pyre_extensions.type_variable_operators import Concatenate Ts = pyre_extensions.ListVariadic("Ts") TReturn = TypeVar("TReturn") def simple_partial_application( f: Callable[[Concatenate[float, Ts]], TReturn] ) -> Callable[[Ts], TReturn]: def inner( *args: Ts) -> TReturn: return f(42.0, *args) return inner @simple_partial_application def foo(x: float, y: str, z: bool) -> int: return 3 reveal_type(foo) # typing.Callable(foo)[[str, bool], int] ``` For more details on ListVariadics, you can read this presentation from the last typing summit (https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variab...) The trade-off here is that by going in and out of this ListVariadic, we lose the names of the arguments, meaning that, for example, `foo(y="A", z=True)` would not be accepted by Pyre in the second example, even though it would work in the runtime. Furthermore, this will not work at all for functions that have keyword-only arguments. However, as far as I'm aware, click and patch work positionally, so ListVariadics should work in that instance. Supporting mutation in the full-fidelity ParameterSpecifications would require rich handling of name collision, which would get very complex very quickly. In my opinion working out the specification/implementation of that isn't worth blocking the rest of this, since it seems like the combination of these two features can get us a lot of the way there.
Currently, Python supports 5 types of arguments:
* positional-only * regular arguments that can be used in both positional and named contexts * keyword-only * var-args: `*args` * keyword var-args: `*kwargs`
The first three can have default arguments, it also affects a signature. I think it makes the proposed object much more complex
This complexity is a the reason why we have to make a purpose-built abstraction for parameter specifications, rather than being able to directly use a combination of other type system extensions like Map and List Variadics. However, I believe that the `TParams.args` and `TParams.kwargs` component model encompasses all of these kinds of arguments since they encode the set of positional arguments and keyword arguments of a single invocation of the signature. When the function has defaults, *args, **kwargs, etc, each argument is always either invoked positionally or by name. This is the reason why the `def f(*args, **kwargs): return g(*args, **kwargs)` pattern works in the runtime, and that's all that we need to be able to statically enforce in order to make the signature forwarding sound.
Hi All, I'm transferring a response over from https://github.com/python/peps/pull/1259 to a comment from Sebastian Kreft to keep our discussion consolidated.
I do have some questions regarding the proposed PEP. Those questions were being written while the PR was merged.
1) I'm missing a brief description on what happens to keyword only (PEP 3102 <https://www.python.org/dev/peps/pep-3102/>) and to positional only (PEP 570 <https://www.python.org/dev/peps/pep-0570/>) arguments.
Will for example the following be accepted or rejected:
def add_logging(f: Callable[Ps, R]) -> Callable[Ps, R]: async def inner(*args: Ps.args, **kwargs: Ps.kwargs) -> R: # log return f(*args, **kwargs) return inner
@add_logging def add(x:int, *, y: int = 0): return x + y
add(1, 2)
Note that today in Mypy, that add call (without the decorator) will be rejected.
I only found there's some talk abut it in the rejected proposals section.
Yes the decorated version will still be correctly rejected under the new proposal. This will be because the entire specification of the parameter of the decorated function will be transferred over, including the keyword-only-ness of `y`. Where do you see a mention of this in the rejected alternatives section? I'd like to clarify that confusion there if possible.
2) What do you mean by anonymous arguments? Are those arguments prefixed by double underscore (__) as in the provided example. Is that rule enforced or suggested?
These are defined in PEP484 (https://www.python.org/dev/peps/pep-0484/#positional-only-arguments), and are enforced by type checkers, not the runtime
3) What is the motivation to require TParams.args and TParams.kwargs to be used together.
Why is the example def baz(*args: TParams.args) -> int rejected?
The reason is that only by using them together do they come together into a callable type that can be actually called. Consider ``` def partial_decorator(f: Callable[TParams, int]) -> ???: def inner(*args: TParams.args) -> int: return f(*args) # This is an unsafe call return inner # What is the type of inner? ``` If you only want access to the positional parameters of a function, you can use a ListVariadic (currently in pyre_extensions, to be standardized in a forthcoming PEP), like so: ``` from typing import Callable, TypeVar from pyre_extensions.type_variable_operators import Concatenate Ts = pyre_extensions.ListVariadic("Ts") def prepend_addition_argument(f: Callable[[Ts], int]) -> Callable[[Concatenate[int, Ts]], str]: def inner(x: int, *args: Ts) -> str: return str(x + f( *args)) return inner @prepend_addition_argument def foo(x: int, y: int) -> int: return x + y reveal_type(foo) # typing.Callable(foo)[[int, int, int], str] ```
Hi, Instead of using the type parameter approach, have you considered using a type operator approach? I'm sure this approach has problems, but thought I'd bring it up anyway. I mean is something like this: ``` from typing import Awaitable, Callable, TypeVar, Parameters TReturn = TypeVar("TReturn") def add_logging( f: Callable[..., TReturn] ) -> Callable[Parameters[f], Awaitable[TReturn]]: async def inner(*args: Parameters[f].args, **kwargs: Parameters[f].kwargs) -> TReturn: await log_to_database() return f(*args, **kwargs) return inner ``` or even, extending Callable a bit to eliminate the TypeVar as well, ``` from typing import Awaitable, Callable, Parameters, ReturnType def add_logging( f: Callable[..., ...] ) -> Callable[Parameters[f], Awaitable[ReturnType[f]]]: async def inner(*args: Parameters[f].args, **kwargs: Parameters[f].kwargs) -> ReturnType[f]: await log_to_database() return f(*args, **kwargs) return inner ``` This is the approach used by TypeScript, at least. They are defined like this[0]: ``` type Parameters<T extends (...args: any) => any> = T extends (...args: infer P) => any ? P : never; type ReturnType<T extends (...args: any) => any> = T extends (...args: any) => infer R ? R : any; ``` [0] https://github.com/microsoft/TypeScript/blob/v3.7.4/src/lib/es5.d.ts#L1473-L... In Python, `Parameters` would return something like a `ParameterSpecification` instance, with the same restrictions described in the PEP. In TypeScript, these type operators appear to use some complex machinery, namely conditional types and the `infer` declaration, which no Python type checker supports AFAIK. But maybe they can be implemented directly. They might also be useful for other cases. For example, I often have a function with a complex set of arguments, and another functions which wraps it and just forwards the arguments. In these cases, I would have loved to be able to write this: ``` def function( with: int, lots: bool, of: string, complex: bytes, *, arguments: float, ): None:... def function_wrapper( *args: Parameters[function].args, **kwargs: Parameters[function].kwargs, ) -> ReturnType[function]: ... ``` Ran
Hi Ran, Thanks for taking a look at the PEP and for taking the time to write up such a thoughtful response! I hadn't considered `ParametersOf` in the context of being able to do local captures as you describe, and the fact that TypeScript is able to support building something like this out of primitives is really cool. However, I think that this alternative is actually less powerful than the proposal. The key issue is that sometimes we'd like to refer to the parameters of a function which has no name yet. One case of this is when the parameter has a type that is more complex than a bare callable ``` def adaptable_decorator(f: Union[Callable[TParams, int], Callable[TParams, str]]) -> Callable[TParams, bool]: ... ``` Another case of this is classes parameterized on ParamSpecs. Consider a class that was constructed around a callable, which it forwards to through a `call` method. With ParamSpecs we can spell that like this: ``` class MyClass(Generic[TParams, TReturn]): f: Callable[TParams, TReturn] def __init__(self, f: Callable[TParams, TReturn]) -> None: self.f = f def call(__self, *args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: f = __self.f # do some logging or something return f(*args, **kwargs) ``` In a `Parameters` operator world, the type of MyClass(some_function) would have to be somehow *more* than just MyClass, but in a way that is not a generic. That all being said, I am personally strongly in favor of a follow-up PEP that would introduce `Parameters[global_function]`, since that is the only full-featured way to spell an instance of a class like MyClass, (e.g. `MyClass[ParametersOf[some_global_function_or_maybe_a_callback_protocol]]`) Thanks again for the feedback! Best, Mark Mendoza
Hi Mark, I have finally looked over your PEP carefully, and I am in favor of accepting it. I have two tiny edits that I will submit as a PR later. I am willing to act as a PEP sponsor. Do you think you will want to get more feedback or should we then just submit it for acceptance to the Steering Council? --Guido On Thu, Dec 19, 2019 at 2:56 PM Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi All, I want to announce the submission of a new typing PEP, for ParameterSpecifications (https://github.com/python/peps/pull/1259). We discussed this concept at the last typing summit (presentation can be found here https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variab...), and there seemed to be general consensus on the utility of this idea, and the appropriateness of the particular solution.
One note that I know there was some disagreement on was the name for the feature itself, with some folks preferring ArgSpec. I wrote up a brief note in the PEP itself why we're not as big fans of that name, but I'd love to hear all of your opinions on this issue, as well as on anything else in the PEP.
Best, Mark Mendoza _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Hi Guido, Thanks for volunteering as a sponsor! I think there is one more issue I'd like to get feedback on: a more thorough discussion of Ran Benita's ParametersOf proposal. After some internal discussion, we realized that the more apt analogy to TypedScript would be of an operator that was defined on other types, not names. If that type is a type variable, you can express everything you can in ParamSpecs in terms of ParametersOf operations. The trick is defining what the bound on the type variable has to be. So far we have been avoiding “late incompatibility”, or errors that occur during type variable instantiation. This means that we need to somehow be able to verify that ParametersOf[X] will actually be able to extract a parameter specification from any X that would be substituted into it. To achieve this, we need a bound for the type variable that includes exactly all callables. Currently there are no types in the type system that fit that bill while not introducing an “Any”. The closest thing we have is Callable[..., object], but our current semantic for that is that we permit it to be called with any arguments, unsoundly. Therefore this proposal requires the definition of a new type, Function. Function is callable with, and only with ParametersOf[F]. ParametersOf can only operate on type variables with precisely this bound. This means we’re just introducing a new bound and a new operator (one more if you also want ReturnType[F], but that’s actually not strictly necessary, see case #3 below), rather than a new kind of type variable. To demonstrate the pros and cons of this alternative, here are some examples of translating between the two forms: In all of the following examples: ``` TParams = ParamSpec("TParams") TReturn = TypeVar("TReturn") F = TypeVar("F", bound=Function) ``` #1 Typing a type-preserving decorator ``` def no_change_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs) return inner ``` vs. ``` def no_change_decorator(f: F) -> F: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return f(*args, **kwargs) return inner ``` #2 Typing a decorator that wraps the return type in a List ``` def wrapping_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, List[TReturn]]: def inner( *args: TParams.args, **kwargs: TParams.kwargs ) -> List[TReturn]: return [f(*args, **kwargs)] return inner ``` vs. ``` def wrapping_decorator( f: F ) -> Callable[ParametersOf[F], List[ReturnType[F]]]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> List[ReturnType[F]]: return [f(*args, **kwargs)] return inner ``` #3 Typing a decorator that unwraps a list return type ``` def unwrapping_decorator( f: Callable[TParams, List[TReturn]] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs)[0] return inner ``` vs. ``` def unwrapping_decorator( f: Callable[ParametersOf[F], List[TReturn]] ) -> Callable[ParametersOf[F], TReturn]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> TReturn: return f(*args, **kwargs)[0] return inner ``` #4 Typing a class that is a container for a callable ``` class FunctionContainer(Generic[TParams, TReturn]): f : F def __init__(self, f: Callable[TParams, TReturn]) -> None: self.f = f def call( self, *args: TParams.args, **kwargs: TParams.kwargs ) -> TReturn: return self.f(*args, **kwargs) ``` vs. ``` class FunctionContainer(Generic[F]): f : F def __init__(self, f: F) -> None: self.f = f def call( self, *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return self.f(*args, **kwargs) ``` In my opinion, case #1 is the best one for the proposed alternative, and case #3 is the worst one. Case #3 is odd to read because F is not actually referring to any callable. It’s just being used as a container for the ParameterSpecification. I think that the weakness in case #3 highlights what exactly the difference in philosophy between these two approaches is. ParamSpec was born out of the existing approach to Python typing which so far has avoided supporting operators, whether user-defined or built-in, in favor of destructuring. ParametersOf was inspired by TypeScript, which has a type system with extensive use of user-defined operators. Case #3 would be much more ergonomic if you could define an operator RemoveList[List[X]] = X and then you could just return Callable[ParametersOf[F], RemoveList[ReturnType[F]]]. Without that, you unfortunately get into a situation where you have to use a F-variable like a ParamSpec, in that you never actually bind the return type. Overall, the oddness of Case #3 in the ParametersOf alternative is the biggest issue with the ParametersOf proposal that we've heard internally. Combining destructuring with operators does seem to genuinely cause some ergonomics/readability issues. Another issue we've heard raised from Python users we've shown this to is that it would potentially be confusing to have multiple ways to spell the same equivalent type i.e. F === Callable[ParametersOf[F], ReturnType[F]]. The big win for the alternative, beyond the concision of case #1, is that it would avoid introducing a new kind of type variable could potentially make further extensions to the language more easy to specify. I can see the merits of both syntaxes, so I'd love to get some feedback from the community about folks' preferences before we proceed with the approval process. Thanks again for your support and sponsorship! Best, Mark Mendoza
Well, the new proposal introduces two new magic operators, versus the old proposal one new category of type variable. I don't think we really have anything similar to those magic type operators, so that seems a bigger change. (Unless you count ClassVar and Final, but those are more special -- they're not just functions from types to types.) But frankly I'd like to see some feedback from Jukka, Ivan or Michael Sullivan. On Wed, Feb 5, 2020 at 7:48 PM Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi Guido, Thanks for volunteering as a sponsor!
I think there is one more issue I'd like to get feedback on: a more thorough discussion of Ran Benita's ParametersOf proposal. After some internal discussion, we realized that the more apt analogy to TypedScript would be of an operator that was defined on other types, not names. If that type is a type variable, you can express everything you can in ParamSpecs in terms of ParametersOf operations.
The trick is defining what the bound on the type variable has to be. So far we have been avoiding “late incompatibility”, or errors that occur during type variable instantiation. This means that we need to somehow be able to verify that ParametersOf[X] will actually be able to extract a parameter specification from any X that would be substituted into it. To achieve this, we need a bound for the type variable that includes exactly all callables. Currently there are no types in the type system that fit that bill while not introducing an “Any”. The closest thing we have is Callable[..., object], but our current semantic for that is that we permit it to be called with any arguments, unsoundly.
Therefore this proposal requires the definition of a new type, Function. Function is callable with, and only with ParametersOf[F]. ParametersOf can only operate on type variables with precisely this bound. This means we’re just introducing a new bound and a new operator (one more if you also want ReturnType[F], but that’s actually not strictly necessary, see case #3 below), rather than a new kind of type variable.
To demonstrate the pros and cons of this alternative, here are some examples of translating between the two forms:
In all of the following examples:
``` TParams = ParamSpec("TParams") TReturn = TypeVar("TReturn") F = TypeVar("F", bound=Function) ```
#1 Typing a type-preserving decorator
``` def no_change_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs) return inner ```
vs.
``` def no_change_decorator(f: F) -> F: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return f(*args, **kwargs) return inner ```
#2 Typing a decorator that wraps the return type in a List
``` def wrapping_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, List[TReturn]]: def inner( *args: TParams.args, **kwargs: TParams.kwargs ) -> List[TReturn]: return [f(*args, **kwargs)] return inner ```
vs.
``` def wrapping_decorator( f: F ) -> Callable[ParametersOf[F], List[ReturnType[F]]]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> List[ReturnType[F]]: return [f(*args, **kwargs)] return inner ```
#3 Typing a decorator that unwraps a list return type
``` def unwrapping_decorator( f: Callable[TParams, List[TReturn]] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs)[0] return inner ```
vs.
``` def unwrapping_decorator( f: Callable[ParametersOf[F], List[TReturn]] ) -> Callable[ParametersOf[F], TReturn]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> TReturn: return f(*args, **kwargs)[0] return inner ```
#4 Typing a class that is a container for a callable
``` class FunctionContainer(Generic[TParams, TReturn]): f : F def __init__(self, f: Callable[TParams, TReturn]) -> None: self.f = f def call( self, *args: TParams.args, **kwargs: TParams.kwargs ) -> TReturn: return self.f(*args, **kwargs) ```
vs.
``` class FunctionContainer(Generic[F]): f : F def __init__(self, f: F) -> None: self.f = f def call( self, *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return self.f(*args, **kwargs) ```
In my opinion, case #1 is the best one for the proposed alternative, and case #3 is the worst one. Case #3 is odd to read because F is not actually referring to any callable. It’s just being used as a container for the ParameterSpecification. I think that the weakness in case #3 highlights what exactly the difference in philosophy between these two approaches is. ParamSpec was born out of the existing approach to Python typing which so far has avoided supporting operators, whether user-defined or built-in, in favor of destructuring. ParametersOf was inspired by TypeScript, which has a type system with extensive use of user-defined operators. Case #3 would be much more ergonomic if you could define an operator RemoveList[List[X]] = X and then you could just return Callable[ParametersOf[F], RemoveList[ReturnType[F]]]. Without that, you unfortunately get into a situation where you have to use a F-variable like a ParamSpec, in that you never actually bind the return type. Overall, the oddness of Case #3 in the ParametersOf alternative is the biggest issue with the ParametersOf proposal that we've heard internally. Combining destructuring with operators does seem to genuinely cause some ergonomics/readability issues.
Another issue we've heard raised from Python users we've shown this to is that it would potentially be confusing to have multiple ways to spell the same equivalent type i.e. F === Callable[ParametersOf[F], ReturnType[F]].
The big win for the alternative, beyond the concision of case #1, is that it would avoid introducing a new kind of type variable could potentially make further extensions to the language more easy to specify.
I can see the merits of both syntaxes, so I'd love to get some feedback from the community about folks' preferences before we proceed with the approval process.
Thanks again for your support and sponsorship!
Best, Mark Mendoza _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I'll try to have feedback in the next week. On Thu, Feb 6, 2020 at 5:21 PM Guido van Rossum <guido@python.org> wrote:
Well, the new proposal introduces two new magic operators, versus the old proposal one new category of type variable. I don't think we really have anything similar to those magic type operators, so that seems a bigger change. (Unless you count ClassVar and Final, but those are more special -- they're not just functions from types to types.)
But frankly I'd like to see some feedback from Jukka, Ivan or Michael Sullivan.
On Wed, Feb 5, 2020 at 7:48 PM Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi Guido, Thanks for volunteering as a sponsor!
I think there is one more issue I'd like to get feedback on: a more thorough discussion of Ran Benita's ParametersOf proposal. After some internal discussion, we realized that the more apt analogy to TypedScript would be of an operator that was defined on other types, not names. If that type is a type variable, you can express everything you can in ParamSpecs in terms of ParametersOf operations.
The trick is defining what the bound on the type variable has to be. So far we have been avoiding “late incompatibility”, or errors that occur during type variable instantiation. This means that we need to somehow be able to verify that ParametersOf[X] will actually be able to extract a parameter specification from any X that would be substituted into it. To achieve this, we need a bound for the type variable that includes exactly all callables. Currently there are no types in the type system that fit that bill while not introducing an “Any”. The closest thing we have is Callable[..., object], but our current semantic for that is that we permit it to be called with any arguments, unsoundly.
Therefore this proposal requires the definition of a new type, Function. Function is callable with, and only with ParametersOf[F]. ParametersOf can only operate on type variables with precisely this bound. This means we’re just introducing a new bound and a new operator (one more if you also want ReturnType[F], but that’s actually not strictly necessary, see case #3 below), rather than a new kind of type variable.
To demonstrate the pros and cons of this alternative, here are some examples of translating between the two forms:
In all of the following examples:
``` TParams = ParamSpec("TParams") TReturn = TypeVar("TReturn") F = TypeVar("F", bound=Function) ```
#1 Typing a type-preserving decorator
``` def no_change_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs) return inner ```
vs.
``` def no_change_decorator(f: F) -> F: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return f(*args, **kwargs) return inner ```
#2 Typing a decorator that wraps the return type in a List
``` def wrapping_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, List[TReturn]]: def inner( *args: TParams.args, **kwargs: TParams.kwargs ) -> List[TReturn]: return [f(*args, **kwargs)] return inner ```
vs.
``` def wrapping_decorator( f: F ) -> Callable[ParametersOf[F], List[ReturnType[F]]]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> List[ReturnType[F]]: return [f(*args, **kwargs)] return inner ```
#3 Typing a decorator that unwraps a list return type
``` def unwrapping_decorator( f: Callable[TParams, List[TReturn]] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs)[0] return inner ```
vs.
``` def unwrapping_decorator( f: Callable[ParametersOf[F], List[TReturn]] ) -> Callable[ParametersOf[F], TReturn]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> TReturn: return f(*args, **kwargs)[0] return inner ```
#4 Typing a class that is a container for a callable
``` class FunctionContainer(Generic[TParams, TReturn]): f : F def __init__(self, f: Callable[TParams, TReturn]) -> None: self.f = f def call( self, *args: TParams.args, **kwargs: TParams.kwargs ) -> TReturn: return self.f(*args, **kwargs) ```
vs.
``` class FunctionContainer(Generic[F]): f : F def __init__(self, f: F) -> None: self.f = f def call( self, *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return self.f(*args, **kwargs) ```
In my opinion, case #1 is the best one for the proposed alternative, and case #3 is the worst one. Case #3 is odd to read because F is not actually referring to any callable. It’s just being used as a container for the ParameterSpecification. I think that the weakness in case #3 highlights what exactly the difference in philosophy between these two approaches is. ParamSpec was born out of the existing approach to Python typing which so far has avoided supporting operators, whether user-defined or built-in, in favor of destructuring. ParametersOf was inspired by TypeScript, which has a type system with extensive use of user-defined operators. Case #3 would be much more ergonomic if you could define an operator RemoveList[List[X]] = X and then you could just return Callable[ParametersOf[F], RemoveList[ReturnType[F]]]. Without that, you unfortunately get into a situation where you have to use a F-variable like a ParamSpec, in that you never actually bind the return type. Overall, the oddness of Case #3 in the ParametersOf alternative is the biggest issue with the ParametersOf proposal that we've heard internally. Combining destructuring with operators does seem to genuinely cause some ergonomics/readability issues.
Another issue we've heard raised from Python users we've shown this to is that it would potentially be confusing to have multiple ways to spell the same equivalent type i.e. F === Callable[ParametersOf[F], ReturnType[F]].
The big win for the alternative, beyond the concision of case #1, is that it would avoid introducing a new kind of type variable could potentially make further extensions to the language more easy to specify.
I can see the merits of both syntaxes, so I'd love to get some feedback from the community about folks' preferences before we proceed with the approval process.
Thanks again for your support and sponsorship!
Best, Mark Mendoza _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
I do not like case #3 at all. Applying a type operator to what feels like an unbound type variable seems very iffy to me. Combined with adding a new concept of type operators, my inclination is for the original proposal. The suggested big win for the type operator approach ("could potentially make further extensions to the language more easy to specify") is very speculative, so I'd want to see more detail along those lines if we wanted to move in that direction. Another potential argument in that direction is that "we are going to (with high probability) want to add type operators anyway for X, Y, Z important reasons". Overall I like this PEP and am fine with the decision to postpone consideration of changes of the arguments with maybe one caveat: I'd like to be at least reasonably confident that this mechanism can be *extended* into handling parameter modification. It would be unfortunate if we end up with a ton of distinct ways to type decorators. On Wed, Feb 5, 2020 at 7:48 PM Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi Guido, Thanks for volunteering as a sponsor!
I think there is one more issue I'd like to get feedback on: a more thorough discussion of Ran Benita's ParametersOf proposal. After some internal discussion, we realized that the more apt analogy to TypedScript would be of an operator that was defined on other types, not names. If that type is a type variable, you can express everything you can in ParamSpecs in terms of ParametersOf operations.
The trick is defining what the bound on the type variable has to be. So far we have been avoiding “late incompatibility”, or errors that occur during type variable instantiation. This means that we need to somehow be able to verify that ParametersOf[X] will actually be able to extract a parameter specification from any X that would be substituted into it. To achieve this, we need a bound for the type variable that includes exactly all callables. Currently there are no types in the type system that fit that bill while not introducing an “Any”. The closest thing we have is Callable[..., object], but our current semantic for that is that we permit it to be called with any arguments, unsoundly.
Therefore this proposal requires the definition of a new type, Function. Function is callable with, and only with ParametersOf[F]. ParametersOf can only operate on type variables with precisely this bound. This means we’re just introducing a new bound and a new operator (one more if you also want ReturnType[F], but that’s actually not strictly necessary, see case #3 below), rather than a new kind of type variable.
To demonstrate the pros and cons of this alternative, here are some examples of translating between the two forms:
In all of the following examples:
``` TParams = ParamSpec("TParams") TReturn = TypeVar("TReturn") F = TypeVar("F", bound=Function) ```
#1 Typing a type-preserving decorator
``` def no_change_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs) return inner ```
vs.
``` def no_change_decorator(f: F) -> F: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return f(*args, **kwargs) return inner ```
#2 Typing a decorator that wraps the return type in a List
``` def wrapping_decorator( f: Callable[TParams, TReturn] ) -> Callable[TParams, List[TReturn]]: def inner( *args: TParams.args, **kwargs: TParams.kwargs ) -> List[TReturn]: return [f(*args, **kwargs)] return inner ```
vs.
``` def wrapping_decorator( f: F ) -> Callable[ParametersOf[F], List[ReturnType[F]]]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> List[ReturnType[F]]: return [f(*args, **kwargs)] return inner ```
#3 Typing a decorator that unwraps a list return type
``` def unwrapping_decorator( f: Callable[TParams, List[TReturn]] ) -> Callable[TParams, TReturn]: def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> TReturn: return f(*args, **kwargs)[0] return inner ```
vs.
``` def unwrapping_decorator( f: Callable[ParametersOf[F], List[TReturn]] ) -> Callable[ParametersOf[F], TReturn]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> TReturn: return f(*args, **kwargs)[0] return inner ```
#4 Typing a class that is a container for a callable
``` class FunctionContainer(Generic[TParams, TReturn]): f : F def __init__(self, f: Callable[TParams, TReturn]) -> None: self.f = f def call( self, *args: TParams.args, **kwargs: TParams.kwargs ) -> TReturn: return self.f(*args, **kwargs) ```
vs.
``` class FunctionContainer(Generic[F]): f : F def __init__(self, f: F) -> None: self.f = f def call( self, *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return self.f(*args, **kwargs) ```
In my opinion, case #1 is the best one for the proposed alternative, and case #3 is the worst one. Case #3 is odd to read because F is not actually referring to any callable. It’s just being used as a container for the ParameterSpecification. I think that the weakness in case #3 highlights what exactly the difference in philosophy between these two approaches is. ParamSpec was born out of the existing approach to Python typing which so far has avoided supporting operators, whether user-defined or built-in, in favor of destructuring. ParametersOf was inspired by TypeScript, which has a type system with extensive use of user-defined operators. Case #3 would be much more ergonomic if you could define an operator RemoveList[List[X]] = X and then you could just return Callable[ParametersOf[F], RemoveList[ReturnType[F]]]. Without that, you unfortunately get into a situation where you have to use a F-variable like a ParamSpec, in that you never actually bind the return type. Overall, the oddness of Case #3 in the ParametersOf alternative is the biggest issue with the ParametersOf proposal that we've heard internally. Combining destructuring with operators does seem to genuinely cause some ergonomics/readability issues.
Another issue we've heard raised from Python users we've shown this to is that it would potentially be confusing to have multiple ways to spell the same equivalent type i.e. F === Callable[ParametersOf[F], ReturnType[F]].
The big win for the alternative, beyond the concision of case #1, is that it would avoid introducing a new kind of type variable could potentially make further extensions to the language more easy to specify.
I can see the merits of both syntaxes, so I'd love to get some feedback from the community about folks' preferences before we proceed with the approval process.
Thanks again for your support and sponsorship!
Best, Mark Mendoza _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
Thanks for the feedback. I guess there's one action item for Mark below: On Tue, Feb 18, 2020 at 2:58 PM Michael Sullivan <sully@msully.net> wrote:
Overall I like this PEP and am fine with the decision to postpone consideration of changes of the arguments with maybe one caveat: I'd like to be at least reasonably confident that this mechanism can be *extended* into handling parameter modification. It would be unfortunate if we end up with a ton of distinct ways to type decorators.
And with parameter modification I presume you mean that eventually we'll want to be able to describe decorators that add/remove parameters, like the original issue (https://github.com/python/mypy/issues/3157). In particular, I could imaging a decorator that turns printf() into fprintf() and another that does the opposite (sorry for the C analogy). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Hi Guido and Michael, Thanks for taking a look at the alternative! I agree with your overall weighing of the pros and cons of both approaches, and think that we should go ahead with the existing proposal (`ParamSpec`). I will put an edited version of my message above into the rejected alternatives section of the PEP, and summarize the points folks have made about why we don't want to go that way. As for parameter modification, I actually can announce that we have a working implementation of basic parameter modification as of today with (https://github.com/facebook/pyre-check/commit/04d4401ee26f92c5b2bb9444d9c570...). I think this is the extent of what kind of parameter modification will be possible without significantly extending ParamSpecs, so it's worth going into what it can do now. Big picture we can now safely support adding and removing anonymous arguments from the front of a ParamSpec. This looks something like this: Adding arguments: ``` from typing import ParamSpec from typing.type_variable_operators import Concatenate TParams = ParamSpec("TParams") def add_on_arguments(f: Callable[TParams, str]) -> Callable[Concatenate[str, bool, TParams], int]: def inner(first: str, second: bool, /, *args: TParams.args, **kwargs: TParams.kwargs) -> int: use(first, second) s = f( *args, **kwargs) return int(s) return inner ``` Removing arguments ``` def remove_arguments(f: Callable[Concatenate[int, bool, TParams], str]) -> Callable[TParams, int]: def inner( *args: TParams.args, **kwargs: TParams.kwargs) -> int: s = f(75, True, *args, **kwargs) return int(s) return inner ``` Transforming the type of a finite number of arguments ``` def change_arguments(f: Callable[Concatenate[int, bool, TParams], str]) -> Callable[Concatenate[float, string, TParams], int]: def inner( first: float, second: string, /, *args: TParams.args, **kwargs: TParams.kwargs) -> int: use(first, second) s = f(75, True, *args, **kwargs) return int(s) return inner ``` What this approach doesn't support is creating new named arguments, or interacting with keyword-only arguments. These are still problematic for the reasons outlined in the rejected alternatives section. With that being said: * Do you think this is sufficient flexibility? * Do you think this behavior should be included in the PEP or postponed to a follow up?
That sounds reasonable to add to the current PEP. I haven’t thought about enough about whether it would be sufficient for most use cases. Does https://github.com/python/mypy/issues/3157 have use cases it couldn’t handle? On Thu, Feb 20, 2020 at 17:36 Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi Guido and Michael, Thanks for taking a look at the alternative! I agree with your overall weighing of the pros and cons of both approaches, and think that we should go ahead with the existing proposal (`ParamSpec`). I will put an edited version of my message above into the rejected alternatives section of the PEP, and summarize the points folks have made about why we don't want to go that way.
As for parameter modification, I actually can announce that we have a working implementation of basic parameter modification as of today with ( https://github.com/facebook/pyre-check/commit/04d4401ee26f92c5b2bb9444d9c570...). I think this is the extent of what kind of parameter modification will be possible without significantly extending ParamSpecs, so it's worth going into what it can do now.
Big picture we can now safely support adding and removing anonymous arguments from the front of a ParamSpec. This looks something like this:
Adding arguments: ``` from typing import ParamSpec from typing.type_variable_operators import Concatenate TParams = ParamSpec("TParams") def add_on_arguments(f: Callable[TParams, str]) -> Callable[Concatenate[str, bool, TParams], int]: def inner(first: str, second: bool, /, *args: TParams.args, **kwargs: TParams.kwargs) -> int: use(first, second) s = f( *args, **kwargs) return int(s) return inner ```
Removing arguments ``` def remove_arguments(f: Callable[Concatenate[int, bool, TParams], str]) -> Callable[TParams, int]: def inner( *args: TParams.args, **kwargs: TParams.kwargs) -> int: s = f(75, True, *args, **kwargs) return int(s) return inner ```
Transforming the type of a finite number of arguments ``` def change_arguments(f: Callable[Concatenate[int, bool, TParams], str]) -> Callable[Concatenate[float, string, TParams], int]: def inner( first: float, second: string, /, *args: TParams.args, **kwargs: TParams.kwargs) -> int: use(first, second) s = f(75, True, *args, **kwargs) return int(s) return inner ```
What this approach doesn't support is creating new named arguments, or interacting with keyword-only arguments. These are still problematic for the reasons outlined in the rejected alternatives section.
With that being said: * Do you think this is sufficient flexibility? * Do you think this behavior should be included in the PEP or postponed to a follow up? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
-- --Guido (mobile)
Mark's basic parameter modification would be sufficient for my use-case, the `@composite` decorator in Hypothesis ( https://hypothesis.readthedocs.io/en/latest/data.html#composite-strategies). It looks like I'd annotate that as # T is a typevar, Ex is a typevar, SearchStrategy is a generic type covariant in Ex Draw = Callable[[SearchStrategy[T]], T] def composite(f: Callable[Concatenate[Draw, TParams], Ex]) -> Callable[[TParams], SearchStrategy[Ex]]: ... If I'm correct, that makes it sufficiently flexible to be actually useful for me (and downstream Hypothesis users), so I'd favor it going into the PEP. Cheers, Zac On Fri, 21 Feb 2020 at 15:22, Guido van Rossum <guido@python.org> wrote:
That sounds reasonable to add to the current PEP. I haven’t thought about enough about whether it would be sufficient for most use cases. Does https://github.com/python/mypy/issues/3157 have use cases it couldn’t handle?
On Thu, Feb 20, 2020 at 17:36 Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
Hi Guido and Michael, Thanks for taking a look at the alternative! I agree with your overall weighing of the pros and cons of both approaches, and think that we should go ahead with the existing proposal (`ParamSpec`). I will put an edited version of my message above into the rejected alternatives section of the PEP, and summarize the points folks have made about why we don't want to go that way.
As for parameter modification, I actually can announce that we have a working implementation of basic parameter modification as of today with ( https://github.com/facebook/pyre-check/commit/04d4401ee26f92c5b2bb9444d9c570...). I think this is the extent of what kind of parameter modification will be possible without significantly extending ParamSpecs, so it's worth going into what it can do now.
Big picture we can now safely support adding and removing anonymous arguments from the front of a ParamSpec. This looks something like this:
Adding arguments: ``` from typing import ParamSpec from typing.type_variable_operators import Concatenate TParams = ParamSpec("TParams") def add_on_arguments(f: Callable[TParams, str]) -> Callable[Concatenate[str, bool, TParams], int]: def inner(first: str, second: bool, /, *args: TParams.args, **kwargs: TParams.kwargs) -> int: use(first, second) s = f( *args, **kwargs) return int(s) return inner ```
Removing arguments ``` def remove_arguments(f: Callable[Concatenate[int, bool, TParams], str]) -> Callable[TParams, int]: def inner( *args: TParams.args, **kwargs: TParams.kwargs) -> int: s = f(75, True, *args, **kwargs) return int(s) return inner ```
Transforming the type of a finite number of arguments ``` def change_arguments(f: Callable[Concatenate[int, bool, TParams], str]) -> Callable[Concatenate[float, string, TParams], int]: def inner( first: float, second: string, /, *args: TParams.args, **kwargs: TParams.kwargs) -> int: use(first, second) s = f(75, True, *args, **kwargs) return int(s) return inner ```
What this approach doesn't support is creating new named arguments, or interacting with keyword-only arguments. These are still problematic for the reasons outlined in the rejected alternatives section.
With that being said: * Do you think this is sufficient flexibility? * Do you think this behavior should be included in the PEP or postponed to a follow up? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
-- --Guido (mobile) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
The only use cases mentioned in that thread that can't be handled are `builtins.map`, which we've decided to go the `ListVariadic` route with, and an analogous mapping function described by @rggjan, that again would be best served by `ListVariaidic`s. The only thing that won't be served by either this kind of modification or ListVariadics, is adding named arguments, which I have not yet seen a request for. I can try to get out a revised version of the PEP today :). Best, Mark Mendoza
Can you link to a description of the ListVariadic proposal? On Fri, Feb 21, 2020 at 10:49 Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
The only use cases mentioned in that thread that can't be handled are `builtins.map`, which we've decided to go the `` route with, and an analogous mapping function described by @rggjan, that again would be best served by `ListVariaidic`s.
The only thing that won't be served by either this kind of modification or ListVariadics, is adding named arguments, which I have not yet seen a request for.
I can try to get out a revised version of the PEP today :).
Best, Mark Mendoza _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
-- --Guido (mobile)
All I have right now is still the talk from the last meetup (https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variab...). I'm planning on working on a PEP for the basic version of that (just including ListVariadic. Map and Concatenate), after this one.
participants (5)
-
Guido van Rossum -
Mark Mendoza -
Michael Sullivan -
Ran Benita -
Zac Hatfield Dodds