Question about PEP 612
I have some functions that from a typing perspective look like: def call_in_special_context(f, *args, **kwargs): # In reality, runs 'f' in a thread or something return f(*args, **kwargs) An example in the stdlib would be https://docs.python.org/3/library/contextvars.html#contextvars.Context.run Of course, right now, there's no good way to add types to this. But with PEP 612, it seems like we could write: Ps = ParameterSpecification("Ps") R = TypeVar("R") def call_in_special_context(f: Callable[Ps, R], *args: Ps.args, **kwargs, Ps.kwargs) -> R: ... None of the examples is the PEP actually look like this though – they all involve referencing Ps.args and Ps.kwargs inside the function/class scope that uses Ps, while here we need to reference them right in the same args list. So, first question: is the example above something that would be supported under PEP 612? (And if so, a suggestion would be to add some examples like that to the text :-).) Second question: there's also a convention that shows up in e.g. asyncio, of passing through *args, while reserving kwargs for the wrapper. For example: # https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.call_s... def call_soon(self, callback, *args, context=None): ... In PEP 612 notation, this method has type: def call_soon(self, callback: Callable[Ps, Any], *args: Ps.args, context=None): But I think that this is *not* allowed by PEP 612, because "These operators can only be used together, as the annotated types for *args and **kwargs", and the 'def baz' example. So even with PEP 612, we still can't type loop.call_soon. Is that correct? Third question: suppose I have a function that captures some kwargs, but passes through unrecognized ones: def call_in_special_context(f, *args, **kwargs, special_kwarg): do_something_else_with(special_kwarg) return f(*args, **kwargs) In this case, you might expect to write: def call_in_special_context(f: Callable[Ps, R], *args: Ps.args, **kwargs, Ps.kwargs, special_kwarg: SomeType) -> R: The semantics would be that at call sites for call_in_special_context, the type checker would check that 'special_kwarg' matches the given type, and that all other kwargs are valid for 'f'. Is this legal/supported? Thanks! -n P.S.: I'm not subscribed to the list; trying out posting through the mailman 3 web interface thing. So if you could CC me on replies that would be helpful :-)
Hi Nathaniel! Thank you for the interest and feedback on the PEP!
they all involve referencing Ps.args and Ps.kwargs inside the function/class scope that uses Ps, while here we need to reference them right in the same args list.
I believe there actually is one instance of this kind of function in the PEP: ``` One additional form that we want to support is functions that pass only a subset of their arguments on to another function. To avoid shadowing a named or keyword only argument in the ParameterSpecification we require that the additional arguments be anonymous arguments that precede the *args and *kwargs def call_n_times( __f: Callable[TParams, None], __n: int, *args: TParams.args, **kwargs: TParams.kwargs, ) -> None: for x in range(__n); __f(*args, **kwargs) ``` I agree this isn't exactly prominent in the flow of the document. In my forthcoming draft, this is highlighted more forcefully, as it forms the basis of the `Concatenate` syntax. Note that this also introduces the constraint that the preceding parameters be positional-only. I'll get more into the details of why when answering your third question.
So even with PEP 612, we still can't type loop.call_soon. Is that correct?
This is correct, but we have a plan for handling these scenarios :). The plan for these is to introduce another kind of variable, the ListVariadic. You can see more details of the plan for these here: https://github.com/facebook/pyre-check/blob/master/docs/Variadic_Type_Variab...
def call_in_special_context(f, *args, **kwargs, special_kwarg):
This example doesn't actually pass the python parser. The rule is that nothing can come after a **kwargs parameter. However, if you consider the corrected spelling: ``` def call_in_special_context(__f: Callable[Ps, R], *args: Ps.args, special_kwarg: SomeType, **kwargs, Ps.kwargs) -> R: ``` This is explicitly banned by the PEP: "we require that the additional arguments be anonymous arguments that precede the *args and *kwargs". The rationale for this is that otherwise we could end up substituting our way into impossible callable types: in this example, if the callable passed in for `__f` also had a `special_kwarg` keyword only parameter." Consider the following example: ``` def wrapper(f: Callable[TParams, int]) -> Callable[TParams, int]: def inner(*args: TParams.args, **kwargs: TParams.args) -> int: # fails at runtime with TypeError: call_in_special_context() got multiple values for keyword argument 'special_kwarg' return call_in_special_context(f, *args, special_kwarg=42, **kwargs) return inner @wrapper def ohno(*, special_kwarg: str) -> int: return -1 ohno("A") ``` We can avoid this situation with positional-only parameters, so those are the only ones we accept. Hope that answers your questions! Best, Mark
[I subscribed to the list now... unfortunately the mailman 3 web interface turns out to be pretty much unusable for me.] Mark Mendoza wrote:
Hi Nathaniel! Thank you for the interest and feedback on the PEP!
Thanks for working on it! :-)
def call_in_special_context(f, *args, **kwargs, special_kwarg): This example doesn't actually pass the python parser. The rule is that nothing can come after a **kwargs parameter. However, if you consider the corrected spelling: def call_in_special_context(__f: Callable[Ps, R], *args: Ps.args, special_kwarg: SomeType, **kwargs, Ps.kwargs) -> R:
This is explicitly banned by the PEP: "we require that the additional arguments be anonymous arguments that precede the args and kwargs".
Note: I think the PEP text is currently pretty unclear about what it means by "anonymous arguments". In 3.8+ we have syntax for those (via PEP 570), but you're not using that syntax here, and it would be unfortunate if this feature were restricted to projects that have adopted 3.8+ only syntax...
The rationale for this is that otherwise we could end up substituting our way into impossible callable types: in this example, if the callable passed in for __f also had a special_kwarg keyword only parameter." Consider the following example: def wrapper(f: Callable[TParams, int]) -> Callable[TParams, int]: def inner(*args: TParams.args, **kwargs: TParams.args) -> int: # fails at runtime with TypeError: call_in_special_context() got multiple values for keyword argument 'special_kwarg' return call_in_special_context(f, *args, special_kwarg=42, **kwargs) return inner
@wrapper def ohno(*, special_kwarg: str) -> int: return -1
ohno("A")
We can avoid this situation with positional-only parameters, so those are the only ones we accept.
Hmmmmmm. I see how that code raises an error, but I don't really understand the connection between that and disallowing additional kwargs. It seems to me that if we follow the rules of the PEP and define: def call_in_special_context(__f: Callable[Ps, R], *args: Ps.args, **kwargs: Ps.kwargs) -> R: ... ...then the PEP does allow your example, but you still get the same error at runtime. Also, even if we fail to catch this error, is that a problem? Python type annotations don't try to guarantee that all errors will be caught statically. The reason I'm asking about this case is that in Trio, we have a lot of 'call_in_special_context' type functions, and are trying to figure out the best convention for how to pass arguments to 'call_in_special_context' vs 'f'. Currently we follow the asyncio convention where positional args go to 'f' while kwargs go to 'call_in_special_context', but this turns out to be unsustainable – it's a constant irritant for users, we get complaints all the time, and it has bad effects like encouraging API designers to avoid using kwargs in their APIs. So we need something else, and most options are pretty awkward. The frontrunner so far is to declare that __dunder__ kwargs modify *how* the function is called, and regular kwargs are passed through to the function, e.g.: def run_in_thread(f, *args, __thread_limit__=None, **kwargs): raise_if_has_unrecognized_dunders(kwargs) ... [This would be clearer if we could write it 'def run_in_thread(f, *args, **kwargs, __thread_limit__=None)', but oh well.] It would be very convenient if we could write: def run_in_thread(f: Callable[TP, TR], *args: TP.args, __thread_limit__: Optional[int]=None, **kwargs: TP.kwargs) -> TR: ... def some_func(x: int): ... run_in_thread(f, x="hello") and get a nice error from the type checker complaining that we passed 'x' as a str but expected int. I feel like 99% of the value of parameter types here is that they let you do *any* type-checking on 'x'. With PEP 570 as currently written, 'run_in_thread' has to be typed with 'Any's everywhere, so you don't catch *any* errors, which seems clearly worse than catching *most* errors but still missing a few exotic cases with kwarg collisions.
Note: I think the PEP text is currently pretty unclear about what it means by "anonymous arguments". In 3.8+ we have syntax for those (via PEP 570), but you're not using that syntax here, and it would be unfortunate if this feature were restricted to projects that have adopted 3.8+ only syntax...
For the record, the syntax for anonymous arguments I'm using here is defined in PEP 484 (https://www.python.org/dev/peps/pep-0484/#positional-only-arguments). I'll try to make that more clear in my new draft. Furthermore, in my forthcoming draft, these are no longer required to be declared as positional-only, but rather just be used as if they were positional only. I don't think that's particularly relevant to the concerns you have, but is worth mentioning. The new draft still bans adding keyword only arguments, and tries to do so a bit more clearly.
...then the PEP does allow your example, but you still get the same error at runtime.
I don't believe that it would be accepted. If you don't add on `special_kwarg` then it would be an error to call the function with it, avoiding the problem. If I'm misunderstanding you, then could you give me a complete example of what would be accepted by the PEP but would still have a TypeError at runtime.
Also, even if we fail to catch this error, is that a problem? Python type annotations don't try to guarantee that all errors will be caught statically.
One of our long term goals with Pyre is to support an expressive yet sound type system so that we can eventually make those guarantees. Therefore, when introducing new features like this, we're expressly trying to not introduce unsoundnesses that we will eventually have to come back and patch out later. This means that we aren't going to be able to support all use cases with these features. Ultimately the issue of adding named parameters onto ParamSpecs has more problems than just soundness issues, there are implementation and syntax concerns. * implementing positional only concatenation is already pretty tricky, and adding names along for the ride makes implementation significantly more challenging. For a PEP that hasn't been implemented by any other type checkers yet, this imposes a significant burden for adoption. * Spelling the type of a function like `run_in_thread` would introduce another scenario where we would require you to drop down into callback protocol syntax, which is already unfortunate and a source of confusion for users. For more details, I have a draft version of the reworked PEP up here: (https://github.com/python/peps/pull/1424), and in a fully-commentable form here: (https://github.com/mrkmndz/peps/pull/1). Overall I think this is a reasonable request, and I think it's possible to soundly support it via "banned name" constraints on ParamSpecs. However I think that it is complex enough to merit a separate PEP.
participants (2)
-
Mark Mendoza -
Nathaniel J. Smith