Mailman 3 June 2021 - Typing-sig

Re: Compact syntax for Callable: `(Arg1, Arg2) -> Ret`
by Daniel Mouritzen June 19, 2021

June 19, 2021

Thanks for the feedback Jelle, Rebecca and Eric. It's true as you all point out that the same callback function could be invoked in multiple different ways, but I don't remember coming across this much in practice. A real-life example would be helpful for this discussion. Anyway I don't think the shorthand syntax needs to cover all possible uses if that means it becomes less useful for the common cases as a result. Wouldn't it make sense to use callback protocols as Eric suggested for these complex cases and use the shorthand for the more common cases? @Jelle > For example, a callback could be called sometimes with named and sometimes with positional arguments; an argument with a default is sometimes omitted and sometimes is not; et cetera. I don't think it's common to call the same callback sometimes with named and sometimes with positional arguments. When adding type annotations to code that does this, the callback invocations can just be updated to be consistent with each other by adding or removing argument names. This causes no loss of generality. As to the arguments with default values, yes I suppose allowing the previously proposed syntax for specifying that an argument must be optional (i.e. (foo: int = ...)) in the hybrid syntax would be needed to cover those cases. @Rebecca > Either way, it feels very inadequate to have large classes of functions, Python's main callable objects, that are inexpressible as Callable types. The proposed syntax can describe any function *invocation*, but no not all valid invocations of a specific function. > This is especially relevant because any sort of function can be used as a value, for example as a default argument, and many untyped or partially-typed programs take a more duck-typed approach; "here's the default function but if you pass something that takes the same arguments that will be used instead." This is exactly why the callable type should describe how the function is *used* not how it might be *defined*. With the hybrid syntax, the simplest valid type annotation will also automatically be the most general one as it defines only the minimal requirements the passed callback needs to fulfill. > Python programs do all sorts of funny things with functions and their arguments, and we should be wary of limiting the types of functions and callables that can be expressed in the Python type system — a limited type system makes many legacy libraries untypable (for example, if it can't provide a more accurate annotation than "Callable") and restricts which "Pythonic" APIs (which tend to be extremely dynamic) can be added to new, typed Python programs. This is why I brought up PEP 612. It tries to address exactly these dynamically defined functions and such. Using the hybrid syntax or stub-like syntax wouldn't change anything about this, neither of them support what you're describing (as they are defined currently). @Eric > I disagree with the premise that *function type annotations* and *callable type annotations* are different or have different requirements. Treating them differently would lead to unnecessary and undesirable inconsistencies and limitations. A type annotated function describes all the valid ways in which that specific function can be invoked. A callable type describes the requirements a function would need to fulfill in order to be used in a given context. In other words, a function type describes all usages of a given function, a callable type describes all functions that match a given usage (or set of usages, point taken). If a callable type was simply a copy of the definition of some specific usable callback function, then the callable type would be overly specific and might exclude other functions that would've worked fine. Unless we decide (as for the stub-like syntax) that e.g. a positional-only argument in a callable can match either a positional-only or positional-or-keyword argument in a function type. And that the names of positional arguments don't matter. And that the function can have additional optional arguments that are not in the callable type. But these decisions change the meaning of the syntax and thus might harm more than the familiar-looking syntax helps. > Yes, a callback annotation specifies how the callback must be called, but that doesn't mean it must limit the caller to only one way of invoking the callback. A callback can be invoked in multiple ways just like a regular function. When a callback is invocable in multiple ways, it is up the a type checker to verify that any supplied callback is compatible with all possible invocation techniques described in the annotation. This is already implemented in type checkers today with [callback protocols]( https://www.python.org/dev/peps/pep-0544/#callback-protocols). Note that none of the proposed syntaxes would be sufficient to support all possible callback usages. The callback could be used in a way that would require overloads. In this case you would need to use a callback protocol, which I think is fine. - Daniel PS: Sorry for starting a new thread with my previous email, I'm new to this mailing list stuff

1 0

Re: Compact syntax for Callable: `(Arg1, Arg2) -> Ret`
by Daniel Mouritzen June 19, 2021

June 19, 2021

I think it's important to be aware of the subtle differences between *function type annotations* and *callable types*. The type annotations on a function definition (or stub) define *all* the possible ways the function can be called. Callable types on the other hand define *one* particular way a function can be called. When you're writing type annotations for a callback argument, you base it on how the callback will be *called*, not how any particular function is *defined*. These differences also mean we have slightly different requirements from the syntax. A function definition can have three types of arguments: positional-or-keyword arguments (the default), positional-only arguments and keyword-only arguments. For a callable type on the other hand there are two types (which correspond to how the function will actually be called): positional arguments and keyword arguments. Due to these differences I don't think the "stub syntax" makes sense. In order to make it unambiguous we would have to disallow positional-or-keyword arguments, so you would always need to use / or * or both. And even then it wouldn't be quite intuitive to newcomers that a callable type like (a: int, /) -> bool would also match a function defined as def foo(bar: int, baz: int = 0) -> bool. It's better to use new intuitive syntax than to use existing syntax in unintuitive ways IMO. The hybrid syntax makes more sense, but I would not allow * and / in this syntax since they have no useful meaning in this context. There is also no point being able to specify that a named argument should be optional - it will either be given or not when calling the function, so it's irrelevant whether the implementation provides a default value. Variadic arguments are still needed, but giving them names would have no meaning. Here are a few options I can think of for how they could be specified: - (*: int, **: Any) - (*int, **Any) - (*int, *: Any) - (*int, **: Any) I think the second option would be preferable, because it matches the syntax for unpacking. On the other hand, the last two keep the convention of using ":" for keyword arguments. To be clear about the meaning of this syntax: Any matching function *has to have* corresponding *args and **kwargs in its definition - variadic arguments in the callable type would never match non-variadic required arguments in the function definition. To give an example of how the hybrid syntax could be used in practice, consider the following function taking a callback argument: ``` def foo(x: int, callback: ??) -> bool: y = "bar" return callback(x, y, asdf=6.5) ``` From this, writing the type annotation for the callback is straightforward: ``` callback: (int, str, asdf: float) -> bool ``` And one with variadic arguments: ``` def foo(x: Sequence[int], callback: (*int, **Any) -> bool, **kwargs: Any) -> bool: return callback(*x, **kwargs) ``` I'm also in favor of the proposed syntax of (...) -> T to cover the case where it is not statically known how the function might be called, e.g. for most decorators. We should also consider how the new syntax will interact with PEP 612 (ParamSpec). Should the following modified example from the PEP be allowed? ``` P = ParamSpec("P") R = TypeVar("R") def add_logging(f: P -> R) -> (P -> Awaitable[R]): ... ``` Or perhaps even something like this? ``` def with_request(f: (Request, *P.args, **P.kwargs) -> R) -> (P -> R): ... ``` - Daniel

4 3

Re: Compact syntax for Callable: `(Arg1, Arg2) -> Ret`
by Mehdi Drissi June 10, 2021

June 10, 2021

I dislike the syntax of (str, int, int -> int). This can be read as (str, int, int) -> int or as tuple[str, int, int -> int]. The second is how some languages (haskell) would read it as. One question is it fine to nest callables and how does associativity/precedence work for this? int -> int -> bool fine? That's a fairly common pattern in function languages and it'd be nice to stay similar to them. I think of -> as binding weakly and that int -> int -> bool == int -> (int -> bool) (right associativity). Right associativity is needed if you want curry/uncurrying to not need a lot of parentheses. I weakly prefer -> over =>, but either is a great improvement over Callable[] stacks. Also while keyword/other callable details would be nice, I'd like some simple solution like this for the most common easy cases. Any solution that supports more complex callables I'd want to allow something as simple as int -> bool to not need more stuff.

3 2

Typing of sentinel objects
by Tal Einat June 9, 2021

June 9, 2021

Hi, I'm working on PEP 661: Sentinel Values [1]. One thing I think it should get right is that it should be possible to have strict type annotations for sentinel values. Many sentinel implementations don't support this. I see that this has come up before on this mailing list [2] and as an issue in the GitHub repo [3]. Using Literal and Final, as suggested in the discussions of the PEP [4] and as suggested by Guido on the aforementioned GitHub issue [5], seems to be a very nice approach. 1. What would be needed to make that work with mypy? 2. What should the process be, in terms of what the PEP defines and what is later implemented in mypy and similar tools? Please note that I'm far from well-versed on typing in Python, so assume I know only the very basics. - Tal Einat [1]: https://www.python.org/dev/peps/pep-0661/ [2]: https://mail.python.org/archives/list/typing-sig@python.org/thread/TI5Y2HTV… [3]: https://github.com/python/typing/issues/689 [4]: https://discuss.python.org/t/pep-661-sentinel-values/9126/2 [5]: https://github.com/python/typing/issues/689#issuecomment-561425237

6 18

Next tensor typing meeting
by Matthew Rahtz June 2, 2021

June 2, 2021

Hi all, We'd tentatively set the date for the next tensor typing meeting to be this coming Monday, the 7th of June. As it happens though, I'm not sure we have much to report/discuss this time - the main thing Pradeep and I were working on since last time was the talk and associated materials for the Typing Summit (great to see so many people there, and thanks again Shannon and Alfonso for organising!). Unless anyone else has something they'd like to talk about, I'd say let's skip meeting this month and push the next meeting out until Tuesday July 7th. Cheers, Matthew

1 0

Introspecting a generic type at runtime
by Paul Moore June 2, 2021

June 2, 2021

I'm trying to write a function that introspects the types of fields in a dataclass (this is to automatically generate SQL table creation statements). However, I have a problem when dealing with generic types. In particular, if I have a field with the type `Optional[int]`, I want to create a SQL column of type `INTEGER`, but without a `NOT NULL` constraint (so nulls are allowed). But I can't work out how to introspect the type of the field, to determine that it's based on `int`. ``` from dataclasses import dataclass, fields from typing import Optional @dataclass class Example: mandatory: int optional: Optional[int] for f in fields(Example): print(f.type) ``` This displays: ``` <class 'int'> typing.Optional[int] ``` But from `f.type`, I can't find any way to confirm it's an optional type (issubclass gives "TypeError: Subscripted generics cannot be used with class and instance checks") and I can't recover the "underlying" integer type. I understand that introspecting types at runtime is not the key use case for annotations, but with the availability of dataclasses in the standard library, it seems like this sort of usage is only going to become more common, as it's an extremely obvious way to handle this type of requirement, and the alternative, using field metadata, is very verbose: ``` >>> @dataclass ... class Example: ... mandatory: int = field(metadata={"sql_type": "INTEGER"}) ... optional: Optional[int] = field(metadata={"sql_type": "INTEGER", "allow_null": True}) ``` Is there a reasonable way to do this? Or is it something that's likely to be added in the foreseeable future?

5 5