note: I wasn't thinking -- typeshed, of course, has a lot more than the standard lib. But it's still a collection of widely used somewhat general purpose libraries. So I think my hypothesis is still valid. -CHB On Sun, Dec 19, 2021 at 8:54 PM Christopher Barker <pythonchb@gmail.com> wrote:

A question that came up for me is:

How common is it to need to use Callable for type hints? particularly complex versions, specifying what parameters the Callable takes? A more compact and easier to read syntax is nice, but not very important if it isn't used much.

My first thought on this was that I can't remember a single time that I wrote production code that took a Callable as a function parameter -- or returned one -- OK maybe a few times, but it's certainly rare in my production code.

So I looked in the PEP to see if that issue was addressed, and indeed it is:

"The Callable type is widely used. For example, as of October 2021 it was the fifth most common complex type in typeshed,"

That did surprise me, but on thinking about it, maybe not so much. It strikes me that Callable is most likely to be used in fairly low level, general purpose functions, like map(), sort(), various functions in itertools, etc. Just the sort of functions that are common in the standard library, but may not so much in production code.

I have no idea how to evaluate how common it is in production code -- maybe type hinting is common enough now that PyPi could be searched -- but even PyPi is a biased sample, as it is full of, by definition, libraries for others' use -- i.e. general purpose tools (less general that the stad lib, but still not specialty production code, which I suspect is the majority of Python code out there).

Perhaps some folks that have been type=hinting their production code bases could provide anecdotal evidence.

Anyway, if my hypothesis is correct, then it's not so bad that not-so-nice syntax is required to type hint general purpose utilities.

-CHB

-- Christopher Barker, PhD (Chris)

Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

OOI, of those 1577 Callable type hints, how many distinct Callable types? On 20/12/2021 7:52 am, Andrew Svetlov wrote:

At my job, we have 1577 Callable type hints scattered in 1063 Python files. For comparison, this codebase also has 2754 dict annotations and 1835 list ones.

On Mon, Dec 20, 2021 at 8:11 AM Christopher Barker <pythonchb@gmail.com <mailto:pythonchb@gmail.com>> wrote:

note: I wasn't thinking -- typeshed, of course, has a lot more than the standard lib.  But it's still a collection of widely used somewhat general purpose libraries. So I think my hypothesis is still valid.

-CHB

On Sun, Dec 19, 2021 at 8:54 PM Christopher Barker <pythonchb@gmail.com <mailto:pythonchb@gmail.com>> wrote:

A question that came up for me is:

How common is it to need to use Callable for type hints? particularly complex versions, specifying what parameters the Callable takes? A more compact and easier to read syntax is nice, but not very important if it isn't used much.

My first thought on this was that I can't remember a single time that I wrote production code that took a Callable as a function parameter -- or returned one -- OK maybe a few times, but it's certainly rare in my production code.

So I looked in the PEP to see if that issue was addressed, and indeed it is:

"The Callable type is widely used. For example, as of October 2021 it was the fifth most common complex type in typeshed,"

That did surprise me, but on thinking about it, maybe not so much. It strikes me that Callable is most likely to be used in fairly low level, general purpose functions, like map(), sort(), various functions in itertools, etc. Just the sort of functions that are common in the standard library, but may not so much in production code.

I have no idea how to evaluate how common it is in production code -- maybe type hinting is common enough now  that PyPi could be searched -- but even PyPi is a biased sample, as it is full of, by definition, libraries for others' use -- i.e. general purpose tools (less general that the stad lib, but still not specialty production code, which I suspect is the majority of Python code out there).

Perhaps some folks that have been type=hinting their production code bases could provide anecdotal evidence.

Anyway, if my hypothesis is correct, then it's not so bad that not-so-nice syntax is required to type hint general purpose utilities.

-CHB

-- Christopher Barker, PhD (Chris)

Python Language Consulting   - Teaching   - Scientific Software Development   - Desktop GUI and Web Development   - wxPython, numpy, scipy, Cython

-- Christopher Barker, PhD (Chris)

Python Language Consulting   - Teaching   - Scientific Software Development   - Desktop GUI and Web Development   - wxPython, numpy, scipy, Cython _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LWIXFDUG... <https://mail.python.org/archives/list/python-dev@python.org/message/LWIXFDUG...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>

-- Thanks, Andrew Svetlov

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JYZGIDEB... Code of Conduct: http://python.org/psf/codeofconduct/

20.12.21 13:42, Mark Shannon пише:

OOI, of those 1577 Callable type hints, how many distinct Callable types?

Around 15-20%. Most of them are in tests which widely use pytest fixtures, so functions taking and returning callables are common. There are around 200 occurrences in non-test code, half of them are distinct types.

As someone with use of this, would you find this useful (i.e. +1, +0)? Serhiy already said "no" in another thread. On Mon, Dec 20, 2021 at 4:38 AM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:

Perhaps Serhiy did more accurate counting, my estimate is very rough.

On Mon, Dec 20, 2021 at 2:15 PM Serhiy Storchaka <storchaka@gmail.com> wrote:

...

20.12.21 13:42, Mark Shannon пише:

...

OOI, of those 1577 Callable type hints, how many distinct Callable types?

Around 15-20%. Most of them are in tests which widely use pytest fixtures, so functions taking and returning callables are common. There are around 200 occurrences in non-test code, half of them are distinct types.

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7YXSPACK... Code of Conduct: http://python.org/psf/codeofconduct/

-- Thanks, Andrew Svetlov _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q4A4ZTJH... Code of Conduct: http://python.org/psf/codeofconduct/

On 20/12/2021 22:34, Serhiy Storchaka wrote:

20.12.21 21:28, Brett Cannon пише:

...

As someone with use of this, would you find this useful (i.e. +1, +0)? Serhiy already said "no" in another thread. In every file we import 5-10 or more names from the typing module. We still does not use PEP 585 and PEP 604 syntax, but are going to do this in near future. It could save as from importing List, Tuple, Dict, Set, Union, Optional, but we still need to import Any, Sequence, Iterable, Iterator, AsyncIterator, Awaitable, TypeVar, and sometimes AsyncContextManager, NewType, cast, overload. Special syntax for callable type hints will not save game. Sequence, Iterable, Iterator, AsyncIterator, Awaitable, AsyncContextManager are all in collections.abc not typing now. Of course it doesn't reduce total imports but it makes it less likely that you'll need to import typing.

On Mon, Dec 20, 2021 at 3:44 AM Mark Shannon <mark@hotpy.org> wrote:

Hi,

Why not make Callable usable as a function decorator?

The motivating example in the PEP is this:

def flat_map( l: list[int], func: Callable[[int], list[int]] ) -> list[int]: ....

Since, as the PEP claims, `Callable[[int], list[int]]` is hard to read, then give it a name and use regular function definition syntax.

@Callable def IntToIntFunc(a:int)->int: pass

def flat_map( l: list[int], func: IntToIntFunc ) -> list[int]: ....

To me, this seems much clearer than the proposed syntax and is more general.

It is a little longer, but unless you are playing code golf, that shouldn't matter.

It's an interesting idea! Both `@overload` and `@final` show there is precedence for having decorators have special meanings to static type checkers: https://docs.python.org/3/library/typing.html#functions-and-decorators.

Cheers, Mark.

...

Hello all,

Thanks everyone for comments on our earlier thread [1] about callable type syntax. We now have a draft PEP [2] proposing an arrow-based syntax for callable types, for example:

``` (int, str) -> bool # equivalent to Callable[[int, str], bool] ```

In support of the PEP we also have: - a reference implementation of the parser [3] to ensure the grammar is correct (tests [5], [6], [7]) - a detailed specification of planned runtime behavior [4], which is not yet in the reference implementation

We'd like to get your feedback about the PEP in general, and especially

On 16/12/2021 5:57 pm, Steven Troxler wrote: details and edge cases we need to consider regarding runtime behavior.

...

Cheers, Steven Troxler

--------- [1] Earlier python-dev thread

https://mail.python.org/archives/list/python-dev@python.org/thread/VBHJOS3LO...

...

[2] PEP 677: https://www.python.org/dev/peps/pep-0677/ [3] Reference implementation of Parser: https://github.com/stroxler/cpython/tree/callable-type-syntax--shorthand [4] Details on the runtime behavior: https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG7...

[5] Ast tests for parser changes:

https://github.com/stroxler/cpython/blob/20eb59fdca0d6d8dbe4efa3b04038c7c220...

...

[6] Easy-read tests of examples from the PEP: https://github.com/stroxler/cpython/blob/callable-type-syntax--shorthand/Lib... [7] Test sanity checking hundreds of examples pulled from typeshed:

https://github.com/stroxler/cpython/blob/callable-type-syntax--shorthand/Lib...

...

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OGACYN2X... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2BKD5YBU... Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Dec 20, 2021 at 12:44 PM <asleep.cult@gmail.com> wrote:

This is such a great idea that I think it deserves its own PEP (to compete with this one?) Let me explain. PEP 677 was created for the sole purpose of replacing typing.Callable, but there are still some other areas where function metadata is required. What if we instead introduced a function prototype that allows you to "declare" a "function" without a body.

typing example:

import typing

@typing.Callable def IntToIntFunc(a: int) -> int

def flat_map( l: list[int], func: IntToIntFunc ) -> list[int]: ...

ctypes example:

import ctypes

@ctypes.CFUNCTYPE def f(x: int) -> bool

But of course this comes with a few issues: should it be an expression and if so, should the name be optional? How can ParamSpec be handled?

Allowing 'def' without a body based on the presence or absence of a decorator sounds like it will just confuse people and cause bizarre errors if people accidentally leave out a body. Let's not go there. However, Lukasz has already proposed a very similar mechanism (with dummy body), without needing a decorator. His proposal is simply: def IntToIntFunc(a: int) -> int: ... def flat_map(l: list[int], func: IntToIntFunc) -> list[int]: # body No need for a `@Callable` decorator! -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

Yeah, making the body optional (without looking at decorators) is not acceptable either. Too easy to do by mistake (I still do this All. The. Time. :-) On Mon, Dec 20, 2021 at 2:19 PM <asleep.cult@gmail.com> wrote:

My proposal wasn't to make the body optional based on the presence of a decorator, but rather to return a "function prototype" iff the body does not exist (I probably should have made my made my own reply instead of piggybacking on his proposal). I also mentioned some form of expression to represent this, similar to lambda. Maybe a friendly error message telling the user to use a function when this thing is called would alleviate some confusion? I'm not sure how one would forget to add the function body anyway. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3HJY6VGL... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

In most of our discussions of this idea, we've assumed that we'd adopt the same semantics that callback protocols use. If we do that, then only `lambda a: 3` will type check. In order to type check both you'd have to make `a` positional-only: ``` def IntToIntFunc(a: int, /) -> int: ... ``` This is one of the reasons I think functions-as-types could be a great idea but not a good substitute for better callable syntax.

I don’t see how this helps much in situations with nested callables which are very common for decorators. I’m also unsure how this will work with situations where input and output are both callables and the signature is modified using paramspec. An example would be a decorator that adds/removes one argument. Is Concatenate[int, InttoIntFunction] a thing? I also would ideally want a very light weight way to handle simple callables like (int) -> bool. This decorator solution of @Callable seems roughly as readable as existing solution of using callback protocols or a type alias. I took a quick scan of my team’s codebase and see about 150 callable vs like 1.5k optional. Optional is fairly high as a lot of our code has many arguments that can be disabled which I think is normal for ml/numerical libraries with tons of tuning settings. The most complex callable signatures tend to be decorators/decorator factories for us. Main place I find this useful is skimming other libraries type signatures and hovering in ides. Nice hover type hints is helpful over large blocks of callable brackets. I do find this syntactic sugar nice. I’m +0.5, mainly as while I think current callable generic is messy, I also think most of the people that write those type annotations are the ones that write a lot of them. Maybe that would change with better readability but right now heavy callable type usage feel like an advanced, uncommon thing. Part of it is also heavy functional style code isn’t that common in python.