
Hello In thread about PEP 677, I (with the help of another proposal) came up with an alternative to typing.Callable called function prototypes. The basic concept is to create a function prototype object when the body of a function is omitted. The thread can be found here: https://mail.python.org/archives/list/python-dev@python.org/thread/OGACYN2X7... Mark Shannon initially proposed that functions be used as types and provided this example: @Callable def IntToIntFunc(a:int)->int: pass def flat_map( l: list[int], func: IntToIntFunc ) -> list[int]: .... I further proposed that we make the body of a function non-mandatory and create a function prototype if it is omitted. I provided these examples: import typing @typing.Callable def IntToIntFunc(a: int) -> int def flat_map( l: list[int], func: IntToIntFunc ) -> list[int]: ... import ctypes @ctypes.CFUNCTYPE def f(x: int) -> bool I have since taken it upon myself to implement this in a fork of cpython: https://github.com/asleep-cult/cpython To remain consistent with function definitions, I have also added an alternative lambda syntax that allows you to annotate arguments and the return type. The syntax requires you to parenthesize the argument list: lambda (a: int, b: int) -> int: ... This new lambda syntax also allows you to create a function prototype by omitting the body. The original example can be rewritten as follows: def flat_map( l: list[int], func: lambda (a: int) -> int ) -> list[int]: ... Problems: - It is not possible to use ParamSpec with this - The lambda prototype syntax might be jarring - Some people might accidentally forget the function body Here is some feedback that I have already collected: "Yeah, making the body optional (without looking at decorators) is not acceptable either. Too easy to do by mistake (I still do this All. The. Time. :-)" - Guido van Rossum "i would strongly prefer this over the existing pep" - Ronny Pfannschmidt "I find this unnecessary and unreadable. Python isn't C or Java." - BundleOfJoysticks (Reddit)

How about instead of omitting the body, it contains a single expression, say: def int(x: float): -> float typing.Prototype It doesn’t totally break protocol to have a function complie differently depending on its content— that’s done with generator functions. -CHB On Thu, Dec 23, 2021 at 7:20 AM <asleep.cult@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Dec 23, 2021 at 03:00:03PM -0000, asleep.cult@gmail.com wrote:
I have to admit, when Mark Shannon initially proposed that as an improvement over both the current and the proposed syntax, I was so taken aback that I initially thought he was being sarcastic and had to read it a second time to realise he was serious :-( The status quo is an in-place declaration using an anonymous type: func: Callable[[int], list[int]] # proposed equivalent func: (int) -> list[int] The anonymous type using Callable requires requires 26 characters, including 7 punctuation characters. The PEP 677 proposal cuts that down to 18 chars (6 punctuation chars) while increasing readability: the arrow syntax is "executable pseudo code". As far back as PEP 484 in 2014 this same syntax was used in type-comments for Python2 straddling code: https://www.python.org/dev/peps/pep-0484/#id50 Extending that to refer to the signature of a function is an obvious step. Many other languages have converged on the same, or very similar, syntax. I've been using similar `param -> result` pseudo-syntax when sketching out code using pencil and paper, or on a whiteboard, for years, and nobody has failed to understand it. In comparison, Mark's version: @Callable def IntToIntFunc(a:int)->int: pass # in the type declaration func: IntToIntFunc uses 54 characters, plus spaces and newlines (including 7 punctuation characters); it takes up three extra lines, plus a blank line. As syntax goes it is double the size of Callable. It separates the type declaration from the point at which it is used, potentially far away from where it is used. It adds a new name to the global namespace, bloating the output of introspection tools like dir(), help() etc. And it *requires* a named (non-anonymous) type where an anonymous type is all that is needed or wanted. Being able to name types using an alias when it helps readability is good. Being required to name them even at the cost of hurting readability is not. Naming is hard, and bad names are worse than no names. Consider Mark's name for the function: "IntToIntFunc", which tells us nothing that the signature (int)->int doesn't already tell us. It is the naming equivalent of the comment: x += 1 # Add 1 to x. Your proposal is slightly more compact than Mark's: you drop the ending colon and the body ("pass"), saving one line and five characters out of the 54. But it suffers from the same major flaws: - verbose and relatively heavy on vertical space; - bloats the output of introspection tools; - separating the definition of the type from where it is used; - requiring a name for something which doesn't need a name. If I had the choice between using the current syntax with Callable[] and the proposed PEP 677 arrow syntax, I would almost always use the arrow syntax. It matches the pseudo-syntax I already use when writing pseudo- code on paper. If I had the choice between Callable[] and this proposed function-as-a- type syntax, I would stick to Callable. If I wanted to give the type a name, for some reason, I would still use Callable, and just write an alias. I cannot imagine any scenario where I would prefer this function- as-a-type syntax over the other two alternatives.
What do you get when the inevitable occurs, and you forget the decorator? If I just write this: def IntToIntFunc(a: int) -> int it will create what sort of object? [...]
At least that brings back the ability to write it as an anonymous type, but at the cost of adding a totally unnecessary keyword "lambda" and an unused, redundant parameter name: func: (int) -> int func: lambda (a: int) -> int -- Steve

Hello and thank you for the much needed feedback. One thing that you must consider is that function prototypes have a few implications beyond typing but it seems like you're only looking at it as a tool for type hinting. The interpreter will create a function prototype object regardless of if you forget your decorator, it needs to pass something to the decorator after all. After reading through your reply, I am seeing that the main concern is the bloat added by the lambda keyword. My decision to use lambda instead of introducing a special syntax was one that required heavy deliberation. I ultimately decided to stick with lambda because it was consistent with the prototype statement form. The fact that lambda is hard to type has been felt by almost everyone who has ever used Python, this isn't just a problem that would be introduced by function prototypes. PEP 677 has taken the lazy approach to solving this issue and has prioritized type hinting over functionality. PEP 667 also suggests the usage of => for lambdas which would likely never be accepted because of the confusion it would cause. As someone who has used typing with Python, I do think that a new callable syntax is needed, but I truly believe that PEP 677 is taking the wrong approach. So what if we broke every Python program in existence by creating a new lambda syntax, how would it look? This question is particularly hard to answer because the body and annotations both need to be optional. Our best bet is an augmented form of the PEP 677 syntax that allows you to add a body. Here is an example: (a: int) -> int: a ** 2 But of course this causes ambiguity when the return annotation and body are both omitted. One thing that I did consider is simply treating it like a tuple if there is no return annotation AND there is no body, but that might lead to confusion. Another thing that I considered is a different prefix than lambda: $(a: int) -> int: a ** 2

My eyes are bleeding with these incomplete function definitions. If you are using decorators and "def", then, please, there is no need for special syntax that would just be a syntax error in "normal"Python., Just add ": pass" to the end. If eyes bleeding is not enough of an argument for you: the new syntax would only be possible be usable in code that would run on Python 3.11 and above. While using the decorator + function declaration syntax, you have the annotations that could readly be checked and code compatible with existing all supported Python versions, right out of the box. On Thu, 23 Dec 2021 at 16:31, MRAB <python@mrabarnett.plus.com> wrote:

On Thu, Dec 23, 2021 at 02:09:50PM -0800, Guido van Rossum wrote:
Without decorator too (that was Lukasz’ idea). Why bother with the decorator (*if* we were to go there)?
So that def func(params): pass creates a function object, and def func(params) makes a Callable type object? I'm going to *love* explaining the difference to beginners when they accidently do this and try to call func. Python is not Scratch and is not intended as a teaching language for kids, but some proposals are actively hostile to beginners, and I think this is one. Have we considered how this would effect something as simple as the meaning of keywords? - `class` creates classes, also known as types; - `def` creates functions, and also types, but not types you can use for anything except type-checking. Given that annotations are optional everywhere else, what happens if you leave out the annotations in the type definition? def SomethingFunction(x) Is that the same as Callable[[Any], Any] or Callable[[None], None]? Or are we going to create a rule that annotations are mandatory in some `def`s but optional in others? -- Steve

On Thu, Dec 23, 2021 at 3:24 PM Steven D'Aprano <steve@pearwood.info> wrote:
No, no, no. That syntax has already been discredited. Mark's proposal was ``` @Callable def func(params): pass ``` My question is, why does it need `@Callable`? Lukasz proposed just using any (undecorated) function, with the convention being that the body is `...` (to which I would add the convention that the function *name* be capitalized, since it is a type). My question (for Mark, or for anyone who supports `@Callable`) is why bother with the decorator. It should be easy to teach a type checker about this: ``` def SomeFn(x: float) -> int: ... def twice(f: SomeFn) -> SomeFn: return lambda x: f(f(x)) ``` -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Dec 23, 2021, at 17:09, Guido van Rossum <guido@python.org> wrote:
That seems pretty intuitive to me. The conventions you mention would be just that though, right? I.e. `pass` could be used, but whatever the body is it would be ignored for type checking `twice()` in this case, right? -Barry

Yes, and yes. (Or the PEP could say it has to be ‘…’ and static checkers could enforce it. But checkers already carry everything they need to check this with them, for checking calls.) Downside of this idea is that it requires you to invent a name for every callable type you use. On Thu, Dec 23, 2021 at 17:37 Barry Warsaw <barry@python.org> wrote:
-- --Guido (mobile)

On Thu, Dec 23, 2021 at 7:38 PM Barry Warsaw <barry@python.org> wrote:
I think this was briefly mentioned in another thread, but it seems to have been lost in the discussion here, so I want to mention it again because I think it's important: aside from the verbosity and hassle of needing to always define callable types out-of-line and give them a name, another significant downside to the function-as-type approach is that generally Python signatures are too specific for intuitive use as a callback type. Note that in the example immediately above, a typechecker should error on this call: ``` def float_to_int(y: float) -> int: return int(y) twice(float_to_int) ``` The intent of the programmer was probably that `twice` should accept any function taking a single float argument and returning an int, but in fact, given the possibility of keyword calls, the names of non-positional-only parameters are part of the function signature too. Since the body of `twice` could call `f(x=...)`, a typechecker must error on the use of `float_to_int` as the callback, since its parameter is not named `x`. In order to correctly express their intent, the programmer must instead ensure that their callable type takes a positional-only argument: ``` def SomeFn(_x: float, /) -> int: ... ``` The need to almost always use the extra `/` in the callable type signature in order to get the desired breadth of signature, and the likelihood of forgetting it the first time around until someone tries to pass a second callback value and gets a spurious error, is in my mind a major negative to functions-as-callable-type. So I agree with Steven: this issue, plus the verbosity and out-of-line-ness, mean that Callable would continue to be preferable for many cases, meaning we'd end up with two ways to do it, neither clearly preferable to the other. I don't think out-of-line function-as-type could ever be an acceptable full replacement for Callable, whereas I think PEP 677 immediately would be. Carl

On Thu, Dec 23, 2021 at 05:09:18PM -0800, Guido van Rossum wrote:
It has? Where? Have I missed something? This thread is about using that syntax as an alternative to Mark's proposal. If it is already ruled out, then somebody should mention it to asleep.cult (the original poster of this thread). I'm curious: what objections to asleep.cult's proposal don't equally apply to Mark's proposal? From what I can see, Mark's original proposal has all the same disadvantages, plus it is even more verbose. Dropping the need for the @Callable decorator reduces the level of verbosity somewhat (one less line, nine fewer characters) but all the other negatives remain. Honestly, I cannot see a single positive to using `def` statements. Sure, we can do it, and an experienced programmer could infer the meaning of it, but given the choice of writing an anonymous type in place where you want it, versus having to pre-declare a function prototype with a name that adds nothing to the readability of the code before using it, why would I prefer the `def` version? This is not a rhetorical question. The fact that the existing feature (Callable) and the PEP 677 arrow syntax are anonymous, and can be written in place rather than needing to be pre-declared with a name, are positives. *Requiring* a name to use this `def` syntax is a point against it. If I need a named type alias, I can already create one, and name it: IntToIntFunc = Callable[[int], int] and while I can see that there are complicated signatures where a named alias would be useful: FileOpener = Callable[ ... ] # complicated signature we can already do that, so the `def` syntax adds nothing. For simple cases we don't need a name. The name IntToIntFunc adds nothing that isn't just as clear, if not more so, in the signature itself. It is like the comment: x += 1 # add one to x
Indeed, and I already wrote a criticism of that proposal. Removing the decorator saves one line and nine characters, but the other criticisms remain.
But without the Callable decorator, it isn't a type, its a function. You're just using it as a type (or to be precise, a function prototype). I'm okay with naming conventions reflecting usage (we sort of already do that, with int, float, etc) but we should be clear about what's really happening. `def` creates a function. -- Steve

On Fri, Dec 24, 2021 at 1:36 PM Steven D'Aprano <steve@pearwood.info> wrote:
My reading of this is that a function IS the type of a function with that signature, just like how None means the type NoneType. Is that correct? Or putting it another way: is this (silly) example legal? def repeat_string(s: str, times: int) -> str: return s * times def decimate_strings(lines: Iterable[str], xfrm: repeat_string) -> List[str]: return [xfrm(l, 10) for l in lines] If def creates a function, and def creates the type of a function, it stands to reason that a function is the type of a function. ChrisA

On Fri, Dec 24, 2021 at 01:54:35PM +1100, Chris Angelico wrote:
That's not the status quo, but I think the idea is that it will be. Except that I think that the preferred terminology is that it is a *function prototype* rather than a type, since it is a prototype, not an actual class you can instantiate. A template, if you will: "Any callable with the same signature (modulo parameter names) as this template / protocol is acceptable."
Or putting it another way: is this (silly) example legal? [snip example]
I think maybe it should be, but right now, mypy reports it as illegal: [steve ~]$ mypy ~/func_prototype.py /home/steve/func_prototype.py:7: error: Function "func_prototype.repeat_string" is not valid as a type If we allow this, I see no reason why functions used as prototypes should be required to have an empty body. (Ellipsis or pass.) I think that actual, useful functions with an implementation should be allowed, as in your example. -- Steve

On Fri, Dec 24, 2021 at 01:28:44PM +1100, Steven D'Aprano wrote:
Hmm, I think I may have come up with one. If we did teach type checkers to use actual functions as prototypes, that would allow the same function object to do double duty as the type-hint (as a prototype) and as an implementation of that prototype. So if you wanted a function that has the same signature as builtin `print`, you could just say something like: def traverse_graph( g: Graph, maxdepth: int = -1, # Same signature as print, defaults to print visitor: print = print, ) -> None: instead of having to replicate print's signature. The duplication `: print = print` is a bit on the nose, but not too much. And maybe type-checkers could infer that if a parameter defaults to a function, its type-hint should clearly be the same as the default? This would require builtins to gain annotations, of course. Which they don't currently have :-( And for the avoidance of doubt, I am not suggesting this be limited to only builtin functions. Any function with annotations would work. So to my mind, that moves Mark's proposal into the category of an independent new feature separate to PEP 677, rather than a competitor or alternative proposal: * Teach type-checkers to use functions made with `def` as function prototypes (regardless of what the body is). * Likewise for classes (use the `__call__` method's signature as the prototype). while PEP 677 remains as a nice-looking shorter syntax for in-place anonymous Callable Types. I remain strongly -1 on `def ...` as an alternative to PEP 677, I don't think that the `def` syntax makes a good alternative to either Callable or the proposed arrow syntax. But being able to use an existing function, complete with implementation in the body, as a prototype, yeah, I'll buy that. -- Steve

There were multiple threads about this (or maybe the thread was split by mailers) and I already stated that the colon-free (and hence body-less) syntax with def is too confusing to consider. Please look it up in the archives. Happy Holidays! On Thu, Dec 23, 2021 at 18:36 Steven D'Aprano <steve@pearwood.info> wrote:
-- --Guido (mobile)

On Thu, Dec 23, 2021 at 5:13 PM Guido van Rossum <guido@python.org> wrote:
My guess is the thinking is to prevent anyone from thinking that the function is actually meant to do anything other than be a type signature for a callable. Otherwise it's very much convention to know what is purposefully a no-op/always-returns-None function meant to be a callable type based either on the naming scheme or some other practice like the ellipsis body versus you just haven't implemented the function yet.
It probably is more useful without the decorator. For instance, if you want to type a function parameter to something that acts like `open()`, then would you want to reproduce that type annotation or just say `opener: builtins.open`? This would actually lean into duck typing even by allowing people to easily say, "I want _something_ that directly quacks like this callable" instead of having to manually duplicate the API with a separate type annotation (which one could still do if they were worried about code drift in regards to expectations). Basically it makes all functions and methods immediately a potential type annotation which could be rather powerful and could lead to more callables being used compared to single-method protocols if there isn't some OOP need to pass an object with a method around.

24.12.21 00:09, Guido van Rossum пише:
Without decorator too (that was Lukasz’ idea). Why bother with the decorator (*if* we were to go there)?
It is errorprone. Some library provide function foo() which returns an instance of private type _Foo and people start using it as a type hint. A new version converts foo() into a class. It is usually a safe backward compatible change, except that now all type hints became wrong. "a: foo" now means an instance of foo, not a callable which returns an instance of _Foo. There are also issues with subclassing.
There are also issues with subscripting: list[typing.Callable[[int], list[int]]]

I like the Callable decorator idea very much. It supports all Python function flavors out of the box, isn't it? Also, what is about allowing to make callable types from existing functions (and even methods maybe) with type hints? def f(a: int, /, b: float) -> str: return str(a*b) F = Callable(f) Could it work? I'm ok with making an explicit Callable type alias first for every usage. But if I can create it from an existing function augmented with type hints without copy-pasting the signature -- it can make my life significantly easier. What do you think? On Fri, Dec 24, 2021 at 11:57 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
-- Thanks, Andrew Svetlov

On Fri, Dec 24, 2021 at 11:53:22AM +0200, Serhiy Storchaka wrote:
Some library provide function foo() which returns an instance of private type _Foo and people start using it as a type hint.
If people want to shoot themselves in the foot, is it our job to stop them? Typically the only promise we make is that the interpreter won't segfault, not to protect coders from making bad decisions in their code. If _Foo is private, you shouldn't be using it as a type hint, not even indirectly through a function prototype. *Everything* about _Foo is subject to change without notice, including its very existence. It seems a bit strange to accept the risk of unpredictable changes to a private implementation detail, while worrying about backwards- incompatible changes to a public function.
It is a safe backwards compatible change only if you don't care about the type of foo. As soon as you use foo as a function prototype, then you now care about its type, just as much as if you inspected it with type() or isinstance(). In other words, it's not *actually* a backwards compatible change in a language like Python where functions are first-class objects. We can just get away with that in a context where the type of the callable usually doesn't matter, just as we usually don't care if the repr() changes. In this case, if we introduce function prototypes, then people will learn that changing a factory function to a class is a breaking change for people who do type-checking, just as changing the repr of objects is a breaking change for people who use doctests. On the other hand, changing a function to a callable instance should not be a breaking change, if we follow my earlier suggestion that objects with a `__call__` method should be usable as prototypes too. -- Steve

I've done some more thinking more about Serhiy's worry about changing a factory function to a class, and how that would change the meaning of type-hints. Say: def spam(x: Eggs, y:Cheese) -> _Aardvark: # actual factory function implementation # later, we use it as a function protocol def myfunction(a: int, callback: spam) -> str: ... If spam changes implementation from a factory function to the actual class of _Aardvark, keeping the name: class spam: # body of _Ardvark goes here that would completely change the meaning of the myfunction type declaration. I still think that is not a scenario we need to care about. In my mind, that counts as a user-applied footgun. But if people disagree, and Serhiy's argument persuades them, then we can still use functions as their own prototype. @Callable def spam(x: Eggs, y: Cheese) -> _Aardvark: # actual factory function implementation The Callable decorator would just flag the function as *permitted* to be used as a prototype, with no other change. Or perhaps we could have @FunctionPrototype. Alternatively, we could write the consumer's annotation like this: def myfunction(a: int, callback: Prototype[spam]) -> str: ... where Prototype[x] uses: - the signature of x if x is a function or method; - the signature of `x.__call__` if x is a callable instance; - the signature of `x.__new__` or `x.__init__` is x is a class; as the prototype. However it is spelled, we might require functions to opt-in before they can be used as prototypes, in other words the decorator is the author's promise that they aren't going to change the function into a class, or change its signature, without the usual deprecation warnings etc. -- Steve

On Thu, Dec 23, 2021 at 07:04:17PM -0000, asleep.cult@gmail.com wrote:
One thing that you must consider is that function prototypes
Just to be clear, by "function prototype", do you mean what PEP 677 calls a Callable Type?
What sort of implications beyond typing? Some questions: 1. What kind of object does `def Func(a:int)->int` create, if you leave out the decorator? Is that what you are calling a "function prototype object"? 2. What does it need the @Callable decorator for? 3. What can you do with it, apart from typing? 4. What happens if you leave out the annotations and just say `def Func(a)` alone? 5. Do you have any response to my other criticisms about this syntax?
After reading through your reply, I am seeing that the main concern is the bloat added by the lambda keyword.
Perhaps you should re-read my reply. The lambda keyword is the *least* part of my dislike of this proposal.
The fact that lambda is hard to type has been felt by almost everyone who has ever used Python,
Such exaggerations don't help your case. I have never felt the need to type a lambda expression, and doubt I'm so very unusual. Annotations have only existed for less than half of Python's existence. Even today, I doubt that as many as half of Python's user base are heavy users of typing. Or even casual users of typing. And many of them would not use lambda, or at least not in places where adding typing to it would add any value. In any case, allowing type-hints in lambda expressions is independent of the question of using the lambda keyword to declare a function prototype. I have no objection in principle to allowing annotations in lambda expressions if such a thing would actually be useful. But that doesn't mean I want to see lambda used as function prototype syntax: def map(func: lambda (obj: Any) -> Any, items: Sequence[Any]) -> Sequence[Any] especially not in preference to just using arrow syntax: def map(func: (Any)-> Any, items: Sequence[Any]) -> Sequence[Any]
What does that mean? What is lazy about it?
Syntactic sugar for lambda is not part of PEP 667, it merely references the fact that people have suggested using => as shorthand for a lambda. For what its worth, I was skeptical about using two different arrows (one for declaring callable types, one for functions) when I first heard the idea (I think it was Guido who mentioned it?). But I've come to believe that whatever confusion there might be in using two arrows "do I use -> or => here? I never remember which is which" will be less, not more, than the confusion due to using the same arrow for both contexts. That is, I think, the experience from other languages. (Kotlin if I remember correctly? Maybe not.) But that's a discussion for when somebody writes a PEP for lambda shortcut syntax.
Creating new syntax is backwards compatible: it doesn't break existing code that is syntactically correct. Only removing, or changing the meaning of, existing syntax will break "every Python program in existence". I doubt the Steering Council would accept such breakage. -- Steve

What sort of implications beyond typing? Here are a few places where "function prototypes" would be useful:
import ctypes @ctypes.CFUNCTYPE def f(a: ctypes.c_int) -> ctypes.c_int import abc class Model(abc.ABC): def get_id(self) -> int By returning a concrete object, I leave it up to the implementation to define its meaning.

Steven D'Aprano wrote:
I think it takes only the characters needed to write the name IntToIntFunc. The @callable def section is a one-time definition, and not logically part of each function definition where it is used. I get that some people prefer an inline lambda to a named function, and others hate naming an infrastructure function, but ... Why are you even bothering to type the callback function? If it is complicated enough to be worth explicitly typing, then it is complicated enough to chunk off with a name. I won't say it is impossible to understand a function signature on the first pass if it takes several lines and whitespace to write ... but it is much easier when the the declaration is short enough to fit on a single line. An @ on the line above complicates the signature parsing, but can be mentally processed separately. The same is true of a named-something-or-other in the middle. Having to switch parsing modes to understand an internal ([int, float, int] -> List[int]), and then to pop that back off the stack is much harder. Hard enough that you really ought to help your reader out with a name, and let them figure out what that names means separately, when their brain's working memory isn't already loaded with the first part of your own function, but still waiting for the last part.
It separates the type declaration from the point at which it is used, potentially far away from where it is used.
The sort of code that passes around functions tends to pass around many functions, but with only a few signatures. If this is really the only time you'll need that signature (not even when you create the functions that will be passed from a calling site?), then ... great. But be nice to your reader anyhow, unless the signature is really so simple that the type-checking software should infer it for you. Then be nice by leaving it out as cruft. [As an aside, I would see some advantage to def myfunc(f:like blobfunc) pointing to an examplar instead of a specifically constructed function-type. You discuss this later as either ... f:blobfunc ... or ... f:blobfunc=blobfunc ... and I would support those, if other issues can be worked out.] -jJ

On Fri, Dec 24, 2021 at 06:24:03PM -0000, Jim J. Jewett wrote:
That's only true if IntToIntFunc is a builtin, otherwise it needs to be defined somewhere. It doesn't just magically happen. If you are using the declaration many times, then I acknowledge that it may be worth the effort of pre-declaration and naming. (But see below.) Particularly if the signature is complicated, although I think that will be relatively rare. But in the worst case, you may only use it once. So the entire cognitive burden of pre-declaration (both writing it and reading it) applies to that one use.
The @callable def section is a one-time definition, and not logically part of each function definition where it is used.
The status quo is that we can use an anonymous type in the annotation without pre-defining it, using Callable. PEP 677 proposes a new, more compact syntax for the same. Any proposal for function prototypes using `def` is directly competing against Callable or arrow syntax for the common case that we want an anonymous, unnamed type written in place. Even in the case that we want to give the type a name that we plan to use repeatedly, this `def` syntax is still competing directly against what is already possible using the status quo: use Callable to create a named type alias. But with the `def` syntax, you can *only* use it as a named, pre-defined object. So half, maybe 90%, of your use-cases disappear. Any time that we have a short, simple Callable that doesn't require a name, why would we bother creating a do-nothing function just so we can use it as a prototype? I don't think many people would. I know I wouldn't. That would be the equivalent of filling your program with trivial plus_one(x) and times_two(y) functions instead of just using `x+1` and `2*y`. So the benefit of the `def` syntax comes from that relatively small subset of cases: - the callable signature is complicated; - we wish to refer it it multiple times; - giving it a name (like "FileOpener", say, not "IntToInt") aids clarity. That's not to be sneered at. But in those circumstances, we don't need the `def` syntax, because we can already use Callable and a type alias. So the `def` syntax adds nothing we don't already have, it is no easier to use, it is more verbose, not less. But if we can use an existing function as the prototype instead of having to declare the prototype, that shifts the balance. If we already have some function, then there is no extra cost in having to declare it and give it a name, it already has been declared and given a name.
I would say the opposite: most callback or key functions have very simple signatures. If my function takes a key function, let's say: def spam(mylist:[str], a: int, b: float, c: bool|None, key: Callable[[str], str], ) -> Eggs: mylist = sorted(mylist, key=key) ... the relevant signature is (str) -> str. Do we really need to give that a predefined named prototype? def StrToStr(s: str) -> str: pass I would argue that very few people would bother. If somebody did, they probably also defined type aliases for ListOfStr and BoolOrNone, and wish they were using Java or Pascal *wink* It seems to me that most callbacks and key functions have short signatures. Certainly all the ones I have written do: they typically take a single argument, of a known type, and return a known type.
I notice that you just used something very close to PEP 677 arrow syntax totally unself-consciously, without any need to explain it. I think this is good evidence that far from being confusing, this is a completely natural syntax that we already interpret as a function prototype.
Hard enough that you really ought to help your reader out with a name,
What are you going to name it? Int_and_Float_and_Int_returns_List_of_Int_Function tells us nothing that (int, float, int) -> list[int] Callable[[int, float, int], list[int]] doesn't already say. Naming functions is hard. Naming function *prototypes* is even harder. Just duplicating the prototype in the name is noise. We don't bloat our code with say-nothing comments: mylist.sort() # sort mylist mylist.append(x) # append x to mylist or at least we hopefully don't do it beyond the initial first few months of learning to program. We let the code speak for itself. But I agree with you, if a type is complex enough that a meaningful name, or even a generic name, helps comprehension, that we should name it. We can already do that with type aliases. -- Steve

Steven D'Aprano wrote:uble the size of Callable.
I think it takes only the characters needed to write the name IntToIntFunc. ... you may only use it once.
Could you provide an example where it is only used once? The only way I can imagine is that you use it here when when defining your complicated function that takes a callback (once), but then you never actually call that complicated function, even from test code, nor do you expect your users to do so.
The status quo is that we can use an anonymous type in the annotation without pre-defining it, using Callable.
OK. I'm not sure it would be a good idea, but we agree it is legal.
PEP 677 proposes a new, more compact syntax for the same.
Does it? I agree that "(int, float) -> bool" is more compact than typing.Callable[...], but that feels like optimizing for the wrong thing. I dislike the PEP's flat_map as an example, because it is the sort of infrastructure function that carries no semantic meaning, but ... I'll use it anyhow. def flat_map(l, func): out = [] for element in l: out.extend(f(element)) return out def wrap(x: int) -> list[int]: return [x] def add(x: int, y: int) -> int: return x + y It is reasonable to add a docstring to flat_map, but I grant that this doesn't work as well with tooling that might involve not actually seeing the function. I agree that adding a long prefix of: from typing import Callable def flat_map( l: list[int], func: Callable[[int], list[int]] ) -> list[int]: is undesirable. But the biggest problem is not that "Callable..." has too many characters; the problem is that "Callable[[...], list[...]]" requires too many levels of sub-parsing. The PEP doesn't actually say what it proposes, [and you've suggested that my earlier attempt was slightly off, which may not bode well for likelihood of typos], but I'll *guess* that you prefer: def flat_map( l: list[int], func: ((int) ->[int]) ) -> list[int]: which is slightly shorter physically, but not much simpler mentally. It therefore creates an attractive nuisance. def flat_map( l: list[int], func: wrap ) -> list[int]: on the other hand, lets you read this definition without having to figure out what "wrap" does at the same time. "wrap" is a particularly bad example (because of the lack of semantic content in this example), but I think it still easily beats the proposed new solution, simply because it creates a "you don't need need to peer below this right now" barrier.
I'm saying that catering to that "common" case is a trap, often leading you to a local optima that is bad globally.
But if we can use an existing function as the prototype instead of having to declare the prototype, that shifts the balance.
I agree that re-using an existing function with the correct signature is better, *even* when that function doesn't make a good default. ...
If you really care about enforcing the str, then yes, it is worth saying key: str_key and defining str_key function as an example def str_key(data:str)->str return str(data)
I would argue that very few people would bother.
Because it would usually be silly to care that the list really contained strings, as opposed to "something sortable". So if you do care, it is worth making your requirement stand out, instead of losing it in a pile of what looks like boilerplate. -jJ

How about instead of omitting the body, it contains a single expression, say: def int(x: float): -> float typing.Prototype It doesn’t totally break protocol to have a function complie differently depending on its content— that’s done with generator functions. -CHB On Thu, Dec 23, 2021 at 7:20 AM <asleep.cult@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Dec 23, 2021 at 03:00:03PM -0000, asleep.cult@gmail.com wrote:
I have to admit, when Mark Shannon initially proposed that as an improvement over both the current and the proposed syntax, I was so taken aback that I initially thought he was being sarcastic and had to read it a second time to realise he was serious :-( The status quo is an in-place declaration using an anonymous type: func: Callable[[int], list[int]] # proposed equivalent func: (int) -> list[int] The anonymous type using Callable requires requires 26 characters, including 7 punctuation characters. The PEP 677 proposal cuts that down to 18 chars (6 punctuation chars) while increasing readability: the arrow syntax is "executable pseudo code". As far back as PEP 484 in 2014 this same syntax was used in type-comments for Python2 straddling code: https://www.python.org/dev/peps/pep-0484/#id50 Extending that to refer to the signature of a function is an obvious step. Many other languages have converged on the same, or very similar, syntax. I've been using similar `param -> result` pseudo-syntax when sketching out code using pencil and paper, or on a whiteboard, for years, and nobody has failed to understand it. In comparison, Mark's version: @Callable def IntToIntFunc(a:int)->int: pass # in the type declaration func: IntToIntFunc uses 54 characters, plus spaces and newlines (including 7 punctuation characters); it takes up three extra lines, plus a blank line. As syntax goes it is double the size of Callable. It separates the type declaration from the point at which it is used, potentially far away from where it is used. It adds a new name to the global namespace, bloating the output of introspection tools like dir(), help() etc. And it *requires* a named (non-anonymous) type where an anonymous type is all that is needed or wanted. Being able to name types using an alias when it helps readability is good. Being required to name them even at the cost of hurting readability is not. Naming is hard, and bad names are worse than no names. Consider Mark's name for the function: "IntToIntFunc", which tells us nothing that the signature (int)->int doesn't already tell us. It is the naming equivalent of the comment: x += 1 # Add 1 to x. Your proposal is slightly more compact than Mark's: you drop the ending colon and the body ("pass"), saving one line and five characters out of the 54. But it suffers from the same major flaws: - verbose and relatively heavy on vertical space; - bloats the output of introspection tools; - separating the definition of the type from where it is used; - requiring a name for something which doesn't need a name. If I had the choice between using the current syntax with Callable[] and the proposed PEP 677 arrow syntax, I would almost always use the arrow syntax. It matches the pseudo-syntax I already use when writing pseudo- code on paper. If I had the choice between Callable[] and this proposed function-as-a- type syntax, I would stick to Callable. If I wanted to give the type a name, for some reason, I would still use Callable, and just write an alias. I cannot imagine any scenario where I would prefer this function- as-a-type syntax over the other two alternatives.
What do you get when the inevitable occurs, and you forget the decorator? If I just write this: def IntToIntFunc(a: int) -> int it will create what sort of object? [...]
At least that brings back the ability to write it as an anonymous type, but at the cost of adding a totally unnecessary keyword "lambda" and an unused, redundant parameter name: func: (int) -> int func: lambda (a: int) -> int -- Steve

Hello and thank you for the much needed feedback. One thing that you must consider is that function prototypes have a few implications beyond typing but it seems like you're only looking at it as a tool for type hinting. The interpreter will create a function prototype object regardless of if you forget your decorator, it needs to pass something to the decorator after all. After reading through your reply, I am seeing that the main concern is the bloat added by the lambda keyword. My decision to use lambda instead of introducing a special syntax was one that required heavy deliberation. I ultimately decided to stick with lambda because it was consistent with the prototype statement form. The fact that lambda is hard to type has been felt by almost everyone who has ever used Python, this isn't just a problem that would be introduced by function prototypes. PEP 677 has taken the lazy approach to solving this issue and has prioritized type hinting over functionality. PEP 667 also suggests the usage of => for lambdas which would likely never be accepted because of the confusion it would cause. As someone who has used typing with Python, I do think that a new callable syntax is needed, but I truly believe that PEP 677 is taking the wrong approach. So what if we broke every Python program in existence by creating a new lambda syntax, how would it look? This question is particularly hard to answer because the body and annotations both need to be optional. Our best bet is an augmented form of the PEP 677 syntax that allows you to add a body. Here is an example: (a: int) -> int: a ** 2 But of course this causes ambiguity when the return annotation and body are both omitted. One thing that I did consider is simply treating it like a tuple if there is no return annotation AND there is no body, but that might lead to confusion. Another thing that I considered is a different prefix than lambda: $(a: int) -> int: a ** 2

My eyes are bleeding with these incomplete function definitions. If you are using decorators and "def", then, please, there is no need for special syntax that would just be a syntax error in "normal"Python., Just add ": pass" to the end. If eyes bleeding is not enough of an argument for you: the new syntax would only be possible be usable in code that would run on Python 3.11 and above. While using the decorator + function declaration syntax, you have the annotations that could readly be checked and code compatible with existing all supported Python versions, right out of the box. On Thu, 23 Dec 2021 at 16:31, MRAB <python@mrabarnett.plus.com> wrote:

On Thu, Dec 23, 2021 at 02:09:50PM -0800, Guido van Rossum wrote:
Without decorator too (that was Lukasz’ idea). Why bother with the decorator (*if* we were to go there)?
So that def func(params): pass creates a function object, and def func(params) makes a Callable type object? I'm going to *love* explaining the difference to beginners when they accidently do this and try to call func. Python is not Scratch and is not intended as a teaching language for kids, but some proposals are actively hostile to beginners, and I think this is one. Have we considered how this would effect something as simple as the meaning of keywords? - `class` creates classes, also known as types; - `def` creates functions, and also types, but not types you can use for anything except type-checking. Given that annotations are optional everywhere else, what happens if you leave out the annotations in the type definition? def SomethingFunction(x) Is that the same as Callable[[Any], Any] or Callable[[None], None]? Or are we going to create a rule that annotations are mandatory in some `def`s but optional in others? -- Steve

On Thu, Dec 23, 2021 at 3:24 PM Steven D'Aprano <steve@pearwood.info> wrote:
No, no, no. That syntax has already been discredited. Mark's proposal was ``` @Callable def func(params): pass ``` My question is, why does it need `@Callable`? Lukasz proposed just using any (undecorated) function, with the convention being that the body is `...` (to which I would add the convention that the function *name* be capitalized, since it is a type). My question (for Mark, or for anyone who supports `@Callable`) is why bother with the decorator. It should be easy to teach a type checker about this: ``` def SomeFn(x: float) -> int: ... def twice(f: SomeFn) -> SomeFn: return lambda x: f(f(x)) ``` -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Dec 23, 2021, at 17:09, Guido van Rossum <guido@python.org> wrote:
That seems pretty intuitive to me. The conventions you mention would be just that though, right? I.e. `pass` could be used, but whatever the body is it would be ignored for type checking `twice()` in this case, right? -Barry

Yes, and yes. (Or the PEP could say it has to be ‘…’ and static checkers could enforce it. But checkers already carry everything they need to check this with them, for checking calls.) Downside of this idea is that it requires you to invent a name for every callable type you use. On Thu, Dec 23, 2021 at 17:37 Barry Warsaw <barry@python.org> wrote:
-- --Guido (mobile)

On Thu, Dec 23, 2021 at 7:38 PM Barry Warsaw <barry@python.org> wrote:
I think this was briefly mentioned in another thread, but it seems to have been lost in the discussion here, so I want to mention it again because I think it's important: aside from the verbosity and hassle of needing to always define callable types out-of-line and give them a name, another significant downside to the function-as-type approach is that generally Python signatures are too specific for intuitive use as a callback type. Note that in the example immediately above, a typechecker should error on this call: ``` def float_to_int(y: float) -> int: return int(y) twice(float_to_int) ``` The intent of the programmer was probably that `twice` should accept any function taking a single float argument and returning an int, but in fact, given the possibility of keyword calls, the names of non-positional-only parameters are part of the function signature too. Since the body of `twice` could call `f(x=...)`, a typechecker must error on the use of `float_to_int` as the callback, since its parameter is not named `x`. In order to correctly express their intent, the programmer must instead ensure that their callable type takes a positional-only argument: ``` def SomeFn(_x: float, /) -> int: ... ``` The need to almost always use the extra `/` in the callable type signature in order to get the desired breadth of signature, and the likelihood of forgetting it the first time around until someone tries to pass a second callback value and gets a spurious error, is in my mind a major negative to functions-as-callable-type. So I agree with Steven: this issue, plus the verbosity and out-of-line-ness, mean that Callable would continue to be preferable for many cases, meaning we'd end up with two ways to do it, neither clearly preferable to the other. I don't think out-of-line function-as-type could ever be an acceptable full replacement for Callable, whereas I think PEP 677 immediately would be. Carl

On Thu, Dec 23, 2021 at 05:09:18PM -0800, Guido van Rossum wrote:
It has? Where? Have I missed something? This thread is about using that syntax as an alternative to Mark's proposal. If it is already ruled out, then somebody should mention it to asleep.cult (the original poster of this thread). I'm curious: what objections to asleep.cult's proposal don't equally apply to Mark's proposal? From what I can see, Mark's original proposal has all the same disadvantages, plus it is even more verbose. Dropping the need for the @Callable decorator reduces the level of verbosity somewhat (one less line, nine fewer characters) but all the other negatives remain. Honestly, I cannot see a single positive to using `def` statements. Sure, we can do it, and an experienced programmer could infer the meaning of it, but given the choice of writing an anonymous type in place where you want it, versus having to pre-declare a function prototype with a name that adds nothing to the readability of the code before using it, why would I prefer the `def` version? This is not a rhetorical question. The fact that the existing feature (Callable) and the PEP 677 arrow syntax are anonymous, and can be written in place rather than needing to be pre-declared with a name, are positives. *Requiring* a name to use this `def` syntax is a point against it. If I need a named type alias, I can already create one, and name it: IntToIntFunc = Callable[[int], int] and while I can see that there are complicated signatures where a named alias would be useful: FileOpener = Callable[ ... ] # complicated signature we can already do that, so the `def` syntax adds nothing. For simple cases we don't need a name. The name IntToIntFunc adds nothing that isn't just as clear, if not more so, in the signature itself. It is like the comment: x += 1 # add one to x
Indeed, and I already wrote a criticism of that proposal. Removing the decorator saves one line and nine characters, but the other criticisms remain.
But without the Callable decorator, it isn't a type, its a function. You're just using it as a type (or to be precise, a function prototype). I'm okay with naming conventions reflecting usage (we sort of already do that, with int, float, etc) but we should be clear about what's really happening. `def` creates a function. -- Steve

On Fri, Dec 24, 2021 at 1:36 PM Steven D'Aprano <steve@pearwood.info> wrote:
My reading of this is that a function IS the type of a function with that signature, just like how None means the type NoneType. Is that correct? Or putting it another way: is this (silly) example legal? def repeat_string(s: str, times: int) -> str: return s * times def decimate_strings(lines: Iterable[str], xfrm: repeat_string) -> List[str]: return [xfrm(l, 10) for l in lines] If def creates a function, and def creates the type of a function, it stands to reason that a function is the type of a function. ChrisA

On Fri, Dec 24, 2021 at 01:54:35PM +1100, Chris Angelico wrote:
That's not the status quo, but I think the idea is that it will be. Except that I think that the preferred terminology is that it is a *function prototype* rather than a type, since it is a prototype, not an actual class you can instantiate. A template, if you will: "Any callable with the same signature (modulo parameter names) as this template / protocol is acceptable."
Or putting it another way: is this (silly) example legal? [snip example]
I think maybe it should be, but right now, mypy reports it as illegal: [steve ~]$ mypy ~/func_prototype.py /home/steve/func_prototype.py:7: error: Function "func_prototype.repeat_string" is not valid as a type If we allow this, I see no reason why functions used as prototypes should be required to have an empty body. (Ellipsis or pass.) I think that actual, useful functions with an implementation should be allowed, as in your example. -- Steve

On Fri, Dec 24, 2021 at 01:28:44PM +1100, Steven D'Aprano wrote:
Hmm, I think I may have come up with one. If we did teach type checkers to use actual functions as prototypes, that would allow the same function object to do double duty as the type-hint (as a prototype) and as an implementation of that prototype. So if you wanted a function that has the same signature as builtin `print`, you could just say something like: def traverse_graph( g: Graph, maxdepth: int = -1, # Same signature as print, defaults to print visitor: print = print, ) -> None: instead of having to replicate print's signature. The duplication `: print = print` is a bit on the nose, but not too much. And maybe type-checkers could infer that if a parameter defaults to a function, its type-hint should clearly be the same as the default? This would require builtins to gain annotations, of course. Which they don't currently have :-( And for the avoidance of doubt, I am not suggesting this be limited to only builtin functions. Any function with annotations would work. So to my mind, that moves Mark's proposal into the category of an independent new feature separate to PEP 677, rather than a competitor or alternative proposal: * Teach type-checkers to use functions made with `def` as function prototypes (regardless of what the body is). * Likewise for classes (use the `__call__` method's signature as the prototype). while PEP 677 remains as a nice-looking shorter syntax for in-place anonymous Callable Types. I remain strongly -1 on `def ...` as an alternative to PEP 677, I don't think that the `def` syntax makes a good alternative to either Callable or the proposed arrow syntax. But being able to use an existing function, complete with implementation in the body, as a prototype, yeah, I'll buy that. -- Steve

There were multiple threads about this (or maybe the thread was split by mailers) and I already stated that the colon-free (and hence body-less) syntax with def is too confusing to consider. Please look it up in the archives. Happy Holidays! On Thu, Dec 23, 2021 at 18:36 Steven D'Aprano <steve@pearwood.info> wrote:
-- --Guido (mobile)

On Thu, Dec 23, 2021 at 5:13 PM Guido van Rossum <guido@python.org> wrote:
My guess is the thinking is to prevent anyone from thinking that the function is actually meant to do anything other than be a type signature for a callable. Otherwise it's very much convention to know what is purposefully a no-op/always-returns-None function meant to be a callable type based either on the naming scheme or some other practice like the ellipsis body versus you just haven't implemented the function yet.
It probably is more useful without the decorator. For instance, if you want to type a function parameter to something that acts like `open()`, then would you want to reproduce that type annotation or just say `opener: builtins.open`? This would actually lean into duck typing even by allowing people to easily say, "I want _something_ that directly quacks like this callable" instead of having to manually duplicate the API with a separate type annotation (which one could still do if they were worried about code drift in regards to expectations). Basically it makes all functions and methods immediately a potential type annotation which could be rather powerful and could lead to more callables being used compared to single-method protocols if there isn't some OOP need to pass an object with a method around.

24.12.21 00:09, Guido van Rossum пише:
Without decorator too (that was Lukasz’ idea). Why bother with the decorator (*if* we were to go there)?
It is errorprone. Some library provide function foo() which returns an instance of private type _Foo and people start using it as a type hint. A new version converts foo() into a class. It is usually a safe backward compatible change, except that now all type hints became wrong. "a: foo" now means an instance of foo, not a callable which returns an instance of _Foo. There are also issues with subclassing.
There are also issues with subscripting: list[typing.Callable[[int], list[int]]]

I like the Callable decorator idea very much. It supports all Python function flavors out of the box, isn't it? Also, what is about allowing to make callable types from existing functions (and even methods maybe) with type hints? def f(a: int, /, b: float) -> str: return str(a*b) F = Callable(f) Could it work? I'm ok with making an explicit Callable type alias first for every usage. But if I can create it from an existing function augmented with type hints without copy-pasting the signature -- it can make my life significantly easier. What do you think? On Fri, Dec 24, 2021 at 11:57 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
-- Thanks, Andrew Svetlov

On Fri, Dec 24, 2021 at 11:53:22AM +0200, Serhiy Storchaka wrote:
Some library provide function foo() which returns an instance of private type _Foo and people start using it as a type hint.
If people want to shoot themselves in the foot, is it our job to stop them? Typically the only promise we make is that the interpreter won't segfault, not to protect coders from making bad decisions in their code. If _Foo is private, you shouldn't be using it as a type hint, not even indirectly through a function prototype. *Everything* about _Foo is subject to change without notice, including its very existence. It seems a bit strange to accept the risk of unpredictable changes to a private implementation detail, while worrying about backwards- incompatible changes to a public function.
It is a safe backwards compatible change only if you don't care about the type of foo. As soon as you use foo as a function prototype, then you now care about its type, just as much as if you inspected it with type() or isinstance(). In other words, it's not *actually* a backwards compatible change in a language like Python where functions are first-class objects. We can just get away with that in a context where the type of the callable usually doesn't matter, just as we usually don't care if the repr() changes. In this case, if we introduce function prototypes, then people will learn that changing a factory function to a class is a breaking change for people who do type-checking, just as changing the repr of objects is a breaking change for people who use doctests. On the other hand, changing a function to a callable instance should not be a breaking change, if we follow my earlier suggestion that objects with a `__call__` method should be usable as prototypes too. -- Steve

I've done some more thinking more about Serhiy's worry about changing a factory function to a class, and how that would change the meaning of type-hints. Say: def spam(x: Eggs, y:Cheese) -> _Aardvark: # actual factory function implementation # later, we use it as a function protocol def myfunction(a: int, callback: spam) -> str: ... If spam changes implementation from a factory function to the actual class of _Aardvark, keeping the name: class spam: # body of _Ardvark goes here that would completely change the meaning of the myfunction type declaration. I still think that is not a scenario we need to care about. In my mind, that counts as a user-applied footgun. But if people disagree, and Serhiy's argument persuades them, then we can still use functions as their own prototype. @Callable def spam(x: Eggs, y: Cheese) -> _Aardvark: # actual factory function implementation The Callable decorator would just flag the function as *permitted* to be used as a prototype, with no other change. Or perhaps we could have @FunctionPrototype. Alternatively, we could write the consumer's annotation like this: def myfunction(a: int, callback: Prototype[spam]) -> str: ... where Prototype[x] uses: - the signature of x if x is a function or method; - the signature of `x.__call__` if x is a callable instance; - the signature of `x.__new__` or `x.__init__` is x is a class; as the prototype. However it is spelled, we might require functions to opt-in before they can be used as prototypes, in other words the decorator is the author's promise that they aren't going to change the function into a class, or change its signature, without the usual deprecation warnings etc. -- Steve

On Thu, Dec 23, 2021 at 07:04:17PM -0000, asleep.cult@gmail.com wrote:
One thing that you must consider is that function prototypes
Just to be clear, by "function prototype", do you mean what PEP 677 calls a Callable Type?
What sort of implications beyond typing? Some questions: 1. What kind of object does `def Func(a:int)->int` create, if you leave out the decorator? Is that what you are calling a "function prototype object"? 2. What does it need the @Callable decorator for? 3. What can you do with it, apart from typing? 4. What happens if you leave out the annotations and just say `def Func(a)` alone? 5. Do you have any response to my other criticisms about this syntax?
After reading through your reply, I am seeing that the main concern is the bloat added by the lambda keyword.
Perhaps you should re-read my reply. The lambda keyword is the *least* part of my dislike of this proposal.
The fact that lambda is hard to type has been felt by almost everyone who has ever used Python,
Such exaggerations don't help your case. I have never felt the need to type a lambda expression, and doubt I'm so very unusual. Annotations have only existed for less than half of Python's existence. Even today, I doubt that as many as half of Python's user base are heavy users of typing. Or even casual users of typing. And many of them would not use lambda, or at least not in places where adding typing to it would add any value. In any case, allowing type-hints in lambda expressions is independent of the question of using the lambda keyword to declare a function prototype. I have no objection in principle to allowing annotations in lambda expressions if such a thing would actually be useful. But that doesn't mean I want to see lambda used as function prototype syntax: def map(func: lambda (obj: Any) -> Any, items: Sequence[Any]) -> Sequence[Any] especially not in preference to just using arrow syntax: def map(func: (Any)-> Any, items: Sequence[Any]) -> Sequence[Any]
What does that mean? What is lazy about it?
Syntactic sugar for lambda is not part of PEP 667, it merely references the fact that people have suggested using => as shorthand for a lambda. For what its worth, I was skeptical about using two different arrows (one for declaring callable types, one for functions) when I first heard the idea (I think it was Guido who mentioned it?). But I've come to believe that whatever confusion there might be in using two arrows "do I use -> or => here? I never remember which is which" will be less, not more, than the confusion due to using the same arrow for both contexts. That is, I think, the experience from other languages. (Kotlin if I remember correctly? Maybe not.) But that's a discussion for when somebody writes a PEP for lambda shortcut syntax.
Creating new syntax is backwards compatible: it doesn't break existing code that is syntactically correct. Only removing, or changing the meaning of, existing syntax will break "every Python program in existence". I doubt the Steering Council would accept such breakage. -- Steve

What sort of implications beyond typing? Here are a few places where "function prototypes" would be useful:
import ctypes @ctypes.CFUNCTYPE def f(a: ctypes.c_int) -> ctypes.c_int import abc class Model(abc.ABC): def get_id(self) -> int By returning a concrete object, I leave it up to the implementation to define its meaning.

Steven D'Aprano wrote:
I think it takes only the characters needed to write the name IntToIntFunc. The @callable def section is a one-time definition, and not logically part of each function definition where it is used. I get that some people prefer an inline lambda to a named function, and others hate naming an infrastructure function, but ... Why are you even bothering to type the callback function? If it is complicated enough to be worth explicitly typing, then it is complicated enough to chunk off with a name. I won't say it is impossible to understand a function signature on the first pass if it takes several lines and whitespace to write ... but it is much easier when the the declaration is short enough to fit on a single line. An @ on the line above complicates the signature parsing, but can be mentally processed separately. The same is true of a named-something-or-other in the middle. Having to switch parsing modes to understand an internal ([int, float, int] -> List[int]), and then to pop that back off the stack is much harder. Hard enough that you really ought to help your reader out with a name, and let them figure out what that names means separately, when their brain's working memory isn't already loaded with the first part of your own function, but still waiting for the last part.
It separates the type declaration from the point at which it is used, potentially far away from where it is used.
The sort of code that passes around functions tends to pass around many functions, but with only a few signatures. If this is really the only time you'll need that signature (not even when you create the functions that will be passed from a calling site?), then ... great. But be nice to your reader anyhow, unless the signature is really so simple that the type-checking software should infer it for you. Then be nice by leaving it out as cruft. [As an aside, I would see some advantage to def myfunc(f:like blobfunc) pointing to an examplar instead of a specifically constructed function-type. You discuss this later as either ... f:blobfunc ... or ... f:blobfunc=blobfunc ... and I would support those, if other issues can be worked out.] -jJ

On Fri, Dec 24, 2021 at 06:24:03PM -0000, Jim J. Jewett wrote:
That's only true if IntToIntFunc is a builtin, otherwise it needs to be defined somewhere. It doesn't just magically happen. If you are using the declaration many times, then I acknowledge that it may be worth the effort of pre-declaration and naming. (But see below.) Particularly if the signature is complicated, although I think that will be relatively rare. But in the worst case, you may only use it once. So the entire cognitive burden of pre-declaration (both writing it and reading it) applies to that one use.
The @callable def section is a one-time definition, and not logically part of each function definition where it is used.
The status quo is that we can use an anonymous type in the annotation without pre-defining it, using Callable. PEP 677 proposes a new, more compact syntax for the same. Any proposal for function prototypes using `def` is directly competing against Callable or arrow syntax for the common case that we want an anonymous, unnamed type written in place. Even in the case that we want to give the type a name that we plan to use repeatedly, this `def` syntax is still competing directly against what is already possible using the status quo: use Callable to create a named type alias. But with the `def` syntax, you can *only* use it as a named, pre-defined object. So half, maybe 90%, of your use-cases disappear. Any time that we have a short, simple Callable that doesn't require a name, why would we bother creating a do-nothing function just so we can use it as a prototype? I don't think many people would. I know I wouldn't. That would be the equivalent of filling your program with trivial plus_one(x) and times_two(y) functions instead of just using `x+1` and `2*y`. So the benefit of the `def` syntax comes from that relatively small subset of cases: - the callable signature is complicated; - we wish to refer it it multiple times; - giving it a name (like "FileOpener", say, not "IntToInt") aids clarity. That's not to be sneered at. But in those circumstances, we don't need the `def` syntax, because we can already use Callable and a type alias. So the `def` syntax adds nothing we don't already have, it is no easier to use, it is more verbose, not less. But if we can use an existing function as the prototype instead of having to declare the prototype, that shifts the balance. If we already have some function, then there is no extra cost in having to declare it and give it a name, it already has been declared and given a name.
I would say the opposite: most callback or key functions have very simple signatures. If my function takes a key function, let's say: def spam(mylist:[str], a: int, b: float, c: bool|None, key: Callable[[str], str], ) -> Eggs: mylist = sorted(mylist, key=key) ... the relevant signature is (str) -> str. Do we really need to give that a predefined named prototype? def StrToStr(s: str) -> str: pass I would argue that very few people would bother. If somebody did, they probably also defined type aliases for ListOfStr and BoolOrNone, and wish they were using Java or Pascal *wink* It seems to me that most callbacks and key functions have short signatures. Certainly all the ones I have written do: they typically take a single argument, of a known type, and return a known type.
I notice that you just used something very close to PEP 677 arrow syntax totally unself-consciously, without any need to explain it. I think this is good evidence that far from being confusing, this is a completely natural syntax that we already interpret as a function prototype.
Hard enough that you really ought to help your reader out with a name,
What are you going to name it? Int_and_Float_and_Int_returns_List_of_Int_Function tells us nothing that (int, float, int) -> list[int] Callable[[int, float, int], list[int]] doesn't already say. Naming functions is hard. Naming function *prototypes* is even harder. Just duplicating the prototype in the name is noise. We don't bloat our code with say-nothing comments: mylist.sort() # sort mylist mylist.append(x) # append x to mylist or at least we hopefully don't do it beyond the initial first few months of learning to program. We let the code speak for itself. But I agree with you, if a type is complex enough that a meaningful name, or even a generic name, helps comprehension, that we should name it. We can already do that with type aliases. -- Steve

Steven D'Aprano wrote:uble the size of Callable.
I think it takes only the characters needed to write the name IntToIntFunc. ... you may only use it once.
Could you provide an example where it is only used once? The only way I can imagine is that you use it here when when defining your complicated function that takes a callback (once), but then you never actually call that complicated function, even from test code, nor do you expect your users to do so.
The status quo is that we can use an anonymous type in the annotation without pre-defining it, using Callable.
OK. I'm not sure it would be a good idea, but we agree it is legal.
PEP 677 proposes a new, more compact syntax for the same.
Does it? I agree that "(int, float) -> bool" is more compact than typing.Callable[...], but that feels like optimizing for the wrong thing. I dislike the PEP's flat_map as an example, because it is the sort of infrastructure function that carries no semantic meaning, but ... I'll use it anyhow. def flat_map(l, func): out = [] for element in l: out.extend(f(element)) return out def wrap(x: int) -> list[int]: return [x] def add(x: int, y: int) -> int: return x + y It is reasonable to add a docstring to flat_map, but I grant that this doesn't work as well with tooling that might involve not actually seeing the function. I agree that adding a long prefix of: from typing import Callable def flat_map( l: list[int], func: Callable[[int], list[int]] ) -> list[int]: is undesirable. But the biggest problem is not that "Callable..." has too many characters; the problem is that "Callable[[...], list[...]]" requires too many levels of sub-parsing. The PEP doesn't actually say what it proposes, [and you've suggested that my earlier attempt was slightly off, which may not bode well for likelihood of typos], but I'll *guess* that you prefer: def flat_map( l: list[int], func: ((int) ->[int]) ) -> list[int]: which is slightly shorter physically, but not much simpler mentally. It therefore creates an attractive nuisance. def flat_map( l: list[int], func: wrap ) -> list[int]: on the other hand, lets you read this definition without having to figure out what "wrap" does at the same time. "wrap" is a particularly bad example (because of the lack of semantic content in this example), but I think it still easily beats the proposed new solution, simply because it creates a "you don't need need to peer below this right now" barrier.
I'm saying that catering to that "common" case is a trap, often leading you to a local optima that is bad globally.
But if we can use an existing function as the prototype instead of having to declare the prototype, that shifts the balance.
I agree that re-using an existing function with the correct signature is better, *even* when that function doesn't make a good default. ...
If you really care about enforcing the str, then yes, it is worth saying key: str_key and defining str_key function as an example def str_key(data:str)->str return str(data)
I would argue that very few people would bother.
Because it would usually be silly to care that the list really contained strings, as opposed to "something sortable". So if you do care, it is worth making your requirement stand out, instead of losing it in a pile of what looks like boilerplate. -jJ
participants (14)
-
Andrew Svetlov
-
asleep.cult@gmail.com
-
Barry Scott
-
Barry Warsaw
-
Brett Cannon
-
Carl Meyer
-
Chris Angelico
-
Christopher Barker
-
Guido van Rossum
-
Jim J. Jewett
-
Joao S. O. Bueno
-
MRAB
-
Serhiy Storchaka
-
Steven D'Aprano