[Python-3000] Draft pre-PEP: function annotations

[Phillip J. Eby]

At 05:58 PM 8/12/2006 +1000, Nick Coghlan wrote:
>Phillip J. Eby wrote:
>>At 03:39 PM 8/12/2006 -0700, Talin <talin at acm.org> wrote:
>>>So programmer C, who wants to incorporate both A and B's work into his
>>>program, has a dilemma - each has a sharing mechanism, but the sharing
>>>mechanisms are different and incompatible. So he is unable to apply both
>>>A-type and B-type metadata to any given signature.
>>Not at all.  A and B need only use overloadable functions,
>
>Stop right there. "A and B need only use overloadable functions"? That 
>sounds an awful lot like placing a constraint on the way annotation 
>libraries are implemented in order to facilitate a single program using 
>multiple annotation libraries - which is exactly what Talin is saying is 
>needed!

You could perhaps look at it that way.  However, I'm simply using 
overloadable functions as a trivial example of how easy this is to handle 
without specifying a single mechanism.  There are numerous overloaded 
function implementations available, for example, including ad-hoc 
registry-based ones (like the ones used by pickle) and other mechanisms 
besides overloaded functions that do the same thing.  PEP 246 adaptation, 
for example, as used by Twisted and Zope.

My point is that:

1. trivial standard extension mechanisms (that are already in use in 
today's Python) allow libraries to offer compatibility between approaches, 
without choosing any blessed implementation or even approach to combination

2. there is no need to define a fixed semantic framework for 
annotations.  Guidelines for combinability (e.g. a standard interpretation 
for tuples or lists) might be a good idea, but it isn't *necessary* to 
mandate a single interpretation.


>(and using overloaded functions for this strikes me as hitting a very 
>small nail with a very large hammer).

Remember: Python is built from the ground up on overloaded 
functions.  len(), iter(), str(), repr(), hash(), int(), ...  You name it 
in builtins or operator, it's pretty much an overloaded function.

These functions differ from "full" overloaded functions in only these respects:

1. There is no framework to let you define new ones

2. They are single-dispatch only (except for the binary arithmetic 
operators, which have a crude double-dispatching protocol)

3. They do not allow third-party registration; classes must define 
__special__ methods to register implementations

(Some other overloaded functions in Python, such as pickle.dump and 
copy.copy, *do* allow third-party registrations, but they have ad-hoc 
implementations rather than using a common base implementation.)

So, saying that overloaded functions are a large hammer may or may not be 
meaningful, but it's certainly true that they are in *enormous* use in 
today's Python, even for very small nails like determining the length of an 
object.  :)

Indeed, the *default* way of doing almost anything in Python that involves 
multiple possible implementations is to define an overloaded function -- 
regardless of how small the nail might be.


>However, what we're really talking about here is a scenario where you're 
>defining your *own* custom annotation processor: you want the first part 
>of the tuple in the expression handled by the type processing library, and 
>the second part handled by the docstring processing library.
>
>Which says to me that the right solution is for the annotation to be split 
>up into its constituent parts before the libraries ever see it.
>
>This could be done as Collin suggests by tampering with 
>__signature__.annotations before calling each decorator, but I think it is 
>cleaner to do it by defining a particular signature for decorators that 
>are intended to process annotations.

Now you're embedding a particular implementation again.  The way to do this 
that imposes the least constraints on users, is to just have an 
'iter_annotations()' overloadable function, and let it iterate over lists 
and tuples, and yield anything else, e.g.:

     @iter_annotations.when(tuple)
     @iter_annotations.when(list)
     def iter_annotation_sequence(annotation):
         for a in annotation:
             for aa in iter_annotations(a):
                 yield aa

Now, if you have some custom annotation type that contains other 
annotations, you need only add a method to iter_annotations, and everything 
works.

In contrast, your approach is too limiting because you're *creating a 
framework* that then everyone has to conform to.  I want annotations to be 
framework-free.  I don't even think that the stdlib needs to provide an 
iter_annotations function, because there's no reason not to just define a 
method similar to the above for the specific operations you're doing.

In fact the general rule of overloadable functions is that the closer to 
the domain semantics the function is, the better.  For example, a 
'generateCodeFor(annotation)' overloaded function that can walk annotation 
sequences itself is a better idea than writing a non-overloaded function 
that uses iter_annotations() and then generates code for individual 
annotations, because it allows for better overloads.

For example, if you have a type that contains something that would 
ordinarily be considered separate annotation objects, but which the code 
generator could combine in some way to produce more optimal code.  Walking 
the annotations and then generating code would rob you of the opportunity 
to define an optimization overload in this case.

And *that* is why I don't think the stdlib should impose any semantics on 
annotations -- semantic imposition doesn't *fix* incompatibility, it 
*creates* it.

How?  Because if somebody needs to do something that doesn't fit within the 
imposed semantics, they are forced to create their own, and they now must 
reinvent everything so it works with their own!

This is the history of Python frameworks in a nutshell, and it's entirely 
avoidable.  We should leave the semantics open, precisely so that it will 
force people to make their code *extensible*.  As a side benefit, it 
provides a nice example of when and how to use overloaded functions 
effectively.