[Python-3000] pep 3124 plans

[Talin]

Phillip J. Eby wrote:
> At 07:49 AM 7/20/2007 -0700, Guido van Rossum wrote:
>> On 7/19/07, Joe Smith <unknown_kev_cat at hotmail.com> wrote:
>>> So the state of the PEP? From the rest of the posts so far,
>>> it sounds like there is no real objection to the basic end user API as
>>> described in the PEP,
>> Actually I want to reserve judgment on that until the PEP is rewritten
>> to explain and document the underlying mechanisms. It is currently
>> impossible (for me, anyway) to understand how the machinery to support
>> the described features could be built. Without that I cannot approve
>> the PEP. Phillip knows this but is too busy to work on it.
> 
> Actually, I was under the impression you didn't want the API 
> described in the PEP, and wanted the following changes in addition to 
> dropping method combination, aspects, and interfaces:

I'd like to clarify these requirements a little bit:

On the issue of method combination, aspects, and interfaces: Guido has 
not made a pronouncement on whether these things may or may not be 
accepted at some time in the future. What he has said is that he doesn't 
*yet* understand the use case for them, and that these should be 
separate PEPs so that we can argue their merits independently. What he's 
strongly against (if my understanding is correct) is a "package deal" 
where he is forced to accept all of the features, or none.

I get the sense that the need for some of these advanced features 
becomes apparent only after having worked with generics for a while. If 
that's the case, then the best hope for including them in the stdlib is 
to get an implementation of generics into the hands of lots of Python 
programmers so that they can become familiar with them.

> * :next_method as a keyword-only argument
> 
> * @somegeneric.overload as the standard decorator (w/no @overload or @when)

You mentioned earlier that there was a design reason for preferring 
@overload and @when vs. the earlier RuleDispatch syntax, but the 
explanation you gave wasn't very clear (to me anyway).

(I personally prefer the @somegeneric.overload, but that's purely an 
aesthetic value judgement - if there's a strong architectural advantage 
of the other syntax, I'd like to hear it.)

> * advance declaration of a function as overloadable (which is also 
> required by the previous change and by your preference not to modify 
> functions in-place)

Right. There are two reasons that I think that post-hoc overloading runs 
into problems. The first, as you mentioned, is that it's difficult to 
implement without some kind of trickery.

The second reason - this is my opinion - is that it too much resembles 
the mythical "comefrom" statement (the opposite of "goto"). The 
"comefrom" statement is intended to be a joke - the worst possible 
language feature from the standpoint of being able to manually trace the 
flow of execution of a program.

I do think that there are use cases for being able to 'decorate' (in the 
broader sense) the execution of a function, in an aspect-like way; But I 
also think that such power should not be used casually, and places where 
its used should stick out in a way that makes them visually obvious and 
searchable.

> Also, I didn't know you wanted an explanation of how the underlying 
> mechanisms work in general.  I thought the only piece you were 
> looking for more explanation of was the method combination machinery 
> -- which would be moot if we're scaling back the API as described by the above.
> 
> Just to be sure I'm clear as to what you want, is that the only 
> mechanism you're unclear on, or is the whole thing unclear?  The 
> whole thing was inspired by your overloading prototype, I've just 
> made all the concrete bits of it more... "generic".

It seems to me that PEPs should only be required to explain their 
mechanisms if there's some doubt or controversy about the 
implementation. It seems to me that this PEP pushes the bounds of what 
is efficiently doable, so some extra explanation is required.

One issue that hasn't been satisfactorily resolved is the handling of 
the 'self' parameter. At least, let me give my explanation of what I 
think the issue is and see if we're on the same page:

Overloading a class method requires special treatment of the 'self' 
parameter because there's an implicit constraint on what types of 
objects can be passed as 'self': for any method defined in any class, 
the 'self' parameter must be an instance of the class (or a subclass) in 
which the method is defined. Now, this would be trivial if we required 
the programmer to explicitly declare the type of 'self', but this 
violates DRY and has the potential to cause mischief if the programmer 
forgets to update the method signature when they change the class.

In order to avoid this syntactical redundancy, there is a desire to be 
able to automatically detect the type of the class in which the overload 
is declared.

This is hard to do, because the "overload" machinery is handled by a 
function decorator, which runs before the class is actually constructed. 
Various methods for deducing the class have been proposed, but they have 
all so far been somewhat problematic, especially in light of "new-style" 
metaclasses.

I can think of only two approaches for solving this cleanly.

The first is that the overload decorator should be given some C-code 
help. Now, I recognize that part of your goal was to make the initial 
prototype a "pure Python" implementation in order to make life easier 
for Jython/IronPython and friends. That is certainly laudable. However, 
if the C-code help is a relatively small function that can be 
reimplemented for the other interpreters, then the impact on portability 
will be small.

The other approach is to somehow defer the work until after the class is 
fully constructed. The question then is when will the work be done - in 
other words, where should the decorator hook its fixup callback?

Even assuming we had some sort of hook that would be triggered when a 
class has finished construction, then the question is what about 
non-member generic functions? Since they are not contained in a class 
body, this hypothetical hook will never be called, and thus the methods 
won't be "finished". (A way around this would be to say that the only 
thing that the class-construction hook does is to add the additional 
type information for 'self', and the method is otherwise finished and 
ready to go as soon as the decorator is completed.)

If it turns out that there's no way to get a callback when the class has 
finished being built, then we may have to defer finishing the 
construction until the first time the generic function is called. This 
wouldn't be too bad, considering that there's a bunch of other stuff 
that is lazily calculated on first call anyway, from what I understand.

> That is, instead of using issubclass or other explicit relationship 
> tests between overload signatures, I use a generic function 
> implies().  Instead of simply storing a method added as an overload, 
> I use a "combine_actions()" generic function to combine it with any 
> method that's already there (possibly including a method type for "No 
> Method Found").  Instead of simply finding the most-specific matching 
> signature on cache misses, I use combine_actions() to combine *all 
> applicable* actions (i.e., all those that the calling signature implies()).
> 
> The combine_actions() function uses another generic function, 
> overrides(), to compare method priorities.  overrides() is defined so 
> that Around beats Before beats After beats regular methods beats no 
> method found.  The overrides() of two methods of the same type is 
> determined by which signature implies() the other, without also being 
> implied *by* the other.
> 
> If there is no overrides() order between two methods, you get an 
> AmbiguousMethod combining the two -- which can be overridden by any 
> method whose signature implies() everything in the AmbiguousMethod.
> 
> All this is pretty much the same as in your prototype, except that 
> it's done by adding these rules to the generic functions, rather than 
> by hardcoding them.  That's why it's bigger than your prototype, but 
> also why it's extensible in terms of adding new method types or ways 
> to specify signatures.
> 
> I then also added the ability to attach different dispatchers to a 
> function, so that you could replace the simple "tuple of types" 
> matching with more sophisticated engines like RuleDispatch's, while 
> still retaining the ability to use the same method combinations and 
> existing overloads registered for a function.
> 
> That is, it lets you keep the same API for defining overloads and 
> method combinations as the basic implementation, while allowing the 
> actual overload targets and dispatching mechanisms to vary.
> 
> That's pretty much it except for Aspects and Interfaces.  I've ended 
> up making my Aspect implementation available separately in the 
> ObjectRoles cheeseshop package, renaming them Roles instead of Aspects.
> 
> (And yes, I will add all the above explanation to the PEP.)
> 
> 
>> AFAIK Phillip has declared that his implementation only uses (or could
>> be made to only use) isinstance()/issubclass(), and the overriding of
>> these two used by the ABCs is actually very convenient for the GF PEP.
> 
> Yep.  The overload of "implies(c1:type, c2:type)" is 
> "issubclass".  "isinstance()" isn't used, since that would render 
> your type-tuple caching strategy unusable.
> 
> _______________________________________________
> Python-3000 mailing list
> Python-3000 at python.org
> http://mail.python.org/mailman/listinfo/python-3000
> Unsubscribe: http://mail.python.org/mailman/options/python-3000/talin%40acm.org
>