[Python-3000] PEP 3124 - Overloading, Generic Functions, Interfaces, etc.

[Phillip J. Eby]

At 04:19 PM 5/14/2007 -0700, Guido van Rossum wrote:
>On 5/14/07, Phillip J. Eby <pje at telecommunity.com> wrote:
>>However, with respect, I didn't go to all the trouble of implementing
>>method combination in RuleDispatch just for the heck of it.  (And it
>>was considerable trouble, doing it the way CLOS implements it, until
>>I figured out an approach more suitable for Python and decorators.)
>
>So you owe us more motivating examples (in addition to the explanatory
>examples), showing how you had a particular problem, and you couldn't
>solve it cleanly using the usual suspects (subclassing, callbacks,
>etc.), and how method combining came to the rescue. Perhaps writing it
>up like a pattern description a la GoF might help.

It's really not that complicated.  If you have only strict precedence 
(i.e., methods with the same signature are ambiguous), you wind up in 
practice needing a way to disambiguate methods when you don't really 
care what order they're executed in (because they're being registered 
independently).

Before and After methods give you that escape, because they're 
assumed to be independent, and thus any number of libraries can thus 
register a before or after method for any given signature, without 
conflicting with each other.

So the "particular problem" I had is simply that when you are using 
GF methods as "observer"-like hooks, you need a way to specify them 
that doesn't result in ambiguities between code that's watching the 
same thing (but is written by different people).  And, the nature of 
these observer-ish use cases is that you sometimes need 
pre-observers, and sometimes you need post-observers.

(For example, a pre-observer like "block the sale if there's a hold 
on the item by a more valuable customer" or a post observer like, 
"send an email to the sales manager if this is an account we got from 
FooCorp.")

Can these use cases be handled with callbacks of some other 
sort?  Sure!  But then, we can and do also get by with implementing 
ad-hoc generic functions using __special__ methods and copy_reg and 
so on.  The point of the PEP was to provide a standardized API for 
generic functions and method combination, so you don't need to 
reinvent or relearn new ways of doing it for every single Python 
library that uses something that follows these patterns.

Indeed, having yet another implementation of generic functions was 
never the point of the PEP, as we already have several of them in the 
language and stdlib, plus several more third-party modules that implement them!

The point, instead, was to standardize an *API* for generic 
functions, so that one need only learn that API once.  A default GF 
implementation is merely necessary for bootstrapping that API, and 
useful for "batteries included"-ness.

So, if the bar is that a feature has to be unsolvable using ad hoc 
techniques, it seems the entire PEP would fail on those grounds.  We 
have plenty of ad hoc techniques for implementing GF's or quasi-GF's 
already, likewise for callbacks and the like.  The point was for you 
to Pronounce on One Obvious API (to Rule Them All).


>>But let me try to get closer to the issue that I have.  I honestly
>>don't see at this moment in time, how to split out most of the
>>features you don't like (mainly before/after/around), in such a way
>>that they can be put back in by a third-party module, without leading
>>to other problems.  For example, I fear that certain of those
>>features (especially before/after/around) require a single "blessed"
>>implementation in order to have a sane/stable base for library
>>inter-op, even if they *could* be separated out and put back
>>in.  That is, even if it's possible to separate the "mechanism", I
>>think that for "policy" reasons, they should have a canonical implementation.
>
>Please share more details, so your readers can understand this too.
>Right now the whole discussion around this appears to be in your head
>only, and what you write is the conclusion *you* have drawn.

Actually, the discussion about method combination precedence has been 
ongoing in several threads here on Py3K, mostly with Greg Ewing and 
Jim Jewett.  These discussions illustrate why having some basic 
operators of known precedence gives the system more stability when 
multiple libraries start playing together.


>But can you at least share enough of the problem so others can look at
>it and either suggest a solution or agree with your conclusion?

Sure.  Take a look at peak.rules.core (while keeping in mind all the 
bits that will be changed per your prior requests):

http://svn.eby-sarna.com/PEAK-Rules/peak/rules/core.py?view=markup

What you'll notice is that the method combination framework (Method, 
MethodList, combine_actions, always_overrides, and merge_by_default, 
if you don't count the places these things get called) is in fact 
most of the code, with relatively little of it being the actual 
implementation of Around, Before, or After (or even generic functions 
themselves!).

In principle, I could pull that framework out and leave just a 
mechanism for adding it back in.  But in practice, that framework 
lays down the principles of "governance" for method combination, as 
far as how to decide what things have precedence over what.

Thus, I'm skeptical of how useful it is in this area to provide 
mechanism but no policy.  It's always possible for someone to create 
their own independent policy within the mechanism -- even if there's 
a default policy.  But One Obvious Way suggests that there should be 
*some* sort of policy in place by default, just like we have a 
standard set of descriptors that implement the conventional forms of 
properties and methods.  You can subclass them or entirely replace 
them, but they cover all the typical use cases, and you can use them 
as examples to understand how to do more exotic things.

Meanwhile, if we didn't have the examples of properties and methods, 
how would we know we were designing descriptor hooks correctly?  If 
we are positing that I know enough to design the hooks correctly, we 
are implicitly positing that I know what the hooks will be used and 
useful *for*.  :)  However, by making various use cases (before, 
after, around, and the custom example) explicit in the PEP, I was 
attempting to provide the motivation and rationale for the design of 
the hooks.  (Although in all fairness, the hooks are not actually 
documented in the PEP yet, aside from a listing of function names.)


>I'm all for hooks. They can take the form of a particular factoring
>into methods that make it easy to override some method; or using GF's
>recursively for some of the implementation, etc.

This is in fact how it works now; all the extension API functions in 
the PEP are either existing GF's in peak.rules.core, or proposed for addition.