[Python-3000] Abilities / Interfaces

[Phillip J. Eby]

At 11:16 AM 11/21/2006 -0800, Guido van Rossum wrote:
>Phillip then argues that he doesn't want to encourage introspectable
>interfaces. I think others have use cases for those though.

Examples?


>  It seems
>that Phillip's approach only hanges well together if everybody totally
>adopts the generic functions religion; I think that's an unlikely
>(since too radical) change of course for Python 3.0.

Is it really radical?

Consider the fact that single-dispatch generic functions are ubiquitous in 
the language and the stdlib, and have been pretty much forever.  They just 
vary in how they're implemented, and there's no One Obvious Way for 
developers to implement new ones, nor is there a standard way to add a 
method to an existing one.

That's because some generic functions use if-isinstance checks (bad), while 
others use custom __special__ methods (not bad, but not great), registries 
(good), adaptation (okay), or generic function libraries.

So, all I am proposing is that we:

    * provide a standard GF implementation for the simple cases
    * ...that is extensible to allow others to handle the more complex 
cases, by having a generic API for manipulating generic functions

So that the "one obvious way" to create new generics is to use the standard 
GF implementation unless you need something else.  And, there would be a 
standard way to add new methods to existing generic functions, perhaps 
using an addmethod() builtin or decorator, and it could be designed so that 
e.g.:

     addmethod(iter, somefunc, sometype)

would actually work by doing 'sometype.__iter__ = somefunc' under the 
hood.  This allows us not to have to change any builtin generics that are 
based on special method names.  In other words, the mechanism isn't 
radical, nor is the idea of having generics.  It's just an improvement in 
ease-of-use: an HCI change, not a comp-sci change!

And, with this relatively simple mechanism, all the fancier forms of 
generic functions (or interfaces and adapters) can be implemented via user 
libraries.  (More on how, below.)


>  It also doesn't
>seem to work for abilities that are tied to an instance instead of to
>a class as Zope allows (see below).

Actually, it still allows for that, as long as the base generic function 
system is extensible.  For example, RuleDispatch generic functions can 
choose implementations based on conditions such as whether an object has a 
particular attribute.  Zope could easily add a generic function type that 
uses their instance-based interface stuff to do the same thing.

I am merely proposing that Python not provide this sort of 
introspection-oriented stuff in the core or stdlib, not that nobody should 
be *allowed* to have it.

Indeed, I explicitly want the Python generic function API (e.g. addmethod, 
hasmethod, or whatever we call them) to *itself* be generic, so that users 
can create their own types that work with the Python-provided operations 
for generic function manipulation.  That is, I should be able to call:

     addmethod(addmethod, adder_for_my_gf_type, my_gf_type)

So that others can then call:

     # this calls adder_for_my_gf_type under the hood:
     addmethod(some_gf, some_method, some_cls)

where 'some_gf' is an instance of 'my_gf_type'.  This allows us not to keep 
the core or stdlib implementation of generic functions quite simple to 
handle the 80% (single dispatch) or 90% (concrete type-based multiple 
dispatch) with ease.  The final 10% (predicate dispatch, fancy interfaces, 
etc.) can then be done by outside libraries, since the use cases in that 
final 10% are more likely to vary than the base 80-90%.

Using the 'simplegeneric' library from PyPI (easy_install simplegeneric):

     from simplegeneric import generic

     @generic
     def addmethod(gf, method, cls):
         """Add a method to a generic function"""
         raise TypeError("Unknown generic function type", gf)

     @addmethod.when_object(iter)
     def add_to_iter(gf, method, cls):
         # XXX should have more error checking here, e.g.
         #     check for a local __iter__ first
         cls.__iter__ = method

     # ... similar declarations for other builtin generics

     @addmethod.when_type(FunctionType)
     def add_to_function(gf, method, cls):
         if hasattr(gf, 'when_type'):
             gf.when_type(cls)(method)
         else:
             raise TypeError("Not a generic function", gf)

And there you are: an extensible way to add new extensible function types, 
while using the same API for all of them.

(If you need multiple or predicate dispatch or method combining, it's easy 
to have decorators that apply to the method itself, so that the same 
three-argument addmethod() can still be used.)


>3. API design -- how do we spell the various concepts? E.g.
>has_ability(x, A) asks whether object x has the ability A;
>provides_ability(C, A) asks whether class C provides the ability A to
>its instances. We could state that provides_ability(C, A) and
>isinstance(x, C) implies has_ability(x, A).

Even if we explicitly have some type of "ability" object, I'd like them to 
be defined in terms of generic functions, so that I could effectively say 
something like:

     sequence_like = (iter & len)

(or perhaps spelled out in some other fashion) to create an "ability" 
representing iter-ability and len-ability.

One of the biggest conceptual/modeling issues with Zope-style interfaces is 
that they don't allow this kind of fine-grained protocol combination.  See, 
for example, zope.interface's long history of trying to nail down various 
Python protocols through elaborate interface inheritance 
hierarchies.  Defining interfaces strictly in terms of operations 
eliminates this issue.


>- How does this interact with generic functions?

The advanced abilities (per-instance/on-the-fly and multi-operation tests) 
will likely affect performance and/or simplicity of the default 
implementation.  A purely type-based system can be implemented efficiently, 
because declaring an interface can simultaneously add a concrete type 
registration for all affected generics.  (And registering an 
interface-linked method in a generic function can pull in all the concrete 
classes.)

Per-instance tests, however, increase code complexity.  Generally speaking, 
RuleDispatch does per-instance tests after class tests, but the way it gets 
reasonable performance is by building decision trees beforehand to manage 
the possible tests and avoid overlap.  If it actually had to call 
individual methods, or call a method more than once (ob.has_ability(Foo), 
ob.has_ability(Bar), etc.)  it would be considerably slower to select an 
option.

Also -- and this is the real kicker -- what do you do if more than one 
ability applies?  Now we have to have precedence rules for what's more 
specific.  The advantage of defining an ability as a set of one or more 
applicable generic functions is that precedence is comparatively 
straightforward: supersets take precedence over subsets, and overlaps are 
ambiguous.  You also have some possibility of being able to implement this 
by registration-time checks, without needing to build a dispatch tree at all.

I don't know of anybody who really uses per-instance interface declarations 
except for Zope.  I used them for a little while with PEAK, and decided 
they were the wrong idea for me; it made more sense to adapt to an 
interface that provides the metadata, or to use a RuleDispatch function 
that just directly introspected for whatever was needed.  I can't think of 
anything in the core or stdlib that would even need to go that far.

Again, my argument is that anything other than isinstance-based single and 
multiple-dispatch is too framework-dependent to be part of the 
language.  Simple things should be simple, complex things should be possible.


>Answering these and similar questions probably requires a
>standardization committed. (Any volunteers?)

Note that Zope has already tried for many years to do exactly this, and 
there is no indication that they have actually succeeded.  When I first 
mentioned PyProtocols on Python-Dev many years ago, Samuele Pedroni argued 
that individual operations (and groups thereof) should be the currency of 
interfaces, and after playing with them a bit, I agreed.  PyProtocols 
defines such interfaces as e.g.:

     protocolForType(file, ['read', 'close'])

to mean "file-like object with 'read' and 'close' methods".  If I define a 
class as supporting that protocol, and somebody wants to adapt to 
'protocolForType(file, ["read"])', then my type will work.

See 
http://peak.telecommunity.com/protocol_ref/protocols-generated-type.html 
for a more complete explanation, including how to get the system to 
actually check for the method names (so as not to require explicit 
declarations).

I think Samuele's idea (i.e., this) works a lot better than trying to 
define a rigid interface hierarchy for builtin-protocols, although I'd just 
as soon be able to do something like:

     file.read & file.close

rather than having to use strings.  But all this assumes that we are 
providing a builtin way to spell abilities and introspect them, which I 
still think is overkill for what Python actually needs to provide as a base.

In short, I think it's adding interface introspection that's the radical 
move, where generic functions are just a bit of polish that brings a bit 
more order to what Python already has.  In contrast, interfaces are a 
foreign religion imported from other languages, while the roots of the 
generic function faith (len, iter, etc.) have already been in place since 
the dawn of time!  :)