[Python-3000] Abilities / Interfaces

[Guido van Rossum]

On 11/21/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 11:16 AM 11/21/2006 -0800, Guido van Rossum wrote:
> >Phillip then argues that he doesn't want to encourage introspectable
> >interfaces. I think others have use cases for those though.
>
> Examples?

I'll wait for others to speak up, but I'd think that automatically
generating appropriate docs for interfaces qualifies. Python in
general has a very proactive approach to introspection -- everythign
is introspectable unless it would violate safety.

> >  It seems
> >that Phillip's approach only hanges well together if everybody totally
> >adopts the generic functions religion; I think that's an unlikely
> >(since too radical) change of course for Python 3.0.
>
> Is it really radical?

Given some of your examples, I'd say so.

Python users are used to providing functionality in classes by
defining special methods.

Java users (and users of many other languages) are used to interfaces
that describe contracts (even if they *appear* only to be syntactic)
and ways to claim that a class fulfills the contract, as part of the
class definition syntax. Optional argument type declarations in Python
would be happy to use such a mechanism too.

These are both quite differently than binding a class and a specific
operation (e.g. flatten) through some call to something named
addmethod(). While the approaches are probably provably equivalent in
expressive power, IMO doing common things is a lot more intuitive
using interfaces. (Always accepting that "intuitive" isn't -- it just
means "familiar from a prevous context".)

While we're discussing familiarity, if you want to continue the
discussion, may I remind you that most people (including myself)
aren't familiar with RuleDispatch, and examples like

  addmethod(addmethod, adder_for_my_gf_type, my_gf_type)

or

  @addmethod.when_object(iter)

are downright off-putting? What on earth is "when_object(iter)"
supposed to mean? HCI indeed! :-)

> Consider the fact that single-dispatch generic functions are ubiquitous in
> the language and the stdlib, and have been pretty much forever.  They just
> vary in how they're implemented, and there's no One Obvious Way for
> developers to implement new ones, nor is there a standard way to add a
> method to an existing one.

Actually the standard way is to have a __special__ method and define
that in your class. We've recently discussed doing that for dir(), for
example (on python-dev, as it can safely go into 2.6).

> That's because some generic functions use if-isinstance checks (bad), while
> others use custom __special__ methods (not bad, but not great), registries
> (good), adaptation (okay), or generic function libraries.

__special__ methods are great for many purposes because they are well
understood. Registries are mostly good when there's a need for an
independent third party to add an ability; IMO this isn't as common in
most places as you seem to think.

> So, all I am proposing is that we:
>
>     * provide a standard GF implementation for the simple cases
>     * ...that is extensible to allow others to handle the more complex
> cases, by having a generic API for manipulating generic functions

I'm all for this. I think it would be more successful if it had
support for interfaces.

> So that the "one obvious way" to create new generics is to use the standard
> GF implementation unless you need something else.

I'd rather not try to teach people that they can't define a __len__
method any more but must update half a dozen generic functions in
order to create a new sequence type. Overriding __special__ methods
works fine for the most part.

The one exception I've seen is binary operations where some types are
too aggressive in rejecting unacceptable 'other' operands, raising
TypeError where they should return NotImplemented. I believe datetime
does this, but I also believe this is mostly to work around issues
with comparisons in 2.x that have been fixed in 3.0 (and datetime has
been taught to be less rude).

Sure, I see a use case for defining an operation that the class
author(s) did *not* foresee, but that's the exception rather than the
rule.

> And, there would be a
> standard way to add new methods to existing generic functions, perhaps
> using an addmethod() builtin or decorator, and it could be designed so that
> e.g.:
>
>      addmethod(iter, somefunc, sometype)

I don't actually understand what you want this example to mean. What
is 'iter' supposed to be? A generic function?

> would actually work by doing 'sometype.__iter__ = somefunc' under the
> hood.

How would addmethod know this?

> This allows us not to have to change any builtin generics that are
> based on special method names.  In other words, the mechanism isn't
> radical, nor is the idea of having generics.  It's just an improvement in
> ease-of-use: an HCI change, not a comp-sci change!

You can repeat that as many times as you want but that doesn't make it so.

At least one poster has already remarked that the head-exploding
capacity of generic functions is greater than that of metaclasses. I
think that's a gross exaggeration, but calling it "simple" and "easy
to use" is an exaggeration too. It's quite deep (otherwise we'd seen
it in Java already ;-).

> And, with this relatively simple mechanism, all the fancier forms of
> generic functions (or interfaces and adapters) can be implemented via user
> libraries.  (More on how, below.)
>
>
> >  It also doesn't
> >seem to work for abilities that are tied to an instance instead of to
> >a class as Zope allows (see below).
>
> Actually, it still allows for that, as long as the base generic function
> system is extensible.

How should it be extensible? Please explain this without making any
use of examples from RuleDispatch.

> For example, RuleDispatch generic functions can
> choose implementations based on conditions such as whether an object has a
> particular attribute.  Zope could easily add a generic function type that
> uses their instance-based interface stuff to do the same thing.

What's a generic function type? How does one create one?

> I am merely proposing that Python not provide this sort of
> introspection-oriented stuff in the core or stdlib, not that nobody should
> be *allowed* to have it.

And I think introspection is too important to leave out of the core.
People will want to introspect and find a way to do it by referring to
implementation details -- just like there's currently code in
inspect.py to reconstruct a function's signature from random
attributes of the code object, including flag bits.

> Indeed, I explicitly want the Python generic function API (e.g. addmethod,
> hasmethod, or whatever we call them) to *itself* be generic, so that users
> can create their own types that work with the Python-provided operations
> for generic function manipulation.  That is, I should be able to call:
>
>      addmethod(addmethod, adder_for_my_gf_type, my_gf_type)

OK, I take it back. My brain just exploded. it *is* worse than
metaclasses. Please find enclosed the bits of brain that I scraped of
my monitors. :-)

> So that others can then call:
>
>      # this calls adder_for_my_gf_type under the hood:
>      addmethod(some_gf, some_method, some_cls)
>
> where 'some_gf' is an instance of 'my_gf_type'.  This allows us not to keep
> the core or stdlib implementation of generic functions quite simple to
> handle the 80% (single dispatch) or 90% (concrete type-based multiple
> dispatch) with ease.  The final 10% (predicate dispatch, fancy interfaces,
> etc.) can then be done by outside libraries, since the use cases in that
> final 10% are more likely to vary than the base 80-90%.
>
> Using the 'simplegeneric' library from PyPI (easy_install simplegeneric):

Hey, another example referring to technology so advanced that for me
and other readers it is indistinguishable from magic. ;-)

>      from simplegeneric import generic
>
>      @generic
>      def addmethod(gf, method, cls):
>          """Add a method to a generic function"""
>          raise TypeError("Unknown generic function type", gf)
>
>      @addmethod.when_object(iter)
>      def add_to_iter(gf, method, cls):
>          # XXX should have more error checking here, e.g.
>          #     check for a local __iter__ first
>          cls.__iter__ = method
>
>      # ... similar declarations for other builtin generics
>
>      @addmethod.when_type(FunctionType)
>      def add_to_function(gf, method, cls):
>          if hasattr(gf, 'when_type'):
>              gf.when_type(cls)(method)
>          else:
>              raise TypeError("Not a generic function", gf)
>
> And there you are: an extensible way to add new extensible function types,
> while using the same API for all of them.

Poof. You disappear in a cloud of orange smoke.

> (If you need multiple or predicate dispatch or method combining, it's easy
> to have decorators that apply to the method itself, so that the same
> three-argument addmethod() can still be used.)
>
>
> >3. API design -- how do we spell the various concepts? E.g.
> >has_ability(x, A) asks whether object x has the ability A;
> >provides_ability(C, A) asks whether class C provides the ability A to
> >its instances. We could state that provides_ability(C, A) and
> >isinstance(x, C) implies has_ability(x, A).
>
> Even if we explicitly have some type of "ability" object, I'd like them to
> be defined in terms of generic functions, so that I could effectively say
> something like:
>
>      sequence_like = (iter & len)
>
> (or perhaps spelled out in some other fashion) to create an "ability"
> representing iter-ability and len-ability.

That's an interesting thought but it works with or without generic
functions -- it's just "ability algebra". I think I've played with
"type algebra" using similar ideas in a blog entry once.

> One of the biggest conceptual/modeling issues with Zope-style interfaces is
> that they don't allow this kind of fine-grained protocol combination.  See,
> for example, zope.interface's long history of trying to nail down various
> Python protocols through elaborate interface inheritance
> hierarchies.  Defining interfaces strictly in terms of operations
> eliminates this issue.

Not really, IMO. The underlying problem is that standard type
hierarchy defies capturing it in interfaces *BECAUSE INTERFACES DIDN'T
EXIST WHEN THEY WERE CREATED*.

> >- How does this interact with generic functions?
>
> The advanced abilities (per-instance/on-the-fly and multi-operation tests)
> will likely affect performance and/or simplicity of the default
> implementation.  A purely type-based system can be implemented efficiently,
> because declaring an interface can simultaneously add a concrete type
> registration for all affected generics.  (And registering an
> interface-linked method in a generic function can pull in all the concrete
> classes.)

This seems to argue *for* interfaces?

> Per-instance tests, however, increase code complexity.  Generally speaking,
> RuleDispatch does per-instance tests after class tests, but the way it gets
> reasonable performance is by building decision trees beforehand to manage
> the possible tests and avoid overlap.  If it actually had to call
> individual methods, or call a method more than once (ob.has_ability(Foo),
> ob.has_ability(Bar), etc.)  it would be considerably slower to select an
> option.

That would make more sense as Foo.implemented_by(ob), right?

> Also -- and this is the real kicker -- what do you do if more than one
> ability applies?  Now we have to have precedence rules for what's more
> specific.  The advantage of defining an ability as a set of one or more
> applicable generic functions is that precedence is comparatively
> straightforward: supersets take precedence over subsets, and overlaps are
> ambiguous.  You also have some possibility of being able to implement this
> by registration-time checks, without needing to build a dispatch tree at all.

Couldn't ability inheritance serve the same purpose?

> I don't know of anybody who really uses per-instance interface declarations
> except for Zope.  I used them for a little while with PEAK, and decided
> they were the wrong idea for me; it made more sense to adapt to an
> interface that provides the metadata, or to use a RuleDispatch function
> that just directly introspected for whatever was needed.  I can't think of
> anything in the core or stdlib that would even need to go that far.
>
> Again, my argument is that anything other than isinstance-based single and
> multiple-dispatch is too framework-dependent to be part of the
> language.  Simple things should be simple, complex things should be possible.

Sure, I can dig that. I'd like to have separate APIs for querying
instances and classes, but I don't mind if the built-in facilities
only let you define abilities for classes. As long as it's possible to
add an ability to a class dynamically (to deal with the problem of
pre-existing 3rd party classes that implement de-facto interfaces but
don't declare them).

> >Answering these and similar questions probably requires a
> >standardization committed. (Any volunteers?)
>
> Note that Zope has already tried for many years to do exactly this, and
> there is no indication that they have actually succeeded.

I tend to blame that on their inability to change the language and the
standard library though. Py3k has that option, and I'm not afraid to
exercise it if it would help.

> When I first
> mentioned PyProtocols on Python-Dev many years ago, Samuele Pedroni argued
> that individual operations (and groups thereof) should be the currency of
> interfaces, and after playing with them a bit, I agreed.  PyProtocols
> defines such interfaces as e.g.:
>
>      protocolForType(file, ['read', 'close'])
>
> to mean "file-like object with 'read' and 'close' methods".  If I define a
> class as supporting that protocol, and somebody wants to adapt to
> 'protocolForType(file, ["read"])', then my type will work.
>
> See
> http://peak.telecommunity.com/protocol_ref/protocols-generated-type.html
> for a more complete explanation, including how to get the system to
> actually check for the method names (so as not to require explicit
> declarations).
>
> I think Samuele's idea (i.e., this) works a lot better than trying to
> define a rigid interface hierarchy for builtin-protocols, although I'd just
> as soon be able to do something like:
>
>      file.read & file.close
>
> rather than having to use strings.  But all this assumes that we are
> providing a builtin way to spell abilities and introspect them, which I
> still think is overkill for what Python actually needs to provide as a base.

Well, I think it would be useful and I'm sure it would be popular.

I'm asking for a committee to lay this to rest by defining the one
true hierarchy for containers. Other languages have succeeded in doing
so (even Java).

> In short, I think it's adding interface introspection that's the radical
> move, where generic functions are just a bit of polish that brings a bit
> more order to what Python already has.  In contrast, interfaces are a
> foreign religion imported from other languages, while the roots of the
> generic function faith (len, iter, etc.) have already been in place since
> the dawn of time!  :)

Python has a long tradition of borrowing ideas from other languages,
and mostly very successful.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)