[Python-3000] Abilities / Interfaces

[Guido van Rossum]

Phillip, please shorten your posts. You're hogging all the bandwidth I
have for thinking about this. Please!

On 11/21/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> The only things I propose to *add*, would be:
>
> 1. functions like 'addmethod()' and 'hasmethod()', themselves to be
> generics in the style of iter() or len().  Whether they are implemented
> using registries or __special__ methods is of no consequence.

Please write a spec for these rather than assuming the reader already
knows what they are supposed to do.

If you could present simplified versions of these that were *not*
generic themselves first that would be a huge boon -- the
self-referentiality is what's head-exploding.

> 2. a 'defop' syntax as shorthand for 'addmethod()', such that e.g.:
>
>      import operator
>
>      class Foo:
>          defop iter(self):
>              ...
>          defop len(self):
>              ...
>          defop operator.getitem(self, key):
>              ...
>
> would produce exactly the same results as the same code using __special__
> methods,

New syntax is radical. Especialy when it's such superficial syntactic sugar.

> except that it could be expanded to add methods for arbitrary
> *new* operations not defined by the language.

I fail to see how we need new syntax to define new __special__
operations.  There are plenty of new __special__ methods being
introduced (e.g. by Zope, or by some library modules, like pickle, at
least originally) without new syntax.

Please spec this without new syntax first. The lack of new syntax is
not what blocks the proposal's acceptance.

> With these two things, Python would have enough foundation to allow any
> number of interface, adaptation, or generic function-based frameworks to
> run on top of it.  And, instead of each framework having to teach people
> new __special__ methods or provide different decorators, they could all
> simply say, "here are the generic functions or interfaces we use".

Again, that sounds mostly like a very thin layer of syntactic sugar.
Let's pretend we don't need that sugar -- at least not in the first
round.

> I submit that this is more than sufficient to cover the 80-90% of common
> use cases that don't need abstract interfaces at all, or which can simply
> use a generic function to stand in as an interface, the way 'iter' can
> stand in for the "iteration interface".

I'm still at a conceptual loss about this particular example. I fail
to see how 'iter' means 'iteration interface'. iter is a callable that
can do two very different things based on how it's called. Pretending
that it means "an object that has an __iter__ method" is too far a
stretch for me, since there is no way to find out that that is its
meaning except by knowing it. This is the problem I have with
non-introspectable interfaces. I don't understand why you find that an
advantage.

> And, in the same way that 'def __iter__(self): return self' means "I
> implement iteration", doing "defop IFoo(self): return self" would mean, "I
> implement IFoo".

Why the 'return self'? Did you mean '...'? Or did you really mean that
the 'return self' is part of the statement "I implement XXX"?

> (Note, by the way, that common usage of interface
> adaptation in Python is already to call IFoo(anObject), so this is *also*
> no change to current usage!)

You're taking some liberties by calling that current usage. What does
"interface adaptation" even mean? It's not part of the current Python
vocabulary.

> So, from my POV, this is actually a very modest proposal thus far.
>
> Now I'll answer those specific points you brought up that aren't covered in
> the above.

Maybe I'll have time to read and respond to those tomorrow.
>
> At 06:07 PM 11/21/2006 -0800, Guido van Rossum wrote:
> >are downright off-putting? What on earth is "when_object(iter)"
> >supposed to mean? HCI indeed! :-)
>
> Simplegeneric allows you to define methods for either types or
> instances.  The doc is at http://cheeseshop.python.org/pypi/simplegeneric
>
>
> > > So, all I am proposing is that we:
> > >
> > >     * provide a standard GF implementation for the simple cases
> > >     * ...that is extensible to allow others to handle the more complex
> > > cases, by having a generic API for manipulating generic functions
> >
> >I'm all for this. I think it would be more successful if it had
> >support for interfaces.
>
> Sure.  See my explanation above for why I think that this *does* support
> interfaces.
>
>
> > > So that the "one obvious way" to create new generics is to use the standard
> > > GF implementation unless you need something else.
> >
> >I'd rather not try to teach people that they can't define a __len__
> >method any more but must update half a dozen generic functions in
> >order to create a new sequence type.
>
> Implementing __len__ (or invoking 'addmethod(len,...)', or 'defop
> len(self):...') will still be sufficient to make a type len-able.  So code
> that uses len() will "just work" in that case.  Same thing for
> operator.getitem() or its [] shorthand syntax.
>
>
> >Overriding __special__ methods works fine for the most part.
>
> We can certainly keep the implementation of built-in generics based on
> __special__ methods.  My example code was intended to demonstrate that it's
> possible even in today's Python to have a uniform interface to defining
> such methods -- not that it's necessarily a desirable way to do it, given
> the absence of dedicated syntax like 'defop' for operator (operation?)
> overloading.
>
> Even *with* a 'defop' syntax, I would still not propose we eliminate
> __special__ methods, as they are very fast when implemented as C slots.  I
> would just say that, like 'apply()', they would no longer be the optimum
> way to do something for which there is a syntactical shortcut.  Note that
> __special__ methods can be misspelled, and the resulting error can be hard
> to find.  Misspell a 'defop', and the failure is immediate.  (Of course,
> you could still defop the *wrong* function, but it's still an improvement.)
>
>
> >Sure, I see a use case for defining an operation that the class
> >author(s) did *not* foresee, but that's the exception rather than the
> >rule.
>
> Right, at least in the sense that team development is exceptional for
> Python.  In "enterprisey" and other team development scenarios (e.g. Zope,
> Chandler, etc.), it's valuable to have separation between domain model code
> and presentation code.  IIRC, the driving use cases for adaptation in Zope
> 3 were to achieve this separation.
>
> In other words, even if you *can* define the methods, that doesn't mean
> it's a good idea to, if you are trying to maximize reuse.  But I freely
> admit that Zope and PEAK at least are exceptional situations: Zope Corp.
> (and my group at Verio during the time I initially created PEAK), were both
> effectively consulting organizations gaining cost-effectiveness through
> reuse of framework code.  This is not exactly the most common use of
> Python...  unless of course you're in an "enterprise" shop or consulting
> organization.
>
>
> > >      addmethod(iter, somefunc, sometype)
> >
> >I don't actually understand what you want this example to mean. What
> >is 'iter' supposed to be? A generic function?
>
> The iter builtin, as an example of the uniform addmethod() being applicable
> to any generic function, including existing builtin ones.  Although I guess
> you already know that from what you read beyond that point.
>
>
> > > would actually work by doing 'sometype.__iter__ = somefunc' under the
> > > hood.
> >
> >How would addmethod know this?
>
> By addmethod itself being a generic function.
>
>
> >At least one poster has already remarked that the head-exploding
> >capacity of generic functions is greater than that of metaclasses. I
> >think that's a gross exaggeration, but calling it "simple" and "easy
> >to use" is an exaggeration too. It's quite deep (otherwise we'd seen
> >it in Java already ;-).
>
> We've already seen it in Python, actually.  Generic operations of this
> nature are at the very soul of the language; we simply provide syntactic
> shorthand for many of them, and builtins for the rest.
>
> So, adding an explicit GF API seems more like type/class unification to me
> (or adding the callable __class__ hook), than a sea change in the nature of
> Python itself.  That is, a nice improvement in uniformity and
> user-extensibility, rather than any actual change.
>
>
> >How should it be extensible? Please explain this without making any
> >use of examples from RuleDispatch.
>
> Implement the API using generic functions.  If you have an addmethod(),
> make it a generic.  If you have a 'hasmethod()', make it generic.
>
> (Please note that 'generic' here refers to the extensibility of the
> function, not to a requirement that it be implemented via a registry!  I
> don't care if we have to give generic functions __addmethod__ and
> __hasmethod__ specials.  The point is merely that there should be *some*
> way to extend the core API.)
>
>
> >What's a generic function type? How does one create one?
>
> Implement a callable object that can be passed to addmethod(), hasmethod(),
> etc.
>
>
> >OK, I take it back. My brain just exploded. it *is* worse than
> >metaclasses. Please find enclosed the bits of brain that I scraped of
> >my monitors. :-)
>
> Ironically, it's *you* who gave me this idea, although I don't think you
> realized it at the time.  It was back when you were prototyping the
> multiple-dispatch implementation using a tuple of types -- something you
> said made me realize that we could have a generic API for manipulating
> generics, and thus allow hypothetical Zope and PEAK generics to live
> alongside Python-provided generics as equal citizens.  I then went away and
> made a proof of concept, before coming back to explode your head.
>
>
> >Poof. You disappear in a cloud of orange smoke.
>
> I guess now it's my turn not to understand what something means.  :)
>
> I was merely trying to show that my idea is trivially
> implementable.  simplegeneric is only about 100 lines of Python added to
> what I wrote.  Throw in a 'defop' syntax that calls addmethod(), and we're
> done.
>
> What I suppose is not obvious to anyone else from what I've written, is
> that this would:
>
> 1. allow competing type systems to coexist and even interoperate with a
> minimum of fuss
> 2. provide a *definition-time checkable* alternative to __special__ method
> names
> 3. ...while still allowing __specials__ to be used for fast lookup under
> the hood
> 4. allow for a "blessed" interface mechanism to either be chosen based on
> actual uses, OR
> 5. avoid the need for having a "blessed" interface mechanism at all, if GF
> introspection or adaptation suffices
>
>
> >That's an interesting thought but it works with or without generic
> >functions -- it's just "ability algebra". I think I've played with
> >"type algebra" using similar ideas in a blog entry once.
>
> Sure -- the difference is that I'm suggesting using the operation objects
> themselves as the basic unit of that algebra.
>
> >Not really, IMO. The underlying problem is that standard type
> >hierarchy defies capturing it in interfaces *BECAUSE INTERFACES DIDN'T
> >EXIST WHEN THEY WERE CREATED*.
>
> That's a reasonable hypothesis.  You might find it works out okay for
> things with few operations (like sequences and "mappings") but not so well
> for things with lots of utility methods ("list-like", "file-like",
> "dict-like" etc.).  I guess we'll see how it works out in practice.
>
>
> > > The advanced abilities (per-instance/on-the-fly and multi-operation tests)
> > > will likely affect performance and/or simplicity of the default
> > > implementation.  A purely type-based system can be implemented efficiently,
> > > because declaring an interface can simultaneously add a concrete type
> > > registration for all affected generics.  (And registering an
> > > interface-linked method in a generic function can pull in all the concrete
> > > classes.)
> >
> >This seems to argue *for* interfaces?
>
> I was pointing out that if you really need interfaces, you can have them,
> without needing to create a distinct interface object.  As you yourself
> pointed out in your blog, an "interface" can just be a generic function
> that returns an object supporting that interface, for a given input
> object.  In this sense, "iter" is an interface, because when you call it on
> an object, you get back an object that supports the "iter" interface.
>
> Thus, I don't see much need to create a special notion of interfaces as a
> first-class citizen.  In the common case, a generic "adapting" function
> (like iter) is sufficient, even if the "interface" includes multiple
> methods (like __iter__ and next()).
>
>
> > > Per-instance tests, however, increase code complexity.  Generally speaking,
> > > RuleDispatch does per-instance tests after class tests, but the way it gets
> > > reasonable performance is by building decision trees beforehand to manage
> > > the possible tests and avoid overlap.  If it actually had to call
> > > individual methods, or call a method more than once (ob.has_ability(Foo),
> > > ob.has_ability(Bar), etc.)  it would be considerably slower to select an
> > > option.
> >
> >That would make more sense as Foo.implemented_by(ob), right?
>
> Um, I guess.  I was emphasizing the dynamic/per-instance aspect of the
> issue.  Whichever way you do it, unless you have a well-defined implication
> or inheritance hierarchy between interfaces, you have the problem of
> ambiguity between opaque interfaces.
>
>
> > > Also -- and this is the real kicker -- what do you do if more than one
> > > ability applies?  Now we have to have precedence rules for what's more
> > > specific.  The advantage of defining an ability as a set of one or more
> > > applicable generic functions is that precedence is comparatively
> > > straightforward: supersets take precedence over subsets, and overlaps are
> > > ambiguous.  You also have some possibility of being able to implement this
> > > by registration-time checks, without needing to build a dispatch tree
> > at all.
> >
> >Couldn't ability inheritance serve the same purpose?
>
> Sure.  The above was part of my argument against *dynamically-determined*
> abilities as a core feature.
>
>
> > > In short, I think it's adding interface introspection that's the radical
> > > move, where generic functions are just a bit of polish that brings a bit
> > > more order to what Python already has.  In contrast, interfaces are a
> > > foreign religion imported from other languages, while the roots of the
> > > generic function faith (len, iter, etc.) have already been in place since
> > > the dawn of time!  :)
> >
> >Python has a long tradition of borrowing ideas from other languages,
> >and mostly very successful.
>
> Of course -- and generic functions have a successful tradition in many
> other languages too, from Common Lisp to Haskell, that IIUC goes back much
> further than "interfaces" as such do.  I was pointing out that you had
> *already* borrowed this tradition many years ago, whether you realized it
> or not.  You just "hid" it by using __special__ method names instead of
> registries.  ;-)
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)