[Python-3000] ABC's, Roles, etc

[Jeff Shell]

On 5/8/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 03:52 PM 5/8/2007 -0600, Jeff Shell wrote:
> >I have a lengthy post that dissects a major issue that I have with
> >ABCs and the Interface definition that I saw in PEP 3124:: it all
> >seems rigidly class and class-instance based.
>
> Hi Jeff; I read your post a few days ago, but your blog doesn't
> support comments, so I've been "getting around to" writing a
> counterpoint on my blog.  But, now I can do it here instead.  :)

Regarding the comments, blame spammers. And CAPTCHA. There's an equal
spot in hell (should I choose to believe in hell) for both. :) I miss
the chance to have conversation, but the weight of gardening or having
to try seven times to differentiate between two funnily-drawn
characters killed that part of my humanity.

> >The cardinal sin I saw
> >in the Interface definition in PEP 3124 (at least, at the time I last
> >viewed it) was the inclusion of 'self' in a method spec.
>
> That's because you're confusing a generic function and a "method
> spec".  PEP 3124 interfaces are not *specifications*; they're
> namespaces for generic functions.  They are much closer in nature to
> ABCs than they are zope.interface-style Interfaces.  The principal
> thing they have in common with zope.interface (aside from the name)
> is the support for "IFoo(ob)"-style adaptation.
>
> Very few of zope.interface's design goals are shared by PEP
> 3124.  Notably, they are not particularly good for type checking or
> verification.  In PEP 3124, for example, IFoo(ob) *always* returns an
> object with the specified attributes; the only way to know whether
> they are actually implemented is to try using them.
>
> Notice that this is diametrically opposed to what zope.interface
> wants to do in such situations -- which is why the PEP makes such a
> big deal about it being possible to use zope.interface *instead*.

Well that puts another fear in my heart about confusing the issue
further - "oh, these kindof sound and look the same but are
diametrically opposed?"

I must admit that I didn't read PEP 3124 in depth - most of it was
fascinating, some of it went way over my head in complexity, and then
suddenly I saw an Interface. It seemed quite out of place, actually,
and it seemed diametrically opposed to the simplicity and power I've
been enjoying.

> That is, my own observation is that different frameworks sometimes
> need different kinds of interfaces.  For example, someone might
> create a framework that wants to verify preconditions and
> postconditions of methods in an interface, rather than merely
> specifying their names and arguments!  Using zope.interface as an
> exclusive basis for interface definition and type annotations would
> block innovation in this area.

FWIW, zope.interface allows 'tagged values' on all Interface Elements,
which is the base class/type of Attribute, Method, and even Interface
(I believe). Tagged values are used to hold invariants, which are code
objects in the specification and can provide the 'if obj.age < 18,
then obj.has_parents_permission must be true' type of logic. The
interface (ha!) for setting and getting tagged attributes aint the
prettiest, but it's the equivalent of type annotations and all other
such things. And like 'invariant', it's not too difficult to write
helper functions that deal with that interface.

I use this in a SQLAlchemy based system that uses zope.schema (which
builds on zope.interface to describe field (attributes) types and
restrictions). The spec looks something like this::

        class ILogins(Interface):
            login = zope.schema.TextLine(...)
            validateUnique(login, column=table.c.login)

I have even used `zope.interface` to stamp out a new abstract class
(of sorts), which it supports::

    schema = field.schema
    if IInterface.providedBy(schema):
        # schema is an Interface, not an implementation; we need a concrete
        # instance.
        schema = schema.deferred()
        directlyProvides(schema, field.schema)

This stamps out a new abstract instance, and declares support for the
interface. This particular use was for view binding (something that
you mention), but it's three lines of code that are very useful in my
system.

This particular use case is for Zope 'Object' fields; like an address
attribute might expect to have a complex type, like 'IAddress'. There
are situations, such as dynamic UI generation, where an empty instance
is needed. This allows the abstract specification to provide just
enough of a concrete implementation to fill in for a real object
that's expected to arrive in the future.

One could envision having that 'deferred()' interface method filling
in stronger implementations. Which means that there's probably more
power, or hooks (at least) in zope.interface than may be realized. The
common uses of it don't cover all possibilities.

> >... 'self' is an internal detail of class-instance implementations.
>
> Again - this is because you're assuming the purpose of a PEP 3124
> interface is to *specify* an interface, when in fact it's much more
> like an ABC, which may also *implement* the interface.  The
> specification and implementation are intentionally unified here.

Hm. I'll have to process this one...

> >It seems to me that Abstract Base Classes and even PEP 3124 are
> >primarily focused on classes. But in Python, "everything is an
> >object", but not everything is class-based.
> >...
> >The rest of this post focuses on what `zope.interface` already
> >provides - a system for specifying behavior and declaring support at
> >both the class and object level - and 'object' really means 'object',
> >which includes modules.
>
> Right -- and *neither* "specifying behavior" nor "declaring support"
> are goals of PEP 3124; they're entirely out of its scope.  The
> Interface object's purpose is to support uniform access to, and
> implementation of, individual operations.  These are somewhat
> parallel concepts, but very different in thrust; zope.interface is
> LBYL, while PEP 3124 is EAFP all the way.

Funny that those things are apparently opposed. 'Look Before You Leap'
brings to mind the concept of "don't dive into an empty pool" or
"don't do a backwards flip onto the pointy rocks"; where as 'Easier to
Ask Forgiveness Than Permission' brings to bind the concept of "sorry
i dove head first into your empty pool and cracked my skull open Mr.
Johnson. If I had asked I'm sure you would have said no! In any case,
even though it's your pool and there was a fence and everything and
you did not give me permission, my parents are going to sue" (OK,
maybe that last bit is the result of a healthy american upbringing...
but still!)

> As a result, PEP 3124 chooses to punt on the issue of individual
> objects.  It's quite possible within the framework to allow
> instance-level checks during dispatching, but it's not going to be in
> the default engine (which is based on type-tuple dispatching; see
> Guido's Py3K overloading prototype).

Huh? I'll try to look at that. types, classes, instances... That does
it, I'm switching to Io. (Honestly - I've recently seen the light
about prototype based object oriented programming; in light of types
of types and classes of classes and classes and instances and oh my,
languages that believe in "there are only objects, and they are only
instances" are sounding sweeter every day)

> zope.interface (and zope.component, IIRC) pay a high price in
> complexity for allowing interfaces to be per-instance rather than
> type-defined.  I implemented the same feature in PyProtocols, but
> over the years I rarely found it useful.  My understanding of its
> usefulness in Zope is that it:
>
> 1. supports specification and testing of module-level Zope APIs

I've had uses for it outside of modules.

> 2. allows views and other wrapping operations to be selected on a dynamic basis

That's an essential (and very powerful) feature in a large system. But
there are uses outside of that.

An area where parts of the zope 3 component architecture DO pay a high
price in complexity is where it has to create dynamic types in order
to satisfy some core requirement. I can't remember where this is, but
I know that I HATE it - suddenly, Zope is playing with MY class
hierarchy. Suddenly I'm in debug mode and have no idea what I'm
looking at. I'd much rather have it augmenting my instance than
mangling my classes, unless I choose to have it mangle my classes by
subclassing from a mangler. Annotate my class, but don't replace it.

> Since #1 falls outside of PEP 3124's goals (i.e., it's not about
> specification or testing), that leaves use case #2.  In my
> experience, it has been more than sufficient to simply give these
> object some *other* interface, such as an IViewTags interface with a
> method to query these dynamic "tag" interfaces.  In other words, my
> experience and opinion supports the view that use case #2 is actually
> a coincidental abuse of interfaces for convenience, rather than the
> "one obvious way" to handle the use case.

Ugh. Yeah, there are 'marker' interfaces, but.. ugh. dynamic "tag"
interfaces. Yuck.

My experience has been otherwise.

But I'm sorry that I confused that section of PEP 3124 to be about
specification and testing. I do, however, think that is a better use
case.

> To put it another way, if you define getView() as a generic function,
> you can always define a dynamic implementation of it for any type
> that you wish to have dynamic view selection capability.  Then, only
> those cases that require a complex solution have to pay for the complexity.
>
> So, that's my rationale for why PEP 3124 doesn't provide any
> instance-based features out of the box; outside of API specs for
> singleton objects, the need for them is mostly an illusion created by
> Zope 3's dynamic view selection hack.
>
>
> >My main focus is on determining what Abstract Base Classes and/or PEP
> >3124's Interfaces do better than `zope.interface` (if anyone else is
> >familiar with that package).
>
> I at least am quite familiar with it, having helped to define some of
> its terminology and API, as well as being the original author of its
> class-decorator emulation for Python versions 2.2 and up.  :)  I also
> argued for its adoption of PEP 246, and wrote PyProtocols to unify
> Twisted and Zope interfaces in a PEP 246-based adaptation framework.
>
> And what PEP 3124 does much better than zope.interface or even PyProtocols is:
>
> 1. Adaptation, especially incomplete adaptation.  You can implement
> only the methods that are actually needed for your use case.  If the
> interface includes generic implementations that are defined in terms
> of other methods in the interface, you need not reimplement
> them.  (Note: I'm well aware that here my definition of "better"
> would be considered "worse" by Jim Fulton, since zope.interface is
> LBYL-oriented.  However, for many users and use cases, EAFP *is*
> better, even if it's not for Zope.)

And for many users and use cases, LBYL is better. Especially for those
of us who get pissed off and start smoking every time we end up on a
spikey rock!

> 2. Interface recombination.  AFAIK, zope.interface doesn't support
> subset interfaces like PyProtocols does.  Neither zope.interface nor
> PyProtocols support method renaming, where two interfaces have a
> method with the same specification but different method names.

Um, then it's a different specification.

eat_pizza() and consume_pizza() are different. They may be the same to
Cookie Monster, but they're not the same for Grover.

> 3. Low mental overhead.  PEP 3124 doesn't even *need* interfaces;
> simple use cases can just use overloaded functions and be on about
> their business.  Use cases that would require an interface and half a
> dozen adapter classes in zope.interface can be met by simply creating
> an overloaded function and adding methods.  And the resulting code
> reads like code in other languages that support overloading or
> generic functions, rather than reading like Java.

The mental overhead in PEP 3124 was pretty high for me, but that may
stem from bias resulting from diametrically opposed interpretations of
the same word :).

> >In my blog post, I also show a dynamically constructed object
> >providing an interface's specified behavior. An instance of an empty
> >class is made, and then methods and other supporting attributes are
> >attached to this specific instance only. Real world examples of this
> >include Zope 2, where a folder may have "Python Scripts" or other
> >callable members that, in effect, make for a totally custom object. It
> >can also provide this same behavior (in fact, I was able to take
> >advantage of this on some old old old Zope 2 projects that started in
> >the web environment and transitioned to regular Python
> >modules/classes).
>
> And how often does this happen outside of Zope?  As I said, I rarely
> found it to be the case anywhere else.  I replicated the ability in
> PyProtocols because I was biased by my prior Zope experience, but
> once I got outside of Zope it almost entirely ceased to be useful.

We have a dynamic data transformation framework that exists outside of
Zope (Zope is basically used for UI). Objects are being dynamically
composed, wrapped, decomposed, rewrapped, filtered, and split -
constantly. Objects, not types. It's all composed of rules. I'm
itching to be able to add rules to apply zope.interface specifications
to the generated objects; if only to then make it much easier to add
other filtering rules later on.

With all of the wrapping and generation going on, we had to add some
basic 'is_a' methods to the base classes. And we do care whether an
object is a wrapper (isinstance) as well as whether the wrapped object
provides the DataSet interface.

It's another complex framework, it's just an outside-of-Zope system as
an example.

I know there's been some talk of ``__isinstance__()`` and
``__issubclass__()`` overriding being allowed, and I guess that's to
take care of the wrapped and wrapped and wrapped object situations?

In any case, it seems that I have long occupied worlds wherein complex
objects could be composed on the fly outside of the type system, and
I'd hade to have one of those constructed objects miss out on passing
a 'is-foo-like' test because they weren't raised by proper upper
middle class type parents.

> Meanwhile, as I said, PEP 3124 is not closed to extension.  It's
> specifically intended that zope.interface (and any other interface
> packages that might arise in future) should be able to play as
> first-class citizens in the proposed API.  However, depending on the
> specific features desired, those packages might have some additional
> integration work to do.
>
> (Note, by the way, that zope.interface is explicitly mentioned three
> times in the PEP, as an example of how other interface types should
> be able to be used for overloading, as long as they register
> appropriate methods with the provided framework.)

Thanks for clearing things up. I'll try to make another pass at
reading the PEP more closely. For me, at this moment, all of this
class/type based stuff is rubbing me the wrong way, and that's a
feeling that's very hard to get past. I'm not sure why. I'll try to
suppress those feelings when I revisit 3119 and 3124.

What's happening with Roles/Traits? That's still the system that I'd
like to see. I'm hoping that hasn't gotten swallowed up by generic
overloaded pre-post wrapped abstract methods. (as long as I never have
to type 'def public final', I'm cool).

I think roles/traits as a core concept (LBYL zope.interface style, if
you prefer to think of it that way) is useful, if not important. And I
still believe that zope.interface already provides a language/API from
which to build.

-- 
Jeff Shell