[Python-Dev] Re: PEP 1, PEP Purpose and Guidelines

Clark C. Evans cce@clarkevans.com
Thu, 22 Mar 2001 05:14:25 -0500 (EST)


Barry,

  If you don't mind, I'd like to apply for one of them
  there PEP numbers.  Sorry for not following the guidelines,
  it won't happen again.

  Also, I believe that this isn't just my work, but rather
  a first pass at concensus on this issue via the vocal and
  silent feeback from those on the main and type special
  interest group.  I hope that I have done their ideas
  and feedback justice (if not, I'm sure I'll hear about it).

Thank you so much,

Clark

...

PEP: XXX
Title: Protocol Checking and Adaptation
Version: $Revision$
Author: Clark Evans
Python-Version: 2.2
Status: Draft
Type: Standards Track
Created: 21-Mar-2001
Updated: 23-Mar-2001

Abstract

    This proposal puts forth a built-in, explicit method for
    the adaptation (including verification) of an object to a 
    context where a specific type, class, interface, or other 
    protocol is expected.  This proposal can leverage existing
    protocols such as the type system and class hierarchy and is
    orthogonal, if not complementary to the pending interface
    mechanism [1] and signature based type-checking system [2]

    This proposal allows an object to answer two questions.  First,
    are you a such and such?  Meaning, does this object have a 
    particular required behavior?  And second, if not, can you give
    me a handle which is?  Meaning, can the object construct an 
    appropriate wrapper object which can provide compliance with
    the protocol expected.  This proposal does not limit what 
    such and such (the protocol) is or what compliance to that
    protocol means, and it allows other query/adapter techniques 
    to be added later and utilized through the same interface 
    and infrastructure introduced here.

Motivation

    Currently there is no standardized mechanism in Python for 
    asking if an object supports a particular protocol. Typically,
    existence of particular methods, particularly those that are 
    built-in such as __getitem__, is used as an indicator of 
    support for a particular protocol.  This technique works for 
    protocols blessed by GvR, such as the new enumerator proposal
    identified by a new built-in __iter__.  However, this technique
    does not admit an infallible way to identify interfaces lacking 
    a unique, built-in signature method.

    More so, there is no standardized way to obtain an adapter 
    for an object.  Typically, with objects passed to a context
    expecting a particular protocol, either the object knows about 
    the context and provides its own wrapper or the context knows 
    about the object and automatically wraps it appropriately.  The 
    problem with this approach is that such adaptations are one-offs,
    are not centralized in a single place of the users code, and 
    are not executed with a common technique, etc.  This lack of
    standardization increases code duplication with the same 
    adapter occurring in more than one place or it encourages 
    classes to be re-written instead of adapted.  In both cases,
    maintainability suffers.

    In the recent type special interest group discussion [3], there
    were two complementary quotes which motivated this proposal:

       "The deep(er) part is whether the object passed in thinks of
        itself as implementing the Foo interface. This means that
        its author has (presumably) spent at least a little time
        about the invariants that a Foo should obey."  GvR [4]

    and

       "There is no concept of asking an object which interface it
        implements. There is no "the" interface it implements. It's
        not even a set of interfaces, because the object doesn't 
        know them in advance. Interfaces can be defined after objects
        conforming to them are created." -- Marcin Kowalczyk [5]

    The first quote focuses on the intent of a class, including 
    not only the existence of particular methods, but more 
    importantly the call sequence, behavior, and other invariants.
    Where the second quote focuses on the type signature of the
    class.  These quotes highlight a distinction between interface
    as a "declarative, I am a such-and-such" construct, as opposed
    to a "descriptive, It looks like a such-and-such" mechanism.

    Four positive cases for code-reuse include:

     a) It is obvious object has the same protocol that
        the context expects.  This occurs when the type or
        class expected happens to be the type of the object
        or class.  This is the simple and easiest case.

     b) When the object knows about the protocol that the
        context requires and knows how to adapt itself 
        appropriately.  Perhaps it already has the methods
        required, or it can make an appropriate wrapper

     c) When the protocol knows about the object and can
        adapt it on behalf of the context.  This is often
        the case with backwards-compatibility cases.

     d) When the context knows about the object and the 
        protocol and knows how to adapt the object so that
        the required protocol is satisfied.

    This proposal should allow each of these cases to be handled,
    however, the proposal only concentrates on the first two cases;
    leaving the latter two cases where the protocol adapts the 
    object and where the context adapts the object to other proposals.
    Furthermore, this proposal attempts to enable these four cases
    in a manner completely neutral to type checking or interface
    declaration and enforcement proposals.  

Specification

    For the purposes of this specification, let the word protocol
    signify any current or future method of stating requirements of 
    an object be it through type checking, class membership, interface 
    examination, explicit types, etc.  Also let the word compliance
    be dependent and defined by each specific protocol.

    This proposal initially supports one initial protocol, the
    type/class membership as defined by isinstance(object,protocol)
    Other types of protocols, such as interfaces can be added through
    another proposal without loss of generality of this proposal.  
    This proposal attempts to keep the first set of protocols small
    and relatively unobjectionable.

    This proposal would introduce a new binary operator "isa".
    The left hand side of this operator is the object to be checked
    ("self"), and the right hand side is the protocol to check this
    object against ("protocol").  The return value of the operator 
    will be either the left hand side if the object complies with 
    the protocol or None.

    Given an object and a protocol, the adaptation of the object is:
     a) self, if the object is already compliant with the protocol,
     b) a secondary object ("wrapper"), which provides a view of the
        object compliant with the protocol.  This is explicitly 
        vague, and wrappers are allowed to maintain their own 
        state as necessary.
     c) None, if the protocol is not understood, or if object 
        cannot be verified compliant with the protocol and/or
        if an appropriate wrapper cannot be constructed.

    Further, a new built-in function, adapt, is introduced.  This
    function takes two arguments, the object being adapted ("obj") 
    and the protocol requested of the object ("protocol").  This
    function returns the adaptation of the object for the protocol,
    either self, a wrapper, or None depending upon the circumstances.
    None may be returned if adapt does not understand the protocol,
    or if adapt cannot verify compliance or create a wrapper.

    For this machinery to work, two other components are required.
    First is a private, shared implementation of the adapt function
    and isa operator.  This private routine will have three 
    arguments: the object being adapted ("self"), the protocol 
    requested ("protocol"), and a flag ("can_wrap").  The flag
    specifies if the adaptation may be a wrapper, if the flag is not
    set, then the adaptation may only be self or None.  This flag is
    required to support the isa operator.  The obvious case 
    mentioned in the motivation, where the object easily complies 
    with the protocol, is implemented in this private routine.  

    To enable the second case mentioned in the motivation, when 
    the object knows about the protocol, a new method slot, __adapt__
    on each object is required.  This optional slot takes three
    arguments, the object being adapted ("self"), the protocol 
    requested ("protocol"), and a flag ("can_wrap").  And, like 
    the other functions, must return an adaptation, be it self, a
    wrapper if allowed, or None.  This method slot allows a class 
    to declare which protocols it supports in addition to those 
    which are part of the obvious case.

    This slot is called first before the obvious cases are examined, 
    if None is returned then the default processing proceeds.  If the
    default processing is wrong, then the AdaptForceNoneException
    can be thrown.  The private routine will catch this specific 
    exception and return None in this case.  This technique allows an
    class to subclass another class, but yet catch the cases where 
    it is considered as a substitutable for the base class.  Since 
    this is the exception, rather than the normal case, an exception 
    is warranted and is used to pass this information along.  The 
    caller of adapt or isa will be unaware of this particular exception
    as the private routine will return None in this particular case.

    Please note two important things.  First, this proposal does not
    preclude the addition of other protocols.  Second, this proposal 
    does not preclude other possible cases where adapter pattern may
    hold, such as the protocol knowing the object or the context 
    knowing the object and the protocol (cases c and d in the 
    motivation).  In fact, this proposal opens the gate for these 
    other mechanisms to be added; while keeping the change in 
    manageable chunks.

Reference Implementation and Example Usage

    -----------------------------------------------------------------
    adapter.py
    -----------------------------------------------------------------
        import types
        AdaptForceNoneException = "(private error for adapt and isa)"

        def interal_adapt(obj,protocol,can_wrap):

            # the obj may have the answer, so ask it about the ident
            adapt = getattr(obj, '__adapt__',None)
            if adapt:
                try:
                    retval = adapt(protocol,can_wrap)
                    # todo: if not can_wrap check retval for None or obj
                except AdaptForceNoneException:
                    return None
                if retval: return retval

            # the protocol may have the answer, so ask it about the obj
            pass

            # the context may have the answer, so ask it about the
            pass

            # check to see if the current object is ok as is
            if type(protocol) is types.TypeType or \
               type(protocol) is types.ClassType:
                if isinstance(obj,protocol):
                    return obj

            # ok... nothing matched, so return None
            return None

        def adapt(obj,protocol):
            return interal_adapt(obj,protocol,1)

        # imagine binary operator syntax
        def isa(obj,protocol):
            return interal_adapt(obj,protocol,0)

    -----------------------------------------------------------------
    test.py
    -----------------------------------------------------------------
        from adapter import adapt
        from adapter import isa
        from adapter import AdaptForceNoneException

        class KnightsWhoSayNi: pass  # shrubbry troubles

        class EggsOnly:  # an unrelated class/interface
            def eggs(self,str): print "eggs!" + str

        class HamOnly:  # used as an interface, no inhertance
            def ham(self,str): pass
            def _bugger(self): pass  # irritating a private member

        class SpamOnly: # a base class, inheritance used
            def spam(self,str): print "spam!" + str

        class EggsSpamAndHam (SpamOnly,KnightsWhoSayNi):
            def ham(self,str): print "ham!" + str
            def __adapt__(self,protocol,can_wrap):
                if protocol is HamOnly:
                    # implements HamOnly implicitly, no _bugger
                    return self
                if protocol is KnightsWhoSayNi:
                    # we are no longer the Knights who say Ni!
                    raise AdaptForceNoneException
                if protocol is EggsOnly and can_wrap:
                    # Knows how to create the eggs!
                    return EggsOnly()

        def test():
            x = EggsSpamAndHam()
            adapt(x,SpamOnly).spam("Ni!")
            adapt(x,EggsOnly).eggs("Ni!")
            adapt(x,HamOnly).ham("Ni!")
            adapt(x,EggsSpamAndHam).ham("Ni!")
            if None is adapt(x,KnightsWhoSayNi): print "IckIcky...!"
            if isa(x,SpamOnly): print "SpamOnly"
            if isa(x,EggsOnly): print "EggsOnly"
            if isa(x,HamOnly): print "HamOnly"
            if isa(x,EggsSpamAndHam): print "EggsAndSpam"
            if isa(x,KnightsWhoSayNi): print "NightsWhoSayNi"

    -----------------------------------------------------------------
    Example Run
    -----------------------------------------------------------------
        >>> import test
        >>> test.test()
        spam!Ni!
        eggs!Ni!
        ham!Ni!
        ham!Ni!
        IckIcky...!
        SpamOnly
        HamOnly
        EggsAndSpam

Relationship To Paul Prescod and Tim Hochbergs Type Assertion method

    The example syntax Paul put forth recently [2] was:

        interface Interface
            def __check__(self,obj)

    Pauls proposal adds the checking part to the third (3)
    case described in motiviation, when the protocol knows
    about the object.  As stated, this could be easily added
    as a step in the interal_adapt function:

            # the protocol may have the answer, so ask it about the obj

                if typ is types.Interface:
                    if typ__check__(obj):
                        return obj

    Further, and quite excitingly, if the syntax for this type 
    based assertion added an extra argument, "can_wrap", then this
    mechanism could be overloaded to also provide adapters to
    objects that the interface knows about.

    In short, the work put forth by Paul and company is great, and
    I dont see any problems why these two proposals couldnt work
    together in harmony, if not be completely complementary.

Relationship to Python Interfaces [1] by Michel Pelletier

    The relationship to this proposal is a bit less clear 
    to me, although an implements(obj,anInterface) built-in
    function was mentioned.  Thus, this could be added naively
    as a step in the interal_adapt function:

        if typ is types.Interface:
            if implements(obj,protocol):
                return obj

    However, there is a clear concern here.  Due to the 
    tight semantics being described in this specification,
    it is clear the isa operator proposed would have to have 
    a 1-1 correspondence with implements function, when the
    type of protocol is an Interface.  Thus, when can_wrap is
    true, __adapt__ may be called, however, it is clear that
    the return value would have to be double-checked.  Thus, 
    a more realistic change would be more like:

        def internal_interface_adapt(obj,interface)
            if implements(obj,interface):
                return obj
            else
                return None

        def interal_adapt(obj,protocol,can_wrap):

            # the obj may have the answer, so ask it about the ident
            adapt = getattr(obj, '__adapt__',None)
            if adapt:
                try:
                    retval = adapt(protocol,can_wrap)
                except AdaptForceNoneException:
                    if type(protocol) is types.Interface:
                        return internal_interface_adapt(obj,protocol)
                    else:
                        return None
                if retval: 
                    if type(protocol) is types.Interface:
                        if can_wrap and implements(retval,protocol):
                            return retval
                        return internal_interface_adapt(obj,protocol)
                    else:
                        return retval

            if type(protocol) is types.Interface:
                return internal_interface_adapt(obj,protocol)

            # remainder of function... 

    It is significantly more complicated, but doable.

Relationship To Iterator Proposal:
 
    The iterator special interest group is proposing a new built-in
    called "__iter__", which could be replaced with __adapt__ if an
    an Interator class is introduced.  Following is an example.

        class Iterator:
            def next(self):
                raise IndexError

        class IteratorTest:
            def __init__(self,max):
                self.max = max
            def __adapt__(self,protocol,can_wrap):
                if protocol is Iterator and can_wrap:
                    class IteratorTestIterator(Iterator):
                        def __init__(self,max):
                            self.max = max
                            self.count = 0
                        def next(self):
                            self.count = self.count + 1
                            if self.count < self.max:
                              return self.count
                            return Iterator.next(self)
                    return IteratorTestIterator(self.max)

Relationships To Microsofts Query Interface:

    Although this proposal may sounds similar to Microsofts 
    QueryInterface, it differs by a number of aspects.  First, 
    there is not a special "IUnknown" interface which can be used
    for object identity, although this could be proposed as one
    of those "special" blessed interface protocol identifiers.
    Second, with QueryInterface, once an object supports a particular
    interface it must always there after support this interface; 
    this proposal makes no such guarantee, although this may be 
    added at a later time. Third, implementations of Microsofts
    QueryInterface must support a kind of equivalence relation. 
    By reflexive they mean the querying an interface for itself 
    must always succeed.  By symmetrical they mean that if one 
    can successfully query an interface IA for a second interface 
    IB, then one must also be able to successfully query the 
    interface IB for IA.  And finally, by transitive they mean if 
    one can successfully query IA for IB and one can successfully
    query IB for IC, then one must be able to successfully query 
    IA for IC.  Ability to support this type of equivalence relation
    should be encouraged, but may not be possible.  Further research 
    on this topic (by someone familiar with Microsoft COM) would be
    helpful in further determining how compatible this proposal is.

Backwards Compatibility

    There should be no problem with backwards compatibility.  
    Indeed this proposal, save an built-in adapt() function, 
    could be tested without changes to the interpreter.

Questions and Answers

    Q:  Why was the name changed from __query__ to __adapt__ ?  

    A:  It was clear that significant QueryInterface assumptions were
        being laid upon the proposal, when the intent was more of an 
        adapter.  Of course, if an object does not need to be adapted
        then it can be used directly and this is the basic premise.

    Q:  Why is the checking mechansim mixed with the adapter
        mechanism.

    A:  Good question.  They could be seperated, however, there
        is significant overlap, if you consider the checking
        protocol as returning a compliant object (self) or
        not a compliant object (None).  In this way, adapting
        becomes a special case of checking, via the can_wrap.

        Really, this could be seperated out, but the two 
        concepts are very related so much duplicate work
        would be done, and the overall mechanism would feel
        quite a bit less unified.

    Q:  This is just a type-coercion proposal.

    A:  No. Certainly it could be used for type-coercion, such
        coercion would be explicit via __adapt__ or adapt function. 
        Of course, if this was used for iterator interface, then the
        for construct may do an implicit __adapt__(Iterator) but
        this would be an exception rather than the rule.

    Q:  Why did the author write this PEP?

    A:  He wanted a simple proposal that covered the "deep part" of
        interfaces without getting tied up in signature woes.  Also, it
        was clear that __iter__ proposal put forth is just an example
        of this type of interface.  Further, the author is doing XML 
        based client server work, and wants to write generic tree based
        algorithms that work on particular interfaces and would
        like these algorithms to be used by anyone willing to make
        an "adapter" having the interface required by the algorithm.

    Q:  Is this in opposition to the type special interest group?

    A:  No.  It is meant as a simple, need based solution that could
        easily complement the efforts by that group.

    Q:  Why was the identifier changed from a string to a class?

    A:  This was done on Michel Pelletiers suggestion.  This mechanism
        appears to be much cleaner than the DNS string proposal, which 
        caused a few eyebrows to rise.  

    Q:  Why not handle the case where instances are used to identify 
        protocols?  In other words, 6 isa 6 (where the 6 on the right
        is promoted to an types.Int

    A:  Sounds like someone might object, lets keep this in a
        separate proposal.

    Q:  Why not let obj isa obj be true?  or class isa baseclass?

    A:  Sounds like someone might object, lets keep this in a
        separate proposal.

    Q:  It seems that a reverse lookup could be used, why not add this?

    A:  There are many other lookup and/or checking mechanisms that
        could be used here.  However, the goal of this PEP is to be 
        small and sweet ... having any more functionality would make
        it more objectionable to some people.  However, this proposal
        was designed in large part to be completely orthogonal to other
        methods, so these mechanisms can be added later if needed

Credits

    This proposal was created in large part by the feedback 
    of the talented individuals on both the main mailing list
    and also the type signature list.  Specific contributors
    include (sorry if I missed someone).

        Robin Thomas, Paul Prescod, Michel Pelletier, 
        Alex Martelli, Jeremy Hylton, Carlos Ribeiro,
        Aahz Maruch, Fredrik Lundh, Rainer Deyke,
        Timothy Delaney, and Huaiyu Zhu

Copyright

    This document has been placed in the public domain.


References and Footnotes

    [1] http://python.sourceforge.net/peps/pep-0245.html
    [2] http://mail.python.org/pipermail/types-sig/2001-March/001223.html
    [3] http://www.zope.org/Members/michel/types-sig/TreasureTrove
    [4] http://mail.python.org/pipermail/types-sig/2001-March/001105.html
    [5] http://mail.python.org/pipermail/types-sig/2001-March/001206.html
    [6] http://mail.python.org/pipermail/types-sig/2001-March/001223.html