[Python-3000] PEP 3124 - Overloading, Generic Functions, Interfaces, etc.

[Chris Monson]

On 4/30/07, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> This is just the first draft (also checked into SVN), and doesn't include
> the details of how the extension API works (so that third-party interfaces
> and generic functions can interoperate using the same decorators,
> annotations, etc.).
>
> Comments and questions appreciated, as it'll help drive better
> explanations
> of both the design and rationales.  I'm usually not that good at guessing
> what other people will want to know (or are likely to misunderstand) until
> I get actual questions.
>
>
> PEP: 3124
> Title: Overloading, Generic Functions, Interfaces, and Adaptation
> Version: $Revision: 55029 $
> Last-Modified: $Date: 2007-04-30 18:48:06 -0400 (Mon, 30 Apr 2007) $
> Author: Phillip J. Eby <pje at telecommunity.com>
> Discussions-To: Python 3000 List <python-3000 at python.org>
> Status: Draft
> Type: Standards Track
> Requires: 3107, 3115, 3119
> Replaces: 245, 246
> Content-Type: text/x-rst
> Created: 28-Apr-2007
> Post-History: 30-Apr-2007


[snip]


>
> "Before" and "After" Methods
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> In addition to the simple next-method chaining shown above, it is
> sometimes useful to have other ways of combining methods.  For
> example, the "observer pattern" can sometimes be implemented by adding
> extra methods to a function, that execute before or after the normal
> implementation.
>
> To support these use cases, the ``overloading`` module will supply
> ``@before``, ``@after``, and ``@around`` decorators, that roughly
> correspond to the same types of methods in the Common Lisp Object
> System (CLOS), or the corresponding "advice" types in AspectJ.
>
> Like ``@when``, all of these decorators must be passed the function to
> be overloaded, and can optionally accept a predicate as well::
>
>      def begin_transaction(db):
>          print "Beginning the actual transaction"
>
>
>      @before(begin_transaction)
>      def check_single_access(db: SingletonDB):
>          if db.inuse:
>              raise TransactionError("Database already in use")
>
>      @after(begin_transaction)
>      def start_logging(db: LoggableDB):
>          db.set_log_level(VERBOSE)



If we are looking at doing Design By Contract using @before and @after
(preconditions and postconditions), shouldn't there be some way of getting
at the return value in functions decorated with @after?  For example, it
seems reasonable to require an extra argument, perhaps at the beginning:

def successor(num):
  return num + 1

@before(successor)
def check_positive(num: int):
  if num < 0:
    raise PreconditionError("Positive integer inputs required")

@after(successor)
def check_successor(returned, num:int):
  if returned != num + 1:
    raise PostconditionError("successor failed to do its job")

Or am I missing something about how @after works?

+1, BTW, on this whole idea.

- C


``@before`` and ``@after`` methods are invoked either before or after
> the main function body, and are *never considered ambiguous*.  That
> is, it will not cause any errors to have multiple "before" or "after"
> methods with identical or overlapping signatures.  Ambiguities are
> resolved using the order in which the methods were added to the
> target function.
>
> "Before" methods are invoked most-specific method first, with
> ambiguous methods being executed in the order they were added.  All
> "before" methods are called before any of the function's "primary"
> methods (i.e. normal ``@overload`` methods) are executed.
>
> "After" methods are invoked in the *reverse* order, after all of the
> function's "primary" methods are executed.  That is, they are executed
> least-specific methods first, with ambiguous methods being executed in
> the reverse of the order in which they were added.
>
> The return values of both "before" and "after" methods are ignored,
> and any uncaught exceptions raised by *any* methods (primary or other)
> immediately end the dispatching process.  "Before" and "after" methods
> cannot have ``__proceed__`` arguments, as they are not responsible
> for calling any other methods.  They are simply called as a
> notification before or after the primary methods.
>
> Thus, "before" and "after" methods can be used to check or establish
> preconditions (e.g. by raising an error if the conditions aren't met)
> or to ensure postconditions, without needing to duplicate any existing
> functionality.
>
>
> "Around" Methods
> ~~~~~~~~~~~~~~~~
>
> The ``@around`` decorator declares a method as an "around" method.
> "Around" methods are much like primary methods, except that the
> least-specific "around" method has higher precedence than the
> most-specific "before" or method.
>
> Unlike "before" and "after" methods, however, "Around" methods *are*
> responsible for calling their ``__proceed__`` argument, in order to
> continue the invocation process.  "Around" methods are usually used
> to transform input arguments or return values, or to wrap specific
> cases with special error handling or try/finally conditions, e.g.::
>
>      @around(commit_transaction)
>      def lock_while_committing(__proceed__, db: SingletonDB):
>          with db.global_lock:
>              return __proceed__(db)
>
> They can also be used to replace the normal handling for a specific
> case, by *not* invoking the ``__proceed__`` function.
>
> The ``__proceed__`` given to an "around" method will either be the
> next applicable "around" method, a ``DispatchError`` instance,
> or a synthetic method object that will call all the "before" methods,
> followed by the primary method chain, followed by all the "after"
> methods, and return the result from the primary method chain.
>
> Thus, just as with normal methods, ``__proceed__`` can be checked for
> ``DispatchError``-ness, or simply invoked.  The "around" method should
> return the value returned by ``__proceed__``, unless of course it
> wishes to modify or replace it with a different return value for the
> function as a whole.
>
>
> Custom Combinations
> ~~~~~~~~~~~~~~~~~~~
>
> The decorators described above (``@overload``, ``@when``, ``@before``,
> ``@after``, and ``@around``) collectively implement what in CLOS is
> called the "standard method combination" -- the most common patterns
> used in combining methods.
>
> Sometimes, however, an application or library may have use for a more
> sophisticated type of method combination.  For example, if you
> would like to have "discount" methods that return a percentage off,
> to be subtracted from the value returned by the primary method(s),
> you might write something like this::
>
>      from overloading import always_overrides, merge_by_default
>      from overloading import Around, Before, After, Method, MethodList
>
>      class Discount(MethodList):
>          """Apply return values as discounts"""
>
>          def __call__(self, *args, **kw):
>              retval = self.tail(*args, **kw)
>              for sig, body in self.sorted():
>                  retval -= retval * body(*args, **kw)
>              return retval
>
>      # merge discounts by priority
>      merge_by_default(Discount)
>
>      # discounts have precedence over before/after/primary methods
>      always_overrides(Discount, Before)
>      always_overrides(Discount, After)
>      always_overrides(Discount, Method)
>
>      # but not over "around" methods
>      always_overrides(Around, Discount)
>
>      # Make a decorator called "discount" that works just like the
>      # standard decorators...
>      discount = Discount.make_decorator('discount')
>
>      # and now let's use it...
>      def price(product):
>          return product.list_price
>
>      @discount(price)
>      def ten_percent_off_shoes(product: Shoe)
>          return Decimal('0.1')
>
> Similar techniques can be used to implement a wide variety of
> CLOS-style method qualifiers and combination rules.  The process of
> creating custom method combination objects and their corresponding
> decorators is described in more detail under the `Extension API`_
> section.
>
> Note, by the way, that the ``@discount`` decorator shown will work
> correctly with any new predicates defined by other code.  For example,
> if ``zope.interface`` were to register its interface types to work
> correctly as argument annotations, you would be able to specify
> discounts on the basis of its interface types, not just classes or
> ``overloading``-defined interface types.
>
> Similarly, if a library like RuleDispatch or PEAK-Rules were to
> register an appropriate predicate implementation and dispatch engine,
> one would then be able to use those predicates for discounts as well,
> e.g.::
>
>      from somewhere import Pred  # some predicate implementation
>
>      @discount(
>          price,
>          Pred("isinstance(product,Shoe) and"
>               " product.material.name=='Blue Suede'")
>      )
>      def forty_off_blue_suede_shoes(product):
>          return Decimal('0.4')
>
> The process of defining custom predicate types and dispatching engines
> is also described in more detail under the `Extension API`_ section.
>
>
> Overloading Inside Classes
> --------------------------
>
> All of the decorators above have a special additional behavior when
> they are directly invoked within a class body: the first parameter
> (other than ``__proceed__``, if present) of the decorated function
> will be treated as though it had an annotation equal to the class
> in which it was defined.
>
> That is, this code::
>
>      class And(object):
>          # ...
>          @when(get_conjuncts)
>          def __conjuncts(self):
>              return self.conjuncts
>
> produces the same effect as this (apart from the existence of a
> private method)::
>
>      class And(object):
>          # ...
>
>      @when(get_conjuncts)
>      def get_conjuncts_of_and(ob: And):
>          return ob.conjuncts
>
> This behavior is both a convenience enhancement when defining lots of
> methods, and a requirement for safely distinguishing multi-argument
> overloads in subclasses.  Consider, for example, the following code::
>
>      class A(object):
>          def foo(self, ob):
>              print "got an object"
>
>          @overload
>          def foo(__proceed__, self, ob:Iterable):
>              print "it's iterable!"
>              return __proceed__(self, ob)
>
>
>      class B(A):
>          foo = A.foo     # foo must be defined in local namespace
>
>          @overload
>          def foo(__proceed__, self, ob:Iterable):
>              print "B got an iterable!"
>              return __proceed__(self, ob)
>
> Due to the implicit class rule, calling ``B().foo([])`` will print
> "B got an iterable!" followed by "it's iterable!", and finally,
> "got an object", while ``A().foo([])`` would print only the messages
> defined in ``A``.
>
> Conversely, without the implicit class rule, the two "Iterable"
> methods would have the exact same applicability conditions, so calling
> either ``A().foo([])`` or ``B().foo([])`` would result in an
> ``AmbiguousMethods`` error.
>
> It is currently an open issue to determine the best way to implement
> this rule in Python 3.0.  Under Python 2.x, a class' metaclass was
> not chosen until the end of the class body, which means that
> decorators could insert a custom metaclass to do processing of this
> sort.  (This is how RuleDispatch, for example, implements the implicit
> class rule.)
>
> PEP 3115, however, requires that a class' metaclass be determined
> *before* the class body has executed, making it impossible to use this
> technique for class decoration any more.
>
> At this writing, discussion on this issue is ongoing.
>
>
> Interfaces and Adaptation
> -------------------------
>
> The ``overloading`` module provides a simple implementation of
> interfaces and adaptation.  The following example defines an
> ``IStack`` interface, and declares that ``list`` objects support it::
>
>      from overloading import abstract, Interface
>
>      class IStack(Interface):
>          @abstract
>          def push(self, ob)
>              """Push 'ob' onto the stack"""
>
>          @abstract
>          def pop(self):
>              """Pop a value and return it"""
>
>
>      when(IStack.push, (list, object))(list.append)
>      when(IStack.pop, (list,))(list.pop)
>
>      mylist = []
>      mystack = IStack(mylist)
>      mystack.push(42)
>      assert mystack.pop()==42
>
> The ``Interface`` class is a kind of "universal adapter".  It accepts
> a single argument: an object to adapt.  It then binds all its methods
> to the target object, in place of itself.  Thus, calling
> ``mystack.push(42``) is the same as calling
> ``IStack.push(mylist, 42)``.
>
> The ``@abstract`` decorator marks a function as being abstract: i.e.,
> having no implementation.  If an ``@abstract`` function is called,
> it raises ``NoApplicableMethods``.  To become executable, overloaded
> methods must be added using the techniques previously described. (That
> is, methods can be added using ``@when``, ``@before``, ``@after``,
> ``@around``, or any custom method combination decorators.)
>
> In the example above, the ``list.append`` method is added as a method
> for ``IStack.push()`` when its arguments are a list and an arbitrary
> object.  Thus, ``IStack.push(mylist, 42)`` is translated to
> ``list.append(mylist, 42)``, thereby implementing the desired
> operation.
>
> (Note: the ``@abstract`` decorator is not limited to use in interface
> definitions; it can be used anywhere that you wish to create an
> "empty" generic function that initially has no methods.  In
> particular, it need not be used inside a class.)
>
>
> Subclassing and Re-assembly
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Interfaces can be subclassed::
>
>      class ISizedStack(IStack):
>          @abstract
>          def __len__(self):
>              """Return the number of items on the stack"""
>
>      # define __len__ support for ISizedStack
>      when(ISizedStack.__len__, (list,))(list.__len__)
>
> Or assembled by combining functions from existing interfaces::
>
>      class Sizable(Interface):
>          __len__ = ISizedStack.__len__
>
>      # list now implements Sizable as well as ISizedStack, without
>      # making any new declarations!
>
> A class can be considered to "adapt to" an interface at a given
> point in time, if no method defined in the interface is guaranteed to
> raise a ``NoApplicableMethods`` error if invoked on an instance of
> that class at that point in time.
>
> In normal usage, however, it is "easier to ask forgiveness than
> permission".  That is, it is easier to simply use an interface on
> an object by adapting it to the interface (e.g. ``IStack(mylist)``)
> or invoking interface methods directly (e.g. ``IStack.push(mylist,
> 42)``), than to try to figure out whether the object is adaptable to
> (or directly implements) the interface.
>
>
> Implementing an Interface in a Class
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> It is possible to declare that a class directly implements an
> interface, using the ``declare_implementation()`` function::
>
>      from overloading import declare_implementation
>
>      class Stack(object):
>          def __init__(self):
>              self.data = []
>          def push(self, ob):
>              self.data.append(ob)
>          def pop(self):
>              return self.data.pop()
>
>      declare_implementation(IStack, Stack)
>
> The ``declare_implementation()`` call above is roughly equivalent to
> the following steps::
>
>      when(IStack.push, (Stack,object))(lambda self, ob: self.push(ob))
>      when(IStack.pop, (Stack,))(lambda self, ob: self.pop())
>
> That is, calling ``IStack.push()`` or ``IStack.pop()`` on an instance
> of any subclass of ``Stack``, will simply delegate to the actual
> ``push()`` or ``pop()`` methods thereof.
>
> For the sake of efficiency, calling ``IStack(s)`` where ``s`` is an
> instance of ``Stack``, **may** return ``s`` rather than an ``IStack``
> adapter.  (Note that calling ``IStack(x)`` where ``x`` is already an
> ``IStack`` adapter will always return ``x`` unchanged; this is an
> additional optimization allowed in cases where the adaptee is known
> to *directly* implement the interface, without adaptation.)
>
> For convenience, it may be useful to declare implementations in the
> class header, e.g.::
>
>      class Stack(metaclass=Implementer, implements=IStack):
>          ...
>
> Instead of calling ``declare_implementation()`` after the end of the
> suite.
>
>
> Interfaces as Type Specifiers
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> ``Interface`` subclasses can be used as argument annotations to
> indicate what type of objects are acceptable to an overload, e.g.::
>
>      @overload
>      def traverse(g: IGraph, s: IStack):
>          g = IGraph(g)
>          s = IStack(s)
>          # etc....
>
> Note, however, that the actual arguments are *not* changed or adapted
> in any way by the mere use of an interface as a type specifier.  You
> must explicitly cast the objects to the appropriate interface, as
> shown above.
>
> Note, however, that other patterns of interface use are possible.
> For example, other interface implementations might not support
> adaptation, or might require that function arguments already be
> adapted to the specified interface.  So the exact semantics of using
> an interface as a type specifier are dependent on the interface
> objects you actually use.
>
> For the interface objects defined by this PEP, however, the semantics
> are as described above.  An interface I1 is considered "more specific"
> than another interface I2, if the set of descriptors in I1's
> inheritance hierarchy are a proper superset of the descriptors in I2's
> inheritance hierarchy.
>
> So, for example, ``ISizedStack`` is more specific than both
> ``ISizable`` and ``ISizedStack``, irrespective of the inheritance
> relationships between these interfaces.  It is purely a question of
> what operations are included within those interfaces -- and the
> *names* of the operations are unimportant.
>
> Interfaces (at least the ones provided by ``overloading``) are always
> considered less-specific than concrete classes.  Other interface
> implementations can decide on their own specificity rules, both
> between interfaces and other interfaces, and between interfaces and
> classes.
>
>
> Non-Method Attributes in Interfaces
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The ``Interface`` implementation actually treats all attributes and
> methods (i.e. descriptors) in the same way: their ``__get__`` (and
> ``__set__`` and ``__delete__``, if present) methods are called with
> the wrapped (adapted) object as "self".  For functions, this has the
> effect of creating a bound method linking the generic function to the
> wrapped object.
>
> For non-function attributes, it may be easiest to specify them using
> the ``property`` built-in, and the corresponding ``fget``, ``fset``,
> and ``fdel`` attributes::
>
>      class ILength(Interface):
>          @property
>          @abstract
>          def length(self):
>              """Read-only length attribute"""
>
>      # ILength(aList).length == list.__len__(aList)
>      when(ILength.length.fget, (list,))(list.__len__)
>
> Alternatively, methods such as ``_get_foo()`` and ``_set_foo()``
> may be defined as part of the interface, and the property defined
> in terms of those methods, but this a bit more difficult for users
> to implement correctly when creating a class that directly implements
> the interface, as they would then need to match all the individual
> method names, not just the name of the property or attribute.
>
>
> Aspects
> -------
>
> The adaptation system provided assumes that adapters are "stateless",
> which is to say that adapters have no attributes or storage apart from
> those of the adapted object.  This follows the "typeclass/instance"
> model of Haskell, and the concept of "pure" (i.e., transitively
> composable) adapters.
>
> However, there are occasionally cases where, to provide a complete
> implementation of some interface, some sort of additional state is
> required.
>
> One possibility of course, would be to attach monkeypatched "private"
> attributes to the adaptee.  But this is subject to name collisions,
> and complicates the process of initialization.  It also doesn't work
> on objects that don't have a ``__dict__`` attribute.
>
> So the ``Aspect`` class is provided to make it easy to attach extra
> information to objects that either:
>
> 1. have a ``__dict__`` attribute (so aspect instances can be stored
>     in it, keyed by aspect class),
>
> 2. support weak referencing (so aspect instances can be managed using
>     a global but thread-safe weak-reference dictionary), or
>
> 3. implement or can be adapt to the ``overloading.IAspectOwner``
>     interface (technically, #1 or #2 imply this)
>
> Subclassing ``Aspect`` creates an adapter class whose state is tied
> to the life of the adapted object.
>
> For example, suppose you would like to count all the times a certain
> method is called on instances of ``Target`` (a classic AOP example).
> You might do something like::
>
>      from overloading import Aspect
>
>      class Count(Aspect):
>          count = 0
>
>      @after(Target.some_method)
>      def count_after_call(self, *args, **kw):
>          Count(self).count += 1
>
> The above code will keep track of the number of times that
> ``Target.some_method()`` is successfully called (i.e., it will not
> count errors).  Other code can then access the count using
> ``Count(someTarget).count``.
>
> ``Aspect`` instances can of course have ``__init__`` methods, to
> initialize any data structures.  They can use either ``__slots__``
> or dictionary-based attributes for storage.
>
> While this facility is rather primitive compared to a full-featured
> AOP tool like AspectJ, persons who wish to build pointcut libraries
> or other AspectJ-like features can certainly use ``Aspect`` objects
> and method-combination decorators as a base for more expressive AOP
> tools.
>
> XXX spec out full aspect API, including keys, N-to-1 aspects, manual
>      attach/detach/delete of aspect instances, and the ``IAspectOwner``
>      interface.
>
>
> Extension API
> =============
>
> TODO: explain how all of these work
>
> implies(o1, o2)
>
> declare_implementation(iface, class)
>
> predicate_signatures(ob)
>
> parse_rule(ruleset, body, predicate, actiontype, localdict, globaldict)
>
> combine_actions(a1, a2)
>
> rules_for(f)
>
> Rule objects
>
> ActionDef objects
>
> RuleSet objects
>
> Method objects
>
> MethodList objects
>
> IAspectOwner
>
>
>
> Implementation Notes
> ====================
>
> Most of the functionality described in this PEP is already implemented
> in the in-development version of the PEAK-Rules framework.  In
> particular, the basic overloading and method combination framework
> (minus the ``@overload`` decorator) already exists there.  The
> implementation of all of these features in ``peak.rules.core`` is 656
> lines of Python at this writing.
>
> ``peak.rules.core`` currently relies on the DecoratorTools and
> BytecodeAssembler modules, but both of these dependencies can be
> replaced, as DecoratorTools is used mainly for Python 2.3
> compatibility and to implement structure types (which can be done
> with named tuples in later versions of Python).  The use of
> BytecodeAssembler can be replaced using an "exec" or "compile"
> workaround, given a reasonable effort.  (It would be easier to do this
> if the ``func_closure`` attribute of function objects was writable.)
>
> The ``Interface`` class has been previously prototyped, but is not
> included in PEAK-Rules at the present time.
>
> The "implicit class rule" has previously been implemented in the
> RuleDispatch library.  However, it relies on the ``__metaclass__``
> hook that is currently eliminated in PEP 3115.
>
> I don't currently know how to make ``@overload`` play nicely with
> ``classmethod`` and ``staticmethod`` in class bodies.  It's not really
> clear if it needs to, however.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
> _______________________________________________
> Python-3000 mailing list
> Python-3000 at python.org
> http://mail.python.org/mailman/listinfo/python-3000
> Unsubscribe:
> http://mail.python.org/mailman/options/python-3000/monpublic%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-3000/attachments/20070508/fcf166af/attachment.html