[Python-3000] Generic functions

[Ian Bicking]

Guido van Rossum wrote:
> On 4/3/06, Ian Bicking <ianb at colorstudy.com> wrote:
>> As an alternative to adaptation, I'd like to propose generic functions.
>>   I think they play much the same role, except they are much simpler to
>> use and think about.
> 
> Given that Phillip Eby is another proponent of generic functions I
> seriously doubt the latter. Certainly your post causes me to produce
> an endless list of questions (some of which I'll interject below).

I don't know; I'm not sure if he's on this list, but I'll copy him 
directly.  Phillip wrote PyProtocols before the generic functions; I 
don't know if he feels as strongly about adaptation after going through 
that process and seeing the result of each.

>> Though RuleDispatch offers considerably more features through predicate
>> dispatch, it would probably be best to just consider type based
>> dispatch, as that's more equivalent to adaptation.
> 
> I may have missed some context (I have so far skipped all threads
> mentioning adaptation), but since you're starting a new thread here,
> could you perhaps explain what RuleDispatch refers to?

It's the generic function (predicate dispatch) package that Phillip 
wrote.  What I described is a vague hand-wavy interpretation of it. 
Installation is described on the PEAK front page 
(http://peak.telecommunity.com/) and an article by David Mertz 
(http://www-128.ibm.com/developerworks/library/l-cppeak2/) and I'm not 
actually sure where the main documentation is...

>> So, the copy_reg module is one example where adaptation has been
>> proposed.  The idea is that you adapt an object to a pickleable object,
>> or something along those lines.
> 
> That definition doesn't sound right; but since you are waving your
> hands and I haven't yet read the thread about adaptation and copy_reg
> I'll leave it at that rather than give me own version of how copy_reg
> could be seen as adaptation (actually I'm not sure yet).
> 
>> It's a little vague, because while you
>> typically adapt an instance coming in, you "adapt" a string to a new
>> instance on the way out.
> 
> You only use adaptation for pickling; for unpickling all the
> information is in the pickle data. Pickling is a thoroughly asymmetric
> API.
> 
>> Or, I dunno, it's not clear to me.  In fact,
>> though that's the typical example, I'm going to bail on that because the
>> pickling protocol is an aside to this and too complex for me to digest
>> right now.  pprint is a lot easier, and conveniently is much nicer with
>> generic functions than adaptation ;)
> 
> Watch it though. it may be a great example to explain generic
> functions. But it may be the only example, and its existence may not
> be enough of a use case to motivate the introduction of gneric
> functions.

I can come up with more, to be sure.  TurboGears has started using 
RuleDispatch in several places internally, and I think I saw something 
about Django using it as well.  I've used it some for form generation 
experiments, which is one of the examples where people have noted 
adaptation (adapting an abstract field to an HTML input widget, for 
instance).

In TurboGears they are using it for JSON serialization (JSON is a 
Javascriptish syntax, as an alternative to XML), where it is kind of 
similar to Pickle (but one way), or more similar to pprint.

>> Anyway, pprint could work like:
>>
>> class PrettyPrinter:
>>      @generic
>>      def pformat(self, object):
>>          """Return the pretty string representation of object"""
>>          return repr(object)
>>      # pformat is now "more" than just a function, it's a callable
>>      # object that does type-based dispatch using an internal registery
>>      # of implementations, with the implementation above as the fallback.
> 
> Whoa! First of all, my gut reaction is already the same as for
> adaptation: having a single global registry somehow feels wrong. (Or
> is it not global? "internal" certainly sounds like that's what you
> meant; but for methods this seems wrong, one would expect a registry
> per class, or something like that.)

No, it's pretty local; the registry is stored in the 
PrettyPrinter.pformat object.  This is one of the rather elegant aspects 
of RuleDispatch, I think -- once you define a function as generic, the 
function itself contains all the methods you need to extend it.

> Second, I'm curious if the fact that the argument name is 'object'
> (which is also a built-in type) is an accident, or has significance.

Just the existing signature.

> Next, I wonder what the purpose of the PrettyPrinter class is. Is it
> just there because the real pprint module defines a class by that
> name? Or does it have some special significance? 

It's there because it is matching the pprint module.  Also it holds some 
state which is useful to keep separate from the rest of the arguments, 
like the current level of indentation.

> Are generic functions
> really methods? Can they be either?

They can be either.  I believe that has required special casing methods, 
because methods and functions act differently.  In my experience I also 
have a hard time writing intelligent decorators that work on both.

>> # It also now can be used as a decorator that registers implementations:
>> @PrettyPrinter.pformat.when(object=list)
>> def pformat_list(self, object):
>>      s = '['
>>      for item in object:
>>          s += (' '*self.indent) + self.pformat(item) + ',\n'
>>      return s + (' '*self.indent) + ']'
> 
> Ah, the infamous "when" syntax again, which has an infinite number of
> alternative calling conventions, each of which is designed to address
> some "but what if...?" objection that might be raised.

Yeah, my use here also clashes with what RuleDispatch uses, so don't pay 
too much attention to that particular.

> If pformat is a method of the quite ordinary class PrettyPrinter, why
> isn't the pformat_list() method/function declared inside that class?

You could, but potentially it could be defined in a different module 
somewhat unrelated to pprint.PrettyPrinter.

> Sooner or later the name conflict between your argument and the
> built-in type is going to cause problems, either because you need to
> access the built-in type or because the reader is confused.

I used "pformat_list" as a name, purely for convenience.  I could have 
used "pformat" or "foo" and it wouldn't matter.  The advantage of 
pformat_list is that it displays better in tracebacks because it has a 
more unique function name.  The function is attached to 
PrettyPrinter.pformat because I use the decorator attached to that object.

> What does when(object=list) mean? Does it do an isinstance() check?

Yes; I think RuleDispatch has a form (though I can't remember what the 
form is -- it isn't .when()).

> Is there any significance to the name pformat_list? Could I have
> called it foobar? Why not just pformat?

Just for tracebacks, and for example to make it greppable.

>> Some things to note:
>>
>> * There's no interface created here.  There's no hidden interface
>> lurking in the background either.
> 
> You seem to be expecting a particular objection that I don't have. :-)
> Never in a thousand years would I have thought of interfaces here.
> What context am I missing?

To do the same with adaptation you'd have to create an interface.  It 
would be something like:

class PrettyPrinter:
     def pformat(self, object):
         printable = IPrettyPrintable(object)
         return printable.pformat(object, self.indent, ...)

At least, that's how I'd presume you'd do the same.  When talking about 
copy_reg and other adaptation cases, there's lots of handwaving about 
adapting something to a pickleable interface or somesuch.  In practice 
you'd actually have to create real interfaces for each of those cases, 
so I think the interfaces are a hidden cost.

>> * It requires cooperation from the original function (pformat -- I'm
>> using "function" and "method" interchangably).
> 
> Thereby not helping the poor reader who doesn't understand all of this
> as well as you and Phillip apparently do.

Hey, you're the one set on functions and methods being interchangeable 
with an explicit self argument!  Mostly everything applies to either case.

>> It does not require any
>> cooperation from classes like list, similar to adaptation and dissimilar
>> to magic methods.
> 
> What do you mean here? Is list used because it appears in the when clause?

It means you don't need to add a magic method to the list type, or 
inject a method into a type to add a magic method.

> I'm guessing that you are contrasting it with len(), which could be
> seen as a special kind of built-in "generic function" if one squints
> enough, but one that requires the argument to provide the __len__
> magic method. But since len() *does* require the magic method, doesn't
> that disqualify it from competing?

Yes, this is in contrast with len(), which achieves its goal because the 
people who write the len() function write the entire language, and can 
put a __len__ on whatever they want ;)  For other cases magic-method 
based systems tend to look like:

def pprint(object):
     if isinstance(object, list): ...
     elif isinstance(object, tuple): ...
     ...
     elif hasattr(object, '__pprint__'):
         object.pprint()
     else:
         print repr(object)

That is, all the built in objects get special-cased and other objects 
define a magic method.

>> Adaptation also requires cooperation from the caller,
>> as the adaptation would be applied inside pformat.
> 
> Huh? Aren't you contradicting yourself here? If the adaptation is done
> inside pformat (being the callee) what does the caller have to do
> (except from the obvious "call pformat")?

Sorry, too many callers.  I don't know what name to give it -- anyway, 
if pretty printing was made more extensible with adaptation, it would 
require modifications to PrettyPrinter.pformat, just like generic 
functions require modifying that same method.  With magic methods, all 
arguments to pformat would have to be extended with a magic method.

>> * The function is mostly self-describing.
> 
> Perhaps once you've wrapped your head around the when() syntax. To me
> it's all magic; I feel like I'm back in the situation again where I'm
> learning a new language and I haven't quite figured out which
> characters are operators, which are separators, which are part of
> identifiers, and which have some other magical meaning. IOW it's not
> describing anything for me, nor (I presume) for most Python users at
> this point.

The function is described in its docstring, like other functions, as 
opposed to being partially documented in an interface.

The implementation of my simplistic form of generic function isn't too 
hard.  Ignoring keyword arguments, it might work like:

class generic(object):
     def __init__(self, func):
         self.func = func
         self.registry = {}
     def __call__(self, *args):
         for pattern, implementation in self.registry.items():
             for passed, expected in zip(args, pattern):
                 # None is a wildcard here:
                 if (expected is not None and
                     not isinstance(passed, expected)):
                     break
             else:
                 return implementation(*args)
         return self.func(*args)
     def when(self, *args):
         def decorator(func):
             self.registry[args] = func
             return func
         return decorator
     def __get__(self, obj, type=None):
         if obj is None:
             return self
         return types.MethodType(self, obj, type)

There's lots of details, and handling keyword arguments, dealing 
intelligently with subclasses, and other things I probably haven't 
thought of.  But anyway, this allows:

class PrettyPrinter:
     def pformat(self, object): ...

# Without keyword arguments I have to give a wildcard for the self
# argument...
@PrettyPrinter.pformat(None, list)
def pformat_list(self, object):
     ...


It *is* a new paradigm, just like adaptation is.  I think it's a pretty 
easy paradigm to get used to, and you don't have to introduce auxiliary 
ideas like interfaces to make it sensible.

>> If it has certain return
>> values, then you state what those values are in documentation; there's
>> no need to be formal about it.  In contrast you have to come up with a
>> whole collection of interfaces to start using adaptation.
>>
>> * The function is the hub of all the registration, which seems very
>> natural, since you are extending the function.
> 
> Tell us more about the registration machinery. Revealing that (perhaps
> simplified) could do a lot towards removing the magical feel.

Hopefully the simple implementation I gave makes it clearer. 
RuleDispatch has something much more clever and faster, and you can give 
it arbitrary expressions.  But in terms of registration it works the same.

>> * Like adaptation, you must import a module that defines extra
>> specializations of the generic function before those are active (just
>> like you have to import adapter declarations).  This strikes me as a
>> significant problem.  I assume ZCML addresses this, but I also assume
>> that's not a reasonable solution for core Python.
> 
> You have to import these modules for their side effects (on the
> registry), not so  much because they make some objects or names
> available that you use directly, right?

Yes, for the side effects.

>> * Magic methods do *not* have this import problem, because once you have
>> an object you have all its methods, including magic methods.
> 
> Well, of course that works only until you need a new magic method.

Yes, pluses and minuses ;)

>> RuleDispatch has a bunch more features than just simple type-based
>> generic functions.  But I think that type-based generic functions would
>> be an easier or more comfortable place to start, and wouldn't preclude a
>> more featureful implementation later.
> 
> If RuleDispatch is the thing defining Phillip's full when() syntax,
> yes, please focus on the simpler rules.
> 
>> Type-based generic functions and adaptation are more-or-less equivalent.
>>   That is, you can express one in terms of the other, at least
>> functionally if not syntactically.
> 
> Could you elaborate this with a concrete example?

Well, in Zope you do something like:

     class IFoo(Interface):
         ...

And adaptation looks like:

     fooish_obj = IFoo(obj)

There's other proposals, like adapt(obj, IFoo), but it doesn't really 
change anything.  Now, Zope's Interface is a funny class, because it's 
not really a class, it's an instance you can subclass.  So... to ignore 
that, lets imagine you actually have a class and interface, because I 
don't want to try to figure out how Zope's Interface class/instance works...

     class IFoo(Interface):
         """Calling IFoo()(obj) coerces the object to IFoo"""
         @generic
         def __call__(self, obj):
             raise NotImplementedError

Now you have a class, and you want to register an adaptation:

     class Bar(object): ...
     class FooBarAdapter(object):
         def __init__(self, bar):
             self.bar = bar
         ... other foo-ish methods ...
     @IFoo.__call__.when(None, Bar)
     def coerce_bars(self, obj):
         return FooBarAdapter(obj)

I gave an example of how you'd use adaptation for pretty printing up above.

>> If you really wanted adaptation,
>> then the interface becomes a things-obeying-this-interface factory --
>> i.e., a generic function.  Generic functions are similer to
>> multi-adaptaters, where you adapt a tuple of objects, similar to the
>> tuple of arguments to a function.  This is technically like generic
>> functions, but syntactically rather awkward.
>>
>> [Predicate-based dispatching goes considerably further, allowing real
>> duck typing, e.g., you could implement a pformat method for everything
>> that has an "__iter__" method and no "next" method (i.e., all iterables,
>> but not iterators which could lead to unintentionally consuming the
>> iterator).]
>>
>> Anyway, I think generic functions are very compatible with Python syntax
>> and style, and Python's greater emphasis on what an object or function
>> can *do*, as opposed to what an object *is*, as well as the use of
>> functions instead of methods for many operations.  People sometimes see
>> the use of functions instead of methods in Python as a weakness; I think
>> generic functions turns that into a real strength.
> 
> Perhaps. The import-for-side-effect requirement sounds like a
> showstopper though.

There's a bunch of places where this is a problem.  This is a problem 
anywhere you want to add functionality to something that doesn't belong 
to you.  The effects of imports is a problem -- but it's a problem we 
already have, not one that is added just with generic functions or 
adaptation.

-- 
Ian Bicking  |  ianb at colorstudy.com  |  http://blog.ianbicking.org