[Python-3000] defop ?

[Guido van Rossum]

[Side remark: I'm sure we would reach understanding and agreement much
quicker if we were all in the same room for a day. Maybe we could try
to have a day at the next PyCon where we hash this out?]

One of the problems that I still have with defop is that it is
presented as pure magic.

On the one hand, it looks like

  defop len(self):
    return 42

is just syntactic sugar for

  def __len__(self):
    return 42

OTOH it seems that it is expected to do something completely different
when someone write

  defop foobar(self):
    ...

where foobar is some user-defined generic function. Can someone who
understands this provide exact syntax and semantics?

Do I understand correctly that the syntax is more like

  'defop' <expr> '(' <args> ')' ':' <suite>

than like

  'defop' NAME '(' <args> ')' ':' <suite>

? (That would need some kind of limitation on the syntax for <expr> so
that it can't contain a call, or else it would be syntactically
ambiguous.)

What kind of object should the expression produce? What methods on it
are we going to call, with what arguments? I'd love to see an
explanation that showed how instead of defop we could also use a
certain decorator. Perhaps

  defop <expr> ( <args> ) : <suite>

is equivalent to

  @defop(<expr>)
  def _ ( <args> ) : <suite>  # The method name is immaterial here

? Then what goes into the defop decorator? And how does this jive with
it being inside a class? (And what would it do if it wasn't?)

I'm assuming that "defop foo.bar (...)" only works if foo is imported
or defined as a global and has a bar attribute.

I'm grasping for understanding here. I don't think Phillip explained
this part of his proposal -- all I can find is examples using built-in
operations (e.g. "defop len" or "defop operator.getitem") and the
handwaving gesture "it could be expanded to add methods for arbitrary
*new* operations not defined by the language". My lack of
understanding is directly related to this support for new operations.

Suppose I wanted to create a flatten operation. What do I put in the
module that defines the 'flatten' generic function? Perhaps


# This is flattening.py
from <somewhere> import generic

@generic
def flatten(obj):
  "Generator that flattens a possibly nested sequence."
  # Default implementation assumes the argument is atomic.
  yield obj

@flatten.when(list)  # Oh how I hate "when" but that's not the point here
def flatten_list(obj):
  for value in obj:
    for v in flatten(value):
      yield v

@flatten[tuple] = flatten_list   # or flatten[list]


Now I have another module where I define a class implementing a binary
tree. I want it to be flattened as the flattening of its items in
in-order.


import flattening

class BinaryTree:

  def __init__(self, value, left=None, right=None):
    # Let's pretend the labels must always be strings
    assert isinstance(value, str)
    # I really want this to be homogeneous, i.e. all nodes are
BinaryTrees or None
    assert left is None or isinstance(left, BinaryTree)
    assert right is None or isinstance(right, BinaryTree)
    self.value = value
    self.left = left
    self.right = right

  def __iter__(self):
    # Recursive in-order iterator
    if self.left is not None:
      for value in self.left: yield value
    yield self.value
    if self.right is not None:
      for value in self.right: yield value

  defop flattening.flatten(self):
    for value in self:
      yield value


I realize that I could also have written

  flattening.flatten[BinaryTree] = flattening.flatten[list]

but I'd like to assume for the sake of argument that the BinaryTree
class wants to define its own flattening code -- for example, since we
know its values are always strings (see the first assert), we don't
need the recursive call to flatten() that's present in flatten_list,
and we really care about performance. Anyway, the point is to show a
class that has its own implementation of some user-defined generic
function.

So I'm still at a loss about how defop works here. I could understand
a different way of defining the flattening of a binary tree:


class BinaryTree:
  ... # as before, but without the defop

# now, outside the class:

import flattening

@flattening.flatten.when(BinaryTree)
def flatten_binary_tree(bt):
  for value in bt:
    yield value


but how on earth is the defop syntax of the @defop decorator going to
generate this?  I can't see any way to implement it without having
access to the class object, but that doesn't exist yet at the time
defop or @defop executes. The best I can think of is for @defop to use
sys._getframe() to access the dict in which the class body is being
evaluated, find or create a list __defop_deferred__ there, and append
the tuple(<expr>, <function>) to it. Then the metaclass (i.e., type)
should look for this list, and if it exists, do the registrations. I'm
guessing you had something like that in mind. I find it pretty fragile
though -- what if defop is used in a different context (not directly
inside a class)? Is that a syntactic error or does it just modify a
random frame's globals, effectively becoming an expensive no-op?

A different line of questioning would be to try to understand how
Phillip's addmethod and hasmethod are supposed to work. I still hope
someone will explain it to me.

--Guido

On 11/22/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Antoine wrote:
> > Is "defop" useful?
> >
> > Why wouldn't it be possible to enable special method lookup with this kind
> > of syntax:
> >
> > iter = generic_function()
> > iter.lookup_special_method("__iter__")
> >
> > And then the classical:
> >
> > class Foo:
> >     def __iter__(self):
> >         pass
>
> A major benefit of the defop syntax lies in the fact that the full Python
> module namespace is available to you for disambiguation of which operation you
> are referring to. So if module A defines an overloaded operation "snap" and
> module B define an overloaded operation "snap", that's OK, because "defop
> A.snap" and "defop B.snap" can be used to make it clear which one you mean.
>
> The magic method namespace, on the other hand, is completely flat: there can
> be only one __snap__ magic method in a given Python application. If two
> different modules both want to use the same magic method name, you're out of
> luck because you can't use the module namespace to disambiguate which one you
> actually want to support (heck, you might even want to support both of them).
>
> A second benefit of defop is that if you have a reference to a generic
> function then you can overload it. With magic methods, you can only overload
> it if you know the appropriate incantation (see the thread about correcting
> the fact that the incantation to overload bool in 2.x is __nonzero__ instead
> of __bool__ - with defop, it could be written simply as "defop bool(self):",
> and issue wouldn't need to rise).
>
> Finally, the defop syntax means a typo will likely result in an exception when
> the class definition is executed instead of at some random later time. For
> example, the typo "def __getiten__(self, key):" will go completely undetected
> when a class definition is executed, but "defop operator.getiten(self, key):"
> would fail with an AttributeError.
>
> That said, it doesn't *have* to be new syntax - an appropriate function
> decorator and a class decorator or metaclass to register the appropriate
> overloads also support this. defop just makes it easier to read and write.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)