[Python-3000] PEP 3124 - more commentary

[Guido van Rossum]

On 5/14/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 12:47 PM 5/14/2007 -0700, Guido van Rossum wrote:
> >> >I realize that @overload is only a shorthand for @when(function). But
> >> >I'd much rather not have @overload at all -- the frame inspection
> >> >makes it really hard for me to explain carefully what happens without
> >> >just giving the code that uses sys._getframe(); and this makes it
> >> >difficult to reason about code using @overload.
> >>
> >>This is why in the very earliest GF discussions here, I proposed a
> >>'defop expr(...)' syntax, as it would eliminate the need for any
> >>getframe hackery.
> >
> >But that would completely kill your "but it's all pure Python code so
> >it's harmless and portable" argument.
>
> Uh, wha?  You lost me completely there.  A 'defop' syntax simply
> eliminates the need to name the target function twice (once in the
> decorator, and again in the 'def').  I don't get what that has to do
> with stuff being harmless or portable or any of that.
>
> Are you perhaps conflating this with the issue of marking functions
> as overloadable?  These are  independent ideas, AFAICT.
>
> >It seems that you're really not interested at all in compromising to
> >accept mandatory marking of the base overloadable function.
>
> Uh, wha?  I already agreed to that a couple of weeks ago:
>
> http://mail.python.org/pipermail/python-3000/2007-May/007205.html
>
> I just haven't updated the PEP yet -- any more than I've updated it
> with anything else that's been in these ongoing threads, like the
> :next_method annotation or splitting the PEP.

Ah, sorry. The way this misunderstanding probably originated was that
I read your "this is why I originally proposed defop, to avoid
getframe hackery" as a maintaining the current need for getframe,
instead of a historical fact.

> >>Anyway, with this, it could also be placed as a keyword
> >>argument.  The main reason for putting it in the first position is
> >>performance.  Allowing it to be anywhere, however, would let the
> >>choice of where be a matter of style.
> >
> >Right. What's the performance issue with the first argument?
>
> Chaining using the first argument can be implemented using a bound
> method object, which gets performance bonuses from the C eval loop
> that partial() objects don't.  (Of course, when RuleDispatch was
> written, partial() objects didn't exist, anyway.)

Sounds like premature optimization to me. We can find a way to do it
fast later; let's first make it right.

> >>However, since we're going to have to have some way for 'super' to
> >>know the class a function is defined in, ISTM that the same magic
> >>should be reusable for the first-argument rule.
> >
> >Perhaps. Though super only needs to know it once the method is being
> >called, while your decorator (presumably) needs to know when the
> >method is being defined, i.e. before the class object is constructed.
>
> Not really; at some point the class object has to be assigned and
> stored somewhere for super to use, so if same process of "assigning"
> can be used to actually perform the registration, we're good to go.

True. So are you working with Tim Delaney on this? Otherwise he may
propose a simpler mechanism that won't allow this re-use of the
mechanism.

> >Also, the similarities between next-method and super are overwhelming.
> >It would be great if you could work with Tim Delaney on a mechanism
> >underlying all three issues, or at least two of the three.
>
> I'm not sure I follow you.  Do you mean, something like using :super
> as the annotation instead of next_method, or are you just talking
> about the implementation mechanics?

super is going to be a keyword with magic properties. Wouldn't it be
great if instead of

@when(...)
def flatten(x: Mapping, nm: next_method):
  ...
  nm(x)

we could write

@when(...)
def flatten(x: Mapping):
  ...
  super.flatten(x)  # or super(x)

or some other permutation of super? Or do you see the need to call
both next-method and super from the same code?

> >> >Forgive me if this is mentioned in the PEP, but what happens with
> >> >keyword args? Can I invoke an overloaded function with (some) keyword
> >> >args, assuming they match the argument names given in the default
> >> >implementation?
> >>
> >>Yes.  That's done with code generation; PEAK-Rules uses direct
> >>bytecode generation, but a sourcecode-based generation is also
> >>possible and would be used for the PEP implementation (it was also
> >>used in RuleDispatch).
> >
> >There's currently no discussion of this.
>
> Well, actually there's this bit:
>
> """The use of BytecodeAssembler can be replaced using an "exec" or "compile"
> workaround, given a reasonable effort.  (It would be easier to do this
> if the ``func_closure`` attribute of function objects was writable.)"""
>
> But the closure bit is irrelevant if we're using @overloadable.

Thanks.

> >Without a good understanding
> >of the implementation I cannot accept the PEP.
>
> The mechanism is exec'ing of a string containing a function
> definition.  The original function's signature is obtained using
> inspect.getargspec(), and the string is exec'd to obtain a new
> function whose signature matches, but whose body contains the generic
> function lookup code.

Do note that e.g. in IronPython (and maybe also in Jython?)
exec/eval/compile are 10-50x slower (relative to the rest of the
system) than in CPython.

It does look like a clever approach though.

> In practice, the actual function definition has to be nested, so that
> argument defaults can be passed in without needing to convert them to
> strings, and so that the needed lookup tables can be seen via closure
> variables.  A string template would look something like:
>
>      def make_the_function(__defaults, __lookup):
>          def $funcname($accept_signature):
>              return __lookup($type_tuple)($call_signature)
>          return $funcname
>
> The $type_tuple bit would expand to something like:
>
>      type(firstargname), type(secondargname), ...
>
> And $accept_signature would expand to the original function's
> signature, with default values replaced by "__defaults[0]",
> "__defaults[1]", etc. in order to make the resulting function have
> the same default values.
>
> The function that would be returned from @overloadable would be the
> result of calling "make_the_function", passing in the original
> function's func_defaults and an appropriate value for __lookup.
>
> A similar approach is used in RuleDispatch currently.
>
>
>
> >> >Also, can we overload different-length signatures (like in C++ or
> >> >Java)? This is very common in those languages; while Python typically
> >> >uses default argument values, there are use cases that don't easily
> >> >fit in that pattern (e.g. the signature of range()).
> >>
> >>I see a couple different possibilities for this.  Could you give an
> >>example of how you'd *like* it to work?
> >
> >In the simplest case (no default argument values) overloading two-arg
> >functions and three-arg functions with the same name should act as if
> >there were two completely separate functions, except for the base
> >(default) function. Example:
> >
> >@overloadable
> >def range(start:int, stop:int, step:int):
> >  ...  # implement xrange
> >
> >@range.overload
> >def range(x): return range(0, x, 1)
> >
> >@range.overload
> >def range(x, y): return range(x, y, 1)
>
> Hm.  I'll need to give some thought to that, but it seems to me that
> it's sort of like having None defaults for the missing arguments, and
> then treating the missing-argument versions as requiring type(None)
> for those arguments.  Except that we'd need something besides None,
> and that the overloads would need wrappers that drop the extra
> arguments.  It certainly seems possible, anyway.
>
> I'm not sure I like it, though.

C++ and Java users use it all the time though.

> It's not obvious from the first
> function's signature that you can call it with fewer arguments, or
> what that would mean.  For example, shouldn't the later signatures be
> "range(stop)" and "range(start,stop)"?  Hm.

I don't know if the arg names for overloadings must match those of the
default function or not -- is that specified by your PEP?

My own trivially simple overloading code (sandbox/overload, and now
also added as an experiment to sandbox/abc, with slightly different
terminology and using issubclass exclusively, as you recommended over
a year ago :-) has no problem with this. Of course it only handles
positional arguments and completely ignores argument names except as
keys into the annotations dict.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)