[Python-3000] Updated and simplified PEP 3141: A Type Hierarchy for Numbers

[Guido van Rossum]

On 5/17/07, Talin <talin at acm.org> wrote:
> Lets therefore assume that the numeric ABCs will use this new
> inheritance mechanism, avoiding the problem of taking an immature class
> hierarchy and setting it in stone. The PEPs in this class would then no
> longer need to have this privileged status; They could be replaced and
> changed at will.
>
> Assuming that this is true, the question then becomes whether these
> classes should be treated like any other standard library submission. In
> other words, shouldn't this PEP be implemented as a separate module, and
> have to prove itself 'in the wild' before being adopted into the stdlib?
> Does this PEP even need to be a PEP at all, or can it just be a
> 3rd-party library that is eventually adopted into Python?

No; I think there's a lot of synergy to be had by making it a standard
library module. For example, the Complex, Real and Integer types
provide a common ground for the built-in types and the types
implemented in numpy. Assuming (some form of) PEP 3124 is accepted, it
would be a shame if we had to specialize GFs on concrete types like
int or float. If we could encourage the habit right from the start to
use the abstract classes in such positions, then numpy integration
would be much easier.

> Now, I *could* see adopting an untried library embodying untested ideas
> into the stdlib if there was a crying need for the features of such a
> library, and those needs were clearly being unfulfilled. However, I am
> not certain that this is the case here.

The ideas here are hardly untested; the proposed hierarchy is taught
in high school math if not before, and many other languages use it
(e.g. Scheme's numeric tower, referenced in the PEP). Some
implementation details are untested, but I doubt that the general idea
sparks much controversy (it hasn't so far).

> At the very least, I think it should be stated in the PEP whether or not
> the ABCs defined here are going to be using traditional or dynamic
> inheritance.

Dynamic inheritance for sure.

> If it is the latter, and we decide that this PEP is going to be part of
> the stdlib, then I propose the following library organization:
>
>     import abc              # Imports the basic ABC mechanics
>     import abc.collections  # MutableSequence and such
>     import abc.math         # The number hierarchy
>     ... and so on

I don't like the idea of creating a ghetto for ABCs. Just like the
ABCs for use with I/O are defined in the io module (read PEP 3161 and
notice that it already uses a thinly disguised form of ABCs), the ABCs
for collections should be in the existing collections module. I'm not
sure where to place the numeric ABCs, but I'd rather have a top-level
numbers module.

> Now, there is another issue that needs to be dicussed.

> The classes in the PEP appear to be written with lots of mixin methods,
> such as __rsub__ and __abs__ and such. Unfortunately, the current
> proposed method for dynamic inheritance does not allow for methods or
> properties to be inherited from the 'virtual' base class. Which means
> that all of the various methods defined in this PEP are utterly
> meaningless other than as documentation - except in the case of a new
> user-created class of numbers which inherit from these ABCs using
> traditional inheritance, which is not something that I expect to happen
> very often at all. For virtually all practical uses, the elaborate
> methods defined in this PEP will be unused and inaccessible.

And yet they are designed for the benefit of new numeric type
implementations so that mixed-mode arithmetic involving two different
3rd party implementations can be defined soundly. If this is deemed
too controversial or unwieldy and unnecessary I'd be okay with
dropping it, though I'm not sure that it hurts. There are some
problems with specifying it so that all the cases work right, however.
In any case we could have separate mix-in classes for this purpose --
while ABCs *can* be dual-purpose (both specification and mix-in),
there's no rule that says the *must* be.

The problem Jeffrey and I were trying to solve with this part of the
spec is what should happen if you have two independently developed 3rd
party types, e.g. MyReal and YourReal, both implementing the Real ABC.
Obviously we want MyReal("3.5") + YourReal("4.5") to return an object
for which isinstance(x, Real) holds and whose value is close to
float("8.0"). But this is not so easy. Typically, classes like this
have methods like these::

class MyReal:
  def __add__(self, other):
    if not isinstance(other, MyReal): return NotImplemented
    return MyReal(...)
  def __radd__(self, other):
    if not isinstance(other, MyReal): return NotImplemented
    return MyReal(...)

but this doesn't support mixed-mode arithmetic at all.

(Reminder of the underlying mechanism: for a+b, first a.__add__(b) is
tried; if that returns NotImplemented, b.__radd__(a) is tried; if that
also returns NotImplemented, the Python VM raises TypeError.
Exceptions raised at any stage cause the remaining steps to be
abandoned, so if e.g. __add__ raises TypeError, __radd__ is never
tried. This is the crux of the problem we're trying to solve.)

Supporting mixed-mode arithmetic with *known* other types like float
is easy: just add either

  if isinstance(other, float): return self + MyReal(other)

or

  if isinstance(other, float): return float(self) + other

to the top of each method. But that still doesn't support MyReal() +
YourReal(). For that to work, at least one of the two classes has to
blindly attempt to cast the other argument to a known class. For
example, instead of returning NotImplemented, __radd__ (which knows it
is being called as a last resort) could return self+float(other), or
float(self)+float(other), under the assumption that all 3rd party Real
types can be converted to the built-in float type with only moderate
loss.

Unfortunately, there are a lot of cases to consider here. E.g. we
could be adding MyComplex() to YourReal(), and then the float cast in
__radd__ would be a disaster (since MyComplex() can't be cast to
float, only to complex). We are trying to make things easier for 3rd
parties that *do* want to use the ABC as a mix-in, by asking them to
call super.__[r]add__(self, other) whenever the other argument is not
something they specifically recognize.

I'm pretty sure that we haven't gotten the logic for this quite right
yet. I'm only moderately sure that we *can* get it right. But in any
case, please don't let this distract you from the specification part
of the numeric ABCs.

> This really highlights what I think is a problem with dynamic
> inheritance, and I think that this inconsistency between traditional and
> dynamic inheritance will eventually come back to haunt us. It has always
> been the case in the past that for every property of class B, if
> isinstance(A, B) == True, then A also has that property, either
> inherited from B, or overridden in A. The fact that this invariant will
> no longer hold true is a problem in my opinion.

Actually there is a school of thought (which used to prevail amongst
Zopistas, I don't know if they've been cured yet) that class
inheritance was purely for implementation inheritance, and that a
subclass was allowed to reverse policies set by the base class. While
I don't endorse this as a general rule, I don't see how (with a
sufficiently broad definition of "property") your invariant can be
assumed even for traditional inheritance. E.g. a base class may be
hashable or or immutable but the subclass may not be (it's trivial to
create mutable subclasses of int, str or tuple).

> I realize that there isn't currently a solution to efficiently allow
> inheritance of properties via dynamic inheritance. As a software
> engineer, however, I generally feel that if a feature is unreliable,
> then it shouldn't be used at all.

That sounds like a verdict against all dynamic properties of the language.

> So if I were designing a class
> hierarchy of ABCs, I would probably make a rule for myself not to define
> any properties or methods in the ABCs at all, and to *only* use ABCs for
> type testing via 'isinstance'.

And that is a fine policy to hold yourself to.

> In other words, if I were writing this PEP,

Hey, you are! Or do you want your name taken off? At the very least I
think the rationale (all your words) needs some ervision in the light
of Benji York's comments in another thread.

> all of those special methods
> would be omitted, simply because as a writer of a subclass I couldn't
> rely on being able to use them.

As the author of a subclass, you control the choice between real
inheritance via inclusion in __bases__ or pseudo-inheritance via
register(). So I don't see your point here.

> The only alternative that I can see is to not use dynamic inheritance at
> all, and instead have the number classes inherit from these ABCs using
> the traditional mechanism. But that brings up all the problems of
> immaturity and requiring them to be built-in that I brought up earlier.

That's off the table already.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)