[Python-Dev] Comments2: PEP 252, PEP 253 and jython issues

[Guido van Rossum]

[SP]
> > > The problem is that with the new mro possibly independently C
> > > defined things get interleaved with Python level code, so I
> > > imagine that the effect should be a bit disciplined ...
> > 
> [GvR]
> > 
> > I don't expect this to happen regularly.  if you count on this I
> > think you're on really thin ice.

[SP]
> No I'm not counting on that, I'm worried about this scenario
> 
> A subclasses object and redefines __getattr__
> B subclasses list
> 
> then class C(B,A): ...
> 
> will have the following mro:
> 
> C B list A object
> 
> In Jython codebase hypothetically but possibly list could contains
> calls like:
> 
>   __getattr__(...)
>   and super.__getattr__

This is Java code, right?

> now A is in between list and object: what is the right thing here?

Just say that this is what happens.  The MRO is inconsistent in this
case.

> I don't expect or invite users to do that, but your rules allow A to
> end up in that position in the mro, your philosophy seems to be
> this is thin ice and so let it be ... 

Exactly.

> > > I saw that list_ass_slice calls list_slice directly, is that OK?
> > 
> > It's certainly safe -- any subclass of list will share the same
> > structure lay-out (this is enforced by the subclassing machinery) so
> > the list_slice call will find the object properly inititialized.
> > 
> > There are lots of places where the C code doesn't bother to look
> > for overridden methods -- the PyDict_* API is a common example.  I
> > have no intention of fixing this.

> Fine with that, but ...
> 
> > > Other OO languages like Smalltalk, Self, or Java allow only to
> > > define primitives or single methods to add lowel-level
> > > functionality, they don't try to allow subtyping at that level,
> > > the previous scenario in Python where you could define just
> > > opaque types, and in particular your C code never/normally got
> > > interleaved with Python
> > >  behaviour was on that line.
> > 
> > Well, if you like that better, maybe I should remove the
> > instructions for subclassing at the C level? :-)

> No, I'm just saying that you are moving from a world were you had
> documentable C black-boxes and clear Python semantics, to a world
> where (but you say that would be rare) you can have Python code
> whose behaviour depends on very fine-grained decision inside C code.

I would recommend that C programmers not indulge in this.  They
should continue to write C types that can be treated as black boxes.
The C-level single inheritance is best seen as code sharing.

> On one side your rules make it very easy to interleave Python and C
> code in the mro, on the other hand I agree that will be rarely done
> by the wary user because it's just surprise-land, don't know about
> the unwary ...

On the third hand, if a C programmer *wants* to write a class with the
proper cooperative super calling, they can do so, it's just a bit
clumsy.  Should I provide a C API to make this easier?  I doubt that
it will be used much.  If there's demand, it can be added later.

> > My intention is to make the maximum number of different paradigms
> > usable in Python.  You can do cooperative multiple inheritance,
> > but only of all your classes are designed with cooperation in
> > mind.  If some class uses C-level (or Java-level) inheritance and
> > is not written cooperatively, you have to be a little careful
> > using it as a base class in a cooperatively written class.  I
> > don't want to enforce cooperative coding at the Python level
> > either: the "BaseClass.method(self, args)" approach is still
> > valid, but you have to beware of the consequences.

> Do you feel is that a good thing, especially if new-style classes become
> the default?

Time will tell.  I expect that some people will design vast
cooperative class hierarchies.  I expect that other people will stick
to single inheritance or very simple-minded uses of multiple
inheritance (for example, they could use the mix-in pattern that was
about the only sensible use of multiple inheritance with classic
classes).

Anyway, I expect new classes won't become the default until Python
3.0.

> It is not a rethoric question, Python is becoming a little single
> dispatch CLOS with a MOP, descriptors etc but CLOS has only
> next-method and strange beasts like method combinators and don't
> allow to interleave low-level coded behaviour in the class mro.

Nothing new: classic classes already allow you to cheat or mess up in
various ways by not calling your superclass method.

> > > At worst you could say that you allow subtyping but not code
> > > sharing, but then even a C method for an object should be very
> > > careful using another overridable method of the very same
> > > object.
> > 
> > I definitely have to support code sharing.  I think subclassing at
> > the C level is useful to create variations of built-in objects
> > with additional properties, e.g. numbers with additional
> > operations or formatting parameters, dictionaries with certain
> > restrictions on the keys, etc.  I don't think such subclasses
> > should be used in complex multiple inheritance lattices.

> Better to enforce that?

Why?  I can think of situations where you have a MRO like this:

  C list B object

but where B doesn't override anything that list might use.  In fact,
this is the typical mix-in situation.  I don't want to forbid this.

> > > OK you could say: that's C level, everybody should care for
> > > herself, but with Java we cannot do the same.
> > 
> > Why not?

> Because users would like writing Jython types to become easier,

Do you mean writing Jython types in Java, or in Python?

> and be possible using Java OOP model, without surprises, or
> undocumented subtlities, any deviation is an hard sell because Java
> has already an OOP model.

Recommend strongly that they stick to single inheritance for types
implemented in Java -- that should do the trick.

> > > > > It would make sense to use Java subclassing to implement
> > > > > type subclassing, at least at layout level this does not
> > > > > clash with the multiple inheritance rule in best_base.
> > > > 
> > > > So you would rule out multiple inheritance of types?  Fine
> > > > with me, but you may regret this a few yearsfrom now (when we
> > > > all code multiple inheritance all the time :-).
> > 
> > > No it is ruled out at Java level, not Jython level.
> > > But it is not also ruled out in CPython at C level?
> > 
> > Mostly because I don't provide a way to create multiple base
> > classes.  That would not be hard to add though -- if there's
> > demand.

> But this is a path that Jython could not follow.

Correct -- another reason why it's not a good idea to encourage this.
What you could do though, even in Jython, is simulate everything that
goes on in a class statement: build your own dict, populate it with
function objects, and then call type.__new__(name, bases, dict).
Voila, instant class. :-)

> > > see above.
> > > 
> > > >  I propose that you just try to live with this.
> > 
> > > Related e.g. to Zope porting projects there have been a lot of
> > > pressure on jython-dev regarding how to code new types and
> > > metaclasses, A Jython type writer expect to be able to use Java
> > > naturally (i.e. use inheritance), for that, so the situation
> > > isn't that easy, is not even easy with the current codebase,
> > > because given the new mro one can even put some Python behaviour
> > > between:
> > > 
> > > PyList PythonLevelCode and PyObject :-( (*)
> > 
> > Just sa no. :-)

> See above.

Sorry, my context stack just overflowed.  If this is still an issue,
please start over. :-)

[snip]

> > > I had the same idea because this would make my life easier.
> > 
> > Sounds good to me.
> > 
> > > A problem: some of the Zope porters reported that there are
> > > classes that inherits from more than a single ExtensionClass in
> > > Zope :-( don't know if that is true. (**)
> > 
> > Me neither.  But the problem is probably shallow -- Zope mostly
> > uses mixins of various sorts, so these classes probably don't
> > override stuff -- they just define new methods.  (I haven't looked
> > at what it would take to replace ExtensionClass with the new
> > metaclasses, but that's definitely on the agenda.)
> > 
> > > > don't know if we should enforce this or just warn about it in
> > > > the documentation.
> > 
> > > It would make our (jython-dev) life easier and avoid some
> > > impredictable problem even with CPython and code sharing, but
> > > see previous note (**)
> > 
> > So enforce it and see how it turns out in practice.

> It's probably the best solution, at least for Jython,
> and could even make sense for CPython (see above points)

Good.  We seem to agree.

> > > > I don't want to force you to call a cooperative super method
> > > > at the Java level -- it would be very slow, I suspect...
> > 
> > > Yes ... very slow and we would have to use that also at the very
> > > core of the hierarchy, see (*) but the problem is more
> > > complicated, in Java given three classes
> > > 
> > > C extends B extends A
> > > and a method m
> > > C.m overrides B.m overrides A.m
> > > 
> > > there is not direct way (even using reflection) 
> > > to apply B.m behaviour to a C object, unless
> > > the programmer has left some hook in C using super
> > > or in B ( a method B_m e.g.).
> > 
> > I guess C.m has to call super.m to invoke B.m, right?

> Yup.

> > Does this mean that the following can't be made to work?
> > 
> > class C(list):
> >     def add_spam(self):
> >         return list.append(self, "spam")
> > 
> > It's legal Python (although in general questionable style of course).

> Yes, it can be made to work (it already works for Java subclassing
> and type subclassing would be some flavor of that), but the point is
> that your Java stuff should be at the top of the mro. Because then
> you can insert some hooks using super.
> 
> regards.

I hope this discussion so far has been helpful -- I have to pack for
my trip and will soon be off to San Francisco until next Wednesday.

--Guido van Rossum (home page: http://www.python.org/~guido/)