[Python-Dev] Return type of alternative constructors

[Guido van Rossum]

On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 9 May 2016 at 08:50, Guido van Rossum <guido at python.org> wrote:
> > On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> P.S. The potential complexity of that is one of the reasons the design
> >> philosophy of "prefer composition to inheritance" has emerged -
> >> subclassing is a powerful tool, but it does mean you often end up
> >> needing to care about more interactions between the subclass and the
> >> base class than you really wanted to.
> >
> > Indeed!
> >
> > We could also consider this a general weakness of the "alternative
> > constructors are class methods" pattern. If instead these alternative
> > constructors were folded into the main constructor (e.g. via special
> keyword
> > args) it would be altogether clearer what a subclass should do.
>
> Unfortunately, even that approach gets tricky when the inheritance
> relationship crosses the boundary between components with independent
> release cycles.
>
> In my experience, this timeline is the main one that causes the pain:
>
> * Base class is released in Component A (e.g. CPython)
> * Subclass is released in Component B (e.g. PyPI module)
> * Component A releases a new base class construction feature
>
> Question: does the new construction feature work with the existing
> subclass in Component B if you combine it with the new version of
> Component A?
>
> When alternate constructors can be implemented as class methods that
> work by creating a default instance and using existing public API
> methods to mutate it, then the answer to that question is "yes", since
> the default constructor hasn't changed, and the new convenience
> constructor isn't relying on any other new features.
>
> The answer is also "yes" for existing subclasses that only add new
> behaviour without adding any new state, and hence just use the base
> class __new__ and __init__ without overriding either of them.
>
> It's when the existing subclasses overrides __new__ or __init__ and
> one or both of the following is true that things can get tricky:
>
> - you're working with an immutable type
> - the API implementing the post-creation mutation is a new one
>
> In both of those cases, the new construction feature of the base class
> probably won't work right without updates to the affected subclass to
> support the new capability (whether that's supporting a new parameter
> in __new__ and __init__, or adding their own implementation of the new
> alternate constructor).
>

OTOH it's not the end of the world -- until B is updated, you can't use the
new construction feature with subclass B, you have to use the old way of
constructing instances of B. Presumably that old way is still supported,
otherwise the change to A has just broken all of B, regardless of the new
construction feature. Which is possible, but it's a choice that A's author
has to make after careful deliberation.

Or maybe the "class construction" machinery in A is so prominent that it
really is part of the interface between A and any of its subclasses, and
then that API had better be documented. Just saying "you can subclass this"
won't be sufficient.


> I'm genuinely unsure that's a solvable problem in the general case -
> it seems to be an inherent consequence of the coupling between
> subclasses and base classes during instance construction, akin to the
> challenges with subclass compatibility of the unpickling APIs when a
> base class adds new state.
>

Yup. My summary of it is that versioning sucks.


> However, from a pragmatic perspective, the following approach seems to
> work reasonably well:
>
> * assume subclasses don't change the signature of __new__ or __init__
>

I still find that a distasteful choice, because *in general* there is no
requirement like that and there are good reasons why subclasses might have
a different __init__/__new__ signature. (For example dict and defaultdict.)


> * note the assumptions about the default constructor signature in the
> alternate constructor docs to let implementors of subclasses that
> change the signature know they'll need to explicitly test
> compatibility and perhaps provide their own implementation of the
> alternate constructor
>

Yup, you end up having to design the API for subclasses carefully and then
document it precisely.

This is what people too often forget when they complain e.g. "why can't I
subclass EventLoop more easily" -- we don't want to have a public API for
that, so we discourage it, but people mistakenly believe that anything
that's a class should be subclassable.


> You *do* still end up with some cases where a subclass needs to be
> upgraded before a new base class feature works properly for that
> particular subclass, but subclasses that *don't* change the
> constructor signature "just work".
>

The key is that there's an API requirement and that you have to design and
document that API with future evolution in mind. If you don't do and let
people write subclasses that just happen to work, you're in a lot of pain.
The interface between a base class and a subclass just is very complex so
must designers of base classes get this wrong initially.


> Cheers,
> Nick.
>
> P.S. It occurs to me that a sufficiently sophisticated typechecker
> might be able to look at all of the calls to "cls(*args, **kwds)" in
> class methods and "type(self)(*args, **kwds)" in instance methods, and
> use those to define a set of type constraints for the expected
> constructor signatures in subclassses, even if the current code base
> never actually invokes those code paths.
>

Could you restate that as a concrete code example? (Examples of the
problems with "construction features" would also be helpful, probably --
abstract descriptions of problems often lead me astray.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160509/cea598ca/attachment.html>