[Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules

[Nick Coghlan]

On 11 July 2016 at 23:11, Steven D'Aprano <steve at pearwood.info> wrote:
> If a core developer wishes to extend the API of partial objects to
> include such things as subclassing, isinstance tests, pickling etc, then
> it is reasonable to insist that both the C and the Python implementation
> both support the same API and both use a class. But at the moment I
> don't think any of those things are promised or supported[1], except by
> accident, so removing the discrepency between them is not a bug fix, but
> adding new features.

functools.partial was originally C only [1], then pickling support was
added [2], then a custom __repr__ was added [3], and only then was a
Python fallback added [4].

When [4] was implemented as a closure rather than as a custom
callable, the test cases for the pure Python version needed to be
customised to skip the custom __repr__ and pickling support tests
added by [2] and [3].

Emanuel's patch eliminates the special casing of the Python version in
the test suite by replacing the closure with a custom callable that
also implements the custom __repr__ and pickling support, and also
adds the test case needed to ensure subclasses of the Python version
work as expected (previously that test case was simply omitted, since
subclassing the Python version wasn't supported).

[1] https://www.python.org/dev/peps/pep-0309/
[2] https://hg.python.org/cpython/rev/184ca6293218
[3] https://hg.python.org/cpython/rev/2baad8bd0b4f
[4] https://hg.python.org/cpython/rev/fcfaca024160

> The more I think about it, the more I feel that it is *unreasonable* to
> mandate that devs must ensure that alternate implementations mirror each
> others *unsupported and accidental features*. Mirror the supported API?
> Absolutely! But not accidental features.

The pickling and custom __repr__ on functools.partial weren't
accidental features - they were features deliberately added to the C
version before the Python version was implemented that were
nevertheless originally deemed not part of the standard library
definition.

The fact the incomplete implementation was simpler and easier to read
was then used as an argument against providing a more complete
fallback implementation of the original API.

> (Aside: why can't closures be pickled?)

Functions and classes are deserialised based on their name, but just
the name isn't sufficient to deserialise a closure - you also need the
information about the cell references and their contents. However, the
internal mechanics of the way closures work in practice aren't
standardised at the language level, which means there isn't an
implementation independent way to represent them as a pickle (that's
not to say such a scheme couldn't be created - it just hasn't been
done to date).

> [1] If I'm wrong about this, and these features are supported, then
> Emanuel has found a hole in the functools test suite and a weakness in
> our testing: it's too hard to ensure that *both* the Python and C code
> is tested.

There's no hole, as the test suite was structured to special case the
Python fallback and only test the affected features for the C version.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia