[Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules

[Nick Coghlan]

On 10 July 2016 at 11:15, Brett Cannon <brett at python.org> wrote:
> On Sat, 9 Jul 2016 at 06:52 Nick Coghlan <ncoghlan at gmail.com> wrote:
>> That issue was opened due to a few things that work with the C
>> implementation that fail with the Python implementation:
>>
>> - the C version can be pickled (and hence used with multiprocessing)
>> - the C version can be subclassed
>> - the C version can be used in "isinstance" checks
>> - the C version behaves as a static method, the Python version as a
>> normal instance method
>>
>> While I'm planning to accept the patch that converts the pure Python
>> version to a full class that matches the semantics of the C version in
>> these areas as well as in its core behaviour, that last case is one
>> where the pure Python version merely exhibits different behaviour from
>> the C version, rather than failing outright.
>>
>> Given that the issues that arose in this case weren't at all obvious
>> up front, what do folks think of the idea of updating PEP 399 to
>> explicitly prohibit class/function mismatches between accelerator
>> modules and their pure Python counterparts?
>
> I think flat-out prohibiting won't work in the Python -> C case as you can
> do things such as closures and such that I don't know if we provide the APIs
> to mimic through the C API. I'm fine saying we "strongly encourage mirroring
> the design between the pure Python and accelerated version for various
> reasons".

I think we should be more specific than that, as the main problem is
that the obvious way to emulate a closure in C is with a custom
callable, and there are some subtleties involved in doing that in a
way that doesn't create future cross-implementation compatibility
traps.

Specifically, if the Python implementation is a closure, then from an
external behaviour perspective, the key behaviours to mimic in a C
implementation would be:

- disable subclassing & isinstance checks against the public API (e.g.
by implementing it as a factory function rather than exposing the
custom type directly)
- either wrap the Python version in staticmethod, or add descriptor
protocol support to the C version
- don't add a custom representation in C without also adding it to the
Python version
- don't add pickling support in C without also adding it to the Python version

Similarly, if an existing C implementation uses a custom callable,
then a closure may not be a sufficiently compatible alternative, even
though it's clean to write and easy to read.

These issues don't tend to arise with normal functions, as the obvious
replacement for a module level function written in Python is a module
level function written in C, and those already tend to behave
similarly in all these respects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia