[Python-Dev] PEP 573 -- Module State Access from C Extension Methods

[Petr Viktorin]

On 04/26/18 09:32, Thomas Wouters wrote:
> 
> Thanks for working on this, Marcel (and Petr). This looks like an 
> ambitious intern project :) Couple of questions and comments in-line.

Indeed, but also a multi-year one. Sometimes it seems like Python is 
moving too fast under us!

Specifically, PEP 573 (Unifying function/method classes) andto a smaller 
extent PEP 567 (Context Variables) seem to be solving similar problems 
as this PEP. I'm inclined to put this PEP temporarily on hold, help 575 
get accepted, and then rebase on top of that.


> On Mon, Apr 23, 2018 at 12:36 PM, Marcel Plch <gmarcel.plch at gmail.com 
> <mailto:gmarcel.plch at gmail.com>> wrote:
>     PEP: 573
[...]
> 
>     Abstract
>     ========
> 
[...]
> 
>     Additionaly, support for easier creation of immutable exception
>     classes is added.
> 
> 
> I'm not a fan of using 'immutable' here, or in the API function name. I 
> understand the types are to some extent immutable (apart from their 
> refcount, I assume), but I think it's going to be too easy to confuse it 
> with types whose *instances* are immutable. (We do occasionally say 
> things like "tuples are an immutable type".) Since the point is that 
> they behave like statically defined ones, perhaps 'Static' would be a 
> reasonable replacement.

That was the first naming choice, but then on the other hand, "static 
heap type" sounds like an oxymoron -- in C, static things don't live on 
the heap.
Naming is hard :(


[...]
>     Slot methods
>     ------------
> 
>     The above changes don't cover slot methods, such as ``tp_iter`` or
>     ``nb_add``.
> 
>     The problem with slot methods is that their C API is fixed, so we can't
>     simply add a new argument to pass in the defining class.
>     Two possible solutions have been proposed to this problem:
> 
>          * Look up the class through walking the MRO.
>            This is potentially expensive, but will be useful if
>     performance is not
>            a problem (such as when raising a module-level exception).
>          * Storing a pointer to the defining class of each slot in a
>     separate table,
>            ``__typeslots__`` [#typeslots-mail]_.  This is technically
>     feasible and fast,
>            but quite invasive.
> 
>     Due to the invasiveness of the latter approach, this PEP proposes
>     adding an MRO walking
>     helper for use in slot method implementations, deferring the more
>     complex alternative
>     as a potential future optimisation. Modules affected by this concern
>     also have the
>     option of using thread-local state or PEP 567 context variables, or
>     else defining their
>     own reload-friendly lookup caching scheme.
> 
> 
> I do not believe walking the MRO is going to work without reworking the 
> implementation of types, specifically how typeobject.c deals with slots 
> of subclasses: in some cases copies the slots from the base class (see 
> inherit_slots() and from where it's called). I believe this would cause 
> problems if, for example, you define type X in module A, subclass it 
> from type Y in module B without overriding the slot, and try to find the 
> module object for A from the slot implementation. I don't think copying 
> slots is a requirement for the desired semantics, but it's going to be 
> fairly involved to rewrite it to do something else. There's also 
> backward-compatibility to consider: third-party libraries can be 
> inheriting from builtin types (e.g. numpy does this extensively) using 
> the same copying-slot mechanism, which means those builtin types can't 
> use the MRO walking to find their module without breaking compatibility 
> with those third-party libraries.

The MRO walking code needs to deal with copied slots. It's not 
straigtforward, but I do think it's possible.


[...]
>     Static exceptions
>     -----------------
> 
>     A new function will be added::
> 
>          int PyErr_PrepareImmutableException(PyTypeObject **exc,
>                                           const char *name,
>                                           const char *doc,
>                                           PyObject *base)
> 
>     Creates an immutable exception type which can be shared
>     across multiple module objects.
> 
> 
> How is this going to deal with type.__subclasses__()? Is re-using the 
> static type object between reloads and sub-interpreters important enough 
> to warrant the different behaviour? What if sub-interpreters end up 
> wanting to disallow sharing objects between them?

Argh. Yes, subclasses seem to be the obvious shortcoming that a fresh 
pair of eyes is bound to find. Thanks!
This sends the proposal back to the drawing board.



[...]
>     Modules Converted in the Initial Implementation
>     -----------------------------------------------
> 
>     To validate the approach, several modules will be modified during
>     the initial implementation:
> 
>     The ``zipimport``, ``_io``, ``_elementtree``, and ``_csv`` modules
>     will be ported to PEP 489 multiphase initialization.
> 
> 
> zipimport currently caches things in C globals. Changing it to use PEP 
> 489 multi-phase initialisation is very likely going to change semantics 
> in subtle ways... Is it really worth the risk?

The whole excercise is to show that existing modules *can* be ported to 
the new approach, without unacceptable changes to semantics -- and in 
cases where they can't, to improve the situation. So even if changes to 
zipimport ultimately aren't accepted, it makes sense to attempt this.