[Import-SIG] On singleton modules, heap types, and subinterpreters

[Petr Viktorin]

On Wed, Jul 29, 2015 at 8:57 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Jul 29, 2015 12:05 PM, "Petr Viktorin" <encukou at gmail.com> wrote:
>>
>> On Wed, Jul 29, 2015 at 7:53 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> > An alternative, could the module intra-dependencies be bound where needed?
>> > For example, with _csv could Error be added to reader.__dict__ (i.e. bound
>> > to reader)?
>>
>> Putting a reference to the module on *classes* is not a problem.
>> Getting a reference to the module from normal methods should also not
>> be that hard.
>>
>> The hard part is special methods, like tp_iter, which only get "self"
>> as an argument, at the C level. There's no information about which
>> (super)class the method is defined in.
>
> The slot methods would have to do `type(self).<"global">`, which is
> what any other method would do.  So in the relevant _csv.reader
> methods we would use the equivalent "type(self).Error" where needed.

We need the class that defines the method. type(self) might return a
subclass of that. So we either need to walk the MRO until the defining
class is found, or use Nick's mechanism and record the defining class
for each relevant special method.

> Let me make sure I understand the problem before I say anything more.
> :)  In Python each function has __globals__ and __closure__ that
> provide externally scoped objects for use within the function.  In C
> extension functions neither exists.  Is that right?  (Or is that sort
> of what PyState_FindModule is supposed to facilitate?)

For extension functions, you can choose from a variety of calling
mechanisms that evolved over the years: positional args only vs.
keyword args; and functions/normal methods/class methods depending on
the "special" argument (instance/class) they get.
Adding an extra class argument wouldn't be too much of a problem, and
neither would populating it.

The problem is with special methods, like tp_iter whose signature is:
    PyObject *iter(PyObject *self)
with no way to pass in the defining class or module. As I said
above, type(self) will not always give the class where tp_iter was
defined – and we need the module of the defining class.
That is the missing link. Getting a module reference to everything
except the special methods is relatively straightforward.

[...]
> However, it sounds like you're suggesting that PyState_FindModule
> should be fixed, replaced, or supplemented, assuming I've understood
> correctly that it's part of the problem here.  I'd say it depends on
> the actual impact of the lack of implicit Python-level scoping/lookup
> in C extension functions/methods.

PyState_FindModule bypasses the need for passing a module reference
around: you give it a PyModuleDef (static data from which a module is
constructed), and from that it looks up the corresponding module in
current subinterpreter state.
This  assumes the same set of modules is loaded in every
subinterpreter. (Or at least that the required module is loaded in the
current subinterpreter.)
If we want to support subinterpreters safely, PyState_FindModule must go.