[Cython] redesigning extension modules, avoiding static global C state

[Stefan Behnel]

Hi,

I recently started a discussion on python-dev to move this topic forward.

http://thread.gmane.org/gmane.comp.python.devel/141262

Basically, the idea is to allow extension modules to implement their own
module type, with all benefits like module level properties, proper garbage
collection, sub-interpreter support, module reloading, etc.

One outcome of that discussion (for me) was that a) there is an interest
over there in doing that and b) it makes sense to start implementing this
in Cython as far as we can already, meaning that we'd move all module
globals into an extension type, turn global functions into methods of that
type, move extension types to the heap and then hide that module type
behind the existing module dict by copying over the references to its types
and bound methods at module init time. PEP 3119 provides an infrastructure
that we can use for now to get at least some of the benefits.

Besides being some work to implement, I suspect that this also has a
visible user impact, sadly. There might be a performance regression when we
move from static globals to pointer indirections, often even multiple
indirections. What would be more annoying is if we had to loosen some
assumptions about globals that we currently make, e.g. regarding nogil
safety or temping, as that could prevent users from doing stuff they
currently do in their code. I don't remember if there were any major
changes here for closures (which are similar), so not sure yet what the
exact impact will be for globals.

I think we should try to keep this out of the compiler itself as much as
possible by using C macros for accessing globals. We'd still need to
calculate the access path to the module state instance for each scope, but
if we can move the rest to C compilation time, we may even be able to keep
up a static-globals mode that modules could use if they value access time
higher than more dynamic module support (e.g. because they cannot be
reloaded or reinstantiated anyways due to external C dependencies not
playing nicely).

I'm not sure yet how much time I can invest into this during the next
months, so if anyone else wants to join in for this, that'd be helpful.

Stefan