[Python-Dev] Strange artifacts with PEP 3121 and monkey-patching sys.modules (in csv, ElementTree and others)

[Eli Bendersky]

n Sat, Aug 10, 2013 at 5:47 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> In a similar vein, Antoine recently noted that the fact the per-module
> state isn't a real PyObject creates a variety of interesting lifecycle
> management challenges.
>
> I'm not seeing an easy solution, either, except to automatically skip
> reinitialization when the module has already been imported.
>
This solution has problems. For example, in the case of ET it would
preclude testing what happens when pyexpat is disabled (remember we were
discussing this...). This is because there would be no real way to create
new instances of such modules (they would all cache themselves in the init
function - similarly to what ET now does in trunk, because otherwise some
of its global-dependent crazy tests fail).

A more radical solution would be to *really* have multiple instances of
state per sub-interpreter. Well, they already exist -- it's
PyState_FindModule which is the problematic one because it only remembers
the last one. But I see that it's only being used by extension modules
themselves, to efficiently find modules they belong to. It feels a bit like
a hack that was made to avoid rewriting lots of code, because in general a
module's objects *can* know which module instance they came from. E.g. it
can be saved as a private field in classes exported by the module.

So a more radical approach would be:

PyState_FindModule can be deprecated, but still exist and be documented to
return the state the *last* module created in this sub-interpreter. stdlib
extension modules that actually use this mechanism can be rewritten to just
remember the module for real, and not rely on PyState_FindModule to fetch
it from a global cache. I don't think this would be hard, and it would make
the good intention of PEP 3121 more real - actual intependent state per
module instance.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130810/5f3333a7/attachment.html>