[Python-Dev] Allowing to run certain regression tests in subprocesses

[Nick Coghlan]

On 4 August 2013 23:40, Eli Bendersky <eliben at gmail.com> wrote:
> On Sat, Aug 3, 2013 at 7:08 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Sure, it's just unusual to have a case where "importing is blocked by adding
>> None to sys.modules" differs from "not actually available", so I'd like to
>> understand the situation better.
>
> I must admit I'm confused by the behavior of import_fresh_module too.
>
> Snippet #1 raises the expected ImportError:
>
> sys.modules['pyexpat'] = None
> import _elementtree
>
> However, snippet #2 succeeds importing:
>
> ET = import_fresh_module('_elementtree', blocked=['pyexpat'])
> print(ET)

/me goes and looks

That function was much simpler when it was first created :P

Still, I'm fairly confident the complexity of that dance isn't
relevant to the problem you're seeing.

> I believe this happens because import_fresh_module does an import of
> the 'name' it's given before even looking at the blocked list. Then,
> it assigns None to sys.modules for the blocked names and re-imports
> the module. So in essence, this is somewhat equivalent to snippet #3:
>
> modname = '_elementtree'
> __import__(modname)
> del sys.modules[modname]
> for m in sys.modules:
>     if modname == m or m.startswith(modname + '.'):
>         del sys.modules[m]
> sys.modules['pyexpat'] = None
> ET = importlib.import_module(modname)
> print(ET)
>
> Which also succeeds.
>
> I fear I'm not familiar enough with the logic of importing to
> understand what's going on, but it has been my impression that this
> problem is occasionally encountered with import_fresh_module and C
> code that imports stuff (the import of pyexpat is done by C code in
> this case).

I had missed it was a C module doing the import. Looking into the
_elementtree.c source, the problem in this case is the fact that a
shared library that doesn't use PEP 3121 style per-module state is
only loaded and initialised once, so reimporting it gets the same
module back (from the extension loading cache), even if the Python
level reference has been removed from sys.modules. Non PEP 3121 C
extension modules thus don't work properly with
test.support.import_fresh_module (as there's an extra level of caching
involved that *can't* be cleared from Python, because it would break
things).

To fix this, _elementree would need to move the pyexpat C API pointer
to per-module state, rather than using a static variable (see
http://docs.python.org/3/c-api/module.html#initializing-c-modules).
With per-module state defined, the import machine should rerun the
init function when the fresh import happens, thus creating a new copy
of the module. However, this isn't an entirely trivial change for
_elementree, since:

1. Getting from the XMLParser instance back to the module where it was
defined in order to retrieve the capsule pointer via
PyModule_GetState() isn't entirely trivial in C. You'd likely do it
once in the init method, store the result in an XMLParser attribute,
and then tweak the EXPAT() using functions to include an appropriate
local variable definition at the start of the method implementation.

2. expat_set_error would need to be updated to accept the pyexpat
capsule pointer as a function parameter

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia