[Python-Dev] is this the fault of import_fresh_module or pickle?

[Nick Coghlan]

On Wed, Jan 9, 2013 at 2:58 AM, Eli Bendersky <eliben at gmail.com> wrote:
>> Handling this case is why having a context-manager form of
>> import_fresh_module was suggested earlier in this meta-thread.  At
>> least, I think that would solve it, I haven't tried it :)
>
>
> Would you mind extracting just this idea into this discussion so we can
> focus on it here? I personally don't see how making import_fresh_module a
> context manager will solve things, unless you add some extra functionality
> to it? AFAIU it doesn't remove modules from sys.modules *before* importing,
> at this point.

Sure it does, that's how it works: the module being imported, as well
as anything requested as a "fresh" module is removed from sys.modules,
anything requested as a "blocked" module is replaced with None (or
maybe 0 - whatever it is that will force ImportError). It then does
the requested import and then *reverts all those changes* to
sys.modules.

It's that last part which is giving you trouble: by the time you run
the actual tests, sys.modules has been reverted to its original state,
so pickle gets confused when it attempts to look things up by name.

Rather than a context manager form of import_fresh_module, what we
really want is a "modules_replaced" context manager:

    @contextmanager
    def modules_replaced(replacements):
        _missing = object()
        saved = {}
        try:
            for name, mod in replacements.items():
                saved[name] = sys.modules.get(name, _missing)
                sys.modules[name] = mod
            yield
        finally:
            for name, mod in saved.items():
                if mod is _missing:
                    del sys.modules[name]
                else:
                    sys.modules[name] = mod

And a new import_fresh_modules function that is like
import_fresh_module, but returns a 2-tuple of the requested module and
a mapping of all the affected modules, rather than just the module
object.

However, there will still be cases where this doesn't work (i.e.
modules with import-time side effects that don't support repeated
execution), and the pickle and copy global registries are a couple of
the places that are notorious for not coping with repeated imports of
a module. In that case, the test case will need to figure out what
global state is being modified and deal with that specifically.

(FWIW, this kind of problem is why import_fresh_module is in
test.support rather than importlib and the reload builtin became
imp.reload in Python 3 - "module level code is executed once per
process" is an assumption engrained in most Python developer's brains,
and these functions deliberately violate it).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia