is this the fault of import_fresh_module or pickle?
Hello,
I'm still having some struggles with the interaction between pickle and
import overriding with import_fresh_module.
_elementtree.TreeBuilder can't be pickled at this point. When I do this:
from test.support import import_fresh_module
import pickle
P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
tb = P.TreeBuilder()
print(pickle.dumps(tb))
Everything works fine. However, if I add import_fresh_module for the C
module:
from test.support import import_fresh_module
import pickle
C = import_fresh_module('xml.etree.ElementTree', fresh=['_elementtree'])
P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
tb = P.TreeBuilder()
print(pickle.dumps(tb))
I get an error from pickle.dumps:
Traceback (most recent call last):
File "mix_c_py_etree.py", line 10, in <module>
print(pickle.dumps(tb))
_pickle.PicklingError: Can't pickle
Eli Bendersky
Everything works fine. However, if I add import_fresh_module for the C module:
from test.support import import_fresh_module import pickle C = import_fresh_module('xml.etree.ElementTree', fresh=['_elementtree']) P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
sys.modules still contains the C version at this point, so: sys.modules['xml.etree.ElementTree'] = P
tb = P.TreeBuilder() print(pickle.dumps(tb))
This interaction only seems to happen with pickle. What's going on here? Can we somehow improve import_fresh_module to avoid this? Perhaps actually deleting previously imported modules with some special keyword flag?
pickle always looks up sys.modules['xml.etree.ElementTree']. Perhaps we could improve something, but this requirement is rather special; personally I'm okay with switching sys.modules explicitly in the tests, because that reminds me of what pickle does. Stefan Krah
On Tue, Jan 8, 2013 at 8:05 AM, Stefan Krah
Eli Bendersky
wrote: Everything works fine. However, if I add import_fresh_module for the C module:
from test.support import import_fresh_module import pickle C = import_fresh_module('xml.etree.ElementTree', fresh=['_elementtree']) P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
sys.modules still contains the C version at this point, so:
sys.modules['xml.etree.ElementTree'] = P
tb = P.TreeBuilder() print(pickle.dumps(tb))
This interaction only seems to happen with pickle. What's going on here? Can we somehow improve import_fresh_module to avoid this? Perhaps actually deleting previously imported modules with some special keyword flag?
pickle always looks up sys.modules['xml.etree.ElementTree']. Perhaps we could improve something, but this requirement is rather special; personally I'm okay with switching sys.modules explicitly in the tests, because that reminds me of what pickle does.
Wouldn’t it be be better if import_fresh_module or some alternative function could do that for you? I mean, wipe out the import cache for certain modules I don't want to be found? Eli
Eli Bendersky
On Tue, Jan 8, 2013 at 8:05 AM, Stefan Krah
wrote: pickle always looks up sys.modules['xml.etree.ElementTree']. Perhaps we could improve something, but this requirement is rather special; personally I'm okay with switching sys.modules explicitly in the tests, because that reminds me of what pickle does.
Wouldn?t it be be better if import_fresh_module or some alternative function could do that for you? I mean, wipe out the import cache for certain modules I don't want to be found?
For a single test, perhaps. ContextAPItests.test_pickle() from test_decimal would look quite strange if import_fresh_module was used repeatedly. Stefan Krah
On Tue, 08 Jan 2013 17:05:50 +0100, Stefan Krah
Eli Bendersky
wrote: Everything works fine. However, if I add import_fresh_module for the C module:
from test.support import import_fresh_module import pickle C = import_fresh_module('xml.etree.ElementTree', fresh=['_elementtree']) P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
sys.modules still contains the C version at this point, so:
sys.modules['xml.etree.ElementTree'] = P
tb = P.TreeBuilder() print(pickle.dumps(tb))
This interaction only seems to happen with pickle. What's going on here? Can we somehow improve import_fresh_module to avoid this? Perhaps actually deleting previously imported modules with some special keyword flag?
pickle always looks up sys.modules['xml.etree.ElementTree']. Perhaps we could improve something, but this requirement is rather special; personally I'm okay with switching sys.modules explicitly in the tests, because that reminds me of what pickle does.
Handling this case is why having a context-manager form of import_fresh_module was suggested earlier in this meta-thread. At least, I think that would solve it, I haven't tried it :) --David
On Tue, Jan 8, 2013 at 8:37 AM, R. David Murray
Eli Bendersky
wrote: Everything works fine. However, if I add import_fresh_module for the C module:
from test.support import import_fresh_module import pickle C = import_fresh_module('xml.etree.ElementTree', fresh=['_elementtree']) P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
sys.modules still contains the C version at this point, so:
sys.modules['xml.etree.ElementTree'] = P
tb = P.TreeBuilder() print(pickle.dumps(tb))
This interaction only seems to happen with pickle. What's going on here? Can we somehow improve import_fresh_module to avoid this? Perhaps actually deleting previously imported modules with some special keyword flag?
pickle always looks up sys.modules['xml.etree.ElementTree']. Perhaps we could improve something, but this requirement is rather special;
On Tue, 08 Jan 2013 17:05:50 +0100, Stefan Krah
wrote: personally I'm okay with switching sys.modules explicitly in the tests, because that reminds me of what pickle does.
Handling this case is why having a context-manager form of import_fresh_module was suggested earlier in this meta-thread. At least, I think that would solve it, I haven't tried it :)
Would you mind extracting just this idea into this discussion so we can focus on it here? I personally don't see how making import_fresh_module a context manager will solve things, unless you add some extra functionality to it? AFAIU it doesn't remove modules from sys.modules *before* importing, at this point. Eli
On Wed, Jan 9, 2013 at 2:58 AM, Eli Bendersky
Handling this case is why having a context-manager form of import_fresh_module was suggested earlier in this meta-thread. At least, I think that would solve it, I haven't tried it :)
Would you mind extracting just this idea into this discussion so we can focus on it here? I personally don't see how making import_fresh_module a context manager will solve things, unless you add some extra functionality to it? AFAIU it doesn't remove modules from sys.modules *before* importing, at this point.
Sure it does, that's how it works: the module being imported, as well as anything requested as a "fresh" module is removed from sys.modules, anything requested as a "blocked" module is replaced with None (or maybe 0 - whatever it is that will force ImportError). It then does the requested import and then *reverts all those changes* to sys.modules. It's that last part which is giving you trouble: by the time you run the actual tests, sys.modules has been reverted to its original state, so pickle gets confused when it attempts to look things up by name. Rather than a context manager form of import_fresh_module, what we really want is a "modules_replaced" context manager: @contextmanager def modules_replaced(replacements): _missing = object() saved = {} try: for name, mod in replacements.items(): saved[name] = sys.modules.get(name, _missing) sys.modules[name] = mod yield finally: for name, mod in saved.items(): if mod is _missing: del sys.modules[name] else: sys.modules[name] = mod And a new import_fresh_modules function that is like import_fresh_module, but returns a 2-tuple of the requested module and a mapping of all the affected modules, rather than just the module object. However, there will still be cases where this doesn't work (i.e. modules with import-time side effects that don't support repeated execution), and the pickle and copy global registries are a couple of the places that are notorious for not coping with repeated imports of a module. In that case, the test case will need to figure out what global state is being modified and deal with that specifically. (FWIW, this kind of problem is why import_fresh_module is in test.support rather than importlib and the reload builtin became imp.reload in Python 3 - "module level code is executed once per process" is an assumption engrained in most Python developer's brains, and these functions deliberately violate it). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Eli Bendersky
-
Nick Coghlan
-
R. David Murray
-
Stefan Krah