On Sat, Dec 29, 2012 at 5:34 PM, Nick Coghlan
On Sun, Dec 30, 2012 at 10:46 AM, Benjamin Peterson
wrote: 2012/12/29 Eli Bendersky
: Hi,
This came up while investigating some test-order-dependency failures in issue 16076.
test___all__ goes over modules that have `__all__` in them and does 'from <module> import *' on them. This leaves a lot of modules in sys.modules, which may interfere with some tests that do fancy things with sys,modules. In particular, the ElementTree tests have trouble with it because they carefully set up the imports to get the C or the Python version of etree (see issues 15083 and 15075).
Would it make sense to save the sys.modules state and restore it in test___all__ so that sys.modules isn't affected by this test?
Sounds reasonable to me.
I've tried this as an inherent property of regrtest before (to resolve some problems with test_pydoc), and it didn't work - we have too many modules with non-local side effects (IIRC, mostly related to the copy and pickle registries).
Given that it checks the whole standard library, test___all__ is likely to run into the same problem.
Yes, I'm running into all kinds of weird problems when saving/restoring sys.modules around test___all__. This is not the first time I get to fight this test-run-dependency problem and it's very frustrating. This may be a naive question, but why don't we run the tests in separate interpreters? For example with -j <N> we do (which creates all kinds of strange intermittent problems depending on which tests got bundled into the same process). Is this a matter of performance? Because that would help get rid of these dependencies between tests, which would probably save core devs some work and headache. After all, since a lot of the interpreter state is global (for example sys.modules), does it not make sense to run each test in a clean environment? Many tests do fancy things with the global environment which makes them difficult to keep clean and separate.
Hence test.support.import_fresh_module - it can ensure you get the module you want, regardless of the preexisting contents of sys.modules. ( http://docs.python.org/dev/library/test#test.support.import_fresh_module)
Yes, this is the solution currently used in test_xml_etree. However, once pickling tests are added things stop working. Pickle uses __import__ to import the module a class belongs to, bypassing all such trickery. So if test___all__ got _elementtree into sys.modules, pickle's __import__ finds it even if all the tests in test_xml_etree manage to ignore it for the Python version because they use import_fresh_module. Eli