[Python-Dev] test___all__ polluting sys.modules?

[Eli Bendersky]

On Sat, Dec 29, 2012 at 5:34 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sun, Dec 30, 2012 at 10:46 AM, Benjamin Peterson <benjamin at python.org>
> wrote:
> > 2012/12/29 Eli Bendersky <eliben at gmail.com>:
> >> Hi,
> >>
> >> This came up while investigating some test-order-dependency failures in
> >> issue 16076.
> >>
> >> test___all__ goes over modules that have `__all__` in them and does
> 'from
> >> <module> import *' on them. This leaves a lot of modules in sys.modules,
> >> which may interfere with some tests that do fancy things with
> sys,modules.
> >> In particular, the ElementTree tests have trouble with it because they
> >> carefully set up the imports to get the C or the Python version of etree
> >> (see issues 15083 and 15075).
> >>
> >> Would it make sense to save the sys.modules state and restore it in
> >> test___all__ so that sys.modules isn't affected by this test?
> >
> > Sounds reasonable to me.
>
> I've tried this as an inherent property of regrtest before (to resolve
> some problems with test_pydoc), and it didn't work - we have too many
> modules with non-local side effects (IIRC, mostly related to the copy
> and pickle registries).
>
> Given that it checks the whole standard library, test___all__ is
> likely to run into the same problem.
>
>
Yes, I'm running into all kinds of weird problems when saving/restoring
sys.modules around test___all__. This is not the first time I get to fight
this test-run-dependency problem and it's very frustrating.

This may be a naive question, but why don't we run the tests in separate
interpreters? For example with -j <N> we do (which creates all kinds of
strange intermittent problems depending on which tests got bundled into the
same process). Is this a matter of performance? Because that would help get
rid of these dependencies between tests, which would probably save core
devs some work and headache.

After all, since a lot of the interpreter state is global (for example
sys.modules), does it not make sense to run each test in a clean
environment? Many tests do fancy things with the global environment which
makes them difficult to keep clean and separate.


> Hence test.support.import_fresh_module - it can ensure you get the
> module you want, regardless of the preexisting contents of
> sys.modules. (
> http://docs.python.org/dev/library/test#test.support.import_fresh_module)
>

Yes, this is the solution currently used in test_xml_etree. However, once
pickling tests are added things stop working. Pickle uses __import__ to
import the module a class belongs to, bypassing all such trickery. So if
test___all__ got _elementtree into sys.modules, pickle's __import__ finds
it even if all the tests in test_xml_etree manage to ignore it for the
Python version because they use import_fresh_module.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20121230/289d8a0a/attachment.html>