[Python-Dev] test___all__ polluting sys.modules?

[Gregory P. Smith]

On Sun, Dec 30, 2012 at 5:33 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
>
> On Sat, Dec 29, 2012 at 5:34 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On Sun, Dec 30, 2012 at 10:46 AM, Benjamin Peterson <benjamin at python.org>
>> wrote:
>> > 2012/12/29 Eli Bendersky <eliben at gmail.com>:
>> >> Hi,
>> >>
>> >> This came up while investigating some test-order-dependency failures in
>> >> issue 16076.
>> >>
>> >> test___all__ goes over modules that have `__all__` in them and does
>> 'from
>> >> <module> import *' on them. This leaves a lot of modules in
>> sys.modules,
>> >> which may interfere with some tests that do fancy things with
>> sys,modules.
>> >> In particular, the ElementTree tests have trouble with it because they
>> >> carefully set up the imports to get the C or the Python version of
>> etree
>> >> (see issues 15083 and 15075).
>> >>
>> >> Would it make sense to save the sys.modules state and restore it in
>> >> test___all__ so that sys.modules isn't affected by this test?
>> >
>> > Sounds reasonable to me.
>>
>> I've tried this as an inherent property of regrtest before (to resolve
>> some problems with test_pydoc), and it didn't work - we have too many
>> modules with non-local side effects (IIRC, mostly related to the copy
>> and pickle registries).
>>
>> Given that it checks the whole standard library, test___all__ is
>> likely to run into the same problem.
>>
>>
> Yes, I'm running into all kinds of weird problems when saving/restoring
> sys.modules around test___all__. This is not the first time I get to fight
> this test-run-dependency problem and it's very frustrating.
>
> This may be a naive question, but why don't we run the tests in separate
> interpreters? For example with -j <N> we do (which creates all kinds of
> strange intermittent problems depending on which tests got bundled into the
> same process). Is this a matter of performance? Because that would help get
> rid of these dependencies between tests, which would probably save core
> devs some work and headache.
>

For test___all__ having it run itself within a subprocess to guarantee it
is in its own interpreter is wise.

Doing that for all tests by default though is a good way to hide problems
with particular tests and underhandedly encourage more bad behavior from
tests messing up the global environment as there are no consequences. We
should aim to keep our tests as clean as possible. That doesn't mean that
they _are_ (we've got a _lot_ of old messy code) but it is a good goal.

-gps


>
> After all, since a lot of the interpreter state is global (for example
> sys.modules), does it not make sense to run each test in a clean
> environment? Many tests do fancy things with the global environment which
> makes them difficult to keep clean and separate.
>
>
>> Hence test.support.import_fresh_module - it can ensure you get the
>> module you want, regardless of the preexisting contents of
>> sys.modules. (
>> http://docs.python.org/dev/library/test#test.support.import_fresh_module)
>>
>
>  Yes, this is the solution currently used in test_xml_etree. However,
> once pickling tests are added things stop working. Pickle uses __import__
> to import the module a class belongs to, bypassing all such trickery. So if
> test___all__ got _elementtree into sys.modules, pickle's __import__ finds
> it even if all the tests in test_xml_etree manage to ignore it for the
> Python version because they use import_fresh_module.
>
> Eli
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20121231/6a7477d2/attachment.html>