<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Dec 30, 2012 at 5:33 AM, Eli Bendersky <span dir="ltr"><<a href="mailto:eliben@gmail.com" target="_blank">eliben@gmail.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote"><div class="im">On Sat, Dec 29, 2012 at 5:34 PM, Nick Coghlan <span dir="ltr"><<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>On Sun, Dec 30, 2012 at 10:46 AM, Benjamin Peterson <<a href="mailto:benjamin@python.org" target="_blank">benjamin@python.org</a>> wrote:<br>
> 2012/12/29 Eli Bendersky <<a href="mailto:eliben@gmail.com" target="_blank">eliben@gmail.com</a>>:<br>
>> Hi,<br>
>><br>
>> This came up while investigating some test-order-dependency failures in<br>
>> issue 16076.<br>
>><br>
>> test___all__ goes over modules that have `__all__` in them and does 'from<br>
>> <module> import *' on them. This leaves a lot of modules in sys.modules,<br>
>> which may interfere with some tests that do fancy things with sys,modules.<br>
>> In particular, the ElementTree tests have trouble with it because they<br>
>> carefully set up the imports to get the C or the Python version of etree<br>
>> (see issues 15083 and 15075).<br>
>><br>
>> Would it make sense to save the sys.modules state and restore it in<br>
>> test___all__ so that sys.modules isn't affected by this test?<br>
><br>
> Sounds reasonable to me.<br>
<br>
</div></div>I've tried this as an inherent property of regrtest before (to resolve<br>
some problems with test_pydoc), and it didn't work - we have too many<br>
modules with non-local side effects (IIRC, mostly related to the copy<br>
and pickle registries).<br>
<br>
Given that it checks the whole standard library, test___all__ is<br>
likely to run into the same problem.<br>
<br></blockquote><div><br></div></div><div>Yes, I'm running into all kinds of weird problems when saving/restoring sys.modules around test___all__. This is not the first time I get to fight this test-run-dependency problem and it's very frustrating. <br>
<br>This may be a naive question, but why don't we run the tests in separate interpreters? For example with -j <N> we do (which creates all kinds of strange intermittent problems depending on which tests got bundled into the same process). Is this a matter of performance? Because that would help get rid of these dependencies between tests, which would probably save core devs some work and headache.<br>
</div></div></div></div></blockquote><div><br></div><div style>For test___all__ having it run itself within a subprocess to guarantee it is in its own interpreter is wise.</div><div style><br></div><div style>Doing that for all tests by default though is a good way to hide problems with particular tests and underhandedly encourage more bad behavior from tests messing up the global environment as there are no consequences. We should aim to keep our tests as clean as possible. That doesn't mean that they _are_ (we've got a _lot_ of old messy code) but it is a good goal.</div>
<div style><br></div><div style>-gps</div><div style> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>
<br></div><div>After all, since a lot of the interpreter state is global (for example sys.modules), does it not make sense to run each test in a clean environment? Many tests do fancy things with the global environment which makes them difficult to keep clean and separate.<br>
</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hence test.support.import_fresh_module - it can ensure you get the<br>
module you want, regardless of the preexisting contents of<br>
sys.modules. (<a href="http://docs.python.org/dev/library/test#test.support.import_fresh_module" target="_blank">http://docs.python.org/dev/library/test#test.support.import_fresh_module</a>)<br></blockquote></div></div><br>
</div>
<div class="gmail_extra">Yes, this is the solution currently used in test_xml_etree. However, once pickling tests are added things stop working. Pickle uses __import__ to import the module a class belongs to, bypassing all such trickery. So if test___all__ got _elementtree into sys.modules, pickle's __import__ finds it even if all the tests in test_xml_etree manage to ignore it for the Python version because they use import_fresh_module.<span class="HOEnZb"><font color="#888888"><br>
<br></font></span></div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra">Eli<br><br></div><div class="gmail_extra"><br></div></font></span></div>
<br>_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org">Python-Dev@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/python-dev" target="_blank">http://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="http://mail.python.org/mailman/options/python-dev/greg%40krypto.org" target="_blank">http://mail.python.org/mailman/options/python-dev/greg%40krypto.org</a><br>
<br></blockquote></div><br></div></div>