[issue15392] Create a unittest framework for IDLE

[Ezio Melotti]

Ezio Melotti added the comment:

A few comments about the patch:
1) I would prefer a more explicit idle_test instead of Itest (assuming "I" means idle);
2) having idle_test in Lib/test would be better if possible (see #10572);
3) the tests should only be in idle_test, and not in the "if __name__ == '__main__':" of the files;
4) I'm not sure the @template file is necessary.  It could be documented somewhere else though (see e.g. http://docs.python.org/3/library/test.html#writing-unit-tests-for-the-test-package).

IOW putting all tests in a separate dir is a good thing, and the dir could either be in Lib/test or in Lib/idlelib.  All the tests should be inside this dir, except for Lib/test/test_idle.py that should be the entry point used to run all the tests in idle_test (see e.g. the tests for ctypes, email, and json).  Individual tests in idle_test can be run on their own, and they should use the "if __name__ == '__main__': unittest.main()" idiom.

> Question: "Unittest supports skipping individual test 
> methods and even whole classes of tests" How about skipping
> (possibly conditionally) an entire file -- especially test_idle,
> which has no classes, or and test_xxx with multiple classes?

You can raise unittest.SkipTest or use support.import_module().  This works fine if you run the tests through regrtest, but be aware that unittest itself only sees this as a skip from 3.4 (see #16935).

> * Exceptions raised outside of self.assertXyz (and self.fail)
> calls are regarded as an error in the test code rather than a
> failure of the code tested (26.3.4. Organizing test code).
> So, there being no 'assertNotRaises' context manager, I added
> "try:...except ...: self.fail()" to test_pathbrowser.py so a
> failure will be reported as such and not as an error. If
> there is a better way to do this, let me know.

If it's supposed to work the try/except is not necessary IMHO.  By this logic every line of code should be wrapped in a try/except :)

> Since 3.x chains exceptions and prints the original traceback,
> no message argument is needed with self.fail to explain the
> failure. For 2.7, I believe one should be added.

If you still want to keep the try/except, I would provide a meaningful failure message in any case.

> Note: to empirically test that a test fails properly, it must
> be run with code that does not work properly. This is a
> problem with writing tests after the fact for code that works
> properly.

It's not difficult to break the code to test that test works though :)

> I checked all 62 idlelib/*.py files for a test to see what we
> have to work with: the answer is 'not much'. Less than half
> the files have a test.

If possible I think these should all be moved to idle_test.  You can also add an additional resource to regrtest to skip the ones that require manual intervention.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue15392>
_______________________________________