[Ironpython-users] test framework

[Pawel Jasinski]

First of all, there is a need to document where the tests are and how
to run them. It took me a moment to figure it out, so it make sense to
share it.

The test config: IronLanguages/Test/IronPython.tests
An individual test can be started by: ./test-ipy.cmd /test:test_zipimport_cpy

I remember at one time compiling TestRunner.
IronLanguages/Test/TestRunner/TestRunner.sln, so if something doesn't
work, try to build it.

The body of the tests is located in many places:
IronLanguages/Languages/IronPython/Tests
IronLanguages/External.LCA_RESTRICTED/Languages/IronPython/27/Lib/test


In addition to ./test-ipy.cmd the test from stdlib can be run with a usual:
ipy -X:FullFrames -m unittest test.test_zipimport_support

I also had 'nose' running at some point, so it is likely to be still ok.


Now the second part, what is not so nice about our test framework and
perhaps how can we avoid the same mistakes in "3".

As Jeff refers to it: "fabulously complicated test infrastructure"
(FCTI?), is a bit of inconvenient to use. I suspect the definition of
what individual test constitutes is not uniform. FCTI things a test is
entire file or even worse a package. I found no way to test a single
function out of a test file without reconfiguring or modifying a lot.

For example:
test_stdconsole.py takes ~3 minutes. But a test where it fails 12 seconds.
so I nuke (or comment out) all other test functions in
test_stdconsole.py to get to fast test cycle.

Second example:
modules_io_related_ipy is a single test, but it includes not only all
functions out of a file but entire package (all files out of
IronLanguages/Languages/IronPython/Tests/modules/io_related). Here I
basically remove all not interesting files and functions.

If anybody has a better way, please share.

--pawel