[Ironpython-users] test framework

[Jeff Hardy]

On Thu, Jul 17, 2014 at 9:52 AM, Pawel Jasinski <pawel.jasinski at gmail.com>
wrote:

> First of all, there is a need to document where the tests are and how
> to run them. It took me a moment to figure it out, so it make sense to
> share it.
>
> The test config: IronLanguages/Test/IronPython.tests
> An individual test can be started by: ./test-ipy.cmd
> /test:test_zipimport_cpy
>
> I remember at one time compiling TestRunner.
> IronLanguages/Test/TestRunner/TestRunner.sln, so if something doesn't
> work, try to build it.
>
> The body of the tests is located in many places:
> IronLanguages/Languages/IronPython/Tests
> IronLanguages/External.LCA_RESTRICTED/Languages/IronPython/27/Lib/test
>
>
> In addition to ./test-ipy.cmd the test from stdlib can be run with a usual:
> ipy -X:FullFrames -m unittest test.test_zipimport_support
>
> I also had 'nose' running at some point, so it is likely to be still ok.
>

Thanks for writing this down! It could go on the github wiki as well, but
at least search engines are likely to find it.


>
>
> Now the second part, what is not so nice about our test framework and
> perhaps how can we avoid the same mistakes in "3".
>
> As Jeff refers to it: "fabulously complicated test infrastructure"
> (FCTI?), is a bit of inconvenient to use. I suspect the definition of
> what individual test constitutes is not uniform. FCTI things a test is
> entire file or even worse a package. I found no way to test a single
> function out of a test file without reconfiguring or modifying a lot.
>

Yep. It's a legacy of the system used at MS (which I believe was called
SNAP); the current TestRunner was the simplest thing Dino could put
together to get it working when the code was permanently moved out of MS.


>
> For example:
> test_stdconsole.py takes ~3 minutes. But a test where it fails 12 seconds.
> so I nuke (or comment out) all other test functions in
> test_stdconsole.py to get to fast test cycle.
>
> Second example:
> modules_io_related_ipy is a single test, but it includes not only all
> functions out of a file but entire package (all files out of
> IronLanguages/Languages/IronPython/Tests/modules/io_related). Here I
> basically remove all not interesting files and functions.
>
> If anybody has a better way, please share.
>

My thinking on testing for 3 can be found at
https://github.com/jdhardy/ironpython3/commits/test-runner.

The main reason I want to use NUnit (or xUnit.Net) to drive it is that they
support multiple platforms, and it saves us having to write test runners
everywhere (keeping in mind things like Android, iOS, Win8, etc.). Doing
so, though, involves rewriting the discovery portions of both iptest and
the CPython test suite, but at least we only have to do it once. Then the
tests can be exposed to any NUnit-capable runner.

Nothing to help much in the short term, but longer term things will get
better.

- Jeff



>
> --pawel
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140717/7c07b498/attachment.html>