"Greg" == Greg Ward gward@python.net writes:
Greg> On 18 September 2000, Berthold Höllmann said: >> I'm thinking about providing a framework for pre-install >> testing for distutils enabled Python packages. I think about a >> new "test" command, which inserts the lib build path into >> PYTHONPATH and runs specified tests. Is anyone else working on >> something like this, and what are your opinions and >> requirements on this?
Greg> I've been thinking about it from Day 1, personally. Here are Greg> some of my opinions/requirements:
Greg> * serious tests don't belong in the module being tested, Greg> because test code tends to be at least as long as the code Greg> being tested
I think it is easier for the developer to have code and tests together. One can leave out the tests from any distribution by using a Manifest file. But for the developer it is helpful to have a set of tests that one can run after changes have been made to the sources to enshure that "little" changes have not unexpected side effects.
Greg> ...
Greg> * the requirements on what a test script may output should Greg> be fairly strict. My favourite is loosely based on Perl's Greg> testing convention, which is a great success largely because Greg> the standard-way-to-build- Perl-modules includes a "make Greg> test" step, and it's always immediately clear if everything Greg> passed. My proposed convention is this: each test script Greg> outputs a line saying how many tests are expected, then a Greg> series of lines that start with either "ok" or "not ok". If Greg> there are any "not ok" lines, the test script fails; if the Greg> number of "ok" lines != the number of expected tests, the Greg> test script fails. Each "ok" or "not ok" may be followed by Greg> ":" and an arbitrary string, which makes it a lot easier to Greg> track down problems. (The Perl convention is "ok 1", "ok 2", Greg> ... "ok N", which makes it really awkward to track down Greg> problems.) Any lines not starting with "ok" or "not ok" are Greg> ignored.
Thats not always an option. I'm writing modules using numerical opreations. These require test results to be looked over very carefully to decide, whether the differences are significant or not. Also the underlaying Fortran code makes different decisions on different platforms. That makes automatic testing nearly impossible, but I want at least a constistant way to generate the needed output, and I want to test everything before I install it.
Greg> ...
Greg> * we might also want to accomodate a Python-style test Greg> suite, where you compare "known good" output of a script Greg> with the current output. I'm not a fan of this methodology, Greg> but sometimes it's easier, and it may have some currency in Greg> the Python community outside of Lib/test. It should be the Greg> developer's choice what style of test suite he wants to Greg> write.
All this can be done with my posted, extremly lightweight test.py extension for the _command_ subdirectory. The test scripts can written to implement/use any testframework you like. Maybe a
python setup.py test
command is only a convenience command but it gives you a starting point when maintaining code you wrote ages ago, or somebody else wrote.
Greetings
Berthold
On 21 September 2000, Berthold Höllmann said:
I think it is easier for the developer to have code and tests together.
Not if the tests are really exhaustive! When I write good tests (or at least what I think are good tests), the test script is roughly the size of the module being tested. I don't want my "if __name__ == '__main__'" block to be as big as the rest of the module. I think the "test" command should support "in-situ" testing, but it's *not* enough for serious tests.
One can leave out the tests from any distribution by using a Manifest file.
Huh? That loses half the point of writing tests, which is to assure people building and installing your modules that they probably work as advertised on their platform.
But for the developer it is helpful to have a set of tests that one can run after changes have been made to the sources to enshure that "little" changes have not unexpected side effects.
Yes! That's the other half of the reason for writing tests.
(You can define "half" here as 40-60, 20-80, whatever.)
Thats not always an option. I'm writing modules using numerical opreations. These require test results to be looked over very carefully to decide, whether the differences are significant or not.
Oooh, yes, in a previous life I struggled (briefly) with automated testing of numerical algorithms. Not fun. But asking end-users to examine the test output and make sure that the difference between 1.0234324 and 1.0234821 is less than the machine epsilon is surely even less fun.
Also the underlaying Fortran code makes different decisions on different platforms. That makes automatic testing nearly impossible, but I want at least a constistant way to generate the needed output, and I want to test everything before I install it.
Here's my extreme position:
If a test isn't fully automated -- i.e., if it takes more than epsilon human intelligence to determine if the test passed or failed -- then it's not a proper test, it's just a demo.
Granted there are real-world situations that cannot meet this tough criterion -- testing a curses module, say, or some (many? most?) numerical algorithms. However, it is a worthy goal that, with some work, can often be achieved. Witness the number of Perl modules on CPAN for which you can run "make test" and immediately be assured that everything Just Works on the current platform and Perl interpreter.
Thus, fully automated testing is to be encouraged! I like the "ok, ok, not ok, ok" method because it's immediately obvious what's going on, fairly easy (if tedious) to write tests with no external support, and shouldn't be too hard to write some sort of test harness framework. The framework would have two purposes: * make it easy for developers to write tests that output "ok, ok, not ok, ok" * support the Distutils "test" command that analyses that output and ensures that exactly the correct number of "ok" lines, and no "not ok" lines, were seen
All this can be done with my posted, extremly lightweight test.py extension for the _command_ subdirectory. The test scripts can written to implement/use any testframework you like. Maybe a
OK, lightweight is good. (Something I occasionally need to be reminded of...) I'll definitely take a look at it.
Greg
participants (2)
-
Berthold Höllmann -
Greg Ward