At 10:10 PM 8/12/2005 +0200, holger krekel wrote:
I agree without having looked deeply into it. Providing TestSuite instances that adapt running tests in a simple way with part of the py.test machinery should be quite feasible. Would probably mean that some of py.test's execution features would be switched off (like stdout/stderr capturing).
You would just need to have each test object do its capturing between its calls to result.startTest and result.stopTest - I don't see that it would require switching anything off entirely. Just treat each test as the place for setup/teardown of any special processing. You might take a look at the doctest module, which provides similar wrappers. Doctests of course capture stdout, but this is done within each "test" from the point of view of the unittest text or GUI test runners.
However, as Ian hints at, py.test itself is already a project independent entry point for running tests in a given directory or for a given project.
Many projects distinguish functional or acceptance tests from their unit tests, so running *all* the tests isn't necessarily a good thing. The point of the current "setup.py test" command in setuptools is to provide a way to define what the default on-installation tests are. The test command also allows you to specify a different starting point as well, with the -s option, so that you can tell a user to run a specific test or suite.
That said, if "python setup.py test ..." was completely equivalent to "py.test ..." then that would be great, because though the interface would be different from projects that use unittest, the entry point would be the same (assuming py.test is installed).
would make sense, i think.
I suppose a package could add an entry point that overrides the normal setuptools test command...?
Yes, i guess something needs to be configurable there as far as i understand the situation. I also presume that "setup.py test" would allow a custom test method to actually perform the execution of tests, not just provide a TestSuite.
It certainly could be done that way; my take on it is that both of the test frameworks in the stdlib support providing TestSuites, making it appear to be the One Obvious Way to do it. Providing TestSuites means that your tests can be run in conjunction with unittest and doctest suites in any unittest-compatible test runner.
If so, it should at best become a simple matter of how a package can signal to setuptools that it wants its tests to be handled by py.test in which case the work would be defered to the (neccessarily) installed py library when "setup.py test" is invoked. It shouldn't be required from each application to provide this glue code as it should be generic.
If py.test exposes a zero-argument function returning a test suite constructed from the default py.test approach, then a user need only specify: test_suite = "py.test.default_suite" in their setup() in order to use it.
actually, how does "setup.py test" work today and which
It loads the suite named in setup() or via the '-s' command line option (using the default unittest test loader), and runs the unittest text-based test runner on it. (Note that this means that project management tools (like an IDE) could also run a GUI test runner against the same suite -- something that's not trivially achievable with tests that aren't TestSuite-compatible.)
applications/projects are integrated with it if i may ask?
The PEAK family of projects (including setuptools itself, PyProtocols, PEAK, RuleDispatch, etc.) have been using it as long as setuptools has existed (~18 months). I'm willing to be flexible about the exact machinery invoked for tests, but I'd very much like to encourage "third-party" test frameworks to consider integrating with unittest (and thereby with all the other test frameworks that do), rather than encourage web-style proliferation of incompatible frameworks. :) doctest already can be called from unittest, so it seems like a good example for other test frameworks to follow. And I know that for me at least, no other framework is going to lure me away from unittest unless it allows me to run its tests as suites; I have way too many unittest-based tests I'm not going to go back and port. (I never even bothered *trying* doctest until it was possible to integrate it with my existing unittest-based suites.)