[Distutils] [py-dev] Re: setuptools presentation
Phillip J. Eby
pje at telecommunity.com
Fri Aug 12 23:14:38 CEST 2005
At 10:10 PM 8/12/2005 +0200, holger krekel wrote:
>I agree without having looked deeply into it. Providing TestSuite
>instances that adapt running tests in a simple way with part
>of the py.test machinery should be quite feasible. Would probably
>mean that some of py.test's execution features would be switched
>off (like stdout/stderr capturing).
You would just need to have each test object do its capturing between its
calls to result.startTest and result.stopTest - I don't see that it would
require switching anything off entirely. Just treat each test as the place
for setup/teardown of any special processing.
You might take a look at the doctest module, which provides similar
wrappers. Doctests of course capture stdout, but this is done within each
"test" from the point of view of the unittest text or GUI test runners.
>However, as Ian hints at, py.test itself is already a project independent
>entry point for running tests in a given directory or for a given project.
Many projects distinguish functional or acceptance tests from their unit
tests, so running *all* the tests isn't necessarily a good thing. The
point of the current "setup.py test" command in setuptools is to provide a
way to define what the default on-installation tests are. The test command
also allows you to specify a different starting point as well, with the -s
option, so that you can tell a user to run a specific test or suite.
> > That said, if "python setup.py test ..." was completely equivalent to
> > "py.test ..." then that would be great, because though the interface
> > would be different from projects that use unittest, the entry point
> > would be the same (assuming py.test is installed).
>would make sense, i think.
> > I suppose a package could add an entry point that overrides
> > the normal setuptools test command...?
>Yes, i guess something needs to be configurable there as far as i
>understand the situation. I also presume that "setup.py test"
>would allow a custom test method to actually perform the execution
>of tests, not just provide a TestSuite.
It certainly could be done that way; my take on it is that both of the test
frameworks in the stdlib support providing TestSuites, making it appear to
be the One Obvious Way to do it. Providing TestSuites means that your
tests can be run in conjunction with unittest and doctest suites in any
unittest-compatible test runner.
>If so, it should at best become a simple matter of how a
>package can signal to setuptools that it wants its tests to be
>handled by py.test in which case the work would be defered to
>the (neccessarily) installed py library when "setup.py test"
>is invoked. It shouldn't be required from each application
>to provide this glue code as it should be generic.
If py.test exposes a zero-argument function returning a test suite
constructed from the default py.test approach, then a user need only specify:
test_suite = "py.test.default_suite"
in their setup() in order to use it.
>actually, how does "setup.py test" work today and which
It loads the suite named in setup() or via the '-s' command line option
(using the default unittest test loader), and runs the unittest text-based
test runner on it. (Note that this means that project management tools
(like an IDE) could also run a GUI test runner against the same suite --
something that's not trivially achievable with tests that aren't
>applications/projects are integrated with it if i may ask?
The PEAK family of projects (including setuptools itself, PyProtocols,
PEAK, RuleDispatch, etc.) have been using it as long as setuptools has
existed (~18 months).
I'm willing to be flexible about the exact machinery invoked for tests, but
I'd very much like to encourage "third-party" test frameworks to consider
integrating with unittest (and thereby with all the other test frameworks
that do), rather than encourage web-style proliferation of incompatible
frameworks. :) doctest already can be called from unittest, so it seems
like a good example for other test frameworks to follow. And I know that
for me at least, no other framework is going to lure me away from unittest
unless it allows me to run its tests as suites; I have way too many
unittest-based tests I'm not going to go back and port. (I never even
bothered *trying* doctest until it was possible to integrate it with my
existing unittest-based suites.)
More information about the Distutils-SIG