[Distutils] [py-dev] Re: setuptools presentation

[Phillip J. Eby]

At 10:10 PM 8/12/2005 +0200, holger krekel wrote:
>I agree without having looked deeply into it.  Providing TestSuite
>instances that adapt running tests in a simple way with part
>of the py.test machinery should be quite feasible.  Would probably
>mean that some of py.test's execution features would be switched
>off (like stdout/stderr capturing).

You would just need to have each test object do its capturing between its 
calls to result.startTest and result.stopTest - I don't see that it would 
require switching anything off entirely.  Just treat each test as the place 
for setup/teardown of any special processing.

You might take a look at the doctest module, which provides similar 
wrappers.  Doctests of course capture stdout, but this is done within each 
"test" from the point of view of the unittest text or GUI test runners.


>However, as Ian hints at, py.test itself is already a project independent
>entry point for running tests in a given directory or for a given project.

Many projects distinguish functional or acceptance tests from their unit 
tests, so running *all* the tests isn't necessarily a good thing.  The 
point of the current "setup.py test" command in setuptools is to provide a 
way to define what the default on-installation tests are.  The test command 
also allows you to specify a different starting point as well, with the -s 
option, so that you can tell a user to run a specific test or suite.


> > That said, if "python setup.py test ..." was completely equivalent to
> > "py.test ..." then that would be great, because though the interface
> > would be different from projects that use unittest, the entry point
> > would be the same (assuming py.test is installed).
>
>would make sense, i think.
>
> > I suppose a package could add an entry point that overrides
> > the normal setuptools test command...?
>
>Yes, i guess something needs to be configurable there as far as i
>understand the situation.  I also presume that "setup.py test"
>would allow a custom test method to actually perform the execution
>of tests, not just provide a TestSuite.

It certainly could be done that way; my take on it is that both of the test 
frameworks in the stdlib support providing TestSuites, making it appear to 
be the One Obvious Way to do it.  Providing TestSuites means that your 
tests can be run in conjunction with unittest and doctest suites in any 
unittest-compatible test runner.


>If so, it should at best become a simple matter of how a
>package can signal to setuptools that it wants its tests to be
>handled by py.test in which case the work would be defered to
>the (neccessarily) installed py library when "setup.py test"
>is invoked.  It shouldn't be required from each application
>to provide this glue code as it should be generic.

If py.test exposes a zero-argument function returning a test suite 
constructed from the default py.test approach, then a user need only specify:

    test_suite = "py.test.default_suite"

in their setup() in order to use it.


>actually, how does "setup.py test" work today and which

It loads the suite named in setup() or via the '-s' command line option 
(using the default unittest test loader), and runs the unittest text-based 
test runner on it.  (Note that this means that project management tools 
(like an IDE) could also run a GUI test runner against the same suite -- 
something that's not trivially achievable with tests that aren't 
TestSuite-compatible.)


>applications/projects are integrated with it if i may ask?

The PEAK family of projects (including setuptools itself, PyProtocols, 
PEAK, RuleDispatch, etc.) have been using it as long as setuptools has 
existed (~18 months).

I'm willing to be flexible about the exact machinery invoked for tests, but 
I'd very much like to encourage "third-party" test frameworks to consider 
integrating with unittest (and thereby with all the other test frameworks 
that do), rather than encourage web-style proliferation of incompatible 
frameworks. :)  doctest already can be called from unittest, so it seems 
like a good example for other test frameworks to follow.  And I know that 
for me at least, no other framework is going to lure me away from unittest 
unless it allows me to run its tests as suites; I have way too many 
unittest-based tests I'm not going to go back and port.  (I never even 
bothered *trying* doctest until it was possible to integrate it with my 
existing unittest-based suites.)