I made a presentation on setuptools & company last night at the Chicago Python user group. For the interested, the slides: http://ianbicking.org/docs/setuptools-presentation/ source: http://ianbicking.org/docs/setuptools-presentation/setuptools-slides.txt
Hit the /O in the corner to get all the HTML in one page. The slides kind of dwindle at the end, because I got tired of collecting stuff. But eh, they're just slides, not documentation. I think it went well. There were about 15 people there.
At 11:13 AM 8/12/2005 -0500, Ian Bicking wrote:
I made a presentation on setuptools & company last night at the Chicago Python user group. For the interested, the slides: http://ianbicking.org/docs/setuptools-presentation/
Nice!
Error on slide 13: you can't use --multi-version with install; you're probably thinking "sudo python setup.py easy_install --multi-version ." there.
Re: slide 31, the commands for installing pieces are used by the various bdist commands (including bdist_egg) to install to a temporary directory before archiving the directory to make an egg, windows installer, or "dumb" zipfile.
Re: slide 33, users only get to turn sdist distributions into bdist ones if they have a C compiler, or the package is Python-only.
Re: slide 38, it's "depends", not "deepends", and I've removed it anyway. :) Also, the 'test' command isn't "limited" to unittest, it just expects a unittest-compatible wrapper. For example, doctest has functions that wrap doctests as unittest suites, so it's quite possible to use that. As far as I'm concerned, unittest suites are the WSGI of testing; if somebody creates a fancy new test framework, they should darn well make it possible to put a unittest suite wrapper around it, so that people can integrate their existing tests. :)
CC'ing to bring together opposing viewpoints... (this is referring to setuptools' "setup.py test" command, which expects to have a TestSuite to run)
Phillip J. Eby wrote:
Also, the 'test' command isn't "limited" to unittest, it just expects a unittest-compatible wrapper. For example, doctest has functions that wrap doctests as unittest suites, so it's quite possible to use that. As far as I'm concerned, unittest suites are the WSGI of testing; if somebody creates a fancy new test framework, they should darn well make it possible to put a unittest suite wrapper around it, so that people can integrate their existing tests. :)
Well, I don't know what exactly my opinion is. At a py.test user I don't have TestSuites for my tests. I've argued py.test should load TestCase-based tests by default, but this is kind of the opposite. I don't think it would be that hard to produce such suites; the test items that py.test collects could just be stuffed into a TestCase.
The result would be acceptable as a sort of "this package thinks it runs okay" test. It's not the frontend I'd like to give to users.
That said, if "python setup.py test ..." was completely equivalent to "py.test ..." then that would be great, because though the interface would be different from projects that use unittest, the entry point would be the same (assuming py.test is installed). I suppose a package could add an entry point that overrides the normal setuptools test command...?
Ian Bicking wrote:
CC'ing to bring together opposing viewpoints... (this is referring to setuptools' "setup.py test" command, which expects to have a TestSuite to run)
Phillip J. Eby wrote:
Also, the 'test' command isn't "limited" to unittest, it just expects a unittest-compatible wrapper. For example, doctest has functions that wrap doctests as unittest suites, so it's quite possible to use that. As far as I'm concerned, unittest suites are the WSGI of testing; if somebody creates a fancy new test framework, they should darn well make it possible to put a unittest suite wrapper around it, so that people can integrate their existing tests. :)
Well, I don't know what exactly my opinion is. At a py.test user I don't have TestSuites for my tests. I've argued py.test should load TestCase-based tests by default, but this is kind of the opposite. I don't think it would be that hard to produce such suites; the test items that py.test collects could just be stuffed into a TestCase.
The result would be acceptable as a sort of "this package thinks it runs okay" test. It's not the frontend I'd like to give to users.
That said, if "python setup.py test ..." was completely equivalent to "py.test ..." then that would be great, because though the interface would be different from projects that use unittest, the entry point would be the same (assuming py.test is installed). I suppose a package could add an entry point that overrides the normal setuptools test command...?
+1 on the latter (make "python setup.py test ..." equivalent to "py.test ...") over the former. I'd hate to see py.test have to conform to the awkward conventions introduced by unittest.
Also, the MEMS folks have that Sancho thing, which I seem to recall is a testing framework. Might want to get their opinion on that matter, as they use that testing convention for Durus, Quixote, and other packages.
On Fri, Aug 12, 2005 at 14:35 -0500, Ian Bicking wrote:
CC'ing to bring together opposing viewpoints... (this is referring to setuptools' "setup.py test" command, which expects to have a TestSuite to run)
Phillip J. Eby wrote:
Also, the 'test' command isn't "limited" to unittest, it just expects a unittest-compatible wrapper. For example, doctest has functions that wrap doctests as unittest suites, so it's quite possible to use that. As far as I'm concerned, unittest suites are the WSGI of testing; if somebody creates a fancy new test framework, they should darn well make it possible to put a unittest suite wrapper around it, so that people can integrate their existing tests. :)
Well, I don't know what exactly my opinion is. At a py.test user I don't have TestSuites for my tests. I've argued py.test should load TestCase-based tests by default, but this is kind of the opposite. I don't think it would be that hard to produce such suites; the test items that py.test collects could just be stuffed into a TestCase.
I agree without having looked deeply into it. Providing TestSuite instances that adapt running tests in a simple way with part of the py.test machinery should be quite feasible. Would probably mean that some of py.test's execution features would be switched off (like stdout/stderr capturing).
However, as Ian hints at, py.test itself is already a project independent entry point for running tests in a given directory or for a given project.
The result would be acceptable as a sort of "this package thinks it runs okay" test. It's not the frontend I'd like to give to users.
unsurprisingly, i agree :-)
That said, if "python setup.py test ..." was completely equivalent to "py.test ..." then that would be great, because though the interface would be different from projects that use unittest, the entry point would be the same (assuming py.test is installed).
would make sense, i think.
I suppose a package could add an entry point that overrides the normal setuptools test command...?
Yes, i guess something needs to be configurable there as far as i understand the situation. I also presume that "setup.py test" would allow a custom test method to actually perform the execution of tests, not just provide a TestSuite.
If so, it should at best become a simple matter of how a package can signal to setuptools that it wants its tests to be handled by py.test in which case the work would be defered to the (neccessarily) installed py library when "setup.py test" is invoked. It shouldn't be required from each application to provide this glue code as it should be generic.
actually, how does "setup.py test" work today and which applications/projects are integrated with it if i may ask?
cheers,
holger
holger krekel wrote:
Well, I don't know what exactly my opinion is. At a py.test user I don't have TestSuites for my tests. I've argued py.test should load TestCase-based tests by default, but this is kind of the opposite. I don't think it would be that hard to produce such suites; the test items that py.test collects could just be stuffed into a TestCase.
I agree without having looked deeply into it. Providing TestSuite instances that adapt running tests in a simple way with part of the py.test machinery should be quite feasible. Would probably mean that some of py.test's execution features would be switched off (like stdout/stderr capturing).
However, as Ian hints at, py.test itself is already a project independent entry point for running tests in a given directory or for a given project.
The result would be acceptable as a sort of "this package thinks it runs okay" test. It's not the frontend I'd like to give to users.
unsurprisingly, i agree :-)
Actually I mistyped -- for developers (or potential developers) I think py.test's full featureset and frontend should be available. However, for mere library users there's a use case where they want to simply know that the tests pass, that the package is ok; failures are not expected. At least, when I was talking about this with some Perl people, they specifically said they like that CPAN automatically runs tests when you download a package.
I think a uniform command-line interface is nice in this case; the user is just look for "ok" or "not ok", they aren't looking for a development tool. And it's possible that "setup.py test" should be that uniform interface. That doesn't exclude the possibility of a separate command like "setup.py pytest".
I suppose a package could add an entry point that overrides the normal setuptools test command...?
Yes, i guess something needs to be configurable there as far as i understand the situation. I also presume that "setup.py test" would allow a custom test method to actually perform the execution of tests, not just provide a TestSuite.
If so, it should at best become a simple matter of how a package can signal to setuptools that it wants its tests to be handled by py.test in which case the work would be defered to the (neccessarily) installed py library when "setup.py test" is invoked. It shouldn't be required from each application to provide this glue code as it should be generic.
actually, how does "setup.py test" work today and which applications/projects are integrated with it if i may ask?
AFAIK no applications are integrated. Well, maybe PEAK... this is all rather new. It's documented here:
http://peak.telecommunity.com/DevCenter/setuptools#test-build-package-and-ru...
However, I believe each command to setup.py is simply an "entry_point". I can't find documentation for that at the moment. But it's something like this in the setup.py file:
entry_points={ 'distutils.commands': [ 'test=test_module:test_command', ], }
And then that test_command object (a class) has a specific interface (based on the distutils.Command class). So potentially a py.test-using project could do:
entry_points={ 'distutils.commands': [ 'test=mypkg.commands:test']}
# in mypkg.commands: from py.test.disutils import test
I think that would override "setup.py test" for just that package.
At 03:30 PM 8/12/2005 -0500, Ian Bicking wrote:
However, I believe each command to setup.py is simply an "entry_point". I can't find documentation for that at the moment. But it's something like this in the setup.py file:
entry_points={ 'distutils.commands': [ 'test=test_module:test_command', ], }
Yes, that's exactly it.
And then that test_command object (a class) has a specific interface (based on the distutils.Command class). So potentially a py.test-using project could do:
entry_points={ 'distutils.commands': [ 'test=mypkg.commands:test']}
# in mypkg.commands: from py.test.disutils import test
I think that would override "setup.py test" for just that package.
Actually, it wouldn't, because the first 'test' command defined on sys.path will get used. This doesn't mean that we can't have define test entry points, though, that the setuptools 'test' command would then use to run other testing frameworks. For example, the builtin 'test' command could look up a 'test' command in the just-built package and run that. The problem is that the master test command will have to define what options it takes, because distutils doesn't delegate option parsing to the commands. :(
But setuptools does offer another solution. In py.test's you could create a 'py.test' setup command, and projects that want to use it can do:
setup( ... setup_requires=['py-test>=someversion'], project_args_for_py_test_here="something", )
They can also define a project alias in setup.cfg:
[aliases] test = py.test
This will make "setup.py test" run the "py.test" command instead.
But to me, the simple and obvious thing to do is to make test suites. If py.test exports a zero-argument function that finds tests in the current directory and returns a suite, then a py.test-using package need only set 'test_suite="py.test.get_suite"' in order to use it, with no need to define an alias, nor would py.test need to define another distutils command. And, for basic "does this work?" testing, users get a uniform interface for the 'test' command.
At 10:10 PM 8/12/2005 +0200, holger krekel wrote:
I agree without having looked deeply into it. Providing TestSuite instances that adapt running tests in a simple way with part of the py.test machinery should be quite feasible. Would probably mean that some of py.test's execution features would be switched off (like stdout/stderr capturing).
You would just need to have each test object do its capturing between its calls to result.startTest and result.stopTest - I don't see that it would require switching anything off entirely. Just treat each test as the place for setup/teardown of any special processing.
You might take a look at the doctest module, which provides similar wrappers. Doctests of course capture stdout, but this is done within each "test" from the point of view of the unittest text or GUI test runners.
However, as Ian hints at, py.test itself is already a project independent entry point for running tests in a given directory or for a given project.
Many projects distinguish functional or acceptance tests from their unit tests, so running *all* the tests isn't necessarily a good thing. The point of the current "setup.py test" command in setuptools is to provide a way to define what the default on-installation tests are. The test command also allows you to specify a different starting point as well, with the -s option, so that you can tell a user to run a specific test or suite.
That said, if "python setup.py test ..." was completely equivalent to "py.test ..." then that would be great, because though the interface would be different from projects that use unittest, the entry point would be the same (assuming py.test is installed).
would make sense, i think.
I suppose a package could add an entry point that overrides the normal setuptools test command...?
Yes, i guess something needs to be configurable there as far as i understand the situation. I also presume that "setup.py test" would allow a custom test method to actually perform the execution of tests, not just provide a TestSuite.
It certainly could be done that way; my take on it is that both of the test frameworks in the stdlib support providing TestSuites, making it appear to be the One Obvious Way to do it. Providing TestSuites means that your tests can be run in conjunction with unittest and doctest suites in any unittest-compatible test runner.
If so, it should at best become a simple matter of how a package can signal to setuptools that it wants its tests to be handled by py.test in which case the work would be defered to the (neccessarily) installed py library when "setup.py test" is invoked. It shouldn't be required from each application to provide this glue code as it should be generic.
If py.test exposes a zero-argument function returning a test suite constructed from the default py.test approach, then a user need only specify:
test_suite = "py.test.default_suite"
in their setup() in order to use it.
actually, how does "setup.py test" work today and which
It loads the suite named in setup() or via the '-s' command line option (using the default unittest test loader), and runs the unittest text-based test runner on it. (Note that this means that project management tools (like an IDE) could also run a GUI test runner against the same suite -- something that's not trivially achievable with tests that aren't TestSuite-compatible.)
applications/projects are integrated with it if i may ask?
The PEAK family of projects (including setuptools itself, PyProtocols, PEAK, RuleDispatch, etc.) have been using it as long as setuptools has existed (~18 months).
I'm willing to be flexible about the exact machinery invoked for tests, but I'd very much like to encourage "third-party" test frameworks to consider integrating with unittest (and thereby with all the other test frameworks that do), rather than encourage web-style proliferation of incompatible frameworks. :) doctest already can be called from unittest, so it seems like a good example for other test frameworks to follow. And I know that for me at least, no other framework is going to lure me away from unittest unless it allows me to run its tests as suites; I have way too many unittest-based tests I'm not going to go back and port. (I never even bothered *trying* doctest until it was possible to integrate it with my existing unittest-based suites.)
participants (4)
-
holger krekel -
Ian Bicking -
Patrick K. O'Brien -
Phillip J. Eby