RFC: Standard Declaration of tests in eggs
Here is a rough draft proposal for declaring tests in eggs: Introduction ============ Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal ======== This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Thoughts? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Here is a rough draft proposal for declaring tests in eggs:
Introduction ============
Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests.
Proposal ========
This proposal aims to be extremely simple. It has 2 parts:
1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites.
How would this work if for example, you're using an alternative testing framework (like py.test) for your test? It would be nice to be able to bootstrap it :-)
2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner.
Great, so at least the testing framework could be declared as a dependency David
David Fraser wrote:
Jim Fulton wrote:
Here is a rough draft proposal for declaring tests in eggs:
Introduction ============
Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests.
Proposal ========
This proposal aims to be extremely simple. It has 2 parts:
1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites.
How would this work if for example, you're using an alternative testing framework (like py.test) for your test?
I'm not familiar with py.test. I guess it's not based on unittest. Why? Couldn't it at least have a unittest wrapper, like the one I wrote for doctest? Even though I use doctest almost exclusively, I view unittest as a common API that various testing frameworks can and should play with. I certainly think there should be some common API like that and see unittest as the incumbant.
It would be nice to be able to bootstrap it :-)
What do you mean by that?
2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner.
Great, so at least the testing framework could be declared as a dependency
My assumption (and my use case) is that you'd run whatever test runner you want and tell it to run tests from some given eggs. So the test runner would already be loaded. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:
Here is a rough draft proposal for declaring tests in eggs:
Introduction ============
Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests.
Proposal ========
This proposal aims to be extremely simple. It has 2 parts:
1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites.
2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner.
Thoughts?
Point #2 is unnecessary, since individual entry points can list extras in square brackets following the module/attribute information. When loading an entry point, these extras automatically get require()'d. Conversely, if the test runner wants to manage the loading process, it can simply inspect the entry point object to determine the names of the extras. Regarding point #1, I'm not sure this is enought to define what's necessary. For example, it should perhaps be stated that the entry point must be in code that will be installed by either the distribution itself, or that is included in the code provided by the entry point's extras. Also, an egg can't provide more than one entry point with the same name, so rather than a 'test_suite' entry point, we probably want an entry point *group*, perhaps something like 'installed.test_suites'. Next, there needs to be reasonable support for dynamic test discovery, such as setuptools' own ScanningLoader. I would suggest that there be an optional entry point for the test loader class to be used to process the other entry points' target objects. The standard unittest protocol for loadTestsFromName() takes two arguments: a name and an object. Passing an empty string for the name, and the object loaded by the test suite entry point, suffices to enable normal behavior for loaders such as ScanningLoader, and I believe 'nose' as well as any other well-behaved unittest extensions. However, I think the spec should try to define what "well-behaved" means in terms of I/O, result reporting, etc. (Obviously, one requirement is that the test loader must be able to take an empty name string and an object in its loadTestsFromName() method, and return a test suite.)
At 05:36 PM 1/5/2007 +0200, David Fraser wrote:
How would this work if for example, you're using an alternative testing framework (like py.test) for your test?
If I understand correctly, py.test simply won't be usable for this scenario, because it assumes tests are files first and modules second, if at all. It's not equipped to discover tests that are inside eggs, which is the whole point of this proposal. Integrating py.test with the "setup.py test" command, on the other hand is possible and relatively simple; just create a unittest-compatible test suite object that wraps py.test, and a test loader class that grabs the necessary arguments from the test_suite argument string. No modifications to setuptools are required, you just provide appropriate values for the test_suite and test_loader arguments to setup(). But the proposal being discussed is for tests that are actually *installed* somewhere, and thus might be in a zipfile.
Great, so at least the testing framework could be declared as a dependency
For "setup.py test", this is already possible via the 'tests_require' argument to setup(). What's being discussed here is a standard for *installed* tests -- that is, tests that a developer has chosen to make part of their projects's installed code. The existing "setup.py test" subsystem requires a *source* distribution, not installed code.
Phillip J. Eby wrote:
At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:
Here is a rough draft proposal for declaring tests in eggs:
Introduction ============
Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests.
Proposal ========
This proposal aims to be extremely simple. It has 2 parts:
1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites.
2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner.
Thoughts?
Point #2 is unnecessary, since individual entry points can list extras in square brackets following the module/attribute information. When loading an entry point, these extras automatically get require()'d. Conversely, if the test runner wants to manage the loading process, it can simply inspect the entry point object to determine the names of the extras.
Ah, good point. Man, setuptools is soooooo rich.
Regarding point #1, I'm not sure this is enought to define what's necessary. For example, it should perhaps be stated that the entry point must be in code that will be installed by either the distribution itself, or that is included in the code provided by the entry point's extras.
OK
Also, an egg can't provide more than one entry point with the same name, so rather than a 'test_suite' entry point, we probably want an entry point *group*, perhaps something like 'installed.test_suites'.
That's what I meant to say. I'll clarify.
Next, there needs to be reasonable support for dynamic test discovery, such as setuptools' own ScanningLoader. I would suggest that there be an optional entry point for the test loader class to be used to process the other entry points' target objects. The standard unittest protocol for loadTestsFromName() takes two arguments: a name and an object. Passing an empty string for the name, and the object loaded by the test suite entry point, suffices to enable normal behavior for loaders such as ScanningLoader, and I believe 'nose' as well as any other well-behaved unittest extensions.
However, I think the spec should try to define what "well-behaved" means in terms of I/O, result reporting, etc. (Obviously, one requirement is that the test loader must be able to take an empty name string and an object in its loadTestsFromName() method, and return a test suite.)
Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 12:24 PM 1/5/2007 -0500, Jim Fulton wrote:
Phillip J. Eby wrote:
At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:
2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner.
Thoughts? Point #2 is unnecessary, since individual entry points can list extras in square brackets following the module/attribute information. When loading an entry point, these extras automatically get require()'d. Conversely, if the test runner wants to manage the loading process, it can simply inspect the entry point object to determine the names of the extras.
Ah, good point. Man, setuptools is soooooo rich.
Rich requirements make rich design. See my recent "Where Zope Leads, Python Follows" blog post. :)
Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me.
Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project. If you're using setuptools' ScanningLoader (which is the default for the "test" command, and which I believe should be the default for this proposal as well), you automatically get all TestCase subclasses in all submodules of the package you target, and if any module has an 'additional_tests()' function, it's called to get a suite that's added in. This lets you just make your entry point a "whatever.tests" package, without writing any extra code (unless you want to throw in some doctests via a few 'additional_tests()' functions). See the setuptools documentation for the 'test_suite' and 'test_loader' arguments to setup() for the full details. Oh, and the 'nose' package is a unittest-compatible extended testing framework that is designed to also work as a 'test_loader' for setuptools. So, between ScanningLoader and 'nose', that would be two already-existing ways of specifying tests that would work with the new proposal, if you could specify a loader entry point. And it would be good if the default loader was compatible with ScanningLoader's test discovery features. (I'm not sure if nose really works with eggs, though; ScanningLoader will discover tests in eggs as long as the source is included, but nose may be strictly file-based for all I know. Changing it probably wouldn't be too difficult, however.)
Phillip J. Eby wrote: ...
Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me.
Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project.
It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too. Besides, I expect most projects will have a single test suite (function) that could just be named directly.
If you're using setuptools' ScanningLoader (which is the default for the "test" command, and which I believe should be the default for this proposal as well), you automatically get all TestCase subclasses in all submodules of the package you target, and if any module has an 'additional_tests()' function, it's called to get a suite that's added in. This lets you just make your entry point a "whatever.tests" package, without writing any extra code (unless you want to throw in some doctests via a few 'additional_tests()' functions).
See the setuptools documentation for the 'test_suite' and 'test_loader' arguments to setup() for the full details.
I've looked at this and it is a bit too magic for my taste. It's also a heck of a lot to get one's head around. (This is true of setuptools in general.) Of course, it doesn't help much with our doctests and for our older unittest tests, we have lots of TestCase subclasses that aren't meant to be run directly, but are base classes for other TestCase classes. It appears that your loader would pick these up incorrectly. Of course, we have our own loader (of sorts) and it has it's own magic conventions. We would likely use that for our larger projects (e.g. ZODB) that have more than a few test suites. Importing some function and invoking it explicitly will not be the least bit onerous for those projects. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote:
Phillip J. Eby wrote: ...
Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project.
It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too.
And it's enormously repetitive if you have a lot of projects.
Besides, I expect most projects will have a single test suite (function) that could just be named directly.
You expect wrongly. :) 'nose' and 'py.test' are popular *precisely because* they do NOT require this. Setuptools changed to emulate them by adding ScanningLoader, even though it originally did the "test suite function" thing you're proposing. See, I originally wrote all my test suites the way you're proposing, but I changed because this is one of those "convention beats configuration" situations. It's easier to just conform to a policy and have a tool that applies the policy. Different projects may have different policies, so providing an option for the loader, allows them to follow it. (And it made my life easier, too, as I began doing more small projects instead of a few large ones.) And, if you want your approach to be widely adopted, it would be best to allow projects to follow their own policies without requiring them to add any code, even if it's a one-liner. So, *technically*, there may be no reason to do this, but from a usability, friendliness, compatiblity, marketing, adoption, etc. format I don't think the standard will be successful without allowing this out. People often have heavy psychological investment in existing tools, and it is hundreds of times easier to rationalize adding a line of configuration to setup() than it is to rationalize adding two or three lines of code. One is merely a packaging change, the other is a programming change.
Phillip J. Eby wrote:
At 05:36 PM 1/5/2007 +0200, David Fraser wrote:
How would this work if for example, you're using an alternative testing framework (like py.test) for your test? If I understand correctly, py.test simply won't be usable for this scenario, because it assumes tests are files first and modules second, if at all. It's not equipped to discover tests that are inside eggs, which is the whole point of this proposal. Integrating py.test with the "setup.py test" command, on the other hand is possible and relatively simple; just create a unittest-compatible test suite object that wraps py.test, and a test loader class that grabs the necessary arguments from the test_suite argument string. No modifications to setuptools are required, you just provide appropriate values for the test_suite and test_loader arguments to setup().
But the proposal being discussed is for tests that are actually *installed* somewhere, and thus might be in a zipfile. So I need to go and code a test enumerator for py.test to discover tests within eggs (which is possible)
Great, so at least the testing framework could be declared as a dependency For "setup.py test", this is already possible via the 'tests_require' argument to setup(). What's being discussed here is a standard for *installed* tests -- that is, tests that a developer has chosen to make part of their projects's installed code. The existing "setup.py test" subsystem requires a *source* distribution, not installed code. OK great, thanks for clarifying...
David
At 06:05 PM 1/8/2007 +0000, John J Lee wrote:
On Mon, 8 Jan 2007, David Fraser wrote: [...]
So I need to go and code a test enumerator for py.test to discover tests within eggs (which is possible) [...]
Or (for new code) use nose, which AIUI is intended to be pretty much py.test implemented within the unittest framework (plus whatever bits and pieces the nose authors wanted to change, it's true).
I'm not sure whether nose handles test discovery in eggs, but I believe that, like py.test, it has pluggable enumeration capabilities. What would be nice is if we could get some folks working on py.test and nose to chime in on the proposal, to make sure that it's sound. At the moment, the proposal (or at least my suggested version of it) allows for: * specification of one "loader" for all tests in an egg * specification of zero or more entry points for locating tests The only way to provide any configuration to the loader is via attributes of the entry points. For example, if the entry point is a module, you could set your options location via attributes of the module. The loader must support a .loadTestFromName(name, obj) call, where name will be an empty string, and obj will be the object specified by a test entry point. The return value must be a unittest.TestSuite. And it must be possible to call the loader multiple times with different 'obj' values.
Phillip J. Eby wrote:
At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote:
Phillip J. Eby wrote: ...
Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project.
It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too.
And it's enormously repetitive if you have a lot of projects.
Besides, I expect most projects will have a single test suite (function) that could just be named directly.
You expect wrongly. :)
I can only speak based on my own experience. Projects like Zope and ZODB certainly want some sort of tool to find and assemble tests. Eggs are allowing and encouraging us to move away from such large projects. As we break up Zope into individual packages, I'm finding that the individual packages have single test suites.
'nose' and 'py.test' are popular *precisely because* they do NOT require this. Setuptools changed to emulate them by adding ScanningLoader, even though it originally did the "test suite function" thing you're proposing.
See, I originally wrote all my test suites the way you're proposing, but I changed because this is one of those "convention beats configuration" situations. It's easier to just conform to a policy and have a tool that applies the policy. Different projects may have different policies, so providing an option for the loader, allows them to follow it. (And it made my life easier, too, as I began doing more small projects instead of a few large ones.)
And, if you want your approach to be widely adopted, it would be best to allow projects to follow their own policies without requiring them to add any code, even if it's a one-liner.
They'll be adding code one way or another. Either they add it to their test modules, or they add it to setup.py. The advantage of the former is that it provides more flexibility without requiring any more work. It is also more explicit.
So, *technically*, there may be no reason to do this, but from a usability, friendliness, compatiblity, marketing, adoption, etc. format I don't think the standard will be successful without allowing this out. People often have heavy psychological investment in existing tools, and it is hundreds of times easier to rationalize adding a line of configuration to setup() than it is to rationalize adding two or three lines of code. One is merely a packaging change, the other is a programming change.
We disagree. I don't have any problem with having tools that can be used to find and assemble tests into a test suite. I find the unittest loader framework to be baroque. I'd rather let people use whatever framework they want to do this and to control this from Python. I withdraw the proposal. I wouldn't object to the proposal you want, if you want to write it. I don't have the time or interest to write it myself. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
At 10:14 AM 1/22/2007 -0500, Jim Fulton wrote:
I'd rather let people use whatever framework they want to do this and to control this from Python.
The approach I suggested certainly allows that to happen, and if the default loader is either unittest's default loader or setuptools' scanning loader, then it works for Zope's use cases without any change from the way you'd like it. I just want it to *also* be possible for people to specify a non-default suite loader. People who are using non-default loaders now, already have them specified in their setup.py. If this were part of the new spec, they would need only to change (or duplicate) the test_loader and test_suite to be entry points instead of setup() keywords, and they would then work with the test declaration system. This is a much lower barrier to entry in both work needed to switch over, and (IMO) in psychological objection to doing so.
Phillip J. Eby wrote:
At 10:14 AM 1/22/2007 -0500, Jim Fulton wrote:
I'd rather let people use whatever framework they want to do this and to control this from Python.
The approach I suggested certainly allows that to happen,
I know. That's why I wouldn't object to it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Mon, 8 Jan 2007, David Fraser wrote: [...]
So I need to go and code a test enumerator for py.test to discover tests within eggs (which is possible) [...]
Or (for new code) use nose, which AIUI is intended to be pretty much py.test implemented within the unittest framework (plus whatever bits and pieces the nose authors wanted to change, it's true). John
participants (4)
-
David Fraser -
Jim Fulton -
John J Lee -
Phillip J. Eby