[Distutils] RFC: Standard Declaration of tests in eggs
Phillip J. Eby
pje at telecommunity.com
Fri Jan 5 18:06:12 CET 2007
At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:
>Here is a rough draft proposal for declaring tests in eggs:
>Software packages should have automated tests. Consumers of
>packages will often want to run these tests. Tools should be able to
>do this automatically. This proposal seeks to provide a way for
>automated tools to discover tests in distributions, including eggs, so
>that tests can be run or so that test runners can be automatically
>created to run the tests.
>This proposal aims to be extremely simple. It has 2 parts:
>1. A 'test_suite' entry point is defined. An egg can provide zero or
> more test_suite entry points. These entry points will define
> callable objects that can be called without arguments and that
> return unittest test suites.
>2. An optional 'tests' extra is defined. When creating test runners
> or dynamically loading distributions to load tests, any
> distributions listed in extra requires for the 'tests' extra shall
> be included in the working set for the test runner.
Point #2 is unnecessary, since individual entry points can list extras in
square brackets following the module/attribute information. When loading
an entry point, these extras automatically get require()'d. Conversely, if
the test runner wants to manage the loading process, it can simply inspect
the entry point object to determine the names of the extras.
Regarding point #1, I'm not sure this is enought to define what's
necessary. For example, it should perhaps be stated that the entry point
must be in code that will be installed by either the distribution itself,
or that is included in the code provided by the entry point's extras.
Also, an egg can't provide more than one entry point with the same name, so
rather than a 'test_suite' entry point, we probably want an entry point
*group*, perhaps something like 'installed.test_suites'.
Next, there needs to be reasonable support for dynamic test discovery, such
as setuptools' own ScanningLoader. I would suggest that there be an
optional entry point for the test loader class to be used to process the
other entry points' target objects. The standard unittest protocol for
loadTestsFromName() takes two arguments: a name and an object. Passing an
empty string for the name, and the object loaded by the test suite entry
point, suffices to enable normal behavior for loaders such as
ScanningLoader, and I believe 'nose' as well as any other well-behaved
However, I think the spec should try to define what "well-behaved" means in
terms of I/O, result reporting, etc. (Obviously, one requirement is that
the test loader must be able to take an empty name string and an object in
its loadTestsFromName() method, and return a test suite.)
More information about the Distutils-SIG