[Distutils] RFC: Standard Declaration of tests in eggs

[Phillip J. Eby]

At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:

>Here is a rough draft proposal for declaring tests in eggs:
>
>Introduction
>============
>
>Software packages should have automated tests.  Consumers of
>packages will often want to run these tests.  Tools should be able to
>do this automatically.  This proposal seeks to provide a way for
>automated tools to discover tests in distributions, including eggs, so
>that tests can be run or so that test runners can be automatically
>created to run the tests.
>
>Proposal
>========
>
>This proposal aims to be extremely simple.  It has 2 parts:
>
>1. A 'test_suite' entry point is defined.  An egg can provide zero or
>     more test_suite entry points.  These entry points will define
>     callable objects that can be called without arguments and that
>     return unittest test suites.
>
>2. An optional 'tests' extra is defined.  When creating test runners
>     or dynamically loading distributions to load tests, any
>     distributions listed in extra requires for the 'tests' extra shall
>     be included in the working set for the test runner.
>
>Thoughts?

Point #2 is unnecessary, since individual entry points can list extras in 
square brackets following the module/attribute information.  When loading 
an entry point, these extras automatically get require()'d.  Conversely, if 
the test runner wants to manage the loading process, it can simply inspect 
the entry point object to determine the names of the extras.

Regarding point #1, I'm not sure this is enought to define what's 
necessary.  For example, it should perhaps be stated that the entry point 
must be in code that will be installed by either the distribution itself, 
or that is included in the code provided by the entry point's extras.

Also, an egg can't provide more than one entry point with the same name, so 
rather than a 'test_suite' entry point, we probably want an entry point 
*group*, perhaps something like 'installed.test_suites'.

Next, there needs to be reasonable support for dynamic test discovery, such 
as setuptools' own ScanningLoader.  I would suggest that there be an 
optional entry point for the test loader class to be used to process the 
other entry points' target objects.  The standard unittest protocol for 
loadTestsFromName() takes two arguments: a name and an object.  Passing an 
empty string for the name, and the object loaded by the test suite entry 
point, suffices to enable normal behavior for loaders such as 
ScanningLoader, and I believe 'nose' as well as any other well-behaved 
unittest extensions.

However, I think the spec should try to define what "well-behaved" means in 
terms of I/O, result reporting, etc.  (Obviously, one requirement is that 
the test loader must be able to take an empty name string and an object in 
its loadTestsFromName() method, and return a test suite.)