[Distutils] RFC: Standard Declaration of tests in eggs

[Jim Fulton]

Phillip J. Eby wrote:
> At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:
> 
>> Here is a rough draft proposal for declaring tests in eggs:
>>
>> Introduction
>> ============
>>
>> Software packages should have automated tests.  Consumers of
>> packages will often want to run these tests.  Tools should be able to
>> do this automatically.  This proposal seeks to provide a way for
>> automated tools to discover tests in distributions, including eggs, so
>> that tests can be run or so that test runners can be automatically
>> created to run the tests.
>>
>> Proposal
>> ========
>>
>> This proposal aims to be extremely simple.  It has 2 parts:
>>
>> 1. A 'test_suite' entry point is defined.  An egg can provide zero or
>>     more test_suite entry points.  These entry points will define
>>     callable objects that can be called without arguments and that
>>     return unittest test suites.
>>
>> 2. An optional 'tests' extra is defined.  When creating test runners
>>     or dynamically loading distributions to load tests, any
>>     distributions listed in extra requires for the 'tests' extra shall
>>     be included in the working set for the test runner.
>>
>> Thoughts?
> 
> Point #2 is unnecessary, since individual entry points can list extras 
> in square brackets following the module/attribute information.  When 
> loading an entry point, these extras automatically get require()'d.  
> Conversely, if the test runner wants to manage the loading process, it 
> can simply inspect the entry point object to determine the names of the 
> extras.

Ah, good point.  Man, setuptools is soooooo rich.

> Regarding point #1, I'm not sure this is enought to define what's 
> necessary.  For example, it should perhaps be stated that the entry 
> point must be in code that will be installed by either the distribution 
> itself, or that is included in the code provided by the entry point's 
> extras.

OK

> Also, an egg can't provide more than one entry point with the same name, 
> so rather than a 'test_suite' entry point, we probably want an entry 
> point *group*, perhaps something like 'installed.test_suites'.

That's what I meant to say. I'll clarify.

> Next, there needs to be reasonable support for dynamic test discovery, 
> such as setuptools' own ScanningLoader.  I would suggest that there be 
> an optional entry point for the test loader class to be used to process 
> the other entry points' target objects.  The standard unittest protocol 
> for loadTestsFromName() takes two arguments: a name and an object.  
> Passing an empty string for the name, and the object loaded by the test 
> suite entry point, suffices to enable normal behavior for loaders such 
> as ScanningLoader, and I believe 'nose' as well as any other 
> well-behaved unittest extensions.
 >
> However, I think the spec should try to define what "well-behaved" means 
> in terms of I/O, result reporting, etc.  (Obviously, one requirement is 
> that the test loader must be able to take an empty name string and an 
> object in its loadTestsFromName() method, and return a test suite.)

Why can't an entry point invoke a test loader itself?
This seems much simpler and more straightforward to me.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org