[Distutils] RFC: Standard Declaration of tests in eggs

[Phillip J. Eby]

At 12:24 PM 1/5/2007 -0500, Jim Fulton wrote:
>Phillip J. Eby wrote:
>>At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote:
>>>2. An optional 'tests' extra is defined.  When creating test runners
>>>     or dynamically loading distributions to load tests, any
>>>     distributions listed in extra requires for the 'tests' extra shall
>>>     be included in the working set for the test runner.
>>>
>>>Thoughts?
>>Point #2 is unnecessary, since individual entry points can list extras in 
>>square brackets following the module/attribute information.  When loading 
>>an entry point, these extras automatically get require()'d.
>>Conversely, if the test runner wants to manage the loading process, it 
>>can simply inspect the entry point object to determine the names of the extras.
>
>Ah, good point.  Man, setuptools is soooooo rich.

Rich requirements make rich design.  See my recent "Where Zope Leads, 
Python Follows" blog post.  :)


>Why can't an entry point invoke a test loader itself?
>This seems much simpler and more straightforward to me.

Because that requires you to write code for something that can adequately 
be expressed through an existing configuration mechanism.  *And* you have 
to write that code in every project.

If you're using setuptools' ScanningLoader (which is the default for the 
"test" command, and which I believe should be the default for this proposal 
as well), you automatically get all TestCase subclasses in all submodules 
of the package you target, and if any module has an 'additional_tests()' 
function, it's called to get a suite that's added in.  This lets you just 
make your entry point a "whatever.tests" package, without writing any extra 
code (unless you want to throw in some doctests via a few 
'additional_tests()' functions).

See the setuptools documentation for the 'test_suite' and 'test_loader' 
arguments to setup() for the full details.

Oh, and the 'nose' package is a unittest-compatible extended testing 
framework that is designed to also work as a 'test_loader' for 
setuptools.  So, between ScanningLoader and 'nose', that would be two 
already-existing ways of specifying tests that would work with the new 
proposal, if you could specify a loader entry point.  And it would be good 
if the default loader was compatible with ScanningLoader's test discovery 
features.

(I'm not sure if nose really works with eggs, though; ScanningLoader will 
discover tests in eggs as long as the source is included, but nose may be 
strictly file-based for all I know.  Changing it probably wouldn't be too 
difficult, however.)