[Python-ideas] Adding a test discovery into Python

[Guido van Rossum]

On Sun, Feb 1, 2009 at 6:14 PM, Brett Cannon <brett at python.org> wrote:
> On Sun, Feb 1, 2009 at 14:40, Guido van Rossum <guido at python.org> wrote:
>> On Sun, Feb 1, 2009 at 2:29 PM, Christian Heimes <lists at cheimes.de> wrote:
>>> Guilherme Polo schrieb:
>>>> So.. is there any chance we can enter in agreement what features would
>>>> be useful in a test discovery that could be included with Python ? I
>>>> for myself do not have fancy wishes for this one, I would be happy
>>>> with something that would collect unittests, inside packages and
>>>> subpackages, with some fixed patterns and let me run them with
>>>> test.test_support.run_unittests, or maybe something that would collect
>>>> unittests and doctests and have something like run_tests in
>>>> test.test_support. But then I believe this wouldn't be good enough to
>>>> substitute any of the current tools, making the addition mostly
>>>> useless.
>>>
>>> I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple
>>> because there are enough frameworks for elaborate unit testing.
>>>
>>> Such a tool should
>>>
>>> - find all modules and packages named 'tests' for a given package name
>>
>> I predict that this part is where you'll have a hard time getting
>> consensus. There are lots of different naming conventions. It would be
>> nice if people could use the new discovery feature without having to
>> move all their tests around.
>>
>
> Guido is right; this cannot be convention-based but instead
> configuration-based. Let me specify in a function call what package to
> look in and what naming convention I used for test files.
>
>>> - load all subclasses of unittest.TestCase from 'tests' module or
>>>  '*/tests/test*.py' files
>>
>> Once you've found the test modules, TestCase subclasses are a good
>> place to start. Though beware -- there's a pattern that defines
>> several alternative classes in a module, e.g. one per platform, and
>> then defines a test suite that dynamically decides which class to use.
>> The stdlib test suite uses this in a number of places, though I can't
>> quite remember where.
>>
>
> Really? I have never come across that before in the standard library,
> but I am sure people do stuff like that.

I'm 100% positive I've seen it. 95% sure it was in the stdlib, but it
could have been somewhere else.

> There is no way any tool will work in all pre-existing situations like
> this where people heavily rely on a test_main function to handle
> decisions on what to call. But as long as it is made clear how to
> restrict what tests are found and it is an easy solution I am sure
> people will adjust.

A convention that the presence of e.g. a test_main function overrides
the default in-module discovery would help too.

>>> - support some basic filtering for test cases or test functions
>>
>> Or module names.
>>
>
> yes please! To be able to specify only the test being worked on would
> be very nice indeed.

The alternative of excluding tests by pattern would also be handy. As
would be pointing the tool to a specific module or class.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)