
Hi, I believe it would be good to include a test discovery into Python, right now I notice all the following packages: bsdbb, ctypes, distutils, email, json, lib2to3 and lib-tk (tkinter in py3k) duplicate some form of test discovery that works for each one of them (there are also sqlite3 tests, but they are not using a real "test discovery"). In the future it is very likely that this "duplication count" increases, since from time to time new modules and packages get into Python. I can also feel the "idlelib" package starting getting tests, just making things worse. External packages would benefit from it too. Right now you either duplicate the test discovery once more (because your project is small enough that you don't want to use something specific for that), or you use nose, trial, py.test or whatever looks better for you. So.. is there any chance we can enter in agreement what features would be useful in a test discovery that could be included with Python ? I for myself do not have fancy wishes for this one, I would be happy with something that would collect unittests, inside packages and subpackages, with some fixed patterns and let me run them with test.test_support.run_unittests, or maybe something that would collect unittests and doctests and have something like run_tests in test.test_support. But then I believe this wouldn't be good enough to substitute any of the current tools, making the addition mostly useless. Thanks for reading, -- -- Guilherme H. Polo Goncalves

On Sun, Feb 1, 2009 at 12:10 PM, Guilherme Polo <ggpolo@gmail.com> wrote:
I think reinventing something which has been implemented ad-nauseam in the community (py.test, nose, etc) would be silly. I'm biased towards nose (really biased) - but I think using the discovery core of one of these might be good. jesse

On Sun, Feb 1, 2009 at 3:21 PM, Jesse Noller <jnoller@gmail.com> wrote:
I didn't want to sound like reinventing code, that is why I put some the word "include" sometimes ;) We are in agreement with using the core of some other project, and I like nose too. -- -- Guilherme H. Polo Goncalves

On Sun, Feb 1, 2009 at 09:10, Guilherme Polo <ggpolo@gmail.com> wrote:
... and importlib.
Yep. I want to be able to state "find the tests in this package and search the entire package top-down looking for tests to run". The only trick is if you store the tests in something other than on the file system directly (e.g. zipfile). That would require a little bit more work like having modules listed in the __all__ value of the package and use that to know which modules to look in. -Brett -Brett

On Sun, Feb 1, 2009 at 6:46 PM, Brett Cannon <brett@python.org> wrote:
My bad, I was even looking at it other day but when I compiled this list of packages I looked only in trunk and forgot about the py3k branch.
To the agreement question ?
This made me remember about another two features I think would be useful to have. One of these features would allow me to restrict from which packages I want to collect tests, this is useful even in "flat packages" (no sub-packages) like tkinter where some of its modules possibly depend on external packages in order to be tested/executed. This adds the possibility to skip tests of single modules instead of entire packages. Other one would be able to check the requirements of a test and decide to collect it or not based on some parameters given. So I could collect all the tests that require a GUI, and others that do not, for example. This adds the possibility to skip single tests, or at least the possibility to divide the tests before running.
-Brett
-- -- Guilherme H. Polo Goncalves

Guilherme Polo schrieb:
I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple because there are enough frameworks for elaborate unit testing. Such a tool should - find all modules and packages named 'tests' for a given package name - load all subclasses of unittest.TestCase from 'tests' module or '*/tests/test*.py' files - support some basic filtering for test cases or test functions Do we need more features? Christian

On Sun, Feb 1, 2009 at 2:29 PM, Christian Heimes <lists@cheimes.de> wrote:
I predict that this part is where you'll have a hard time getting consensus. There are lots of different naming conventions. It would be nice if people could use the new discovery feature without having to move all their tests around.
- load all subclasses of unittest.TestCase from 'tests' module or '*/tests/test*.py' files
Once you've found the test modules, TestCase subclasses are a good place to start. Though beware -- there's a pattern that defines several alternative classes in a module, e.g. one per platform, and then defines a test suite that dynamically decides which class to use. The stdlib test suite uses this in a number of places, though I can't quite remember where.
- support some basic filtering for test cases or test functions
Or module names.
Do we need more features?
I'd look at some of the popular alternatives to unittest.py and see what wisdom or conventions they have to offer. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Feb 1, 2009 at 14:40, Guido van Rossum <guido@python.org> wrote:
Guido is right; this cannot be convention-based but instead configuration-based. Let me specify in a function call what package to look in and what naming convention I used for test files.
Really? I have never come across that before in the standard library, but I am sure people do stuff like that. There is no way any tool will work in all pre-existing situations like this where people heavily rely on a test_main function to handle decisions on what to call. But as long as it is made clear how to restrict what tests are found and it is an easy solution I am sure people will adjust.
- support some basic filtering for test cases or test functions
Or module names.
yes please! To be able to specify only the test being worked on would be very nice indeed. -Brett

On Sun, Feb 1, 2009 at 6:14 PM, Brett Cannon <brett@python.org> wrote:
I'm 100% positive I've seen it. 95% sure it was in the stdlib, but it could have been somewhere else.
A convention that the presence of e.g. a test_main function overrides the default in-module discovery would help too.
The alternative of excluding tests by pattern would also be handy. As would be pointing the tool to a specific module or class. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Still, there should be one way to do it, so that future projects can start to use a common pattern. I actually think the selection of such a pattern can be completely arbitrary, as it will be impossible to get a clear vote on this. Obviously, the OWTDI does not lift the requirement that the test finder must support alternate patterns to make it work smoothly with existing test suites. It's just meant to avoid the configuration overhead if you do it 'the right way'. Stefan

On Wed, Mar 11, 2009 at 4:49 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
A little unrelated to your reply but thanks for "reviving" the thread. I still have the intention to do the proposed idea, I just happened to have very busy weeks, month, etc.. new house and others.
Stefan
Regards, -- -- Guilherme H. Polo Goncalves

[Christian Heimes]
I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple because there are enough frameworks for elaborate unit testing.
Test discovery is not the interesting part of the problem. I'm strongly for offering tools that make it easier to write the tests in the first place. The syntax used by py.test and nose is vastly superior to the one used by unittest.py, a module that is more Javathonic than Pythonic. Even if we never adopt that syntax for our own test suite (because we like to run tests with and without -O), it would still be a good service to our users to offer a tool with a lighter weight syntax for writing tests. Raymond P.S. I'm not a partisan on this one. I've been a *heavy* user of unittest.py, doctest.py, py.test, and some personal tools that I wrote long ago in awk. Extensive use of each makes merits of the py.test and nose approaches self-evident. Axiom: The more work involved in writing tests, the fewer tests that will get written. Factoid of the Day: In Py2.7's test_datetime module, the phrase self.assertEqual occurs 578 times.

On Wed, Mar 11, 2009 at 11:47 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
That depends on how well all the other tests (those that *don't* use assertEquals) fit in doctest's mold. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Feb 01, 2009, Christian Heimes wrote:
Depends on whether the above features find doctests. In my previous job (got laid off a month ago), I'd say that about eighty percent of test suites were doctests (harder to guess the percentage by line counts), and many of these were substantial. I think that doctests are one of the killer Python features that make it easy to do testing, and I think it's critical that all core testing tools support them. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

On Sun, Feb 1, 2009 at 12:10 PM, Guilherme Polo <ggpolo@gmail.com> wrote:
I think reinventing something which has been implemented ad-nauseam in the community (py.test, nose, etc) would be silly. I'm biased towards nose (really biased) - but I think using the discovery core of one of these might be good. jesse

On Sun, Feb 1, 2009 at 3:21 PM, Jesse Noller <jnoller@gmail.com> wrote:
I didn't want to sound like reinventing code, that is why I put some the word "include" sometimes ;) We are in agreement with using the core of some other project, and I like nose too. -- -- Guilherme H. Polo Goncalves

On Sun, Feb 1, 2009 at 09:10, Guilherme Polo <ggpolo@gmail.com> wrote:
... and importlib.
Yep. I want to be able to state "find the tests in this package and search the entire package top-down looking for tests to run". The only trick is if you store the tests in something other than on the file system directly (e.g. zipfile). That would require a little bit more work like having modules listed in the __all__ value of the package and use that to know which modules to look in. -Brett -Brett

On Sun, Feb 1, 2009 at 6:46 PM, Brett Cannon <brett@python.org> wrote:
My bad, I was even looking at it other day but when I compiled this list of packages I looked only in trunk and forgot about the py3k branch.
To the agreement question ?
This made me remember about another two features I think would be useful to have. One of these features would allow me to restrict from which packages I want to collect tests, this is useful even in "flat packages" (no sub-packages) like tkinter where some of its modules possibly depend on external packages in order to be tested/executed. This adds the possibility to skip tests of single modules instead of entire packages. Other one would be able to check the requirements of a test and decide to collect it or not based on some parameters given. So I could collect all the tests that require a GUI, and others that do not, for example. This adds the possibility to skip single tests, or at least the possibility to divide the tests before running.
-Brett
-- -- Guilherme H. Polo Goncalves

Guilherme Polo schrieb:
I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple because there are enough frameworks for elaborate unit testing. Such a tool should - find all modules and packages named 'tests' for a given package name - load all subclasses of unittest.TestCase from 'tests' module or '*/tests/test*.py' files - support some basic filtering for test cases or test functions Do we need more features? Christian

On Sun, Feb 1, 2009 at 2:29 PM, Christian Heimes <lists@cheimes.de> wrote:
I predict that this part is where you'll have a hard time getting consensus. There are lots of different naming conventions. It would be nice if people could use the new discovery feature without having to move all their tests around.
- load all subclasses of unittest.TestCase from 'tests' module or '*/tests/test*.py' files
Once you've found the test modules, TestCase subclasses are a good place to start. Though beware -- there's a pattern that defines several alternative classes in a module, e.g. one per platform, and then defines a test suite that dynamically decides which class to use. The stdlib test suite uses this in a number of places, though I can't quite remember where.
- support some basic filtering for test cases or test functions
Or module names.
Do we need more features?
I'd look at some of the popular alternatives to unittest.py and see what wisdom or conventions they have to offer. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Feb 1, 2009 at 14:40, Guido van Rossum <guido@python.org> wrote:
Guido is right; this cannot be convention-based but instead configuration-based. Let me specify in a function call what package to look in and what naming convention I used for test files.
Really? I have never come across that before in the standard library, but I am sure people do stuff like that. There is no way any tool will work in all pre-existing situations like this where people heavily rely on a test_main function to handle decisions on what to call. But as long as it is made clear how to restrict what tests are found and it is an easy solution I am sure people will adjust.
- support some basic filtering for test cases or test functions
Or module names.
yes please! To be able to specify only the test being worked on would be very nice indeed. -Brett

On Sun, Feb 1, 2009 at 6:14 PM, Brett Cannon <brett@python.org> wrote:
I'm 100% positive I've seen it. 95% sure it was in the stdlib, but it could have been somewhere else.
A convention that the presence of e.g. a test_main function overrides the default in-module discovery would help too.
The alternative of excluding tests by pattern would also be handy. As would be pointing the tool to a specific module or class. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Still, there should be one way to do it, so that future projects can start to use a common pattern. I actually think the selection of such a pattern can be completely arbitrary, as it will be impossible to get a clear vote on this. Obviously, the OWTDI does not lift the requirement that the test finder must support alternate patterns to make it work smoothly with existing test suites. It's just meant to avoid the configuration overhead if you do it 'the right way'. Stefan

On Wed, Mar 11, 2009 at 4:49 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
A little unrelated to your reply but thanks for "reviving" the thread. I still have the intention to do the proposed idea, I just happened to have very busy weeks, month, etc.. new house and others.
Stefan
Regards, -- -- Guilherme H. Polo Goncalves

[Christian Heimes]
I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple because there are enough frameworks for elaborate unit testing.
Test discovery is not the interesting part of the problem. I'm strongly for offering tools that make it easier to write the tests in the first place. The syntax used by py.test and nose is vastly superior to the one used by unittest.py, a module that is more Javathonic than Pythonic. Even if we never adopt that syntax for our own test suite (because we like to run tests with and without -O), it would still be a good service to our users to offer a tool with a lighter weight syntax for writing tests. Raymond P.S. I'm not a partisan on this one. I've been a *heavy* user of unittest.py, doctest.py, py.test, and some personal tools that I wrote long ago in awk. Extensive use of each makes merits of the py.test and nose approaches self-evident. Axiom: The more work involved in writing tests, the fewer tests that will get written. Factoid of the Day: In Py2.7's test_datetime module, the phrase self.assertEqual occurs 578 times.

On Wed, Mar 11, 2009 at 11:47 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
That depends on how well all the other tests (those that *don't* use assertEquals) fit in doctest's mold. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Feb 01, 2009, Christian Heimes wrote:
Depends on whether the above features find doctests. In my previous job (got laid off a month ago), I'd say that about eighty percent of test suites were doctests (harder to guess the percentage by line counts), and many of these were substantial. I think that doctests are one of the killer Python features that make it easy to do testing, and I think it's critical that all core testing tools support them. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.
participants (8)
-
Aahz
-
Brett Cannon
-
Christian Heimes
-
Guido van Rossum
-
Guilherme Polo
-
Jesse Noller
-
Raymond Hettinger
-
Stefan Behnel