Location of tests for packages
Right now, the tests for the unittest package are under the package directory instead of Lib/test where we have most of the other tests. There are some other packages that do the same thing, each for their own reason. I think we should develop a strong preference for tests going under Lib/test unless there is a very compelling reason. We already have a similar preference for all Docs going under Doc/ and that has not proved to be an issue with any package maintainer. * The Windows distro has an install option to exclude Lib/test. The currrent situation with unittest works against it. * The commingling of tests with the regular code is making it more difficult to grep code while excluding tests. * Having packages create their little worlds within world is making it more difficult to find things. * For regrtest to work, there still needs to be some file in Lib/test that dispatches to the alternate test directory. This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo. I recommend moving it under Lib/test before everything is set in stone. Raymond P.S. I've discussed this with Michael and his preference is against going back to the Py3.1 style where the tests were under Lib/test. He thinks the current tree makes it easier to sync with Py2.7 and the unittest2 third-party module. Also, he likes grepping the regular source and tests all at once.
Right now, the tests for the unittest package are under the package directory instead of Lib/test where we have most of the other tests.
There are some other packages that do the same thing, each for their own reason.
The corresponding bug report is #10572 (opened by Michael Foord). R. David Murray was +1 for moving email tests, Barry deferred to him, Brett was +0 for importlib, and I was ±0 for distutils. Maintainers of ctypes, json, lib2to3 and sqlite3 haven’t yet expressed themselves. Regards
2011/1/24 Éric Araujo
Right now, the tests for the unittest package are under the package directory instead of Lib/test where we have most of the other tests.
There are some other packages that do the same thing, each for their own reason.
The corresponding bug report is #10572 (opened by Michael Foord).
R. David Murray was +1 for moving email tests, Barry deferred to him, Brett was +0 for importlib, and I was ±0 for distutils. Maintainers of ctypes, json, lib2to3 and sqlite3 haven’t yet expressed themselves.
I prefer lib2to3 tests to stay in lib2to3/. -- Regards, Benjamin
On Mon, Jan 24, 2011 at 2:46 PM, Raymond Hettinger
P.S. I've discussed this with Michael and his preference is against going back to the Py3.1 style where the tests were under Lib/test. He thinks the current tree makes it easier to sync with Py2.7 and the unittest2 third-party module. Also, he likes grepping the regular source and tests all at once.
I'm with Michael on this. -1 on pushing all the tests into Lib/test/. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A storm broke loose in my mind." --Albert Einstein
On Jan 24, 2011, at 11:46 AM, Raymond Hettinger wrote:
P.S. I've discussed this with Michael and his preference is against going back to the Py3.1 style where the tests were under Lib/test. He thinks the current tree makes it easier to sync with Py2.7 and the unittest2 third-party module. Also, he likes grepping the regular source and tests all at once.
Which seem like compelling reasons to keep things the way they are for unittest, in addition to the fact that we're already in RC for 3.2, so you would need RM approval to make such a change this late in the process. I agree that it's not ideal, but for certain packages that are also distributed separately, it can be much easier to keep the tests with the code, and I'm inclined to defer to the primary maintainer's preference. -Barry
On Mon, 24 Jan 2011 11:46:45 -0800
Raymond Hettinger
This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo.
Well can we stop being melodramatic? Tests are not part of the API and so they are free to move whenever we want. No need to hold a release candidate for that. Regards Antoine.
This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo.
Well can we stop being melodramatic? Tests are not part of the API and so they are free to move whenever we want. No need to hold a release candidate for that.
Of course there is. Any addition or removal of files at this point has the chance of breaking the release process, which may fail to pick up files, or break in trying to pick up files that it expected to be there. This has happened *many* times during the alpha and beta releases of 3.2, so it's not at all a theoretical problem. After the next release candidate, I'd prefer to see no changes whatsoever to the tree (but it's Georg's decision, of course). Regards, Martin
Le mardi 25 janvier 2011 à 00:14 +0100, "Martin v. Löwis" a écrit :
This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo.
Well can we stop being melodramatic? Tests are not part of the API and so they are free to move whenever we want. No need to hold a release candidate for that.
Of course there is. Any addition or removal of files at this point has the chance of breaking the release process, which may fail to pick up files, or break in trying to pick up files that it expected to be there. This has happened *many* times during the alpha and beta releases of 3.2, so it's not at all a theoretical problem.
My point was that these changes can take place after 3.2 (both final and rc). Regards Antoine.
On 24/01/2011 23:14, "Martin v. Löwis" wrote:
This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo. Well can we stop being melodramatic? Tests are not part of the API and so they are free to move whenever we want. No need to hold a release candidate for that. Of course there is. Any addition or removal of files at this point has the chance of breaking the release process, which may fail to pick up files, or break in trying to pick up files that it expected to be there. This has happened *many* times during the alpha and beta releases of 3.2, so it's not at all a theoretical problem.
After the next release candidate, I'd prefer to see no changes whatsoever to the tree (but it's Georg's decision, of course).
What Antoine meant is that we could make the change for 3.2.1 and don't need to delay 3.2. Michael
Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
Am 25.01.2011 00:14, schrieb "Martin v. Löwis":
This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo.
Well can we stop being melodramatic? Tests are not part of the API and so they are free to move whenever we want. No need to hold a release candidate for that.
Yes, let's postpone this for after the final release.
Of course there is. Any addition or removal of files at this point has the chance of breaking the release process, which may fail to pick up files, or break in trying to pick up files that it expected to be there. This has happened *many* times during the alpha and beta releases of 3.2, so it's not at all a theoretical problem.
After the next release candidate, I'd prefer to see no changes whatsoever to the tree (but it's Georg's decision, of course).
I agree with both of you. Ideally there shouldn't be any but cosmetic changes after rc2, otherwise I'd be inclined to add an rc3 to the release schedule. Georg
On 24/01/2011 19:46, Raymond Hettinger wrote:
Right now, the tests for the unittest package are under the package directory instead of Lib/test where we have most of the other tests.
There are some other packages that do the same thing, each for their own reason.
I think we should develop a strong preference for tests going under Lib/test unless there is a very compelling reason. We already have a similar preference for all Docs going under Doc/ and that has not proved to be an issue with any package maintainer.
* The Windows distro has an install option to exclude Lib/test. The currrent situation with unittest works against it. * The commingling of tests with the regular code is making it more difficult to grep code while excluding tests. * Having packages create their little worlds within world is making it more difficult to find things. * For regrtest to work, there still needs to be some file in Lib/test that dispatches to the alternate test directory.
This isn't a critical issue (nothing is broken) but we're a week from another release candidate, so the new Py3.2 package organization (unittest was flat in Py3.1 and its test were under Lib/test) is about to become a de-facto decision that will be hard to undo.
The tests are already under unittest in 2.7 so that change isn't "new". Moving the tests now makes it harder to maintain them (patches to 3.2 won't apply to 2.7). This is discussed in issue 10572. http://bugs.python.org/issue10572 It isn't just unittest, it seems that all *test packages* are in their respective package and not Lib/test except for the json module where Raymond already moved the tests: distutils/tests email/test ctypes/test importlib/test lib2to3/tests sqlite3/test tkinter/test So I'm a little confused as to why the focus on the *unittest* test suite. Brett has expressed a willingness to move the importlib tests under Lib/test and R David Murray would *like* to move the email tests there (but hasn't). Barry is -0 and so am I. It generally makes a few things slightly harder for me but not much. If we make a general policy decision to move all package tests out of their packages and into Lib/test (and actually do it) then fine, but I'm not overjoyed with a unilateral decision that unittest is special in this regard... :-) All the best, Michael
I recommend moving it under Lib/test before everything is set in stone.
Raymond
P.S. I've discussed this with Michael and his preference is against going back to the Py3.1 style where the tests were under Lib/test. He thinks the current tree makes it easier to sync with Py2.7 and the unittest2 third-party module. Also, he likes grepping the regular source and tests all at once. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Jan 24, 2011, at 3:40 PM, Michael Foord wrote:
It isn't just unittest, it seems that all *test packages* are in their respective package and not Lib/test except for the json module where Raymond already moved the tests:
distutils/tests email/test ctypes/test importlib/test lib2to3/tests sqlite3/test tkinter/test
So I'm a little confused as to why the focus on the *unittest* test suite.
There's not a focus on unittest. Importlib should also move under Lib/test and when email is ready, it too should fully join the organization of the overall project (Doc, Lib, Lib/test, Modules, Objects, Tools). ISTM, ctypes and disutils could almost be viewed as separate projects. We could ship Python without ctypes for example and we've got a policy against implementing the rest of library using ctypes. The same goes for tkinter (it is not uncommon to have builds with it). And sqlite3 is close to being completely third-party maintained. In contrast, the unittest module and importlib belong with the core distro. So, I'm thinking that there were some precedents in cases where there was a really good reason for separating the project (we don't even include tkinter docs in our doc build), but that we should maintain a strong preference for keeping the overall project organization intact. ElementTree was fully folded into the project. I think we should follow that precedent and avoid balkanizing the python source into many little project subtrees (worlds within a world). Raymond
Am 25.01.2011 02:19, schrieb Raymond Hettinger:
On Jan 24, 2011, at 3:40 PM, Michael Foord wrote:
It isn't just unittest, it seems that all *test packages* are in their respective package and not Lib/test except for the json module where Raymond already moved the tests:
distutils/tests email/test ctypes/test importlib/test lib2to3/tests sqlite3/test tkinter/test
So I'm a little confused as to why the focus on the *unittest* test suite.
There's not a focus on unittest. Importlib should also move under Lib/test and when email is ready, it too should fully join the organization of the overall project (Doc, Lib, Lib/test, Modules, Objects, Tools).
I'm +0 on moving all tests under Lib/test -- I think the respective maintainers of the libraries in question should have the final word, because...
ISTM, ctypes and disutils could almost be viewed as separate projects. We could ship Python without ctypes for example and we've got a policy against implementing the rest of library using ctypes. The same goes for tkinter (it is not uncommon to have builds with it). And sqlite3 is close to being completely third-party maintained.
this weakens the argument of having a consistent organization of test modules: if one or two are allowed to have the test suite intra-package, it doesn't matter so much any more for others. Georg
On Mon, Jan 24, 2011 at 17:19, Raymond Hettinger
On Jan 24, 2011, at 3:40 PM, Michael Foord wrote:
It isn't just unittest, it seems that all *test packages* are in their respective package and not Lib/test except for the json module where Raymond already moved the tests:
distutils/tests email/test ctypes/test importlib/test lib2to3/tests sqlite3/test tkinter/test
So I'm a little confused as to why the focus on the *unittest* test suite.
There's not a focus on unittest. Importlib should also move under Lib/test and when email is ready, it too should fully join the organization of the overall project (Doc, Lib, Lib/test, Modules, Objects, Tools).
Just to clarify my position since importlib keeps getting brought up as an example, I'm fine with a move but I won't be putting the work in to do the move if there is actually consensus to make this a stdlib-wide policy. And I am assuming that the directory will be moved wholesale to Lib/test/importlib (with proper fixes for any relative imports) along with verification that importlib.test.__main__ continues to work (naming it test.importlib_tests seems rather redundant compared to test.importlib). While I'm for consistency, obviously a trend was started by ctypes and sqlite3 that the rest of us who created full packages followed up to this point. If we move some modules and not others purely because some distros choose not to ship e.g., ctypes and sqlite3, that will get annoying w/o some very clear explanation/delineation as to why some packages have a special rule to follow (I'm guessing "packages that have external dependencies" would be it).
On Tue, Jan 25, 2011 at 12:38 PM, Brett Cannon
.. If we move some modules and not others purely because some distros choose not to ship e.g., ctypes and sqlite3
I don't see why this is a problem. Regrtest already has a mechanism that allows skipping tests based on various criteria. This mechanism works for both packages and flat modules that can be optionally installed. FWIW, I am +0 on consolidating tests under Lib/test. One of the reasons that I have not seen mentioned is that it is well-known that test package is not part of the official stdlib API and can be changes/restructured in backward incompatible ways. It is not obvious whether the same applies to say lib2to3.tests or ctypes.test. If you are interested to see what it takes to move tests from a package, I moved json tests to Lib/test/json_tests in r86875. It is not hard, but does require some changes to imports.
On Wed, Jan 26, 2011 at 4:16 AM, Alexander Belopolsky
FWIW, I am +0 on consolidating tests under Lib/test. One of the reasons that I have not seen mentioned is that it is well-known that test package is not part of the official stdlib API and can be changes/restructured in backward incompatible ways. It is not obvious whether the same applies to say lib2to3.tests or ctypes.test.
I am +0 for the same reason as Alexander. The test subpackages should either be moved under the test package, or, for packages with PyPI distributed backports for previous versions, they should be prefixed with a leading underscore to make it clear that they're private implementation details and backwards compatibility guarantees don't apply. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (12)
-
"Martin v. Löwis" -
Alexander Belopolsky -
Antoine Pitrou -
Barry Warsaw -
Benjamin Peterson -
Brett Cannon -
Fred Drake -
Georg Brandl -
Michael Foord -
Nick Coghlan -
Raymond Hettinger -
Éric Araujo