Official site-packages/test directory
Hello, I wonder if we could get an official site-packages/test directory. Currently it seems to be problematic to distribute tests if they are outside the package directory. Here is a nice overview of the two main layout possibilities: http://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/directory_stru... I like the outside-the-package approach, mostly for reasons described very eloquently here: http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_t... CPython itself of course also uses Lib/foo.py and Lib/test/test_foo.py, so it would make sense to have site-packages/foo.py and site-packages/test/test_foo.py. For me, this is the natural layout. Stefan Krah
IIUC another common layout is to have folders named test or tests inside each package. This would avoid requiring any changes to the site-packages layout. On Fri, Jan 19, 2018 at 6:27 AM, Stefan Krah <stefan@bytereef.org> wrote:
-- --Guido van Rossum (python.org/~guido)
Another common approach is to not ship tests as part of your (runtime) package at all - they are in the sdist but not the wheels nor are they deployed with "setup.py install". In my experience, this is the usual approach projects take if they don't have the tests in the package directory. (I don't think I've *ever* seen a project try to install tests except by including them in the package directory...) Paul On 19 January 2018 at 16:10, Guido van Rossum <guido@python.org> wrote:
On Fri, Jan 19, 2018 at 04:23:23PM +0000, Paul Moore wrote:
Yes, given the current situation not shipping is definitely the best approach in that case. I just thought that if we did have something like site-packages/stest (Guido correctly noted that "test" wouldn't work), people might use it. But it is all very speculative and I'm not really sure myself. Stefan Krah
On 19 January 2018 at 17:08, Stefan Krah <stefan@bytereef.org> wrote:
To be usable, tools like pip, wheel, setuptools, flit, etc, would all need to be updated to take into account this option, as well as the relevant standards (the wheel spec for one). Add to that the changes needed to places like the sysconfig package to allow introspecting the location of the new test directory. Would there be a test directory in user-site as well? What about in virtual environments? (If only in site-packages, then it'll likely be read-only in a lot of environments). Also, would we need to reserve the directory name chosen to prohibit 3rd party packages using it? As we've seen the stdlib test package clashes with the original proposal, who's to say there's nothing on PyPI that uses stest? The idea isn't a bad one in principle - there's a proposal from some time back on distutils-sig that Python packaging support more "target locations" matching the POSIX style locations - for docs, config, etc. A test directory would fit in with this idea. But it's a pretty big change in practice, and no-one has yet done much beyond talk about it. And the proposal would likely have put the test directory *outside* site-packages, which avoids the name clash problem. I'd think that the idea of a site-packages/stest directory would need a much more compelling use case to justify it. Paul PS There's nothing stopping a (distribution) package FOO from installing (Python) packages foo and foo-tests. It's not common, and probably violates people's expectations, but it's not *illegal* (the setuptools distribution installs pkg_resources as well as setuptools, for a well-known example). So in theory, if people wanted this enough, they could have implemented it right now, without needing any change to Python or the packaging ecosystem.
On Fri, Jan 19, 2018 at 05:30:43PM +0000, Paul Moore wrote: [cut]
I'd think that the idea of a site-packages/stest directory would need a much more compelling use case to justify it.
Thanks for the detailed explanation! It sounds that there's much more work involved than I thought, so it's probably better to drop this proposal.
If people don't come with pitchforks, that's a good solution. I suspected that people would complain both if foo-tests were installed automatically like pkg_resources but also if foo-tests were a separate optional package (too much hassle). Stefan Krah
hmm, I've struggled for ages with this problem -- I have some packages with REALLY big test suites. so I don't put the tests in the package. But their are also numerous issues with building and installing the package (C code, lots of dependencies, etc), so it would be really nice to have a way to test the actual installed package after the fact (and, be able to properly test conda packages as well -- easy if the tests are in the package, hard if not...) So I like the idea of having a standard way / place to install tests. However, somehow I never thought to make a my_package_tests package -- d'uh! seems the obvious way to handle teh "optionally install the tests" problem. I still like the idea of separate location, but Paul is right that it's a change that would have to filter out through a lot of infrastructure, so maybe not practical. So maybe the way to go is to come up with recommendations for a standard way to do it -- maybe published by PyPa? -CHB On Fri, Jan 19, 2018 at 10:19 AM, Stefan Krah <stefan@bytereef.org> wrote:
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 20 January 2018 at 04:39, Chris Barker <chris.barker@noaa.gov> wrote:
So maybe the way to go is to come up with recommendations for a standard way to do it -- maybe published by PyPa?
I don't think the trade-offs here are clear enough for us to add an opinionated guide to packaging.python.org, but it could be an appropriate topic for a discussion page (e.g. "Publishing Test Suites") that frames the problem, and lays out some of the options for handling it: - tests published as part of the package (generally not ideal, but can make post-install testing easier) - tests in the sdist only (most suitable for unit tests) - tests published as a separate package (often suitable for integration tests) One of the things that makes the 3rd option a bit awkward currently is that a lot of tools assume "1 repo -> 1 sdist", so splitting your test suite out to its own sdist can be fairly annoying in practice. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 January 2018 at 18:19, Stefan Krah <stefan@bytereef.org> wrote:
Personally, I prefer packages that don't install their tests (I'm just about willing to tolerate the tests-inside-the package-approach) so I actually dislike this option myself - I was just saying it's possible. Paul
FWIW, I've had very good experience with putting tests for package `foo` in a directory/package called `test_foo`. This combines the best of both worlds -- it can be easily separated for distribution (like `tests`), and it doesn't cause name conflicts (like `foo.tests`). On 01/19/2018 05:23 PM, Paul Moore wrote:
On Fri, Jan 19, 2018 at 5:23 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I personally include them in psutil distribution so that users can test the installation with "python -m psutil.test". I have even this documented as I think it's an added value. -- Giampaolo - http://grodola.blogspot.com
On Fri, Jan 19, 2018 at 11:21 AM, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
or: pytest --pyargs pkg_name It Is really handy, and sometimes required to test the distribution / installation itself. So I do that most of the time these days -- but it gets ugly if the tests get really huge. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Guido van Rossum wrote:
That's what I do for all my personal code. Yes, it means the test directories are shipped with the sdist, but really who cares? I don't think I've had a single complaint about it, even with large-ish projects like Mailman. I can see you wanting to do something different if your project has truly gargantuan test suites, but even with 100% coverage (or nearly so), I think size just isn't usually a big deal. In another message, Giampaolo describes being able to run tests with -m psutil.test. That's a neat idea which I haven't tried. But I do think including the tests can be instructive, and I know that on more than one occasion, I've cracked open a project's test suite to get a better sense of the semantics and usage of a particular API. Finally, I'll disagree with pytest's recommendation to not put __init__.py files in your test directories. Although I'm not a heavy pytest user (we use it exclusive at work, but I don't use it much with my own stuff), having __init__.py files can be useful, especially if you also have test data you want to access through pkg_resources, or now, importlib_resources (importlib.resources in Python 3.7). Cheers, -Barry
Hi Barry, Sometimes, I need to read the tests of a package because I don't understand the usage of a function/method/class and unfortunately, there is no documentation. In this case, and only in this case, I will try to find the tests and in the worst case, download the source and try to understand with the 'tests' directory. my 0.000002 € ;-) Stephane On 01/26, Barry Warsaw wrote:
-- Stéphane Wirtel - http://wirtel.be - @matrixise
On 01/19/2018 03:27 PM, Stefan Krah wrote:
I think that's a really nice idea. With an official site-packages/test directory there could be pip support for optionally installing tests alongside a package if its layout allows it. So end users could just install things without tests, but developers could do: pip install <package> --with-tests or something to get everything? Wolfgang
On Fri, Jan 19, 2018 at 8:30 AM, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
Oh, I just realized there's another problem here. The existing 'test' package (which is not a namespace package) would hide the site-packages/test directory. -- --Guido van Rossum (python.org/~guido)
On 01/19/2018 05:48 PM, Guido van Rossum wrote:
Well, that shouldn't be a big obstacle since one could just as well choose another name ( __tests__ for example?). Alternatively, package-specific test directories could exist *inside* site-packages. So much like today's <package>.dist-info directories there could be <package>.test dirs?
IIUC another common layout is to have folders named test or tests inside each package. This would avoid requiring any changes to the site-packages layout. On Fri, Jan 19, 2018 at 6:27 AM, Stefan Krah <stefan@bytereef.org> wrote:
-- --Guido van Rossum (python.org/~guido)
Another common approach is to not ship tests as part of your (runtime) package at all - they are in the sdist but not the wheels nor are they deployed with "setup.py install". In my experience, this is the usual approach projects take if they don't have the tests in the package directory. (I don't think I've *ever* seen a project try to install tests except by including them in the package directory...) Paul On 19 January 2018 at 16:10, Guido van Rossum <guido@python.org> wrote:
On Fri, Jan 19, 2018 at 04:23:23PM +0000, Paul Moore wrote:
Yes, given the current situation not shipping is definitely the best approach in that case. I just thought that if we did have something like site-packages/stest (Guido correctly noted that "test" wouldn't work), people might use it. But it is all very speculative and I'm not really sure myself. Stefan Krah
On 19 January 2018 at 17:08, Stefan Krah <stefan@bytereef.org> wrote:
To be usable, tools like pip, wheel, setuptools, flit, etc, would all need to be updated to take into account this option, as well as the relevant standards (the wheel spec for one). Add to that the changes needed to places like the sysconfig package to allow introspecting the location of the new test directory. Would there be a test directory in user-site as well? What about in virtual environments? (If only in site-packages, then it'll likely be read-only in a lot of environments). Also, would we need to reserve the directory name chosen to prohibit 3rd party packages using it? As we've seen the stdlib test package clashes with the original proposal, who's to say there's nothing on PyPI that uses stest? The idea isn't a bad one in principle - there's a proposal from some time back on distutils-sig that Python packaging support more "target locations" matching the POSIX style locations - for docs, config, etc. A test directory would fit in with this idea. But it's a pretty big change in practice, and no-one has yet done much beyond talk about it. And the proposal would likely have put the test directory *outside* site-packages, which avoids the name clash problem. I'd think that the idea of a site-packages/stest directory would need a much more compelling use case to justify it. Paul PS There's nothing stopping a (distribution) package FOO from installing (Python) packages foo and foo-tests. It's not common, and probably violates people's expectations, but it's not *illegal* (the setuptools distribution installs pkg_resources as well as setuptools, for a well-known example). So in theory, if people wanted this enough, they could have implemented it right now, without needing any change to Python or the packaging ecosystem.
On Fri, Jan 19, 2018 at 05:30:43PM +0000, Paul Moore wrote: [cut]
I'd think that the idea of a site-packages/stest directory would need a much more compelling use case to justify it.
Thanks for the detailed explanation! It sounds that there's much more work involved than I thought, so it's probably better to drop this proposal.
If people don't come with pitchforks, that's a good solution. I suspected that people would complain both if foo-tests were installed automatically like pkg_resources but also if foo-tests were a separate optional package (too much hassle). Stefan Krah
hmm, I've struggled for ages with this problem -- I have some packages with REALLY big test suites. so I don't put the tests in the package. But their are also numerous issues with building and installing the package (C code, lots of dependencies, etc), so it would be really nice to have a way to test the actual installed package after the fact (and, be able to properly test conda packages as well -- easy if the tests are in the package, hard if not...) So I like the idea of having a standard way / place to install tests. However, somehow I never thought to make a my_package_tests package -- d'uh! seems the obvious way to handle teh "optionally install the tests" problem. I still like the idea of separate location, but Paul is right that it's a change that would have to filter out through a lot of infrastructure, so maybe not practical. So maybe the way to go is to come up with recommendations for a standard way to do it -- maybe published by PyPa? -CHB On Fri, Jan 19, 2018 at 10:19 AM, Stefan Krah <stefan@bytereef.org> wrote:
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 20 January 2018 at 04:39, Chris Barker <chris.barker@noaa.gov> wrote:
So maybe the way to go is to come up with recommendations for a standard way to do it -- maybe published by PyPa?
I don't think the trade-offs here are clear enough for us to add an opinionated guide to packaging.python.org, but it could be an appropriate topic for a discussion page (e.g. "Publishing Test Suites") that frames the problem, and lays out some of the options for handling it: - tests published as part of the package (generally not ideal, but can make post-install testing easier) - tests in the sdist only (most suitable for unit tests) - tests published as a separate package (often suitable for integration tests) One of the things that makes the 3rd option a bit awkward currently is that a lot of tools assume "1 repo -> 1 sdist", so splitting your test suite out to its own sdist can be fairly annoying in practice. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 January 2018 at 18:19, Stefan Krah <stefan@bytereef.org> wrote:
Personally, I prefer packages that don't install their tests (I'm just about willing to tolerate the tests-inside-the package-approach) so I actually dislike this option myself - I was just saying it's possible. Paul
FWIW, I've had very good experience with putting tests for package `foo` in a directory/package called `test_foo`. This combines the best of both worlds -- it can be easily separated for distribution (like `tests`), and it doesn't cause name conflicts (like `foo.tests`). On 01/19/2018 05:23 PM, Paul Moore wrote:
On Fri, Jan 19, 2018 at 5:23 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I personally include them in psutil distribution so that users can test the installation with "python -m psutil.test". I have even this documented as I think it's an added value. -- Giampaolo - http://grodola.blogspot.com
On Fri, Jan 19, 2018 at 11:21 AM, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
or: pytest --pyargs pkg_name It Is really handy, and sometimes required to test the distribution / installation itself. So I do that most of the time these days -- but it gets ugly if the tests get really huge. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Guido van Rossum wrote:
That's what I do for all my personal code. Yes, it means the test directories are shipped with the sdist, but really who cares? I don't think I've had a single complaint about it, even with large-ish projects like Mailman. I can see you wanting to do something different if your project has truly gargantuan test suites, but even with 100% coverage (or nearly so), I think size just isn't usually a big deal. In another message, Giampaolo describes being able to run tests with -m psutil.test. That's a neat idea which I haven't tried. But I do think including the tests can be instructive, and I know that on more than one occasion, I've cracked open a project's test suite to get a better sense of the semantics and usage of a particular API. Finally, I'll disagree with pytest's recommendation to not put __init__.py files in your test directories. Although I'm not a heavy pytest user (we use it exclusive at work, but I don't use it much with my own stuff), having __init__.py files can be useful, especially if you also have test data you want to access through pkg_resources, or now, importlib_resources (importlib.resources in Python 3.7). Cheers, -Barry
Hi Barry, Sometimes, I need to read the tests of a package because I don't understand the usage of a function/method/class and unfortunately, there is no documentation. In this case, and only in this case, I will try to find the tests and in the worst case, download the source and try to understand with the 'tests' directory. my 0.000002 € ;-) Stephane On 01/26, Barry Warsaw wrote:
-- Stéphane Wirtel - http://wirtel.be - @matrixise
On 01/19/2018 03:27 PM, Stefan Krah wrote:
I think that's a really nice idea. With an official site-packages/test directory there could be pip support for optionally installing tests alongside a package if its layout allows it. So end users could just install things without tests, but developers could do: pip install <package> --with-tests or something to get everything? Wolfgang
On Fri, Jan 19, 2018 at 8:30 AM, Wolfgang Maier < wolfgang.maier@biologie.uni-freiburg.de> wrote:
Oh, I just realized there's another problem here. The existing 'test' package (which is not a namespace package) would hide the site-packages/test directory. -- --Guido van Rossum (python.org/~guido)
On 01/19/2018 05:48 PM, Guido van Rossum wrote:
Well, that shouldn't be a big obstacle since one could just as well choose another name ( __tests__ for example?). Alternatively, package-specific test directories could exist *inside* site-packages. So much like today's <package>.dist-info directories there could be <package>.test dirs?
participants (10)
-
Barry Warsaw -
Chris Barker -
Giampaolo Rodola' -
Guido van Rossum -
Nick Coghlan -
Paul Moore -
Petr Viktorin -
Stefan Krah -
Stephane Wirtel -
Wolfgang Maier