tests split off from scipy package in conda-forge
Hi all, This is an FYI that in conda-forge, thanks to the hard work of @h-vetinari [1], the `scipy` package now no longer contains tests. Instead, those are split off into a new `scipy-tests` package which, when you install that, puts all the test files back inside the `scipy` package tree. The gain of that exercise was a ~30% size reduction, which seems worth it. We already had plans to do something similar in SciPy itself, by putting the heavy data files from the test suite under `scipy.datasets`. The gain of that will be less, probably 15-20% of the total size of wheels (excluding vendored openblas et al.). The conda-forge packages are a nice test case - if we receive no complaints from that, it should be pretty safe to move forward with that plan. Cheers, Ralf [1] https://github.com/conda-forge/scipy-feedstock/pull/184
FWIW, I'm in favor of moving large data assets out, but my preference would be to always have the sources of tests in the main repo. The debates about this aren't super interesting/fun, but anyway that's where I land on it, even if the project goes in another direction. Best wishes, Tyler On Sat, 13 May 2023 at 05:07, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi all,
This is an FYI that in conda-forge, thanks to the hard work of @h-vetinari [1], the `scipy` package now no longer contains tests. Instead, those are split off into a new `scipy-tests` package which, when you install that, puts all the test files back inside the `scipy` package tree.
The gain of that exercise was a ~30% size reduction, which seems worth it. We already had plans to do something similar in SciPy itself, by putting the heavy data files from the test suite under `scipy.datasets`. The gain of that will be less, probably 15-20% of the total size of wheels (excluding vendored openblas et al.). The conda-forge packages are a nice test case - if we receive no complaints from that, it should be pretty safe to move forward with that plan.
Cheers, Ralf
[1] https://github.com/conda-forge/scipy-feedstock/pull/184 _______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
I think the common ground might be using python "optional dependencies" [1] such that only interested parties install it and the rest can skip the extra network payload. You can then install "pip/conda install scipy[full]" or whatever the syntax might be. [1] : https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#op... On Sun, May 28, 2023 at 10:00 PM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
FWIW, I'm in favor of moving large data assets out, but my preference would be to always have the sources of tests in the main repo. The debates about this aren't super interesting/fun, but anyway that's where I land on it, even if the project goes in another direction.
Best wishes, Tyler
On Sat, 13 May 2023 at 05:07, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi all,
This is an FYI that in conda-forge, thanks to the hard work of @h-vetinari [1], the `scipy` package now no longer contains tests. Instead, those are split off into a new `scipy-tests` package which, when you install that, puts all the test files back inside the `scipy` package tree.
The gain of that exercise was a ~30% size reduction, which seems worth it. We already had plans to do something similar in SciPy itself, by putting the heavy data files from the test suite under `scipy.datasets`. The gain of that will be less, probably 15-20% of the total size of wheels (excluding vendored openblas et al.). The conda-forge packages are a nice test case - if we receive no complaints from that, it should be pretty safe to move forward with that plan.
Cheers, Ralf
[1] https://github.com/conda-forge/scipy-feedstock/pull/184 _______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
_______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: ilhanpolat@gmail.com
I think that's pretty much the same thing as removing the test sources from default distribution completely, which I'm probably -1 on, but happy to lose that debate if the majority prefer it... I just don't see the bandwidth benefit as justified for non-data assets, but I realize others disagree. On Sun, 28 May 2023 at 14:10, Ilhan Polat <ilhanpolat@gmail.com> wrote:
I think the common ground might be using python "optional dependencies" [1] such that only interested parties install it and the rest can skip the extra network payload. You can then install "pip/conda install scipy[full]" or whatever the syntax might be.
[1] : https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#op...
On Sun, May 28, 2023 at 10:00 PM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
FWIW, I'm in favor of moving large data assets out, but my preference would be to always have the sources of tests in the main repo. The debates about this aren't super interesting/fun, but anyway that's where I land on it, even if the project goes in another direction.
Best wishes, Tyler
On Sat, 13 May 2023 at 05:07, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi all,
This is an FYI that in conda-forge, thanks to the hard work of @h-vetinari [1], the `scipy` package now no longer contains tests. Instead, those are split off into a new `scipy-tests` package which, when you install that, puts all the test files back inside the `scipy` package tree.
The gain of that exercise was a ~30% size reduction, which seems worth it. We already had plans to do something similar in SciPy itself, by putting the heavy data files from the test suite under `scipy.datasets`. The gain of that will be less, probably 15-20% of the total size of wheels (excluding vendored openblas et al.). The conda-forge packages are a nice test case - if we receive no complaints from that, it should be pretty safe to move forward with that plan.
Cheers, Ralf
[1] https://github.com/conda-forge/scipy-feedstock/pull/184 _______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
_______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: ilhanpolat@gmail.com
_______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
No, not really :) I really don't have any opinions on it. I like how FastAPI uses that machinery to avoid full installations hence my comment. On Sun, May 28, 2023 at 10:15 PM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
I think that's pretty much the same thing as removing the test sources from default distribution completely, which I'm probably -1 on, but happy to lose that debate if the majority prefer it... I just don't see the bandwidth benefit as justified for non-data assets, but I realize others disagree.
On Sun, 28 May 2023 at 14:10, Ilhan Polat <ilhanpolat@gmail.com> wrote:
I think the common ground might be using python "optional dependencies" [1] such that only interested parties install it and the rest can skip the extra network payload. You can then install "pip/conda install scipy[full]" or whatever the syntax might be.
[1] : https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#op...
On Sun, May 28, 2023 at 10:00 PM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
FWIW, I'm in favor of moving large data assets out, but my preference would be to always have the sources of tests in the main repo. The debates about this aren't super interesting/fun, but anyway that's where I land on it, even if the project goes in another direction.
Best wishes, Tyler
On Sat, 13 May 2023 at 05:07, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi all,
This is an FYI that in conda-forge, thanks to the hard work of @h-vetinari [1], the `scipy` package now no longer contains tests. Instead, those are split off into a new `scipy-tests` package which, when you install that, puts all the test files back inside the `scipy` package tree.
The gain of that exercise was a ~30% size reduction, which seems worth it. We already had plans to do something similar in SciPy itself, by putting the heavy data files from the test suite under `scipy.datasets`. The gain of that will be less, probably 15-20% of the total size of wheels (excluding vendored openblas et al.). The conda-forge packages are a nice test case - if we receive no complaints from that, it should be pretty safe to move forward with that plan.
Cheers, Ralf
[1] https://github.com/conda-forge/scipy-feedstock/pull/184 _______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
_______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: ilhanpolat@gmail.com
_______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
_______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: ilhanpolat@gmail.com
And to be clear, by "main repo," I also mean sdists and wheels include test sources by default (in my preferred approach). I realize others don't see the point, but there are always tradeoffs on convenience, etc. On Sun, 28 May 2023 at 13:59, Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
FWIW, I'm in favor of moving large data assets out, but my preference would be to always have the sources of tests in the main repo. The debates about this aren't super interesting/fun, but anyway that's where I land on it, even if the project goes in another direction.
Best wishes, Tyler
On Sat, 13 May 2023 at 05:07, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi all,
This is an FYI that in conda-forge, thanks to the hard work of @h-vetinari [1], the `scipy` package now no longer contains tests. Instead, those are split off into a new `scipy-tests` package which, when you install that, puts all the test files back inside the `scipy` package tree.
The gain of that exercise was a ~30% size reduction, which seems worth it. We already had plans to do something similar in SciPy itself, by putting the heavy data files from the test suite under `scipy.datasets`. The gain of that will be less, probably 15-20% of the total size of wheels (excluding vendored openblas et al.). The conda-forge packages are a nice test case - if we receive no complaints from that, it should be pretty safe to move forward with that plan.
Cheers, Ralf
[1] https://github.com/conda-forge/scipy-feedstock/pull/184 _______________________________________________ SciPy-Dev mailing list -- scipy-dev@python.org To unsubscribe send an email to scipy-dev-leave@python.org https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: tyler.je.reddy@gmail.com
participants (3)
-
Ilhan Polat -
Ralf Gommers -
Tyler Reddy