Declaring numpy and scipy dependencies?
Hi all. I think since the pipy ecosystem improved a lot, we should properly declare the scipy and numpy dependencies, so that ``pip install scikit-learn`` works properly. The argument why we did not do this previously was that this would try to download and build numpy and scipy, which are pretty much guaranteed to fail / result in a very slow installation. This is no longer true. Any opinions? Andy
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? E.g., if you run pip install scikit-learn --upgrade it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run pip install scikit-learn --upgrade --no-deps unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
On Jul 28, 2016, at 2:44 PM, Andreas Mueller <t3kcit@gmail.com> wrote:
Hi all. I think since the pipy ecosystem improved a lot, we should properly declare the scipy and numpy dependencies, so that ``pip install scikit-learn`` works properly. The argument why we did not do this previously was that this would try to download and build numpy and scipy, which are pretty much guaranteed to fail / result in a very slow installation. This is no longer true.
Any opinions?
Andy _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
On 07/28/2016 02:55 PM, Sebastian Raschka wrote:
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? Well so far the compiling was more the issue. E.g., if you run
pip install scikit-learn --upgrade
it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run
pip install scikit-learn --upgrade --no-deps
unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
Well, but that's a pip usability bug. Also, with binary wheels, this shouldn't be a big deal.
On Thu, Jul 28, 2016 at 7:55 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? E.g., if you run
pip install scikit-learn --upgrade
it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run
pip install scikit-learn --upgrade --no-deps
unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
Also - the install will work fine for platforms with wheels, but is still bad for platforms without - like the Raspberry Pi. Cheers, Matthew
On 07/28/2016 03:04 PM, Matthew Brett wrote:
On Thu, Jul 28, 2016 at 7:55 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? E.g., if you run
pip install scikit-learn --upgrade
it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run
pip install scikit-learn --upgrade --no-deps
unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
Also - the install will work fine for platforms with wheels, but is still bad for platforms without - like the Raspberry Pi. Hm... so these would be ARM wheels? Or Raspberry Pi specific ones? Do you know if there are plans? Not sure how I feel about this. Do all platforms need to have wheels before we can rely on them?
On Thu, Jul 28, 2016 at 8:10 PM, Andreas Mueller <t3kcit@gmail.com> wrote:
On 07/28/2016 03:04 PM, Matthew Brett wrote:
On Thu, Jul 28, 2016 at 7:55 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? E.g., if you run
pip install scikit-learn --upgrade
it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run
pip install scikit-learn --upgrade --no-deps
unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
Also - the install will work fine for platforms with wheels, but is still bad for platforms without - like the Raspberry Pi.
Hm... so these would be ARM wheels? Or Raspberry Pi specific ones?
No, they'd have to be Raspberry Pi specific ones because no-one has worked out a general ARM-wide specification, as we have for Intel Linux = manylinux1.
Do you know if there are plans? Not sure how I feel about this. Do all platforms need to have wheels before we can rely on them?
I'm not sure either - just throwing it out there... Matthew
My main issue with the upgrade is that if there was a slightly newer version of numpy/scipy it would try to upgrade my numpy/scipy linked against MKL/blas to a vanilla version downloaded from the cheese shop. It was a huge pain. On Thu, 28 Jul 2016 at 21:17 Matthew Brett <matthew.brett@gmail.com> wrote:
On Thu, Jul 28, 2016 at 8:10 PM, Andreas Mueller <t3kcit@gmail.com> wrote:
On 07/28/2016 03:04 PM, Matthew Brett wrote:
On Thu, Jul 28, 2016 at 7:55 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? E.g., if you run
pip install scikit-learn --upgrade
it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run
pip install scikit-learn --upgrade --no-deps
unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
Also - the install will work fine for platforms with wheels, but is still bad for platforms without - like the Raspberry Pi.
Hm... so these would be ARM wheels? Or Raspberry Pi specific ones?
No, they'd have to be Raspberry Pi specific ones because no-one has worked out a general ARM-wide specification, as we have for Intel Linux = manylinux1.
Do you know if there are plans? Not sure how I feel about this. Do all platforms need to have wheels before we can rely on them?
I'm not sure either - just throwing it out there...
Matthew _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
On Thu, Jul 28, 2016 at 8:30 PM, federico vaggi <vaggi.federico@gmail.com> wrote:
My main issue with the upgrade is that if there was a slightly newer version of numpy/scipy it would try to upgrade my numpy/scipy linked against MKL/blas to a vanilla version downloaded from the cheese shop. It was a huge pain.
The current Linux wheels are linked against recent OpenBLAS, and OSX wheels link against Accelerate, and both should be pretty fast, but I take your point. Matthew
On 07/28/2016 03:30 PM, federico vaggi wrote:
My main issue with the upgrade is that if there was a slightly newer version of numpy/scipy it would try to upgrade my numpy/scipy linked against MKL/blas to a vanilla version downloaded from the cheese shop. It was a huge pain.
You mean with binary wheels or without? Now the cheese shop will give you OpenBlas.
On 07/28/2016 03:16 PM, Matthew Brett wrote:
On Thu, Jul 28, 2016 at 8:10 PM, Andreas Mueller <t3kcit@gmail.com> wrote:
On 07/28/2016 03:04 PM, Matthew Brett wrote:
On Thu, Jul 28, 2016 at 7:55 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
I think that should work fine for the `pip install scikit-learn`, however, I think the problem was with upgrading, right? E.g., if you run
pip install scikit-learn --upgrade
it would try to upgrade numpy and scipy as well, which may not be desired. I think the only workaround would be to run
pip install scikit-learn --upgrade --no-deps
unless they changed the behavior recently. I mean, it’s not really a problem, but many users may not know about the --no-deps flag.
Also - the install will work fine for platforms with wheels, but is still bad for platforms without - like the Raspberry Pi. Hm... so these would be ARM wheels? Or Raspberry Pi specific ones?
No, they'd have to be Raspberry Pi specific ones because no-one has worked out a general ARM-wide specification, as we have for Intel Linux = manylinux1.
Following up on this thread, I'm trying to write better installation instructions. https://github.com/scikit-learn/scikit-learn/pull/7313 What's the best-practice for cases when there are no wheels? I imagine there's also no conda channel for Raspberry Pi. So is it the package manager? Andy
I would be +1 to add the dependencies to numpy and scipy on the binary wheels only. We don't have the tools yet but this could be implemented in the auditwheel tool that is already used to generate the manylinux1 compatible wheels for Linux. -- Olivier
participants (5)
-
Andreas Mueller -
federico vaggi -
Matthew Brett -
Olivier Grisel -
Sebastian Raschka