[Numpy-discussion] Should I use pip install numpy in linux?

[Oscar Benjamin]

On 8 January 2016 at 23:27, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Jan 8, 2016 at 1:58 PM, Robert McGibbon <rmcgibbo at gmail.com> wrote:
>>
>> I'm not sure if this is the right path for numpy or not,
>
>
> probably not -- AFAICT, the PyPa folks aren't interested in solving teh
> problems we have in the scipy community -- we can tweak around the edges,
> but we wont get there without a commitment to really solve the issues -- and
> if pip did that, it would essentially be conda -- non one wants to
> re-impliment conda.

I think that's a little unfair to the PyPA people. They would like to
solve all of these problems is just a question of priority and
expertise. As always in open source you have to scratch your own itch
and those guys are working on other things like the security,
stability, scalability of the infrastructure, consistency of pip's
version handling and dependency resolution etc.

Linux wheels is a problem that has been discussed on distutils-sig.
The reason it hasn't happened is that it's a lower priority than
wheels for OSX/Windows because:
1) Most distros already package this stuff i.e. apt-get numpy
2) On Linux it's much easier to get the appropriate compilers so that
pip can build e.g. numpy.
3) The average Linux user is more capable of solving these problems.
4) Getting binary distribution to work across all Linux distributions
is significantly harder than for Windows/OSX because of the myriad
different distros/versions.

Considering point 2 pip install numpy etc. already works for a lot of
Linux users even if it is slow because of the time taken to compile (3
minutes on my system). Depending on your use case that problem is
partially solved by wheel caching. So if Linux wheels were allowed but
didn't always work then that would be a regression for many users. On
OSX binary wheels for numpy are already available and work fine AFAIK.
The absence of binary numpy wheels for Windows is not down to PyPA.

Considering point 4 the idea of compiling on an old base Linux system
has been discussed on distutils-sig before and it seems likely to
work. The problem is really about the external non-libc dependencies
though. The reason progress there has stalled is not because the PyPA
folks don't want to solve it but rather because they have other
priorities and are hoping that people with more expertise in that area
will step up to address those problems. Most of the issues stem from
the scientific Python community so ideally someone from the scientific
Python community would address how to solve those problems.

Recently Nathaniel brought some suggestions to distutils-sig to
address the problem of build-requires which is a particular pain
point. I think that people there appreciated the effort from someone
who understands the needs of hard-to-build packages to improve the way
that pip/PyPI works in that area. There was a lot of confusion from
people not understanding each others needs but ultimately I thought
there was agreement on how to move forward. (Although what happened to
that in the end?)

The same can happen with other problems like Linux wheels. If  you
guys here have a clear idea of how to solve the external dependency
problem then I'm sure they'll be receptive. Personally I think the
best approach is the pyopenblas approach: internalise the external
dependency so that pip can work with it. This is precisely what
Anaconda does and there's actually no need to make substantive changes
to the way pip/pypi/wheel works in order to achieve that. It just
needs someone to package the external dependencies as sdist/wheel (and
for PyPI to allow Linux wheels).

--
Oscar