[Numpy-discussion] Issue with setup_requires and 1.16 release candidates

[Thomas Robitaille]

Hi all,

Back in December I started getting failures in continuous integration
as well as reports of failures from users of installation issues for a
couple of packages. The problem can be easily reproduced in a Docker
container with:

   FROM ubuntu:16.04
   RUN apt-get update
   RUN apt-get install -y python3 python3-dev python3-pip python3-wheel
   RUN pip3 install fast-histogram
   RUN python3 -c 'import fast_histogram'

Doing this results in the following traceback:

   ImportError: No module named 'numpy.core._multiarray_umath'
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py",
line 1, in <module>
       from .histogram import *
     File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py",
line 7, in <module>
       from ._histogram_core import (_histogram1d,
   ImportError: numpy.core.multiarray failed to import

I've seen similar issues with other packages too. The key is that the
fast-histogram package defines:

   setup_requires=['numpy']

in the setup.py (since the package has a C extension that uses the
Numpy C API) and numpy is needed before the install_requires
dependencies are installed:

   https://github.com/astrofrog/fast-histogram/blob/master/setup.py

Now this normally works fine, but the issues I saw started when the
first 1.16 RC was made available, when installing into an environment
in which numpy is not already installed.

My understanding is that setup_requires is honored by easy_install
(even if installing the main package with pip), which doesn't ignore
pre-releases. Thus, the package is built against the 1.16 RC but then
1.15 is installed due to:

   install_requires=['numpy']

which is honored by pip.

I think that the correct solution is to make sure that:

   [build-system]
   requires = ["setuptools", "wheel", "numpy"]

is added to the pyproject.toml file (as per PEP 518). This then works
properly with recent versions of pip (>10 I think). I think removing
setup_requires then makes sense because it'd be better to have an
error with older versions of pip that numpy is not installed rather
than having the package built against the wrong version.

I know this is a temporary issue in the sense that it will go away
once the final version of 1.16 is out, but I just wanted to share this
as a heads-up in case you get reports of issues from other people, and
also to check whether there are any other solutions/workarounds to be
aware of? (to avoid a similar situation in future).

Thanks,
Tom