[Numpy-discussion] Issue with setup_requires and 1.16 release candidates

[Thomas Robitaille]

Hi Ralf,

On Sun, 6 Jan 2019 at 22:06, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille <thomas.robitaille at gmail.com> wrote:
>>
>> Hi all,
>>
>> Back in December I started getting failures in continuous integration
>> as well as reports of failures from users of installation issues for a
>> couple of packages. The problem can be easily reproduced in a Docker
>> container with:
>>
>>    FROM ubuntu:16.04
>>    RUN apt-get update
>>    RUN apt-get install -y python3 python3-dev python3-pip python3-wheel
>>    RUN pip3 install fast-histogram
>>    RUN python3 -c 'import fast_histogram'
>>
>> Doing this results in the following traceback:
>>
>>    ImportError: No module named 'numpy.core._multiarray_umath'
>>    Traceback (most recent call last):
>>      File "<string>", line 1, in <module>
>>      File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py",
>> line 1, in <module>
>>        from .histogram import *
>>      File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py",
>> line 7, in <module>
>>        from ._histogram_core import (_histogram1d,
>>    ImportError: numpy.core.multiarray failed to import
>>
>> I've seen similar issues with other packages too. The key is that the
>> fast-histogram package defines:
>>
>>    setup_requires=['numpy']
>>
>> in the setup.py (since the package has a C extension that uses the
>> Numpy C API) and numpy is needed before the install_requires
>> dependencies are installed:
>>
>>    https://github.com/astrofrog/fast-histogram/blob/master/setup.py
>>
>> Now this normally works fine, but the issues I saw started when the
>> first 1.16 RC was made available, when installing into an environment
>> in which numpy is not already installed.
>>
>> My understanding is that setup_requires is honored by easy_install
>> (even if installing the main package with pip), which doesn't ignore
>> pre-releases. Thus, the package is built against the 1.16 RC but then
>> 1.15 is installed due to:
>>
>>    install_requires=['numpy']
>>
>> which is honored by pip.
>
>
> Oh fun. Thanks for explaining this!
>
>>
>> I think that the correct solution is to make sure that:
>>
>>    [build-system]
>>    requires = ["setuptools", "wheel", "numpy"]
>>
>> is added to the pyproject.toml file (as per PEP 518).
>
>
> To clarify: you're talking about fast-histogram's pyproject.toml file right?

Yes that's right. Just for the record, here's the pull request to
fast-histogram to implement the fix:

https://github.com/astrofrog/fast-histogram/pull/33

Cheers,
Tom

>
>> This then works
>> properly with recent versions of pip (>10 I think). I think removing
>> setup_requires then makes sense because it'd be better to have an
>> error with older versions of pip that numpy is not installed rather
>> than having the package built against the wrong version.
>
>
> This is a pretty hacky solution, needed because pip doesn't support setup_requires, but yes it should work and I can't think of a better way.
>
> Cheers,
> Ralf
>
>
>>
>> I know this is a temporary issue in the sense that it will go away
>> once the final version of 1.16 is out, but I just wanted to share this
>> as a heads-up in case you get reports of issues from other people, and
>> also to check whether there are any other solutions/workarounds to be
>> aware of? (to avoid a similar situation in future).
>>
>> Thanks,
>> Tom
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion