[Distutils] pip can't find header file for extension module, but `python setup.py install` works fine

[Erik Bray]

On Tue, Jun 2, 2015 at 3:41 AM, Robert Collins
<robertc at robertcollins.net> wrote:
> On 2 June 2015 at 02:51, Erik Bray <erik.m.bray at gmail.com> wrote:
>> On Sun, May 31, 2015 at 5:07 PM, AJ Friend <ajfriend at gmail.com> wrote:
>>> Hi,
>>>
>>> I'm trying to write a new `setup.py` file for an extension module to
>>> wrap a C library (https://github.com/cvxgrp/scs).
>>>
>>> The current `setup.py` file imports numpy. I'm trying to delay that
>>> import statement until setuptools has a chance to install numpy if
>>> it's not already installed. I'm trying to do that with this bit of
>>> code:
>>>
>>> from setuptools.command.build_ext import build_ext as _build_ext
>>> class build_ext(_build_ext):
>>>     def finalize_options(self):
>>>         _build_ext.finalize_options(self)
>>>         # Prevent numpy from thinking it is still in its setup process:
>>>         __builtins__.__NUMPY_SETUP__ = False
>>>         import numpy
>>>         self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
>>>
>>> Running `python setup.py install` seems to work fine on my OSX
>>> machine, but when I run `pip install .` in the directory with
>>> `setup.py`, I get a clang error that it can't find one of the header
>>> files.
>>>
>>> Any idea why that would be happening? Could it have anything to do
>>> with the relative path I'm giving for the include directories?
>>>
>>> Also, I had trouble finding good documentation on subclassing
>>> build_ext. Does anyone know if setting self.include_dirs overwrites or
>>> appends to the include_dirs attribute of an Extension object defined
>>> later in setup.py?
>>>
>>> For the curious, my current attempt at setup.py is
>>> athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The
>>> original can be found in the same directory.
>>>
>>> More generally, since I'm new to python packaging, I'm not sure how
>>> well or correctly I've written my `setup.py` file. Any feedback on
>>> doing things correctly would be appreciated.
>>
>> Hi AJ,
>>
>> For a lot of things in Python packaging there is not, sadly, One Right
>> Way to Do It.  Your setup.py looks okay though.
>>
>> You may want to have a look at the get_numpy_include_path utility here:
>> https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66
>>
>> It's similar to what you're already doing, but maybe a little more
>> 'robust'.  In particular, I think the reload of the numpy module may
>> be important.
>
> It is, because what numpy is doing is importing from its own tree
> before setup has completed - which is in general unsafe.
>
> And numpy may be in process depending on exactly what setuptools
> setup_requires easy-install code has done, which can lead to the
> symptoms you see. This should be better once we get pip interpreting
> setup_requires for you, but a perhaps shorter term fix would be to
> factor out the distutils support glue from numpy (which seems pretty
> self contained) into a out of package file - either adjacent to
> setup.py, or, if numpy is willing to make the dependency on setuptools
> a hard dep (which I recommend), then as a separate package which can
> be setup_require'd.
>
> That would avoid the bootstrapping complexity, resulting in numpy not
> being imported until it has actually been installed, and things should
> be rosy for folk downstream of numpy from that point on.

That might not be a bad idea.  This is more or less what we did for
Astropy with astropy-helpers (this was done also so that other
software distributions in the Astropy affiliated package ecosystem
could take advantage of it).

I think maybe splitting out much of the stuff in numpy.distutils into
its own package (though unfortunately it would need a new name since I
don't think we could make numpy into a namespace package) might help.
Though I'm not sure how that would work for numpy.get_includes().
Would it just figure out where the include files *would* be installed
to if one were installing Numpy into the currently active environment?

Erik