pip can't find header file for extension module, but `python setup.py install` works fine
Hi, I'm trying to write a new `setup.py` file for an extension module to wrap a C library (https://github.com/cvxgrp/scs). The current `setup.py` file imports numpy. I'm trying to delay that import statement until setuptools has a chance to install numpy if it's not already installed. I'm trying to do that with this bit of code: from setuptools.command.build_ext import build_ext as _build_ext class build_ext(_build_ext): def finalize_options(self): _build_ext.finalize_options(self) # Prevent numpy from thinking it is still in its setup process: __builtins__.__NUMPY_SETUP__ = False import numpy self.include_dirs += ext['include_dirs'] + [numpy.get_include()] Running `python setup.py install` seems to work fine on my OSX machine, but when I run `pip install .` in the directory with `setup.py`, I get a clang error that it can't find one of the header files. Any idea why that would be happening? Could it have anything to do with the relative path I'm giving for the include directories? Also, I had trouble finding good documentation on subclassing build_ext. Does anyone know if setting self.include_dirs overwrites or appends to the include_dirs attribute of an Extension object defined later in setup.py? For the curious, my current attempt at setup.py is athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The original can be found in the same directory. More generally, since I'm new to python packaging, I'm not sure how well or correctly I've written my `setup.py` file. Any feedback on doing things correctly would be appreciated. Thanks, AJ
Hi AJ,
On 1 Jun 2015, at 7:07 am, AJ Friend <ajfriend@gmail.com> wrote: I'm trying to write a new `setup.py` file for an extension module to wrap a C library (https://github.com/cvxgrp/scs).
I have been looking for the best way to do this too and I am keen to hear from distutils-sig. I can show you my currently favourite way, although I am not sure it’s applicable to your case. class BuildExtensions(build_ext): def run(self): # At this point we can be sure pip has already installed numpy numpy_incl = pkg_resources.resource_filename('numpy', 'core/include') for ext in self.extensions: if (hasattr(ext, 'include_dirs') and numpy_incl not in ext.include_dirs): ext.include_dirs.append(numpy_incl) build_ext.run(self) and then of course pass requires=[“numpy”] to setup. This avoids running any of numpy’s code. It should work because as of version 6.1.0 pip installs project dependencies in the right (topological) order. This should be enough for your version but the original repo was importing numpy also to access other informations. from numpy.distutils.system_info import get_info, BlasNotFoundError and I am not sure how to access that without importing numpy.
Any idea why that would be happening? Could it have anything to do with the relative path I'm giving for the include directories?
This is impossible to say without a build log. Can you upload one somewhere (gist/pastebin/etc) ? Andrea -- Andrea Bedini @andreabedini, http://www.andreabedini.com use https://keybase.io/andreabedini to send me private messages
On Sun, May 31, 2015 at 5:07 PM, AJ Friend <ajfriend@gmail.com> wrote:
Hi,
I'm trying to write a new `setup.py` file for an extension module to wrap a C library (https://github.com/cvxgrp/scs).
The current `setup.py` file imports numpy. I'm trying to delay that import statement until setuptools has a chance to install numpy if it's not already installed. I'm trying to do that with this bit of code:
from setuptools.command.build_ext import build_ext as _build_ext class build_ext(_build_ext): def finalize_options(self): _build_ext.finalize_options(self) # Prevent numpy from thinking it is still in its setup process: __builtins__.__NUMPY_SETUP__ = False import numpy self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
Running `python setup.py install` seems to work fine on my OSX machine, but when I run `pip install .` in the directory with `setup.py`, I get a clang error that it can't find one of the header files.
Any idea why that would be happening? Could it have anything to do with the relative path I'm giving for the include directories?
Also, I had trouble finding good documentation on subclassing build_ext. Does anyone know if setting self.include_dirs overwrites or appends to the include_dirs attribute of an Extension object defined later in setup.py?
For the curious, my current attempt at setup.py is athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The original can be found in the same directory.
More generally, since I'm new to python packaging, I'm not sure how well or correctly I've written my `setup.py` file. Any feedback on doing things correctly would be appreciated.
Hi AJ, For a lot of things in Python packaging there is not, sadly, One Right Way to Do It. Your setup.py looks okay though. You may want to have a look at the get_numpy_include_path utility here: https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1... It's similar to what you're already doing, but maybe a little more 'robust'. In particular, I think the reload of the numpy module may be important. Erik
On 2 June 2015 at 02:51, Erik Bray <erik.m.bray@gmail.com> wrote:
On Sun, May 31, 2015 at 5:07 PM, AJ Friend <ajfriend@gmail.com> wrote:
Hi,
I'm trying to write a new `setup.py` file for an extension module to wrap a C library (https://github.com/cvxgrp/scs).
The current `setup.py` file imports numpy. I'm trying to delay that import statement until setuptools has a chance to install numpy if it's not already installed. I'm trying to do that with this bit of code:
from setuptools.command.build_ext import build_ext as _build_ext class build_ext(_build_ext): def finalize_options(self): _build_ext.finalize_options(self) # Prevent numpy from thinking it is still in its setup process: __builtins__.__NUMPY_SETUP__ = False import numpy self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
Running `python setup.py install` seems to work fine on my OSX machine, but when I run `pip install .` in the directory with `setup.py`, I get a clang error that it can't find one of the header files.
Any idea why that would be happening? Could it have anything to do with the relative path I'm giving for the include directories?
Also, I had trouble finding good documentation on subclassing build_ext. Does anyone know if setting self.include_dirs overwrites or appends to the include_dirs attribute of an Extension object defined later in setup.py?
For the curious, my current attempt at setup.py is athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The original can be found in the same directory.
More generally, since I'm new to python packaging, I'm not sure how well or correctly I've written my `setup.py` file. Any feedback on doing things correctly would be appreciated.
Hi AJ,
For a lot of things in Python packaging there is not, sadly, One Right Way to Do It. Your setup.py looks okay though.
You may want to have a look at the get_numpy_include_path utility here: https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1...
It's similar to what you're already doing, but maybe a little more 'robust'. In particular, I think the reload of the numpy module may be important.
It is, because what numpy is doing is importing from its own tree before setup has completed - which is in general unsafe. And numpy may be in process depending on exactly what setuptools setup_requires easy-install code has done, which can lead to the symptoms you see. This should be better once we get pip interpreting setup_requires for you, but a perhaps shorter term fix would be to factor out the distutils support glue from numpy (which seems pretty self contained) into a out of package file - either adjacent to setup.py, or, if numpy is willing to make the dependency on setuptools a hard dep (which I recommend), then as a separate package which can be setup_require'd. That would avoid the bootstrapping complexity, resulting in numpy not being imported until it has actually been installed, and things should be rosy for folk downstream of numpy from that point on. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On Tue, Jun 2, 2015 at 3:41 AM, Robert Collins <robertc@robertcollins.net> wrote:
On 2 June 2015 at 02:51, Erik Bray <erik.m.bray@gmail.com> wrote:
On Sun, May 31, 2015 at 5:07 PM, AJ Friend <ajfriend@gmail.com> wrote:
Hi,
I'm trying to write a new `setup.py` file for an extension module to wrap a C library (https://github.com/cvxgrp/scs).
The current `setup.py` file imports numpy. I'm trying to delay that import statement until setuptools has a chance to install numpy if it's not already installed. I'm trying to do that with this bit of code:
from setuptools.command.build_ext import build_ext as _build_ext class build_ext(_build_ext): def finalize_options(self): _build_ext.finalize_options(self) # Prevent numpy from thinking it is still in its setup process: __builtins__.__NUMPY_SETUP__ = False import numpy self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
Running `python setup.py install` seems to work fine on my OSX machine, but when I run `pip install .` in the directory with `setup.py`, I get a clang error that it can't find one of the header files.
Any idea why that would be happening? Could it have anything to do with the relative path I'm giving for the include directories?
Also, I had trouble finding good documentation on subclassing build_ext. Does anyone know if setting self.include_dirs overwrites or appends to the include_dirs attribute of an Extension object defined later in setup.py?
For the curious, my current attempt at setup.py is athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The original can be found in the same directory.
More generally, since I'm new to python packaging, I'm not sure how well or correctly I've written my `setup.py` file. Any feedback on doing things correctly would be appreciated.
Hi AJ,
For a lot of things in Python packaging there is not, sadly, One Right Way to Do It. Your setup.py looks okay though.
You may want to have a look at the get_numpy_include_path utility here: https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1...
It's similar to what you're already doing, but maybe a little more 'robust'. In particular, I think the reload of the numpy module may be important.
It is, because what numpy is doing is importing from its own tree before setup has completed - which is in general unsafe.
And numpy may be in process depending on exactly what setuptools setup_requires easy-install code has done, which can lead to the symptoms you see. This should be better once we get pip interpreting setup_requires for you, but a perhaps shorter term fix would be to factor out the distutils support glue from numpy (which seems pretty self contained) into a out of package file - either adjacent to setup.py, or, if numpy is willing to make the dependency on setuptools a hard dep (which I recommend), then as a separate package which can be setup_require'd.
That would avoid the bootstrapping complexity, resulting in numpy not being imported until it has actually been installed, and things should be rosy for folk downstream of numpy from that point on.
That might not be a bad idea. This is more or less what we did for Astropy with astropy-helpers (this was done also so that other software distributions in the Astropy affiliated package ecosystem could take advantage of it). I think maybe splitting out much of the stuff in numpy.distutils into its own package (though unfortunately it would need a new name since I don't think we could make numpy into a namespace package) might help. Though I'm not sure how that would work for numpy.get_includes(). Would it just figure out where the include files *would* be installed to if one were installing Numpy into the currently active environment? Erik
participants (4)
-
AJ Friend
-
Andrea Bedini
-
Erik Bray
-
Robert Collins