How to add GDAL as a dependency to a Python package
Dear all, I am trying to package a Python script for PyPI that uses GDAL. I started by including a direct reference in my setup.py: install_requires=['GDAL==1.11.2'], This way the package failed to install in my test virtual environment: extensions/gdal_wrap.cpp:2855:22: fatal error: cpl_port.h: No such file or directory #include "cpl_port.h" ^ compilation terminated. error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 I then tried it with a reference to pygdal, since it is marked as a virtualenv friendly version: install_requires=['pygdal'], This way the installation finishes without errors (but with the usual load of compilation warnings). However, when then I invoke the script I get this error back: Traceback (most recent call last): File "/home/desouslu/.virtualenvs/test_p3/bin/hasc2gml", line 5, in <module> from pkg_resources import load_entry_point File "/home/desouslu/.virtualenvs/test_p3/lib/python3.4/site-packages/pkg_resources.py", line 2716, in <module> working_set.require(__requires__) File "/home/desouslu/.virtualenvs/test_p3/lib/python3.4/site-packages/pkg_resources.py", line 685, in require needed = self.resolve(parse_requirements(requirements)) File "/home/desouslu/.virtualenvs/test_p3/lib/python3.4/site-packages/pkg_resources.py", line 588, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: pygdal What is the correct way of setting GDAL as a dependency? Thank you, Luís Sent from [ProtonMail](https://protonmail.ch), encrypted email based in Switzerland.
setting gdal as a dependency is only going to work if one of these is true: * There is a binary wheel on PyPi for gdal (which is very hard to do, though with teh manylinux effort, maybe there will be some day) * the system on which you are trying to install is all set up to compile gdal -- which is a major effort, and indeed, I'm not sure is possible (I haven't looked) - The GDAL source ships with the python bindings, so the usual thing is for gdal itself and the pyton bindings to all get compiled and installed at once. I"m not srue that anyone has made a setup.py that will do that. This kind of thing is exactly why conda exists -- python packages with complex non-python dependencies are not well handled by pip / pypi -- if I were you, I"d give it a try :-) -CHB On Tue, Apr 12, 2016 at 5:24 AM, Luís de Sousa <distutils-sig@python.org> wrote:
Dear all, I am trying to package a Python script for PyPI that uses GDAL. I started by including a direct reference in my setup.py:
install_requires=['GDAL==1.11.2'],
This way the package failed to install in my test virtual environment:
extensions/gdal_wrap.cpp:2855:22: fatal error: cpl_port.h: No such file or directory #include "cpl_port.h" ^ compilation terminated. error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
I then tried it with a reference to pygdal, since it is marked as a virtualenv friendly version:
install_requires=['pygdal'],
This way the installation finishes without errors (but with the usual load of compilation warnings). However, when then I invoke the script I get this error back:
Traceback (most recent call last): File "/home/desouslu/.virtualenvs/test_p3/bin/hasc2gml", line 5, in <module> from pkg_resources import load_entry_point File "/home/desouslu/.virtualenvs/test_p3/lib/python3.4/site-packages/pkg_resources.py", line 2716, in <module> working_set.require(__requires__) File "/home/desouslu/.virtualenvs/test_p3/lib/python3.4/site-packages/pkg_resources.py", line 685, in require needed = self.resolve(parse_requirements(requirements)) File "/home/desouslu/.virtualenvs/test_p3/lib/python3.4/site-packages/pkg_resources.py", line 588, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: pygdal
What is the correct way of setting GDAL as a dependency? Thank you,
Luís
*Sent from ProtonMail <https://protonmail.ch>, encrypted email based in Switzerland.*
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Thank you for the reply Chris. I just tried to install pygdal directly from PiPY on a stock Ubuntu distribution and it fails [0], even if I supposedly have all the necessary headers installed. I will try easier ways before going with Conda, but thanks for the suggestion in any case. Luís [0] http://pastebin.com/gzrimLvB Sent from [ProtonMail](https://protonmail.ch), encrypted email based in Switzerland. -------- Original Message -------- Subject: Re: [Distutils] How to add GDAL as a dependency to a Python package Local Time: April 12, 2016 6:45 PM UTC Time: April 12, 2016 4:45 PM From: chris.barker@noaa.gov To: luis.de.sousa@protonmail.ch CC: distutils-sig@python.org setting gdal as a dependency is only going to work if one of these is true: * There is a binary wheel on PyPi for gdal (which is very hard to do, though with teh manylinux effort, maybe there will be some day) * the system on which you are trying to install is all set up to compile gdal -- which is a major effort, and indeed, I'm not sure is possible (I haven't looked) - The GDAL source ships with the python bindings, so the usual thing is for gdal itself and the pyton bindings to all get compiled and installed at once. I"m not srue that anyone has made a setup.py that will do that. This kind of thing is exactly why conda exists -- python packages with complex non-python dependencies are not well handled by pip / pypi -- if I were you, I"d give it a try :-) -CHB
On Wed, Apr 13, 2016 at 12:02 AM, Luís de Sousa < luis.de.sousa@protonmail.ch> wrote:
Thank you for the reply Chris.
I just tried to install pygdal directly from PiPY on a stock Ubuntu distribution and it fails [0], even if I supposedly have all the necessary headers installed.
exactly -- not a pip issue :-( are there not Ubuntu packages for GDAL? -CHB I will try easier ways before going with Conda, but thanks for the
suggestion in any case.
Luís
[0] http://pastebin.com/gzrimLvB
*Sent from ProtonMail <https://protonmail.ch>, encrypted email based in Switzerland.*
-------- Original Message -------- Subject: Re: [Distutils] How to add GDAL as a dependency to a Python package Local Time: April 12, 2016 6:45 PM UTC Time: April 12, 2016 4:45 PM From: chris.barker@noaa.gov To: luis.de.sousa@protonmail.ch CC: distutils-sig@python.org
setting gdal as a dependency is only going to work if one of these is true:
* There is a binary wheel on PyPi for gdal (which is very hard to do, though with teh manylinux effort, maybe there will be some day)
* the system on which you are trying to install is all set up to compile gdal -- which is a major effort, and indeed, I'm not sure is possible (I haven't looked) - The GDAL source ships with the python bindings, so the usual thing is for gdal itself and the pyton bindings to all get compiled and installed at once. I"m not srue that anyone has made a setup.py that will do that.
This kind of thing is exactly why conda exists -- python packages with complex non-python dependencies are not well handled by pip / pypi -- if I were you, I"d give it a try :-)
-CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, Apr 13, 2016 at 12:02 AM, Luís de Sousa < luis.de.sousa@protonmail.ch> wrote:
Thank you for the reply Chris.
I just tried to install pygdal directly from PiPY on a stock Ubuntu distribution and it fails [0], even if I supposedly have all the necessary headers installed.
I took a quick look, and it looks like the buld/install failed on numpy -- numpy is a dependency of gdal. so you may want to get numpy installed first. (that surprises me -- I thought numpy built out of the box pretty easily -- if not with an optimized BLAS) and it looks like it did build, but somehow failed to install.... anyway, this is all why Anaconda and Canopy exist... -CHB
I will try easier ways before going with Conda, but thanks for the suggestion in any case.
Luís
[0] http://pastebin.com/gzrimLvB
*Sent from ProtonMail <https://protonmail.ch>, encrypted email based in Switzerland.*
-------- Original Message -------- Subject: Re: [Distutils] How to add GDAL as a dependency to a Python package Local Time: April 12, 2016 6:45 PM UTC Time: April 12, 2016 4:45 PM From: chris.barker@noaa.gov To: luis.de.sousa@protonmail.ch CC: distutils-sig@python.org
setting gdal as a dependency is only going to work if one of these is true:
* There is a binary wheel on PyPi for gdal (which is very hard to do, though with teh manylinux effort, maybe there will be some day)
* the system on which you are trying to install is all set up to compile gdal -- which is a major effort, and indeed, I'm not sure is possible (I haven't looked) - The GDAL source ships with the python bindings, so the usual thing is for gdal itself and the pyton bindings to all get compiled and installed at once. I"m not srue that anyone has made a setup.py that will do that.
This kind of thing is exactly why conda exists -- python packages with complex non-python dependencies are not well handled by pip / pypi -- if I were you, I"d give it a try :-)
-CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, Apr 13, 2016 at 11:31 AM, Chris Barker <chris.barker@noaa.gov> wrote:
On Wed, Apr 13, 2016 at 12:02 AM, Luís de Sousa <luis.de.sousa@protonmail.ch> wrote:
Thank you for the reply Chris.
I just tried to install pygdal directly from PiPY on a stock Ubuntu distribution and it fails [0], even if I supposedly have all the necessary headers installed.
I took a quick look, and it looks like the buld/install failed on numpy -- numpy is a dependency of gdal. so you may want to get numpy installed first.
Yeah, due to various infelicities in the distutils stack, automated dependency installation when building stuff is pretty wonky, and a lot of things need numpy to be installed before you try building them. So doing 'pip install numpy' first is a good idea before trying to install any of the rest of the numerical stack. (Part of the motivation for PEP 516/517 is to fix this.)
(that surprises me -- I thought numpy built out of the box pretty easily -- if not with an optimized BLAS) and it looks like it did build, but somehow failed to install....
Thanks to Matthew Brett's tireless efforts (hat tip to Zhang Xianyi for fixing a bunch of bugs in OpenBLAS just for us), there are now[1] numpy linux wheels with optimized BLAS up on PyPI, so if you have pip
= 8.1.0, then 'pip install numpy' should Just Work on linux without needing a compiler at all (and likewise on windows and osx).
-n [1] now = "since ~1 hour ago" ;-) -- Nathaniel J. Smith -- https://vorpus.org
On 13 April 2016 at 21:33, Nathaniel Smith <njs@pobox.com> wrote:
Thanks to Matthew Brett's tireless efforts (hat tip to Zhang Xianyi for fixing a bunch of bugs in OpenBLAS just for us), there are now[1] numpy linux wheels with optimized BLAS up on PyPI, so if you have pip
= 8.1.0, then 'pip install numpy' should Just Work on linux without needing a compiler at all (and likewise on windows and osx).
W00t. That's really good news :-) Thanks to all who worked on making that possible. Paul
On 14 April 2016 at 06:33, Nathaniel Smith <njs@pobox.com> wrote:
Thanks to Matthew Brett's tireless efforts (hat tip to Zhang Xianyi for fixing a bunch of bugs in OpenBLAS just for us), there are now[1] numpy linux wheels with optimized BLAS up on PyPI, so if you have pip
= 8.1.0, then 'pip install numpy' should Just Work on linux without needing a compiler at all (and likewise on windows and osx).
Woohoo! Congrats to all involved in making that possible :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thank you all for the replies. I have tried various tests on Ubuntu, and even installing numpy beforehand, apt has to come into play at some point (as has sudo). I guess it is simpler to leave dependencies up to the user. And perhaps provide a package hooked up to python-dev and python-gdal in Debian based distros. Regards, Luís
participants (5)
-
Chris Barker -
Luís de Sousa -
Nathaniel Smith -
Nick Coghlan -
Paul Moore