Suggested action for "Python.h not found"
Hi, Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest an action to install the headers ? I found http://trac.cython.org/ticket/247 which is quite old, and makes the error message easier to find, but doesn't recommend a solution. `sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like: if sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.") elif sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.") This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done. Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ? [1] - https://pypi.python.org/pypi/distro
+1, want to submit a pull request? On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari <abdealikothari@gmail.com> wrote:
Hi,
Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest an action to install the headers ?
I found http://trac.cython.org/ticket/247 which is quite old, and makes the error message easier to find, but doesn't recommend a solution.
`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:
if sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")
elif sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")
This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.
Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?
[1] - https://pypi.python.org/pypi/distro
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?
I think having this information on an installation instruction page is definitely a good idea. I don't think it makes sense to put in every single error message that users should consult the installation instructions; that should be a given. This would help so that non-developers can use cython based packages easily It is my belief that non-developers should not be manually building software. Nor should developers be doing that; if you're saying "./configure" yourself, it's like programming assembly code. There are many ways to install Cython without manually building: * Use your Linux distro's Cython, if you're happy with that. This is often not ideal. Because the Linux distro you're using, or forced to use by your IT department, doesn't contain the compilers / Python / etc. needed for your project. Or upgrading your Linux ends up breaking your applications that depend on it. Best choice is to not depend on your distro for software you really care about. * If you're on a Mac, install with MacPorts. * If you're on Linux (and maybe Mac too), install with Spack. This can be as easy as typing: spack install py-cython https://github.com/llnl/spack Spack gives you complete control over your compiler. This is important if you're building Python extensions, especially C++-based extensions, which must be built with the same compilers use to build Python. I use Spack to build my complete software stack: the application software I work on requires about 50 dependencies to build, and another ~20 Python dependencies to analyze the output. That includes the full Numpy/Scipy stack. Spack makes it easy for me to use consistent compiler and libraries for Python and my libraries. I've had success making Spack-based install instructions that others (non-developers) can follow and get my software stack installed on their systems: https://github.com/citibeth/icebin --- Elizabeth On Wed, Jun 8, 2016 at 1:25 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
+1, want to submit a pull request?
Hi,
Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest an action to install the headers ?
I found http://trac.cython.org/ticket/247 which is quite old, and makes
On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari <abdealikothari@gmail.com> wrote: the
error message easier to find, but doesn't recommend a solution.
`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:
if sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")
elif sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")
This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.
Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?
[1] - https://pypi.python.org/pypi/distro
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer <elizabeth.fischer@columbia.edu> wrote:
Spack gives you complete control over your compiler. This is important if you're building Python extensions, especially C++-based extensions, which must be built with the same compilers use to build Python.
Just to hopefully avoid confusing people -- you mostly only need to worry about this on Windows. On OS X and Linux, I can't think of any situation where using a different compiler for your extension and for Python will cause a problem. The one thing to watch out for these days is that if you have two different C++ libraries that are being linked together (so this doesn't apply to CPython itself -- it's not a C++ library), and the interface between the libraries uses std::string or std::list, and you're on Linux, then you need to be careful about the GCC 5 / C++11 ABI transition. The short version is that if you consistently use versions of GCC <5 OR consistently use versions of GCC >=5, then you should be fine; the long version can be found by googling :-). -n -- Nathaniel J. Smith -- https://vorpus.org
On Thu, Jun 9, 2016 at 2:49 AM, Nathaniel Smith <njs@vorpus.org> wrote:
On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer <elizabeth.fischer@columbia.edu> wrote:
Spack gives you complete control over your compiler. This is important if you're building Python extensions, especially C++-based extensions, which must be built with the same compilers use to build Python.
Just to hopefully avoid confusing people -- you mostly only need to worry about this on Windows. On OS X and Linux, I can't think of any situation where using a different compiler for your extension and for Python will cause a problem.
It could cause a problem if your extension is a wrapper around a library that you wrote in C++ or Fortran --- whether you wrote your extension by hand, or especially if you used the Cython variant that generates C++ code (a logical choice, if you're interfacing to a C++ library). If your C++ Cython extension requires a different compiler than that used to build Cython, you will also find that standard Cython ways to building your extension break. Maybe you're right --- maybe the extension itself is built in C, and all C++ elements are safely "encapsulated" away from Python. Maybe you can coax Cython into building extensions with your chosen compiler. But who wants to spend time finding out exactly which heterogeneous compilers work together, and how to avoid the standard Cython build paths? The problem is more general than compilers --- it can actually bite you ANY time you use two different versions of a shared library in a build that you ultimately link together. Suppose your Python stack was built with NetCDF4 but you built your extension with NetCDF3? In a software stack of 50 packages, the possibilities for this kind of problem are endless --- or more precisely, there are 50 opportunities for this kind of problem. And they are almost impossible to prevent if you're doing things by hand. Sometimes you can get away with it, and sometimes these problems result in strange, time-sucking bugs. Spack ensures that only ONE version of each package is linked in your final binary. and the
interface between the libraries uses std::string or std::list, and you're on Linux, then you need to be careful about the GCC 5 / C++11 ABI transition. The short version is that if you consistently use versions of GCC <5 OR consistently use versions of GCC >=5, then you should be fine; the long version can be found by googling :-).
That has never been my experience. My actual experience is every time I upgraded my MacPorts, I had to specifically re-build all my C++ libraries. This was after running into strange, time-sucking bugs. Advise on GCC's C++ supports this: http://stackoverflow.com/questions/23895081/can-you-mix-c-compiled-with-diff... And then there's Fortran... in which the .mod files are incompatible between every release. In theory, theory and practice are the same. In practice, the only things that work are those than are used commonly. People test that software stacks build and work together with a single compiler. They don't test random heterogeneous combinations of compilers, libraries, etc. At the end of the day, the advice is simple: build your entire software stack with the same compiler (and same set of libraries too). When you upgrade compilers, re-build your entire software stack. If you use an auto-builder like Spack or MacPorts, this is easier than upgrading to Windows 10 ;-). -- Elizabeth
On Jun 9, 2016 6:28 AM, "Elizabeth A. Fischer" < elizabeth.fischer@columbia.edu> wrote:
On Thu, Jun 9, 2016 at 2:49 AM, Nathaniel Smith <njs@vorpus.org> wrote:
On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer <elizabeth.fischer@columbia.edu> wrote:
Spack gives you complete control over your compiler. This is
you're building Python extensions, especially C++-based extensions, which must be built with the same compilers use to build Python.
Just to hopefully avoid confusing people -- you mostly only need to worry about this on Windows. On OS X and Linux, I can't think of any situation where using a different compiler for your extension and for Python will cause a problem.
It could cause a problem if your extension is a wrapper around a library
important if that you wrote in C++ or Fortran --- whether you wrote your extension by hand, or especially if you used the Cython variant that generates C++ code (a logical choice, if you're interfacing to a C++ library). If your C++ Cython extension requires a different compiler than that used to build Cython, you will also find that standard Cython ways to building your extension break. Right, if you're compiling cython to C++, and then that C++ is calling into a different piece of C++, then you have two C++ modules that are calling each other, and my caveat about that applies.
Maybe you're right --- maybe the extension itself is built in C, and all C++ elements are safely "encapsulated" away from Python.
They are: Python itself has no C++ interfaces, so the interface between Python and your code is totally stable and compiler-independent in all cases I know of on Linux and OS X.
Maybe you can coax Cython into building extensions with your chosen compiler. But who wants to spend time finding out exactly which heterogeneous compilers work together, and how to avoid the standard Cython build paths?
The problem is more general than compilers --- it can actually bite you ANY time you use two different versions of a shared library in a build that you ultimately link together. Suppose your Python stack was built with NetCDF4 but you built your extension with NetCDF3? In a software stack of 50 packages, the possibilities for this kind of problem are endless --- or more precisely, there are 50 opportunities for this kind of problem. And
Sure, if you have the means to easily use the same compiler everywhere then it's a reasonable choice and definitely safe. But there are also lots of people who can't build everything from source all the time or who wish to distribute binaries to users who can't build from source all the time, and I want to make sure those people have accurate information available. Lots of urban legends tend to accrete around this stuff. For example, it's widely believed that on Windows, msvc uses the ".lib" format for describing .dll interfaces, and mingw-w64 uses the ".a" format, and so every time you want to use mingw-w64 and link against a msvc-compiled library, you have to throw away the .lib that shipped shipped with the .dll and write some nasty code to regenerate it from scratch in .a format. It turns out that this is totally wrong: the truth is that both msvc and mingw-w64 are happy to work with both .lib and .a files; the whole legend mostly arose because of a tiny bug in how mingw-w64 was handling .lib files (literally there was a flag field that could have two possible values but it only checked for one of them -- fixing it was a ~6 line patch). But this bug persisted for a decade because all the users who hit this bug decided that it wasn't supposed to work so they never filed a bug report. they are almost impossible to prevent if you're doing things by hand. Sometimes you can get away with it, and sometimes these problems result in strange, time-sucking bugs. Spack ensures that only ONE version of each package is linked in your final binary. Yeah, all major platforms make it *possible* to handle this, but it's tricky and requires some platform specific knowledge that's very very poorly documented. Definitely the easiest approach if you control your whole stack is to just not use multiple different libraries versions together. But if someone is reading this message who has this problem and doesn't control their entire stack, then they should come ask on the wheel-builders@python.org mailing list and we can help them figure out how to make things work right. :-)
and the interface between the libraries uses std::string or std::list, and you're on Linux, then you need to be careful about the GCC 5 / C++11 ABI transition. The short version is that if you consistently use versions of GCC <5 OR consistently use versions of GCC >=5, then you should be fine; the long version can be found by googling :-).
That has never been my experience. My actual experience is every time I upgraded my MacPorts, I had to specifically re-build all my C++ libraries. This was after running into strange, time-sucking bugs. Advise on GCC's C++ supports this:
And then there's Fortran... in which the .mod files are incompatible between every release.
In theory, theory and practice are the same. In practice, the only
http://stackoverflow.com/questions/23895081/can-you-mix-c-compiled-with-diff... Ah interesting -- the issue cited there is the same one as I was talking about, except that I hadn't realized that they accidentally released the breaking change in 4.7.{0,1} (for code that requested the non-default C++11 mode), before reverting it in 4.7.2. So, yeah, fair point, you also should avoid 4.7.{0,1} :-). I'm not saying that dealing with C++ modules that link against each other is easy -- it's actually super annoying right now. But it's definitely doable. And it doesn't apply to pure C interfaces, like the python abi. things that work are those than are used commonly. People test that software stacks build and work together with a single compiler. They don't test random heterogeneous combinations of compilers, libraries, etc. But they do... every single person using binary wheels or conda on Linux or OS X is testing exactly this case every day. (At least the heterogenous compilers part, and often the heterogenous libraries part too.) It works fine for millions of people. -n
On Thu, Jun 9, 2016 at 7:37 AM, Elizabeth A. Fischer <elizabeth.fischer@columbia.edu> wrote:
Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?
I think having this information on an installation instruction page is definitely a good idea. I don't think it makes sense to put in every single error message that users should consult the installation instructions; that should be a given.
I'm inclined to agree that putting instructions for several different systems in the message is overkill, and is a slippery slope, but a friendlier error message with links to documentation (and maybe general, non-Cython-specific docs on building Python extension modules too).
It is my belief that non-developers should not be manually building software. Nor should developers be doing that; if you're saying "./configure" yourself, it's like programming assembly code.
There are many ways to install Cython without manually building:
Unfortunately sometimes, especially with Python software, the line between "developer" and "non-developer" is blurry. This is especially true of scientific software where Cython finds its most use. It's not uncommon, for example, for Cython to be pulled in as a build dependency of something else. Of course as pointed out downthread this is less of an issue on most platforms now that binary wheels are available for so many (including now manylinux!). So I guess point taken that when it comes to just building Cython itself this is less of an issue than it used to be.
* If you're on Linux (and maybe Mac too), install with Spack. This can be as easy as typing: spack install py-cython
Spack gives you complete control over your compiler. This is important if you're building Python extensions, especially C++-based extensions, which must be built with the same compilers use to build Python.
I use Spack to build my complete software stack: the application software I work on requires about 50 dependencies to build, and another ~20 Python dependencies to analyze the output. That includes the full Numpy/Scipy stack. Spack makes it easy for me to use consistent compiler and libraries for Python and my libraries. I've had success making Spack-based install instructions that others (non-developers) can follow and get my software stack installed on their systems: https://github.com/citibeth/icebin
This one's new to me. Will have to investigate--this could be relevant to my interests. Thanks, Erik
On Wed, Jun 8, 2016 at 1:25 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
+1, want to submit a pull request?
On Tue, Jun 7, 2016 at 11:28 PM, Abdeali Kothari <abdealikothari@gmail.com> wrote:
Hi,
Currently, when a user installs cython, if the Python headers are not found, an error message saying "Python.h: no such file or directory" is shown (Example: https://justpaste.it/v0gz). Would it be possible to suggest an action to install the headers ?
I found http://trac.cython.org/ticket/247 which is quite old, and makes the error message easier to find, but doesn't recommend a solution.
`sys.platform`, `platform.linux_distribution` or `distro`[1] can be used to find which OS, platform, etc is and use that to suggest a command like:
if sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python-dev` should install the Python headers.")
elif sys.version_info < (3, ) and sys.platform.startswith("linux") and platform.linux_distribution()[0] in ("ubuntu", "debian"): print("Python headers were not found. On Debian/Ubuntu, `sudo apt-get install python3-dev` should install the Python3 headers.")
This would help so that non-developers can use cython based packages easily as it provides helpful instructions on what to do. Currently, you essentially search for the error message and go to one of the forums/blogs explaining what can be done.
Or maybe by default it can just point the user to an Installation Page which explains in detail what needs to be done to get those headers for various systems ?
[1] - https://pypi.python.org/pypi/distro
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
participants (5)
-
Abdeali Kothari -
Elizabeth A. Fischer -
Erik Bray -
Nathaniel Smith -
Robert Bradshaw