[Cython] Suggested action for "Python.h not found"

[Nathaniel Smith]

On Jun 9, 2016 6:28 AM, "Elizabeth A. Fischer" <
elizabeth.fischer at columbia.edu> wrote:
>
>
>
> On Thu, Jun 9, 2016 at 2:49 AM, Nathaniel Smith <njs at vorpus.org> wrote:
>>
>> On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
>> <elizabeth.fischer at columbia.edu> wrote:
>> > Spack gives you complete control over your compiler.  This is
important if
>> > you're  building Python extensions, especially C++-based extensions,
which
>> > must be built with the same compilers use to build Python.
>>
>> Just to hopefully avoid confusing people -- you mostly only need to
>> worry about this on Windows. On OS X and Linux, I can't think of any
>> situation where using a different compiler for your extension and for
>> Python will cause a problem.
>
>
> It could cause a problem if your extension is a wrapper around a library
that you wrote in  C++ or Fortran --- whether you wrote your extension by
hand, or especially if you used the Cython variant that generates  C++ code
(a logical choice, if you're interfacing to a C++ library).  If your C++
Cython extension requires a different compiler than that used to build
Cython, you will also find that standard  Cython ways to building your
extension break.

Right, if you're compiling cython to C++, and then that C++ is calling into
a different piece of C++, then you have two C++ modules that are calling
each other, and my caveat about that applies.

> Maybe you're right --- maybe the extension itself is built in C, and all
 C++ elements are safely "encapsulated" away from Python.

They are: Python itself has no C++ interfaces, so the interface between
Python and your code is totally stable and compiler-independent in all
cases I know of on Linux and OS X.

> Maybe you can coax Cython into building extensions with your chosen
compiler.  But who wants to spend time finding out exactly which
heterogeneous compilers work together, and how to avoid the standard Cython
build paths?

Sure, if you have the means to easily use the same compiler everywhere then
it's a reasonable choice and definitely safe. But there are also lots of
people who can't build everything from source all the time or who wish to
distribute binaries to users who can't build from source all the time, and
I want to make sure those people have accurate information available.

Lots of urban legends tend to accrete around this stuff. For example, it's
widely believed that on Windows, msvc uses the ".lib" format for describing
.dll interfaces, and mingw-w64 uses the ".a" format, and so every time you
want to use mingw-w64 and link against a msvc-compiled library, you have to
throw away the .lib that shipped shipped with the .dll and write some nasty
code to regenerate it from scratch in .a format.

It turns out that this is totally wrong: the truth is that both msvc and
mingw-w64 are happy to work with both .lib and .a files; the whole legend
mostly arose because of a tiny bug in how mingw-w64 was handling .lib files
(literally there was a flag field that could have two possible values but
it only checked for one of them -- fixing it was a ~6 line patch). But this
bug persisted for a decade because all the users who hit this bug decided
that it wasn't supposed to work so they never filed a bug report.

> The problem is more general than compilers --- it can actually bite you
ANY time you use two different versions of a shared library in a build that
you ultimately link together.  Suppose your Python stack was built with
NetCDF4 but you built your extension with NetCDF3?  In a software stack of
50 packages, the possibilities for this kind of problem are endless --- or
more precisely, there are 50 opportunities for this kind of problem.  And
they are almost impossible to prevent if you're doing things by hand.
Sometimes you can get away with it, and sometimes these problems result in
strange, time-sucking bugs.  Spack ensures that only ONE version of each
package is linked in your final binary.

Yeah, all major platforms make it *possible* to handle this, but it's
tricky and requires some platform specific knowledge that's very very
poorly documented. Definitely the easiest approach if you control your
whole stack is to just not use multiple different libraries versions
together. But if someone is reading this message who has this problem and
doesn't control their entire stack, then they should come ask on the
wheel-builders at python.org mailing list and we can help them figure out how
to make things work right. :-)

>> and the
>> interface between the libraries uses std::string or std::list, and
>> you're on Linux, then you need to be careful about the GCC 5 / C++11
>> ABI transition. The short version is that if you consistently use
>> versions of GCC <5 OR consistently use versions of GCC >=5, then you
>> should be fine; the long version can be found by googling :-).
>
>
> That has never been my experience.  My actual experience is every time I
upgraded my MacPorts, I had to specifically re-build all my C++ libraries.
This was after running into strange, time-sucking bugs.  Advise on GCC's
C++ supports this:
>
>
http://stackoverflow.com/questions/23895081/can-you-mix-c-compiled-with-different-versions-of-the-same-compiler

Ah interesting -- the issue cited there is the same one as I was talking
about, except that I hadn't realized that they accidentally released the
breaking change in 4.7.{0,1} (for code that requested the non-default C++11
mode), before reverting it in 4.7.2. So, yeah, fair point, you also should
avoid 4.7.{0,1} :-). I'm not saying that dealing with C++ modules that link
against each other is easy -- it's actually super annoying right now. But
it's definitely doable. And it doesn't apply to pure C interfaces, like the
python abi.

> And then there's Fortran... in which the .mod files are incompatible
between every release.
>
> In theory, theory and practice are the same.  In practice, the only
things that work are those than are used commonly.  People test that
software stacks build and work together with a single compiler.  They don't
test random heterogeneous combinations of compilers, libraries, etc.

But they do... every single person using binary wheels or conda on Linux or
OS X is testing exactly this case every day. (At least the heterogenous
compilers part, and often the heterogenous libraries part too.) It works
fine for millions of people.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160609/ea96516d/attachment.html>