[Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 11 October 2014 19:32, Nathaniel Smith <njs at pobox.com> wrote:
> The bigger problem is that getting a usable DLL at all is a serious
> challenge. Some of the issues we deal with: (a) the classic, stable mingw
> has no 64-bit support, (b) the only portable way to compile fortran (f2c)
> only works for the ancient fortran 77, (c) getting even mingw-w64 to use a
> msvc-compatible ABI is not trivial (I have no idea why this is the case, but
> it is), (d) mingw-built dlls normally depend on the mingw runtime dlls.
> Because these aren't shipped globally with Python, they have to be either
> linked statically or else a separate copy of them has to be placed into
> every directory that contains any mingw-compiled extension module.

These are all genuine and difficult issues. I have some knowledge of
the mingw environment, although only as a user, and I can confirm that
it is not in an ideal state. As you mention, the core project only
offers 32-bit compilers, and the 64-bit versions are separate, not
always reliable, projects. It can be the case that getting a
successful build relies on using a precise distributed archive of the
relevant mingw binaries.

By default, mingw uses msvcrt, for essentially licensing reasons (to
do with the fact that msvcrt is the only version they can consider as
being "shipped with the OS" which is relevant to their use of the
GPL). To get it to link to an alternative VC runtime requires a symbol
library for that runtime, which needs someone to build it. I don't
know whether mingw includes runtime symbol libraries for later than
msvcr10[1] - I asked for that to be added when Python switched to
VC10, and it was fairly difficult to get that done, even at that point
when the mingw community was much less fragmented than now.

It is possible to build C extensions with mingw in such a way that
they don't depend on the mingw runtime DLLs - at least the distutils
support made that happen for the average extension when I last worked
on it which was pre-Python 3... I'm pretty sure that C++ needs runtime
support DLLs which would be tricky to avoid, and I have no idea what
sorts of difficulty Fortran might add (although your comments sound
about what I expect).

> All the runtime and ABI issues do mean that it would be much easier to use
> mingw(-w64) to build extension modules if Python itself were built with
> mingw(-w64). Obviously this would in turn make it harder to build extensions
> with MSVC, though, which would be a huge transition. I don't know whether
> gcc's advantages (support for more modern C, better cross-platform
> compatibility, better accessibility to non-windows-experts, etc.) would
> outweigh the transition and other costs.

As mentioned above, I don't think the mingw environment is that
reliable, which would be an issue if Python were built with it. Would
it really be a positive step if we had to say "to build Python you
need to download this precise personal build from a specific mingw64
spin-off project"? And yes, I have code for which that is *precisely*
what I need to do.

> As an intermediate step, there are almost certainly things that could be
> done to make it easier to use mingw-w64 to build python extensions, e.g.
> teaching setuptools about how to handle the ABI issues. Maybe it would even
> be possible to ship the mingw runtimes in some globally available location.

As I've said a number of times now, I think this is much more likely
to be a useful avenue. For example, shipping the appropriate
libmsvcrXXX.a and static libraries for the relevant runtimes with
distutils, instead of relying on the user having a version of mingw
that does so. And testing (and fixing if necessary) the distutils
MingwCompiler class with a wider range of mingw builds. Note that
where I say distutils here, it would ideally be something that we do
in setuptools, so that it won't be tied to the stdlib release cycles.
But AFAIK, setuptools doesn't yet include any compiler classes, so
that'd be a bigger change. I have no idea what the most appropriate
direction to take would be here.

By the way, Steve Dower - distutils\cygwinccompiler will need the new
MSVC runtime added to get_msvcr() for 3.5/VC15. It won't help
unless/until mingw ships the runtime symbol library, of course, but if
it's not added to the shipped Python 3.5, it'll be a pain to add
later... It doesn't seem to be in your VC14 branch.

Paul

[1] I just checked and it seems that there's a msvcr110 library
shipped with the mingw I have, but nothing later.