Nathaniel Smith wrote:

You may want to get in touch with Carl Kleffner -- he's done a bunch of work lately on getting a mingw-based toolchain to the point where it can build numpy and scipy.

To build *Python extensions*, one can use Carl's toolchain or the VC9 compiler for Python 2.7 that Microsoft just released. To build *Python* you need Visual Studio, Visual Studio Express, Windows SDK, or Cygwin because there is no other build process available on Windows. Python cannot be built with MinGW. The official 64-bit Python installer from Python.org is built with the Windows SDK compiler, not Visual Studio. The Windows SDK is a free download. The 32-bit installer is built with Visual Studio. Sturla

Paul Moore wrote:

Having said that, I'm personally not interested in this, as I am happy with MSVC Express. Python 3.5 will be using MSVC 14, where the express edition supports both 32 and 64 bit.

If you build Python yourself, you can (more or less) use whichever version of Visual Studio you want. There is nothing that prevents you from building Python 2.7 or 3.4 with MSVC 14. But then you have to build all Python extensions with this version of Visual Studio as well. Sturla

On 10/10/2014 08:07 AM, Paul Moore wrote:

...

What about the Python stable ABI? Would it be broken if we use a different compiler?

What about third party Python extensions?

What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.? The key point for me is that any supported build on Windows supports

On 10 October 2014 01:29, Victor Stinner wrote: the exact same ABI.

Just to make something clear that may not be clear to non-Windows developers: the C library is implicitly part of the ABI. So unless these other compilers bend over backwards to generate code / have a C library that behaves *exactly* like MSVC, their ABI will be different, and therefore shared libraries compiled with one compiler won't work with the next. Heck, even shared libraries compiled with one version of MSVC won't work with any other version! (This is something apparently being fixed by MSVC 15; apparently they are designing the ABI for forwards compatibility. Huzzah!) So if CPython officially said "we support MSVC and Compiler X", I worry that we'd have third-party modules compiled with either one or the other, leaving users unable to mix and match third-party extensions as they do today. ("I want to use library X, which is only available compiled by MSVC. I also want to use library Y, which is only available compiled by Compiler X. What should I do?" "... install Linux?") Here's my perspective. Having your code compilable by more compilers is good. But a maze of #ifdefs is bad. We still have #ifdef's for Borland C--I'd be very surprised if anyone was compiling Python 3 with Borland C. IMO the benefit from supporting other compilers on Windows is negligible, but the costs in maintaining these other compilers is tangible. Or, worse, we accept changes to support these other compilers, but the support is incomplete, or goes unmaintained and breaks (and nobody notices). So as a practical matter I think I'd prefer if we continued to only support MSVC. In fact I'd prefer it if we removed support for other Windows compilers, instead asking those maintainers to publish their own patches / repos, in the way that Stackless does. //arry/

Victor Stinner wrote:

Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough.

Not out of the box. See: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain Sturla

On 10.10.2014 14:05, Paul Moore wrote:

On 10 October 2014 10:50, Victor Stinner wrote:

...

Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough. I guess that a full ABI compatibility means more than just using the C library, calling convention and much more.

MinGW can be made to build ABI-compatible extensions. Whether this will continue with MSVC 15 I don't know, as it requires a change to add an interface library for the relevant msvcrXX runtime. And the MinGW community is somewhat fragmented these days, with the core project not supporting 64-bit and various external projects doing so.

Having said all this, it *is* possible with some effort to use MinGW to build Python extensions. As noted, the numpy developers have done a lot of work on this as some of the libraries they need must be built with mingw. And the state of distutils support for mingw is very sad, as well, IIRC (last time I looked there were a number of open bugs, and very little movement on them).

Rather than put effort into more build options for CPython, I think it would be much more beneficial to the Windows community if effort was put into: .... 2. Looking at ways to support cross-compiling Windows extensions from Linux using mingw. I've no idea how practical this would be, but if Linux developers could provide Windows builds without having to maintain a Windows environment, that would be great.

It is practical. Numpy Windows binaries are built on linux using mingw 3.4.5 and wine. The (vagrant based) setup which is currently used is available here: https://github.com/juliantaylor/numpy-vendor For the next release we do want to look into providing "official" win64 binaries based on the mingw64 toolchain that has been mentioned a few times already. An attempt to do so in the last released failed due to test issues and there were no experienced debuggers available to solve them.

From my perspective cross building for windows is easier than cross building for mac, but thats probably just because I never seriously looked into that.

Larry Hastings wrote:

Just to make something clear that may not be clear to non-Windows developers: the C library is implicitly part of the ABI.

MacOS X also has this issue, but it less known amon Mac developers! There tends to be multiple versions of the C library, one for each SDK version. If you link the wrong one when building a Python extension it can crash. For example if you have Python built with the 10.6 SDK (e.g. Enthought) but has XCode with the 10.9 SDK as default, you need to build with the flag -mmacosx-version-min=10.6, and for C++ also -stdlib=libstdc++. Not doing so will cause all sorts of mysterious errors. Two other ABI problems on Windows is the stack alignment and the MinGW runtime: On 32-bit applications, MSVC use 16 bit stack alignment whereas MinGW uses 32-bit alignment. This is a common cause of segfaults for Python extensions built with MinGW. Most developers just assume it is sufficient to link the same CRT as Python. Another problem is the MinGW runtime (mingw32.a or mingw32.dll) which conflicts with MSCV and can cause segfaults unless it is statically linked. The vanlilla MinGW distro defaults to dynamic linkage for this library. Because of this a special MinGW toolchain was created for building SciPy on Windows: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain Sturla

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/10/2014 05:26 AM, Larry Hastings wrote:

IMO the benefit from supporting other compilers on Windows is negligible

Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, raising the priority of mingw-buildable extensions for numerical work? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlQ37vYACgkQ+gerLs4ltQ6V9ACffcq9Cn+kHzDZxaWJ63NVb6Uk SasAoK5kNfBrCupcBz3FcRsUjs2aZxvu =LFqg -----END PGP SIGNATURE-----

On 10 October 2014 15:36, Tres Seaver wrote:

On 10/10/2014 05:26 AM, Larry Hastings wrote:

...

IMO the benefit from supporting other compilers on Windows is negligible

Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, raising the priority of mingw-buildable extensions for numerical work?

The proposal here is to make it possible to build *python* with a different compiler. Being able to compile extensions with mingw is independent. It used to be relatively easy to build extensions with mingw, although Python's support for that has bitrotted to the point where it's not worth it unless you have a strong need (which is why the SciPy folks are working on it). In my view, anyone capable of modifying the Python build process to support other compilers would also be more than capable of improving the distutils support for building extensions with mingw in a way that is compatible with the current MSVC-built Python. While I can't dictate where anyone else chooses to spend their time, it seems to me that working on allowing non-MSVC builds of Python is a sad waste of limited resources while the mingw extension building issues remain. Paul

Larry Hastings wrote:

CPython doesn't require OpenBLAS. Not that I am not receptive to the needs of the numeric community... but, on the other hand, who in the hell releases a library with Windows support that doesn't work with MSVC?!

It uses AT&T assembly syntax instead of Intel assembly syntax. Sturla

Antoine Pitrou wrote:

But you can compile OpenBLAS with one compiler and then link it to Python using another compiler, right? There is a single C ABI.

BLAS and LAPACK are actually Fortran, which does not have a single C ABI. The ABI depends on the Fortran compiler. g77 and gfortran will produce different C ABIs. This is a consistent source of PITA in any scientific programming that combines C and Fortran. There is cblas though, which is a C API, but it does not include LAPACK. Another thing is that libraries are different. MSVC wants a .lib file, but MinGW produces .a files like GCC does on Linux. Perhaps you can generate a .lib file from a .a file, but I have never tried. Sturla

Antoine Pitrou wrote:

It sound like whatever MSVC produces should be the defacto standard under Windows.

Yes, and that is what Clang does on Windows. It is not as usable as MinGW yet, but soon it will be. Clang also suffers fronthe lack of a Fortran compiler, though. Sturla

Steve Dower wrote:

Is there some reason the Fortran part can't be separated out into a DLL?

DLL hell, I assume. Using the Python extension module loader makes it less of a problem. If we stick with .pyd files where everything is statically linked we can rely on the Python dev team to make sure that DLL hell does not bite us. Most of the contributors to projects like NumPy and SciPy are not computer scientists. So the KISS principle is important, which is why scientific programmers often use Fortran in the first place. Making sure DLLs are resolved and loaded correctly, or using stuff like COM or .NET to mitigate DLL hell, is just in a different league. That is for computer engineers to take care of, but we are trained as physicists, matematicians, astronomers, chemists, biologists, or what ever... I am sure that engineers at Microsoft could do this correctly, but we are not the kind of guys you would hire :-) OT: Contrary to common belief, there is no speed advantage of using Fortran on a modern CPU, because the long pipeline and the hierarchical memory alleviates the problems with pointer aliasing. C code tends to run faster then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster than C. In 2014, Fortran is only used because it is easier to program for non-specialists. And besides, correctness is far more important than speed, which is why we prefer Python or MATLAB in the first place. If you ever see the argument that Fortran is used because of pointer aliasing, please feel free to ignore it. Sturla

On 12 Oct 2014 04:20, "Steve Dower" wrote:

DLLs linked by import library at compile time (ie. not using LoadLibrary

calls) and placed in the same directory as the .pyd should be exempt from DLL hell - Python already creates an activation context when importing pyds to let them load their own dependencies. Multiple CRTs are also okay as long as they don't try and share state (such as file descriptors) across boundaries.

I do understand the lack of knowledge and experience though. I've helped

out in the past but I personally don't have the bandwidth to contribute more, nor the persuasiveness to get someone else to. The current key phrase in getting potential corporate sponsors interested in the scientific Python stack is "big data analytics". The professional programming world is actually quite bad at using modern mathematical techniques effectively - the scientific community are way ahead of us. Python is relatively unique in being a language used extensively by both professional programmers *and* research scientists, so we're well positioned to serve as a basis for effective communication between the two groups. AMPLab at Berkeley are one of the groups at the forefront of that. MS are sponsors, so if you can find the group behind that sponsorship, you may find folks in a position to help out with the scientific Python toolchain challenges. Cheers, Nick.

Top-posted from my Windows Phone ________________________________ From: Sturla Molden Sent: ‎10/‎11/‎2014 9:59

To: python-dev@python.org Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Steve Dower wrote:

...

Is there some reason the Fortran part can't be separated out into a

DLL?

DLL hell, I assume. Using the Python extension module loader makes it less of a problem. If we stick with .pyd files where everything is statically linked we can rely on the Python dev team to make sure that DLL hell does not bite us. Most of the contributors to projects like NumPy and SciPy are not computer scientists. So the KISS principle is important, which is why scientific programmers often use Fortran in the first place. Making sure DLLs are resolved and loaded correctly, or using stuff like COM or .NET to mitigate DLL hell, is just in a different league. That is for computer engineers to take care of, but we are trained as physicists,

matematicians,

astronomers, chemists, biologists, or what ever... I am sure that engineers at Microsoft could do this correctly, but we are not the kind of guys you would hire :-)

OT: Contrary to common belief, there is no speed advantage of using Fortran on a modern CPU, because the long pipeline and the hierarchical memory alleviates the problems with pointer aliasing. C code tends to run faster then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster than C. In 2014, Fortran is only used because it is easier to program for non-specialists. And besides, correctness is far more important than speed, which is why we prefer Python or MATLAB in the first place. If you ever see the argument that Fortran is used because of pointer aliasing, please feel free to ignore it.

Sturla

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 11 October 2014 19:32, Nathaniel Smith wrote:

The bigger problem is that getting a usable DLL at all is a serious challenge. Some of the issues we deal with: (a) the classic, stable mingw has no 64-bit support, (b) the only portable way to compile fortran (f2c) only works for the ancient fortran 77, (c) getting even mingw-w64 to use a msvc-compatible ABI is not trivial (I have no idea why this is the case, but it is), (d) mingw-built dlls normally depend on the mingw runtime dlls. Because these aren't shipped globally with Python, they have to be either linked statically or else a separate copy of them has to be placed into every directory that contains any mingw-compiled extension module.

These are all genuine and difficult issues. I have some knowledge of the mingw environment, although only as a user, and I can confirm that it is not in an ideal state. As you mention, the core project only offers 32-bit compilers, and the 64-bit versions are separate, not always reliable, projects. It can be the case that getting a successful build relies on using a precise distributed archive of the relevant mingw binaries. By default, mingw uses msvcrt, for essentially licensing reasons (to do with the fact that msvcrt is the only version they can consider as being "shipped with the OS" which is relevant to their use of the GPL). To get it to link to an alternative VC runtime requires a symbol library for that runtime, which needs someone to build it. I don't know whether mingw includes runtime symbol libraries for later than msvcr10[1] - I asked for that to be added when Python switched to VC10, and it was fairly difficult to get that done, even at that point when the mingw community was much less fragmented than now. It is possible to build C extensions with mingw in such a way that they don't depend on the mingw runtime DLLs - at least the distutils support made that happen for the average extension when I last worked on it which was pre-Python 3... I'm pretty sure that C++ needs runtime support DLLs which would be tricky to avoid, and I have no idea what sorts of difficulty Fortran might add (although your comments sound about what I expect).

All the runtime and ABI issues do mean that it would be much easier to use mingw(-w64) to build extension modules if Python itself were built with mingw(-w64). Obviously this would in turn make it harder to build extensions with MSVC, though, which would be a huge transition. I don't know whether gcc's advantages (support for more modern C, better cross-platform compatibility, better accessibility to non-windows-experts, etc.) would outweigh the transition and other costs.

As mentioned above, I don't think the mingw environment is that reliable, which would be an issue if Python were built with it. Would it really be a positive step if we had to say "to build Python you need to download this precise personal build from a specific mingw64 spin-off project"? And yes, I have code for which that is *precisely* what I need to do.

As an intermediate step, there are almost certainly things that could be done to make it easier to use mingw-w64 to build python extensions, e.g. teaching setuptools about how to handle the ABI issues. Maybe it would even be possible to ship the mingw runtimes in some globally available location.

As I've said a number of times now, I think this is much more likely to be a useful avenue. For example, shipping the appropriate libmsvcrXXX.a and static libraries for the relevant runtimes with distutils, instead of relying on the user having a version of mingw that does so. And testing (and fixing if necessary) the distutils MingwCompiler class with a wider range of mingw builds. Note that where I say distutils here, it would ideally be something that we do in setuptools, so that it won't be tied to the stdlib release cycles. But AFAIK, setuptools doesn't yet include any compiler classes, so that'd be a bigger change. I have no idea what the most appropriate direction to take would be here. By the way, Steve Dower - distutils\cygwinccompiler will need the new MSVC runtime added to get_msvcr() for 3.5/VC15. It won't help unless/until mingw ships the runtime symbol library, of course, but if it's not added to the shipped Python 3.5, it'll be a pain to add later... It doesn't seem to be in your VC14 branch. Paul [1] I just checked and it seems that there's a msvcr110 library shipped with the mingw I have, but nothing later.

On 11 Oct 2014 14:42, "Antoine Pitrou" wrote:

On Sat, 11 Oct 2014 00:30:51 +0000 (UTC) Sturla Molden wrote:

...

Larry Hastings wrote:

...

CPython doesn't require OpenBLAS. Not that I am not receptive to the needs of the numeric community... but, on the other hand, who in the hell releases a library with Windows support that doesn't work with

MSVC?!

...

It uses AT&T assembly syntax instead of Intel assembly syntax.

But you can compile OpenBLAS with one compiler and then link it to Python using another compiler, right? There is a single C ABI.

In theory, yes, but this is pretty misleading. The theory has been known for years. In practice we've only managed to pull this off for the first time within the last few months, and it requires one specific build of one specific mingw fork with one specific set of build options, and only one person understands the details. Hopefully all those things will continue to be available and there aren't any showstopper bugs we haven't noticed yet... (Before that we've spent the last 5+ years using a carefully preserved build of an ancient 32 bit mingw and substandard BLAS .dll that were handed down from our ancestors; we've had no capability to produce 64 bit official builds at all. And a large proportion of users have been using 3rd party proprietary builds.) -n

Merlijn van Deen wrote:

VC++ 2008/2010 EE do not *bundle* a 64-bit compiler,

Actually it does, but it is not available from the UI. You can use it from the command line, though. Sturla

On 10 October 2014 17:28, Mark Lawrence wrote:

There are 55 open issues on the bug tracker with mingw in the title.

It's not easy to tell, but on a spot check a fair proportion of them seem to be about distutils/extension builds. And a lot of the rest are related to http://bugs.python.org/issue3871 which is a rejected issue about adding build support for mingw. Paul

Steve Dower wrote:

I don't have any official confirmation, but my guess would be that the 64-bit compilers were omitted from the VC 2008 Express to save space (bearing in mind that WinXP was the main target at that time, which had poor 64-bit support, and very few people cared about building 64-bit binaries) and were not available in the IDE for VC 2010 Express by mistake. For building extensions, the former is resolved by the package at http://aka.ms/vcpython27, and the latter works fine since the 64-bit compiler is there, just not exposed in the IDE. Neither of these will be an issue with VC14 - 64-bit is far too important these days.

The 64-bit compiler is in VC 2008 Express as well, just not exposed in the IDE. I know this because when I got the Absoft Fortran compiler I was told to download VC 2008 Express, because Absoft uses the VC9 linker. And indeed there was a 64-bit compiler in VC 2008 Express as well, just not available from the IDE. If I remeber correctly, some fiddling with vcvars.bat was required to turn it on. I never tried to build Python extensions with it, though. In the beginning I thought Absoft had given me the wrong product, because I had ordered a 64-bit Fortran compiler and I "knew" VC 2008 Express was only 32-bit. But they assured me the 64-bit VC9 compiler was there as well, and indeed it was. Sturla

On Sun, Oct 12, 2014 at 9:43 AM, Roumen Petrov wrote:

Victor Stinner wrote:

...

Hi,

[SKIP]

...

=== MinGW

Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw

We even got some patches: http://bugs.python.org/issue3871 (rejected)

[SNIP]

As "all in one" patch it was rejected , but you could find splits: 17605 - mingw-meta: build interpeter core 18653 - mingw-meta: build core modules

Lot of people post links to possible issues using GCC windows compiler. A lot of them are not real issues for CPython.

In addition for those why would like to cross compile C-extensions for MS Windows either from linux of cygwin then could use this set: 18654 - modernize mingw&cygwin compiler classes

I could step in as maintainer for Cygwin and builds based on GCC using mingw* APIs.

+1 for Roumen maintaining GCC cross builds using mingw*. As Rafael Villar Burke mentioned, the MSYS2 project has Native Windows Python builds (for both 3.4.2 and 2.7.8). We use Roumen's split patches (and then our own on top): https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python3 and https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2. To install MSYS2, build then test 32bit and 64bit mingw-w64-python3 on a fresh 64bit Windows installation: Download http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-2014100... Run msys2-x86_64-20141003.exe and install to a (short) path without spaces or non-ascii characters (C:\msys64 is good), keep "Run MSYS2 64bit now." ticked. The remaining commands are to be entered in the MSYS2 mintty shell. # Install the packages necessary to build mingw-w64-python* # using Pacman package manager (answer Y or press enter when prompted): pacman -S base-devel mingw-w64-x86_64-toolchain mingw-w64-i686-toolchain # Download the source recipes and patches # that are used to build all of MSYS2's mingw-w64 packages: git clone https://github.com/Alexpux/MINGW-packages cd MINGW-packages/mingw-w64-python3 # Build it: # (s == sync (install) necessary {make,}dependencies # L == write log files) # answer Y or press enter when prompted # (remove --nocheck if you want to run the testsuite before packaging) makepkg-mingw -sL --nocheck # To install the newly built packages: pacman -U mingw-w64-*.xz # To run them, you should add /mingw64/bin or /mingw32/bin to your PATH # (or launch a new shell via mingw32_shell.bat or mingw64_shell.bat) # Of course, if you don't want to build it from source you can simply issue: pacman -S mingw-w64-python3 .. all of the above applies equally to mingw-w64-python2. If anyone would like to help us to get our work into shape and then merged we would be extremely grateful. Unfortunately Python is one of our most patched packages. In response to Steve Dower's request for discussion: Having an alternative, fully Open Source build system for Python on Windows using a stable Win32 ABI which is compatible all the way back to Windows XP SP3 and Windows XP 64 and can interoperate out of the box with many other tools and libraries (numpy, GNU Fortran - yes we have this, pyQt, pyGTK etc) is something many people would dearly like. We don't wish to usurp Visual Studio as the recommended build system for Windows, we simply want to enable an Open Source choice. For philosophical and practical reasons there are many people who wish to limit their exposure to proprietary, closed build tools such as Visual Studio. That Windows has always been a much more difficult platform for Open Source development is not something that the Open Source community should accept and then work around, rather something we should try to fix. For information on contributing to MSYS2 please see https://sourceforge.net/p/msys2/wiki/Contributing%20to%20MSYS2/ Finally, this thread has contained many references to mingw, care should be taken to be explicit about which of MinGW-w64 or mingw is being referred to, since they are two different projects. MinGW-w64 supports 64bit and a lot of work is being done to support ARM. Best regards, Ray Donnelly.

Regards, Roumen Petrov

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com