Status of C compilers for Python on Windows
Hi, Windows is not the primary target of Python developers, probably because most of them work on Linux. Official Python binaries are currently built by Microsoft Visual Studio. Even if Python developers get free licenses thanks for Microsoft, I would prefer to use an open source compiler if it would be possible. So *anyone* can build Python from scatch. I don't like the requirement of having a license to build Python. The free version (Visual Studio Express) only supports 32-bit and doesn't support PGO build (Profile-Guided Optimizations, which are disabled if I remember correctly because of compiler bugs). I know that it's hard to replace Visual Studio. I don't want to do it right now, but I would like to discuss that with you. === Open Watcom Jeffrey Armstrong is working on the Python support of OpenWatcom(v2), see: http://lightningpython.org/ https://bitbucket.org/ArmstrongJ/lightning-python This compiler was initially written on MS-DOS in 32-bit, but it now supports Windows and Linux as well. The 64-bit mode is new and experimental. The Open Watcom "v2" project is actively developed at: https://github.com/open-watcom/open-watcom-v2/ On Linux, Open Watcom don't support dynamic linking. On Windows, it uses its own C library. I'm not sure that Open Watcom is the best choice to build Python on Windows. === MinGW Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw We even got some patches: http://bugs.python.org/issue3871 (rejected) See also: https://stackoverflow.com/questions/15365249/build-python-with-mingw-and-gcc MinGW reuses the Microsoft C library and it is based on GCC which is very stable, actively developed, supports a lot of archiectures, etc. I guess that it should be possible to reuse third party GCC tools like the famous GDB debugger? === Cywin Cygwin was written compile POSIX applications on Windows and it provides a DLL for that. Python doesn't need that, it uses directly the Windows native API. I don't think that we should use Cygwin. === Clang I have no idea of the support of Clang on Windows. Which C library is used? I found some pages: http://clang.llvm.org/docs/MSVCCompatibility.html http://blog.llvm.org/2014/07/clangllvm-on-windows-update.html This article starts with "It’s time for an update on Clang’s support for building native Windows programs, compatible with Visual C++!". Good. I see binaries for Windows. === Other compilers? Do you know other C compiler which can be used to build Python? === Requirements A compiler alone is not enough. To develop, we need tools to automate the compilation, we need a good debugger, and other similar tools. (Personally, I don't need an IDE. I mostly write code on Linux and only run Windows to ensure that my code works on Windows before pushing it.) IMO 64-bit support is simply required (we currently provide 64-bit binaries on Windows). Supporting ARM can be interesting for Windows 8 and Windows 9. Some parts of Python are low-level like ctypes with libffi. The _decimal module uses libmpdec which is highly optimized. In short, the goal is to have a full working standard Python library. It's probably better to reuse the Microsoft C library instead of having to embed a different C library. What do you think? What about the Python stable ABI? Would it be broken if we use a different compiler? What about third party Python extensions? What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.? Victor
On Fri, Oct 10, 2014 at 1:29 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
Windows is not the primary target of Python developers, probably because most of them work on Linux. Official Python binaries are currently built by Microsoft Visual Studio. Even if Python developers get free licenses thanks for Microsoft, I would prefer to use an open source compiler if it would be possible. So *anyone* can build Python from scatch. I don't like the requirement of having a license to build Python. The free version (Visual Studio Express) only supports 32-bit and doesn't support PGO build (Profile-Guided Optimizations, which are disabled if I remember correctly because of compiler bugs).
I know that it's hard to replace Visual Studio. I don't want to do it right now, but I would like to discuss that with you.
=== Open Watcom
Jeffrey Armstrong is working on the Python support of OpenWatcom(v2), see: http://lightningpython.org/ https://bitbucket.org/ArmstrongJ/lightning-python
This compiler was initially written on MS-DOS in 32-bit, but it now supports Windows and Linux as well. The 64-bit mode is new and experimental. The Open Watcom "v2" project is actively developed at:
https://github.com/open-watcom/open-watcom-v2/
On Linux, Open Watcom don't support dynamic linking. On Windows, it uses its own C library. I'm not sure that Open Watcom is the best choice to build Python on Windows.
=== MinGW
Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw
We even got some patches: http://bugs.python.org/issue3871 (rejected)
See also: https://stackoverflow.com/questions/15365249/build-python-with-mingw-and-gcc
MinGW reuses the Microsoft C library and it is based on GCC which is very stable, actively developed, supports a lot of archiectures, etc. I guess that it should be possible to reuse third party GCC tools like the famous GDB debugger?
You may want to get in touch with Carl Kleffner -- he's done a bunch of work lately on getting a mingw-based toolchain to the point where it can build numpy and scipy. (This is pretty urgent for us because (a) numerical work requires a BLAS library and the main competitive open-source one -- OpenBLAS -- cannot be built by msvc because of asm syntax issues, (b) msvc's fortran support is even worse than its C99 support.) Getting this working is non-trivial, since by default mingw-compiled code depends on the GCC runtime libraries, the default ABI doesn't match msvc, etc. But apparently these issues are all fixable. General info: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain The built toolchains etc.: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads Readme: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/readme.txt The patch to the numpy sources -- this in particular includes the various distutils hacks needed to enable the crucial ABI-compatibility switches: https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/numpy.patch (Unfortunately he doesn't seem to have posted the build recipe for the toolchain itself -- I'm sure he'd be happy to if you asked though.) AFAICT the end result is a single free compiler toolchain that can spit out 32- and 64-bit binaries using whichever MSVC runtime you prefer. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
Nathaniel Smith <njs@pobox.com> wrote:
You may want to get in touch with Carl Kleffner -- he's done a bunch of work lately on getting a mingw-based toolchain to the point where it can build numpy and scipy.
To build *Python extensions*, one can use Carl's toolchain or the VC9 compiler for Python 2.7 that Microsoft just released. To build *Python* you need Visual Studio, Visual Studio Express, Windows SDK, or Cygwin because there is no other build process available on Windows. Python cannot be built with MinGW. The official 64-bit Python installer from Python.org is built with the Windows SDK compiler, not Visual Studio. The Windows SDK is a free download. The 32-bit installer is built with Visual Studio. Sturla
On 10 October 2014 01:29, Victor Stinner <victor.stinner@gmail.com> wrote:
What about the Python stable ABI? Would it be broken if we use a different compiler?
What about third party Python extensions?
What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.?
The key point for me is that any supported build on Windows supports the exact same ABI. It is difficult for Windows users to set up a build environment (and changing the compiler will not alter that fact) so Windows users will rely on binary builds. If multiple ABIs exist, users will have the problem of projects shipping only one ABI binary, and if it doesn't match their Python, they are out of luck. It's critical that we don't double the number of binary builds projects need to ship. Having said that, I'm personally not interested in this, as I am happy with MSVC Express. Python 3.5 will be using MSVC 14, where the express edition supports both 32 and 64 bit. The licensing doesn't bother me personally, and the compiler is easy to install for people who want it. Any competing build environment would have to be as easy to install and use as MSVC Express to be worthwhile, IMO. The only advantage[1] to a new compiler would be if it trivially supported cross-compiling on Linux, as that would allow Limux developers to easily ship Windows binaries. Paul [1] I am not commenting on philosophical advantages like licensing here.
Paul Moore <p.f.moore@gmail.com> wrote:
Having said that, I'm personally not interested in this, as I am happy with MSVC Express. Python 3.5 will be using MSVC 14, where the express edition supports both 32 and 64 bit.
If you build Python yourself, you can (more or less) use whichever version of Visual Studio you want. There is nothing that prevents you from building Python 2.7 or 3.4 with MSVC 14. But then you have to build all Python extensions with this version of Visual Studio as well. Sturla
2014-10-10 11:18 GMT+02:00 Sturla Molden <sturla.molden@gmail.com>:
If you build Python yourself, you can (more or less) use whichever version of Visual Studio you want. There is nothing that prevents you from building Python 2.7 or 3.4 with MSVC 14.
Python 2.7 provides project files (PCbuild/*) for Visual Studio 2008. Python 3.4 provides project files (PCbuild/*) for Visual Studio 2010. Does it mean that VC 2010 supports VC 2008 project files? If I remember correctly, I got a wizzard proposing to convert the project files. I didn't try it, and I installed VS 2008 instead. I guess that VS 2014 requires a similar conversion wizzard for VS 2010 and VS 2008 project files. Victor
On 10/10/2014 08:07 AM, Paul Moore wrote:
What about the Python stable ABI? Would it be broken if we use a different compiler?
What about third party Python extensions?
What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.? The key point for me is that any supported build on Windows supports
On 10 October 2014 01:29, Victor Stinner <victor.stinner@gmail.com> wrote: the exact same ABI.
Just to make something clear that may not be clear to non-Windows developers: the C library is implicitly part of the ABI. So unless these other compilers bend over backwards to generate code / have a C library that behaves *exactly* like MSVC, their ABI will be different, and therefore shared libraries compiled with one compiler won't work with the next. Heck, even shared libraries compiled with one version of MSVC won't work with any other version! (This is something apparently being fixed by MSVC 15; apparently they are designing the ABI for forwards compatibility. Huzzah!) So if CPython officially said "we support MSVC and Compiler X", I worry that we'd have third-party modules compiled with either one or the other, leaving users unable to mix and match third-party extensions as they do today. ("I want to use library X, which is only available compiled by MSVC. I also want to use library Y, which is only available compiled by Compiler X. What should I do?" "... install Linux?") Here's my perspective. Having your code compilable by more compilers is good. But a maze of #ifdefs is bad. We still have #ifdef's for Borland C--I'd be very surprised if anyone was compiling Python 3 with Borland C. IMO the benefit from supporting other compilers on Windows is negligible, but the costs in maintaining these other compilers is tangible. Or, worse, we accept changes to support these other compilers, but the support is incomplete, or goes unmaintained and breaks (and nobody notices). So as a practical matter I think I'd prefer if we continued to only support MSVC. In fact I'd prefer it if we removed support for other Windows compilers, instead asking those maintainers to publish their own patches / repos, in the way that Stackless does. //arry/
Hi, Paul Moore wrote:
The key point for me is that any supported build on Windows supports the exact same ABI.
It looks like ABI compatibility is a goal of Clang on Windows: http://clang.llvm.org/docs/MSVCCompatibility.html http://blog.llvm.org/2014/07/clangllvm-on-windows-update.html I don't know the status of the compatibility for the C ABI with VS 2008 and VS 2010. (These articles look to be focused on C++.) OpenWatcom and Cygwin are not compatible with VS. Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough. I guess that a full ABI compatibility means more than just using the C library, calling convention and much more. Clang documentation mentions for example debug symbols compatible with the Microsoft debugger.
... therefore shared libraries compiled with one compiler won't work with the next.
I noticed this issue when I provided wheel packages for Python 2.7 and 3.3 using the same Windows SDK (7.1)... Python 2.7 and 3.3 from python.org are built with different versions of VS, and so require a different version of the Windows SDK (7.0 for Python 2.7, 7.1 for Python 3.3).
So if CPython officially said "we support MSVC and Compiler X", I worry that we'd have third-party modules compiled with either one or the other, leaving users unable to mix and match third-party extensions as they do today.
Ok, I understand and I agree. Currently, VS is the defacto standard, at least for Python.
We still have #ifdef's for Borland C--I'd be very surprised if anyone was compiling Python 3 with Borland C.
I opened an issue yesterday to drop support of this compiler! Please write your comment there to support my patch. http://bugs.python.org/issue22592
IMO the benefit from supporting other compilers on Windows is negligible, but the costs in maintaining these other compilers is tangible. Or, worse, we accept changes to support these other compilers, but the support is incomplete, or goes unmaintained and breaks (and nobody notices).
If we decide to support officially a C compiler different than VS on Windows, it should be a real support. It should be possible to build Python without any patch, and we should have a buildbot. And someone should maintain the support for this compiler (fix all bugs). Untested code always break (later). Victor
Victor Stinner <victor.stinner@gmail.com> wrote:
Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough.
Not out of the box. See: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain Sturla
On 10 October 2014 10:50, Victor Stinner <victor.stinner@gmail.com> wrote:
Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough. I guess that a full ABI compatibility means more than just using the C library, calling convention and much more.
MinGW can be made to build ABI-compatible extensions. Whether this will continue with MSVC 15 I don't know, as it requires a change to add an interface library for the relevant msvcrXX runtime. And the MinGW community is somewhat fragmented these days, with the core project not supporting 64-bit and various external projects doing so. Having said all this, it *is* possible with some effort to use MinGW to build Python extensions. As noted, the numpy developers have done a lot of work on this as some of the libraries they need must be built with mingw. And the state of distutils support for mingw is very sad, as well, IIRC (last time I looked there were a number of open bugs, and very little movement on them). Rather than put effort into more build options for CPython, I think it would be much more beneficial to the Windows community if effort was put into: 1. Improving support for compiling *extensions* with mingw and its 64-bit cousins, in a way that is compatible with a standard MSVC-built Python. This would involve sorting through and applying some of the distutils changes, as well as working with the mingw projects to ensure they will offer support for the MSVC15 runtime ina timely manner. 2. Looking at ways to support cross-compiling Windows extensions from Linux using mingw. I've no idea how practical this would be, but if Linux developers could provide Windows builds without having to maintain a Windows environment, that would be great. As I say, I have no personal interest here, as I use MSVC, but I know the above two options are commonly asked for. Paul
On 10.10.2014 14:05, Paul Moore wrote:
On 10 October 2014 10:50, Victor Stinner <victor.stinner@gmail.com> wrote:
Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough. I guess that a full ABI compatibility means more than just using the C library, calling convention and much more.
MinGW can be made to build ABI-compatible extensions. Whether this will continue with MSVC 15 I don't know, as it requires a change to add an interface library for the relevant msvcrXX runtime. And the MinGW community is somewhat fragmented these days, with the core project not supporting 64-bit and various external projects doing so.
Having said all this, it *is* possible with some effort to use MinGW to build Python extensions. As noted, the numpy developers have done a lot of work on this as some of the libraries they need must be built with mingw. And the state of distutils support for mingw is very sad, as well, IIRC (last time I looked there were a number of open bugs, and very little movement on them).
Rather than put effort into more build options for CPython, I think it would be much more beneficial to the Windows community if effort was put into: .... 2. Looking at ways to support cross-compiling Windows extensions from Linux using mingw. I've no idea how practical this would be, but if Linux developers could provide Windows builds without having to maintain a Windows environment, that would be great.
It is practical. Numpy Windows binaries are built on linux using mingw 3.4.5 and wine. The (vagrant based) setup which is currently used is available here: https://github.com/juliantaylor/numpy-vendor For the next release we do want to look into providing "official" win64 binaries based on the mingw64 toolchain that has been mentioned a few times already. An attempt to do so in the last released failed due to test issues and there were no experienced debuggers available to solve them.
From my perspective cross building for windows is easier than cross building for mac, but thats probably just because I never seriously looked into that.
On 10.10.2014 11:26, Larry Hastings wrote:
On 10/10/2014 08:07 AM, Paul Moore wrote:
What about the Python stable ABI? Would it be broken if we use a different compiler?
What about third party Python extensions?
What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.? The key point for me is that any supported build on Windows supports
On 10 October 2014 01:29, Victor Stinner <victor.stinner@gmail.com> wrote: the exact same ABI.
Just to make something clear that may not be clear to non-Windows developers: the C library is implicitly part of the ABI. So unless these other compilers bend over backwards to generate code / have a C library that behaves *exactly* like MSVC, their ABI will be different, and therefore shared libraries compiled with one compiler won't work with the next. Heck, even shared libraries compiled with one version of MSVC won't work with any other version! (This is something apparently being fixed by MSVC 15; apparently they are designing the ABI for forwards compatibility. Huzzah!)
So if CPython officially said "we support MSVC and Compiler X", I worry that we'd have third-party modules compiled with either one or the other, leaving users unable to mix and match third-party extensions as they do today. ("I want to use library X, which is only available compiled by MSVC. I also want to use library Y, which is only available compiled by Compiler X. What should I do?" "... install Linux?")
Here's my perspective. Having your code compilable by more compilers is good. But a maze of #ifdefs is bad. We still have #ifdef's for Borland C--I'd be very surprised if anyone was compiling Python 3 with Borland C. IMO the benefit from supporting other compilers on Windows is negligible, but the costs in maintaining these other compilers is tangible. Or, worse, we accept changes to support these other compilers, but the support is incomplete, or goes unmaintained and breaks (and nobody notices).
So as a practical matter I think I'd prefer if we continued to only support MSVC. In fact I'd prefer it if we removed support for other Windows compilers, instead asking those maintainers to publish their own patches / repos, in the way that Stackless does.
I don't think this is special to the Windows platform. We already do support quite a few compilers in CPython and for multiple platforms, so keeping support for e.g. MinGW or adding Intel C support wouldn't really make much difference in the overall #ifdef picture ;-) That said, We do need maintainers for this support, so if there are no people willing to support these compilers in CPython, we should use the external port hosting approach for these, IMO. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Larry Hastings <larry@hastings.org> wrote:
Just to make something clear that may not be clear to non-Windows developers: the C library is implicitly part of the ABI.
MacOS X also has this issue, but it less known amon Mac developers! There tends to be multiple versions of the C library, one for each SDK version. If you link the wrong one when building a Python extension it can crash. For example if you have Python built with the 10.6 SDK (e.g. Enthought) but has XCode with the 10.9 SDK as default, you need to build with the flag -mmacosx-version-min=10.6, and for C++ also -stdlib=libstdc++. Not doing so will cause all sorts of mysterious errors. Two other ABI problems on Windows is the stack alignment and the MinGW runtime: On 32-bit applications, MSVC use 16 bit stack alignment whereas MinGW uses 32-bit alignment. This is a common cause of segfaults for Python extensions built with MinGW. Most developers just assume it is sufficient to link the same CRT as Python. Another problem is the MinGW runtime (mingw32.a or mingw32.dll) which conflicts with MSCV and can cause segfaults unless it is statically linked. The vanlilla MinGW distro defaults to dynamic linkage for this library. Because of this a special MinGW toolchain was created for building SciPy on Windows: https://github.com/numpy/numpy/wiki/Mingw-static-toolchain Sturla
Larry Hastings <larry@hastings.org> wrote:
So as a practical matter I think I'd prefer if we continued to only support MSVC. In fact I'd prefer it if we removed support for other Windows compilers, instead asking those maintainers to publish their own patches / repos, in the way that Stackless does.
The scientific community needs to use MinGW or Intel compilers because of Fortran. So some support for other compilers will be good, at least for building C extensions. Sturla
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/10/2014 05:26 AM, Larry Hastings wrote:
IMO the benefit from supporting other compilers on Windows is negligible
Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, raising the priority of mingw-buildable extensions for numerical work? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlQ37vYACgkQ+gerLs4ltQ6V9ACffcq9Cn+kHzDZxaWJ63NVb6Uk SasAoK5kNfBrCupcBz3FcRsUjs2aZxvu =LFqg -----END PGP SIGNATURE-----
I don't think this is exactly on the same axis. Being able Python to build with a free compiler won't change this issue. Scientific Python won't be only the free compiler version, Visual Studio would remain the main citizen. It may fragment a little bit more the environment with people needing to pay attention to which compiler/Python version the Python extension/module is compatible with. And jettisonning Visual Studio for building Python is not an option because of the amount of applications relying on both (Visual Studio and Python) in companies. Cheers, Matthieu 2014-10-10 15:36 GMT+01:00 Tres Seaver <tseaver@palladion.com>:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 10/10/2014 05:26 AM, Larry Hastings wrote:
IMO the benefit from supporting other compilers on Windows is negligible
Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, raising the priority of mingw-buildable extensions for numerical work?
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAlQ37vYACgkQ+gerLs4ltQ6V9ACffcq9Cn+kHzDZxaWJ63NVb6Uk SasAoK5kNfBrCupcBz3FcRsUjs2aZxvu =LFqg -----END PGP SIGNATURE-----
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/matthieu.brucher%40gmail....
-- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/
On 10 October 2014 15:36, Tres Seaver <tseaver@palladion.com> wrote:
On 10/10/2014 05:26 AM, Larry Hastings wrote:
IMO the benefit from supporting other compilers on Windows is negligible
Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, raising the priority of mingw-buildable extensions for numerical work?
The proposal here is to make it possible to build *python* with a different compiler. Being able to compile extensions with mingw is independent. It used to be relatively easy to build extensions with mingw, although Python's support for that has bitrotted to the point where it's not worth it unless you have a strong need (which is why the SciPy folks are working on it). In my view, anyone capable of modifying the Python build process to support other compilers would also be more than capable of improving the distutils support for building extensions with mingw in a way that is compatible with the current MSVC-built Python. While I can't dictate where anyone else chooses to spend their time, it seems to me that working on allowing non-MSVC builds of Python is a sad waste of limited resources while the mingw extension building issues remain. Paul
On 10/10/2014 03:36 PM, Tres Seaver wrote:
On 10/10/2014 05:26 AM, Larry Hastings wrote:
IMO the benefit from supporting other compilers on Windows is negligible Did you miss the OP's point that OpenBLAS cannot be compiled with MSVC, raising the priority of mingw-buildable extensions for numerical work?
CPython doesn't require OpenBLAS. Not that I am not receptive to the needs of the numeric community... but, on the other hand, who in the hell releases a library with Windows support that doesn't work with MSVC?! //arry/
Larry Hastings <larry@hastings.org> wrote:
CPython doesn't require OpenBLAS. Not that I am not receptive to the needs of the numeric community... but, on the other hand, who in the hell releases a library with Windows support that doesn't work with MSVC?!
It uses AT&T assembly syntax instead of Intel assembly syntax. Sturla
On Sat, 11 Oct 2014 00:30:51 +0000 (UTC) Sturla Molden <sturla.molden@gmail.com> wrote:
Larry Hastings <larry@hastings.org> wrote:
CPython doesn't require OpenBLAS. Not that I am not receptive to the needs of the numeric community... but, on the other hand, who in the hell releases a library with Windows support that doesn't work with MSVC?!
It uses AT&T assembly syntax instead of Intel assembly syntax.
But you can compile OpenBLAS with one compiler and then link it to Python using another compiler, right? There is a single C ABI. Regards Antoine.
Antoine Pitrou <solipsis@pitrou.net> wrote:
But you can compile OpenBLAS with one compiler and then link it to Python using another compiler, right? There is a single C ABI.
BLAS and LAPACK are actually Fortran, which does not have a single C ABI. The ABI depends on the Fortran compiler. g77 and gfortran will produce different C ABIs. This is a consistent source of PITA in any scientific programming that combines C and Fortran. There is cblas though, which is a C API, but it does not include LAPACK. Another thing is that libraries are different. MSVC wants a .lib file, but MinGW produces .a files like GCC does on Linux. Perhaps you can generate a .lib file from a .a file, but I have never tried. Sturla
On Sat, 11 Oct 2014 13:59:52 +0000 (UTC) Sturla Molden <sturla.molden@gmail.com> wrote:
Antoine Pitrou <solipsis@pitrou.net> wrote:
But you can compile OpenBLAS with one compiler and then link it to Python using another compiler, right? There is a single C ABI.
BLAS and LAPACK are actually Fortran, which does not have a single C ABI. The ABI depends on the Fortran compiler. g77 and gfortran will produce different C ABIs. This is a consistent source of PITA in any scientific programming that combines C and Fortran.
But is that CPython's fault? I don't think so.
Another thing is that libraries are different. MSVC wants a .lib file, but MinGW produces .a files like GCC does on Linux.
It sound like whatever MSVC produces should be the defacto standard under Windows. If Microsoft released a (GNU/)Linux compiler which produced incompatible library files, I don't think anyone would regard them very highly. Regards Antoine.
Antoine Pitrou <solipsis@pitrou.net> wrote:
It sound like whatever MSVC produces should be the defacto standard under Windows.
Yes, and that is what Clang does on Windows. It is not as usable as MinGW yet, but soon it will be. Clang also suffers fronthe lack of a Fortran compiler, though. Sturla
Is there some reason the Fortran part can't be separated out into a DLL? That's the C ABI Antoine was referring to, and most compilers can generate import libraries from binaries, even if the original compiler produced then in a different format. Top-posted from my Windows Phone ________________________________ From: Sturla Molden<mailto:sturla.molden@gmail.com> Sent: 10/11/2014 7:22 To: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows Antoine Pitrou <solipsis@pitrou.net> wrote:
It sound like whatever MSVC produces should be the defacto standard under Windows.
Yes, and that is what Clang does on Windows. It is not as usable as MinGW yet, but soon it will be. Clang also suffers fronthe lack of a Fortran compiler, though. Sturla _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
Steve Dower <Steve.Dower@microsoft.com> wrote:
Is there some reason the Fortran part can't be separated out into a DLL?
DLL hell, I assume. Using the Python extension module loader makes it less of a problem. If we stick with .pyd files where everything is statically linked we can rely on the Python dev team to make sure that DLL hell does not bite us. Most of the contributors to projects like NumPy and SciPy are not computer scientists. So the KISS principle is important, which is why scientific programmers often use Fortran in the first place. Making sure DLLs are resolved and loaded correctly, or using stuff like COM or .NET to mitigate DLL hell, is just in a different league. That is for computer engineers to take care of, but we are trained as physicists, matematicians, astronomers, chemists, biologists, or what ever... I am sure that engineers at Microsoft could do this correctly, but we are not the kind of guys you would hire :-) OT: Contrary to common belief, there is no speed advantage of using Fortran on a modern CPU, because the long pipeline and the hierarchical memory alleviates the problems with pointer aliasing. C code tends to run faster then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster than C. In 2014, Fortran is only used because it is easier to program for non-specialists. And besides, correctness is far more important than speed, which is why we prefer Python or MATLAB in the first place. If you ever see the argument that Fortran is used because of pointer aliasing, please feel free to ignore it. Sturla
DLLs linked by import library at compile time (ie. not using LoadLibrary calls) and placed in the same directory as the .pyd should be exempt from DLL hell - Python already creates an activation context when importing pyds to let them load their own dependencies. Multiple CRTs are also okay as long as they don't try and share state (such as file descriptors) across boundaries. I do understand the lack of knowledge and experience though. I've helped out in the past but I personally don't have the bandwidth to contribute more, nor the persuasiveness to get someone else to. Top-posted from my Windows Phone ________________________________ From: Sturla Molden<mailto:sturla.molden@gmail.com> Sent: 10/11/2014 9:59 To: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows Steve Dower <Steve.Dower@microsoft.com> wrote:
Is there some reason the Fortran part can't be separated out into a DLL?
DLL hell, I assume. Using the Python extension module loader makes it less of a problem. If we stick with .pyd files where everything is statically linked we can rely on the Python dev team to make sure that DLL hell does not bite us. Most of the contributors to projects like NumPy and SciPy are not computer scientists. So the KISS principle is important, which is why scientific programmers often use Fortran in the first place. Making sure DLLs are resolved and loaded correctly, or using stuff like COM or .NET to mitigate DLL hell, is just in a different league. That is for computer engineers to take care of, but we are trained as physicists, matematicians, astronomers, chemists, biologists, or what ever... I am sure that engineers at Microsoft could do this correctly, but we are not the kind of guys you would hire :-) OT: Contrary to common belief, there is no speed advantage of using Fortran on a modern CPU, because the long pipeline and the hierarchical memory alleviates the problems with pointer aliasing. C code tends to run faster then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster than C. In 2014, Fortran is only used because it is easier to program for non-specialists. And besides, correctness is far more important than speed, which is why we prefer Python or MATLAB in the first place. If you ever see the argument that Fortran is used because of pointer aliasing, please feel free to ignore it. Sturla _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
On 12 Oct 2014 04:20, "Steve Dower" <Steve.Dower@microsoft.com> wrote:
DLLs linked by import library at compile time (ie. not using LoadLibrary
calls) and placed in the same directory as the .pyd should be exempt from DLL hell - Python already creates an activation context when importing pyds to let them load their own dependencies. Multiple CRTs are also okay as long as they don't try and share state (such as file descriptors) across boundaries.
I do understand the lack of knowledge and experience though. I've helped
out in the past but I personally don't have the bandwidth to contribute more, nor the persuasiveness to get someone else to. The current key phrase in getting potential corporate sponsors interested in the scientific Python stack is "big data analytics". The professional programming world is actually quite bad at using modern mathematical techniques effectively - the scientific community are way ahead of us. Python is relatively unique in being a language used extensively by both professional programmers *and* research scientists, so we're well positioned to serve as a basis for effective communication between the two groups. AMPLab at Berkeley are one of the groups at the forefront of that. MS are sponsors, so if you can find the group behind that sponsorship, you may find folks in a position to help out with the scientific Python toolchain challenges. Cheers, Nick.
Top-posted from my Windows Phone ________________________________ From: Sturla Molden Sent: 10/11/2014 9:59
To: python-dev@python.org Subject: Re: [Python-Dev] Status of C compilers for Python on Windows
Steve Dower <Steve.Dower@microsoft.com> wrote:
Is there some reason the Fortran part can't be separated out into a
DLL?
DLL hell, I assume. Using the Python extension module loader makes it less of a problem. If we stick with .pyd files where everything is statically linked we can rely on the Python dev team to make sure that DLL hell does not bite us. Most of the contributors to projects like NumPy and SciPy are not computer scientists. So the KISS principle is important, which is why scientific programmers often use Fortran in the first place. Making sure DLLs are resolved and loaded correctly, or using stuff like COM or .NET to mitigate DLL hell, is just in a different league. That is for computer engineers to take care of, but we are trained as physicists,
matematicians,
astronomers, chemists, biologists, or what ever... I am sure that engineers at Microsoft could do this correctly, but we are not the kind of guys you would hire :-)
OT: Contrary to common belief, there is no speed advantage of using Fortran on a modern CPU, because the long pipeline and the hierarchical memory alleviates the problems with pointer aliasing. C code tends to run faster then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster than C. In 2014, Fortran is only used because it is easier to program for non-specialists. And besides, correctness is far more important than speed, which is why we prefer Python or MATLAB in the first place. If you ever see the argument that Fortran is used because of pointer aliasing, please feel free to ignore it.
Sturla
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
I'm not at all an expert on Fortran ABIs, but I think there are two distinct issues being conflated here. The first is that there is no standard way to look at some Fortran source code and figure out the corresponding C API. When trying to call a Fortran routine from C, then different Fortran compilers require different sorts of name mangling, different ways of mapping Fortran concepts like "output arguments" onto C concepts like "pointers", etc., so your C code needs to have explicit knowledge of which Fortran compiler is in use. That's what Sturla was referring to with the differences between g77 versus gfortran etc. This is all very annoying, but it isn't a *deep* bad - it can be solved by unilateral action by whatever project wants to link the Fortran library. The bigger problem is that getting a usable DLL at all is a serious challenge. Some of the issues we deal with: (a) the classic, stable mingw has no 64-bit support, (b) the only portable way to compile fortran (f2c) only works for the ancient fortran 77, (c) getting even mingw-w64 to use a msvc-compatible ABI is not trivial (I have no idea why this is the case, but it is), (d) <https://github.com/rust-lang/rust/issues/1768#issuecomment-4007553>mingw-built dlls normally depend on the mingw runtime dlls. Because these aren't shipped globally with Python, they have to be either linked statically or else a separate copy of them has to be placed into every directory that contains any mingw-compiled extension module. All the runtime and ABI issues do mean that it would be much easier to use mingw(-w64) to build extension modules if Python itself were built with mingw(-w64). Obviously this would in turn make it harder to build extensions with MSVC, though, which would be a huge transition. I don't know whether gcc's advantages (support for more modern C, better cross-platform compatibility, better accessibility to non-windows-experts, etc.) would outweigh the transition and other costs. As an intermediate step, there are almost certainly things that could be done to make it easier to use mingw-w64 to build python extensions, e.g. teaching setuptools about how to handle the ABI issues. Maybe it would even be possible to ship the mingw runtimes in some globally available location. -n On 11 Oct 2014 17:07, "Steve Dower" <Steve.Dower@microsoft.com> wrote:
Is there some reason the Fortran part can't be separated out into a DLL? That's the C ABI Antoine was referring to, and most compilers can generate import libraries from binaries, even if the original compiler produced then in a different format.
Top-posted from my Windows Phone ------------------------------ From: Sturla Molden <sturla.molden@gmail.com> Sent: 10/11/2014 7:22 To: python-dev@python.org Subject: Re: [Python-Dev] Status of C compilers for Python on Windows
Antoine Pitrou <solipsis@pitrou.net> wrote:
It sound like whatever MSVC produces should be the defacto standard under Windows.
Yes, and that is what Clang does on Windows. It is not as usable as MinGW yet, but soon it will be. Clang also suffers fronthe lack of a Fortran compiler, though.
Sturla
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
On 11 October 2014 19:32, Nathaniel Smith <njs@pobox.com> wrote:
The bigger problem is that getting a usable DLL at all is a serious challenge. Some of the issues we deal with: (a) the classic, stable mingw has no 64-bit support, (b) the only portable way to compile fortran (f2c) only works for the ancient fortran 77, (c) getting even mingw-w64 to use a msvc-compatible ABI is not trivial (I have no idea why this is the case, but it is), (d) mingw-built dlls normally depend on the mingw runtime dlls. Because these aren't shipped globally with Python, they have to be either linked statically or else a separate copy of them has to be placed into every directory that contains any mingw-compiled extension module.
These are all genuine and difficult issues. I have some knowledge of the mingw environment, although only as a user, and I can confirm that it is not in an ideal state. As you mention, the core project only offers 32-bit compilers, and the 64-bit versions are separate, not always reliable, projects. It can be the case that getting a successful build relies on using a precise distributed archive of the relevant mingw binaries. By default, mingw uses msvcrt, for essentially licensing reasons (to do with the fact that msvcrt is the only version they can consider as being "shipped with the OS" which is relevant to their use of the GPL). To get it to link to an alternative VC runtime requires a symbol library for that runtime, which needs someone to build it. I don't know whether mingw includes runtime symbol libraries for later than msvcr10[1] - I asked for that to be added when Python switched to VC10, and it was fairly difficult to get that done, even at that point when the mingw community was much less fragmented than now. It is possible to build C extensions with mingw in such a way that they don't depend on the mingw runtime DLLs - at least the distutils support made that happen for the average extension when I last worked on it which was pre-Python 3... I'm pretty sure that C++ needs runtime support DLLs which would be tricky to avoid, and I have no idea what sorts of difficulty Fortran might add (although your comments sound about what I expect).
All the runtime and ABI issues do mean that it would be much easier to use mingw(-w64) to build extension modules if Python itself were built with mingw(-w64). Obviously this would in turn make it harder to build extensions with MSVC, though, which would be a huge transition. I don't know whether gcc's advantages (support for more modern C, better cross-platform compatibility, better accessibility to non-windows-experts, etc.) would outweigh the transition and other costs.
As mentioned above, I don't think the mingw environment is that reliable, which would be an issue if Python were built with it. Would it really be a positive step if we had to say "to build Python you need to download this precise personal build from a specific mingw64 spin-off project"? And yes, I have code for which that is *precisely* what I need to do.
As an intermediate step, there are almost certainly things that could be done to make it easier to use mingw-w64 to build python extensions, e.g. teaching setuptools about how to handle the ABI issues. Maybe it would even be possible to ship the mingw runtimes in some globally available location.
As I've said a number of times now, I think this is much more likely to be a useful avenue. For example, shipping the appropriate libmsvcrXXX.a and static libraries for the relevant runtimes with distutils, instead of relying on the user having a version of mingw that does so. And testing (and fixing if necessary) the distutils MingwCompiler class with a wider range of mingw builds. Note that where I say distutils here, it would ideally be something that we do in setuptools, so that it won't be tied to the stdlib release cycles. But AFAIK, setuptools doesn't yet include any compiler classes, so that'd be a bigger change. I have no idea what the most appropriate direction to take would be here. By the way, Steve Dower - distutils\cygwinccompiler will need the new MSVC runtime added to get_msvcr() for 3.5/VC15. It won't help unless/until mingw ships the runtime symbol library, of course, but if it's not added to the shipped Python 3.5, it'll be a pain to add later... It doesn't seem to be in your VC14 branch. Paul [1] I just checked and it seems that there's a msvcr110 library shipped with the mingw I have, but nothing later.
Sturla Molden <sturla.molden@gmail.com> wrote:
BLAS and LAPACK are actually Fortran, which does not have a single C ABI. The ABI depends on the Fortran compiler. g77 and gfortran will produce different C ABIs. This is a consistent source of PITA in any scientific programming that combines C and Fortran.
There is cblas though, which is a C API, but it does not include LAPACK.
Another thing is that libraries are different. MSVC wants a .lib file, but MinGW produces .a files like GCC does on Linux. Perhaps you can generate a .lib file from a .a file, but I have never tried.
And not to mention that the Fortran run-time depends on the C runtime... What Carl Keffner did for SciPy was to use a static libgfortran, which is not liked against any specific CRT, so it could be linked with msvcr90.dll when the Python extension is built. The vanilla libgfortran.dll from MinGW is linked with msvcrt.dll. However, not linking with msvcrt.dll broke the pthreads library, which in turn broke OpenMP, so he had to patch the pthreads library for this... This just shows some of the difficulties of trying to combine the GNU and Microsoft compilers. There are many others, like different stack alignment, differenr exception handling, and the mingw runtime (which causes segfaults when linked dynamically to MSVC executables). It's not just getting the CRT right. Sturla
On 11 Oct 2014 14:42, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
On Sat, 11 Oct 2014 00:30:51 +0000 (UTC) Sturla Molden <sturla.molden@gmail.com> wrote:
Larry Hastings <larry@hastings.org> wrote:
CPython doesn't require OpenBLAS. Not that I am not receptive to the needs of the numeric community... but, on the other hand, who in the hell releases a library with Windows support that doesn't work with
MSVC?!
It uses AT&T assembly syntax instead of Intel assembly syntax.
But you can compile OpenBLAS with one compiler and then link it to Python using another compiler, right? There is a single C ABI.
In theory, yes, but this is pretty misleading. The theory has been known for years. In practice we've only managed to pull this off for the first time within the last few months, and it requires one specific build of one specific mingw fork with one specific set of build options, and only one person understands the details. Hopefully all those things will continue to be available and there aren't any showstopper bugs we haven't noticed yet... (Before that we've spent the last 5+ years using a carefully preserved build of an ancient 32 bit mingw and substandard BLAS .dll that were handed down from our ancestors; we've had no capability to produce 64 bit official builds at all. And a large proportion of users have been using 3rd party proprietary builds.) -n
On 10 October 2014 02:29, Victor Stinner <victor.stinner@gmail.com> wrote:
The free version (Visual Studio Express) only supports 32-bit
VC++ 2008/2010 EE do not *bundle* a 64-bit compiler, but it's certainly possible to build 64-bit applications by using the compiler in the (also free) Windows SDK: http://jenshuebel.wordpress.com/2009/02/12/visual-c-2008-express-edition-and... https://stackoverflow.com/questions/1865069/how-to-compile-a-64-bit-applicat...
Victor Stinner <victor.stinner <at> gmail.com> writes:
Hi,
Windows is not the primary target of Python developers, probably because most of them work on Linux. Official Python binaries are currently built by Microsoft Visual Studio. Even if Python developers get free licenses thanks for Microsoft, I would prefer to use an open source compiler if it would be possible.
=== Other compilers?
I'm happily using MSYS2 (sourceforge.net/projects/msys2/), which handles Python and many Python and GNOME related projects using the mingw64 compiler (which can target 32 and 64 bit). MSYS2 provides a POSIX like environment to build native applications on windows (using the win32 api). It features a source level package manager, pacman, ported from ArchLinux, so lots of *nix applications and libraries are available (see https://github.com/Alexpux/MINGW-packages and also https://github.com/Alexpux/MSYS2-packages/ for core libs and apps). I've used it with success in apps that use Python + GTK+3 + Numpy + SciPy + Matplotlib. Hope this information is useful. Regards, Rafael
On Thu, Oct 9, 2014 at 7:29 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
Windows is not the primary target of Python developers, probably because most of them work on Linux. Official Python binaries are currently built by Microsoft Visual Studio. Even if Python developers get free licenses thanks for Microsoft, I would prefer to use an open source compiler if it would be possible. So *anyone* can build Python from scatch. I don't like the requirement of having a license to build Python. The free version (Visual Studio Express) only supports 32-bit and doesn't support PGO build (Profile-Guided Optimizations, which are disabled if I remember correctly because of compiler bugs).
I know that it's hard to replace Visual Studio. I don't want to do it right now, but I would like to discuss that with you.
Although I'm not very active around here much anymore, I was primarily working on Windows things within the last few years. While we have a lot of Windows users, we don't have a lot of Windows contributors. The huge amount of churn necessary to make a change away from VS, or the more likely move to make it possible to support both VS and <insert other compiler>, seems like a large amount of work that doesn't turn up much of a benefit. Especially for a platform with constrained developer availability, working software trumps all, so I don't expect that a project like this is going to see the regular contributors shifting their focus away from improving Python as it is. With that said, I do see the benefit of being able to build Python with a free compiler. It would be great for us to be able to say it's always built with free tools, but I'm not sure who's going to make this happen...
On 10/10/2014 01:29, Victor Stinner wrote:
=== MinGW
Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw
We even got some patches: http://bugs.python.org/issue3871 (rejected)
There are 55 open issues on the bug tracker with mingw in the title.
See also: https://stackoverflow.com/questions/15365249/build-python-with-mingw-and-gcc
MinGW reuses the Microsoft C library and it is based on GCC which is very stable, actively developed, supports a lot of archiectures, etc. I guess that it should be possible to reuse third party GCC tools like the famous GDB debugger?
-- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On 10 October 2014 17:28, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
There are 55 open issues on the bug tracker with mingw in the title.
It's not easy to tell, but on a spot check a fair proportion of them seem to be about distutils/extension builds. And a lot of the rest are related to http://bugs.python.org/issue3871 which is a rejected issue about adding build support for mingw. Paul
Paul Moore wrote:
On 10 October 2014 17:28, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
There are 55 open issues on the bug tracker with mingw in the title.
It's not easy to tell, but on a spot check a fair proportion of them seem to be about distutils/extension builds. And a lot of the rest are related to http://bugs.python.org/issue3871 which is a rejected issue about adding build support for mingw.
Rejection is for "all in one". It was requested to split in separate patches to be able developers to review them more easy.
Paul
Roumen
From Victor Stinner:
I know that it's hard to replace Visual Studio. I don't want to do it right now, but I would like to discuss that with you.
I have read the rest of the thread, but I want to start from this point. I'm probably going to run off in random directions since there are a lot of issues raised here (all of which I've heard before and given serious thought and discussion to), so please excuse the brain dump. As one of the driving forces behind keeping MSVC a viable choice for building CPython, I'd be disappointed if we were to change away from it (as would my employer). That said, I'm involved with Python because I want to be involved - I was developing in Python long before Microsoft hired me and I'm incredibly lucky that my day job permits me to be involved in this community - and it's not going to directly hurt my career if everyone decides to move away from MSVC, so I don't believe I'm conflicted here. The main way I'm currently trying to keep MSVC viable is by porting CPython 3.5 to build with the latest version (VC14 - yet to be released, but there are previews available). My progress is in my sandbox at https://hg.python.org/sandbox/steve.dower (VC14 branch), I've been regularly testing with the latest (internal) builds, fixing issues in Python and getting issues fixed in OpenSSL, Tcl and VC. The known PGO bugs have been fixed for VC14, and one of my plans is to do some thorough testing to see if it's safe and worth re-enabling. Some answers to points that were raised in this discussion: * VC 14 Express for Desktop can build both 32-bit and 64-bit versions of CPython - I'm also making changes to Python's VC projects so they can build correctly without VS, though I don't know if we'll have a compiler only package for VC14 * The main advantage of VC14 is that the C runtime will be binary compatible into the future (rather than *mostly* source compatible) - most of the macros are now function calls and opaque types like FILE* are genuinely opaque * Extensions built with VC14 or later will work with CPython built with VC14 or later - If other compilers begin to support the new CRT ABI and can match the calling convention, then it won't matter which compiler is used for extensions - The MS CRT has public sources - they ship with any version of VC14 - which should help * There are plenty of issues that prevent you from building with VC14 currently, just like with any other compiler that isn't VC10. - Platforms and compilers need dedicated maintainers I don't have any official confirmation, but my guess would be that the 64-bit compilers were omitted from the VC 2008 Express to save space (bearing in mind that WinXP was the main target at that time, which had poor 64-bit support, and very few people cared about building 64-bit binaries) and were not available in the IDE for VC 2010 Express by mistake. For building extensions, the former is resolved by the package at http://aka.ms/vcpython27, and the latter works fine since the 64-bit compiler is there, just not exposed in the IDE. Neither of these will be an issue with VC14 - 64-bit is far too important these days. Cross compilation is a valid issue, but I hope that build services like Appveyor also help out here. There is regular talk about the PSF/PyPI providing something similar, though I have doubts about its feasibility under any model other than renting a preconfigured VM. I don't see any reason why projects couldn't apply to the PSF for a grant to cover Appveyor costs or the expenses of similar services. As for extensions with Fortran or BLAS dependencies - the Python ABI requirements only extend to the Python interface. You can easily define a second interface within your project and build dependencies with whatever tools you like (and in theory if your compiler emits COFF modules, you can link objects from separate compilers, though that's probably easier said than done). When you control both sides of the ABI, the tooling is less important. It's just unfortunate that right now the ABI for Python is defined by the tooling - hopefully the VC14 CRT will help stabilise this. Currently the official CPython 2.7 build from python.org is built with VS 2008 Ultimate SP1 and I assume 3.4 is built with VS 2010 SP1, since I know Martin has access to the full version of VS. As Paul mentioned a few times, I'd also be quite happy to see all the effort towards building CPython with other compilers go towards building extensions with other compilers. On Windows, very very few people ever build their own binaries for anything - it's a totally different culture to *nix - so it's far more important that the development teams can build their own product. In this scenario, it's more important that extension developers know exactly what the ABI is so they can match it, than whether it's exactly the same as what their tools use by default. I don't think there's anything actionable in there, but here's a few things we could do (discussion points - I'm not necessarily +1 on all of them): * Deprecate/remove support for compiling CPython itself with compilers other than MSVC on Windows * Add preprocessor magic to pyconfig.h to fail extension builds with mismatched ABIs * Improve distutils to automatically detect and support more compilers and cross-compilation where possible * Help out on distutils-sig to finish the PEPs that are holding up improvements to binary distribution * Reach out to companies to help specific projects (e.g. ask Company X if they're willing/able to produce official Y builds on Windows on behalf of the Y team) * Help projects apply to the PSF for grants for Appveyor/similar services Cheers, Steve
On Oct 10, 2014, at 6:59 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Cross compilation is a valid issue, but I hope that build services like Appveyor also help out here. There is regular talk about the PSF/PyPI providing something similar, though I have doubts about its feasibility under any model other than renting a preconfigured VM. I don't see any reason why projects couldn't apply to the PSF for a grant to cover Appveyor costs or the expenses of similar services.
I know very little about Windows, but we can spin up arbitrary Windows boxes, build something on them, and then shut them down as long as there’s some way to automate logging into a Windows box and running some commands… Normally I’d do this with SSH but I don’t know if Windows has anything like that. IOW we can totally spin up preconfigured VMs for a Windows build service. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Steve Dower <Steve.Dower@microsoft.com> wrote:
I don't have any official confirmation, but my guess would be that the 64-bit compilers were omitted from the VC 2008 Express to save space (bearing in mind that WinXP was the main target at that time, which had poor 64-bit support, and very few people cared about building 64-bit binaries) and were not available in the IDE for VC 2010 Express by mistake. For building extensions, the former is resolved by the package at http://aka.ms/vcpython27, and the latter works fine since the 64-bit compiler is there, just not exposed in the IDE. Neither of these will be an issue with VC14 - 64-bit is far too important these days.
The 64-bit compiler is in VC 2008 Express as well, just not exposed in the IDE. I know this because when I got the Absoft Fortran compiler I was told to download VC 2008 Express, because Absoft uses the VC9 linker. And indeed there was a 64-bit compiler in VC 2008 Express as well, just not available from the IDE. If I remeber correctly, some fiddling with vcvars.bat was required to turn it on. I never tried to build Python extensions with it, though. In the beginning I thought Absoft had given me the wrong product, because I had ordered a 64-bit Fortran compiler and I "knew" VC 2008 Express was only 32-bit. But they assured me the 64-bit VC9 compiler was there as well, and indeed it was. Sturla
Victor Stinner wrote:
Hi, [SKIP] === MinGW
Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw
We even got some patches: http://bugs.python.org/issue3871 (rejected) [SNIP]
As "all in one" patch it was rejected , but you could find splits: 17605 - mingw-meta: build interpeter core 18653 - mingw-meta: build core modules Lot of people post links to possible issues using GCC windows compiler. A lot of them are not real issues for CPython. In addition for those why would like to cross compile C-extensions for MS Windows either from linux of cygwin then could use this set: 18654 - modernize mingw&cygwin compiler classes I could step in as maintainer for Cygwin and builds based on GCC using mingw* APIs. Regards, Roumen Petrov
On Sun, Oct 12, 2014 at 9:43 AM, Roumen Petrov <bugtrack@roumenpetrov.info> wrote:
Victor Stinner wrote:
Hi,
[SKIP]
=== MinGW
Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw
We even got some patches: http://bugs.python.org/issue3871 (rejected)
[SNIP]
As "all in one" patch it was rejected , but you could find splits: 17605 - mingw-meta: build interpeter core 18653 - mingw-meta: build core modules
Lot of people post links to possible issues using GCC windows compiler. A lot of them are not real issues for CPython.
In addition for those why would like to cross compile C-extensions for MS Windows either from linux of cygwin then could use this set: 18654 - modernize mingw&cygwin compiler classes
I could step in as maintainer for Cygwin and builds based on GCC using mingw* APIs.
+1 for Roumen maintaining GCC cross builds using mingw*. As Rafael Villar Burke mentioned, the MSYS2 project has Native Windows Python builds (for both 3.4.2 and 2.7.8). We use Roumen's split patches (and then our own on top): https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python3 and https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2. To install MSYS2, build then test 32bit and 64bit mingw-w64-python3 on a fresh 64bit Windows installation: Download http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-2014100... Run msys2-x86_64-20141003.exe and install to a (short) path without spaces or non-ascii characters (C:\msys64 is good), keep "Run MSYS2 64bit now." ticked. The remaining commands are to be entered in the MSYS2 mintty shell. # Install the packages necessary to build mingw-w64-python* # using Pacman package manager (answer Y or press enter when prompted): pacman -S base-devel mingw-w64-x86_64-toolchain mingw-w64-i686-toolchain # Download the source recipes and patches # that are used to build all of MSYS2's mingw-w64 packages: git clone https://github.com/Alexpux/MINGW-packages cd MINGW-packages/mingw-w64-python3 # Build it: # (s == sync (install) necessary {make,}dependencies # L == write log files) # answer Y or press enter when prompted # (remove --nocheck if you want to run the testsuite before packaging) makepkg-mingw -sL --nocheck # To install the newly built packages: pacman -U mingw-w64-*.xz # To run them, you should add /mingw64/bin or /mingw32/bin to your PATH # (or launch a new shell via mingw32_shell.bat or mingw64_shell.bat) # Of course, if you don't want to build it from source you can simply issue: pacman -S mingw-w64-python3 .. all of the above applies equally to mingw-w64-python2. If anyone would like to help us to get our work into shape and then merged we would be extremely grateful. Unfortunately Python is one of our most patched packages. In response to Steve Dower's request for discussion: Having an alternative, fully Open Source build system for Python on Windows using a stable Win32 ABI which is compatible all the way back to Windows XP SP3 and Windows XP 64 and can interoperate out of the box with many other tools and libraries (numpy, GNU Fortran - yes we have this, pyQt, pyGTK etc) is something many people would dearly like. We don't wish to usurp Visual Studio as the recommended build system for Windows, we simply want to enable an Open Source choice. For philosophical and practical reasons there are many people who wish to limit their exposure to proprietary, closed build tools such as Visual Studio. That Windows has always been a much more difficult platform for Open Source development is not something that the Open Source community should accept and then work around, rather something we should try to fix. For information on contributing to MSYS2 please see https://sourceforge.net/p/msys2/wiki/Contributing%20to%20MSYS2/ Finally, this thread has contained many references to mingw, care should be taken to be explicit about which of MinGW-w64 or mingw is being referred to, since they are two different projects. MinGW-w64 supports 64bit and a lot of work is being done to support ARM. Best regards, Ray Donnelly.
Regards, Roumen Petrov
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
participants (19)
-
Antoine Pitrou -
Brian Curtin -
Donald Stufft -
Julian Taylor -
Larry Hastings -
M.-A. Lemburg -
Mark Lawrence -
Matthieu Brucher -
Merlijn van Deen -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Rafael Villar Burke -
Ray Donnelly -
Roumen Petrov -
Steve Dower -
Sturla Molden -
Tres Seaver -
Victor Stinner