[Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

From Victor Stinner:
> I know that it's hard to replace Visual Studio. I don't want to do it right now, but I would like to discuss that with you.


I have read the rest of the thread, but I want to start from this point. I'm probably going to run off in random directions since there are a lot of issues raised here (all of which I've heard before and given serious thought and discussion to), so please excuse the brain dump.

As one of the driving forces behind keeping MSVC a viable choice for building CPython, I'd be disappointed if we were to change away from it (as would my employer). That said, I'm involved with Python because I want to be involved - I was developing in Python long before Microsoft hired me and I'm incredibly lucky that my day job permits me to be involved in this community - and it's not going to directly hurt my career if everyone decides to move away from MSVC, so I don't believe I'm conflicted here.

The main way I'm currently trying to keep MSVC viable is by porting CPython 3.5 to build with the latest version (VC14 - yet to be released, but there are previews available). My progress is in my sandbox at https://hg.python.org/sandbox/steve.dower (VC14 branch), I've been regularly testing with the latest (internal) builds, fixing issues in Python and getting issues fixed in OpenSSL, Tcl and VC. The known PGO bugs have been fixed for VC14, and one of my plans is to do some thorough testing to see if it's safe and worth re-enabling.

Some answers to points that were raised in this discussion:

* VC 14 Express for Desktop can build both 32-bit and 64-bit versions of CPython
 - I'm also making changes to Python's VC projects so they can build correctly without VS, though I don't know if we'll have a compiler only package for VC14

* The main advantage of VC14 is that the C runtime will be binary compatible into the future (rather than *mostly* source compatible)
 - most of the macros are now function calls and opaque types like FILE* are genuinely opaque

* Extensions built with VC14 or later will work with CPython built with VC14 or later
 - If other compilers begin to support the new CRT ABI and can match the calling convention, then it won't matter which compiler is used for extensions
 - The MS CRT has public sources - they ship with any version of VC14 - which should help 

* There are plenty of issues that prevent you from building with VC14 currently, just like with any other compiler that isn't VC10.
 - Platforms and compilers need dedicated maintainers

I don't have any official confirmation, but my guess would be that the 64-bit compilers were omitted from the VC 2008 Express to save space (bearing in mind that WinXP was the main target at that time, which had poor 64-bit support, and very few people cared about building 64-bit binaries) and were not available in the IDE for VC 2010 Express by mistake. For building extensions, the former is resolved by the package at http://aka.ms/vcpython27, and the latter works fine since the 64-bit compiler is there, just not exposed in the IDE. Neither of these will be an issue with VC14 - 64-bit is far too important these days.

Cross compilation is a valid issue, but I hope that build services like Appveyor also help out here. There is regular talk about the PSF/PyPI providing something similar, though I have doubts about its feasibility under any model other than renting a preconfigured VM. I don't see any reason why projects couldn't apply to the PSF for a grant to cover Appveyor costs or the expenses of similar services.

As for extensions with Fortran or BLAS dependencies - the Python ABI requirements only extend to the Python interface. You can easily define a second interface within your project and build dependencies with whatever tools you like (and in theory if your compiler emits COFF modules, you can link objects from separate compilers, though that's probably easier said than done). When you control both sides of the ABI, the tooling is less important. It's just unfortunate that right now the ABI for Python is defined by the tooling - hopefully the VC14 CRT will help stabilise this.

Currently the official CPython 2.7 build from python.org is built with VS 2008 Ultimate SP1 and I assume 3.4 is built with VS 2010 SP1, since I know Martin has access to the full version of VS. As Paul mentioned a few times, I'd also be quite happy to see all the effort towards building CPython with other compilers go towards building extensions with other compilers. On Windows, very very few people ever build their own binaries for anything - it's a totally different culture to *nix - so it's far more important that the development teams can build their own product. In this scenario, it's more important that extension developers know exactly what the ABI is so they can match it, than whether it's exactly the same as what their tools use by default. 

I don't think there's anything actionable in there, but here's a few things we could do (discussion points - I'm not necessarily +1 on all of them):

* Deprecate/remove support for compiling CPython itself with compilers other than MSVC on Windows
* Add preprocessor magic to pyconfig.h to fail extension builds with mismatched ABIs
* Improve distutils to automatically detect and support more compilers and cross-compilation where possible
* Help out on distutils-sig to finish the PEPs that are holding up improvements to binary distribution
* Reach out to companies to help specific projects (e.g. ask Company X if they're willing/able to produce official Y builds on Windows on behalf of the Y team)
* Help projects apply to the PSF for grants for Appveyor/similar services

Cheers,
Steve