[Python-Dev] Status of C compilers for Python on Windows

[Ray Donnelly]

On Sun, Oct 26, 2014 at 1:45 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> Ray Donnelly wrote:
>> On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> On 25 October 2014 23:22, Chris Angelico <rosuav at gmail.com> wrote:
>>>> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>>> My point is that your "Windows build" would not have the same behaviour
>>>>> as a MSVC-produced Windows build, and so testing it with it would not
>>>>> certify that your code would actually be compatible with genuine
>>>>> MSVC builds of CPython, which we will not stop supporting.
>>>>>
>>>>
>>>> So you're saying it's impossible to support two compilers?
>>>
>>> No, rather that Windows currently only has a single supported compiler
>>> (excluding cygwin, which is essentially a different OS). Adding a
>>> second compiler doesn't just involve adding support for it - which is
>>> all that the people offering mingw patches are doing - but also
>>> involves going through the whole CPython ecosystem locating the places
>>> where there is an implicit assumption that "all Windows builds use the
>>> same compiler" and fixing them. I've already pointed out where this is
>>> a question for pip and wheel. Whoever wants to add support for a
>>> second compiler needs to be willing to do this part of the job as
>>> well.
>>>
>>> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.
>>
>> The msvcrt.dlls that MinGW-w64 depends on are those dating back to
>> Windows XP SP3 / XP64. Ironically, the official Windows CPython
>> doesn't come with any such crt guarantees and you must ensure that the
>> same msvcr??.dll is used for *all* extensions. This puts considerable
>> strain on extension developers to use the correct (or any) version of
>> Visual Studio to build their extensions for CPython on Windows.
>
> We're well aware of this, and it's a big part of why I'm currently migrating CPython to build with VC14, which will not have the same binary compatibility issues. For VC14, the entire CRT has been cleaned up and mostly hidden behind calls into DLLs, so provided the calling conventions match (which they must or everything would crash very quickly), it should be relatively easy to build compatible extensions with MinGW-w64.

Compatibility going forwards though, right? Still it's great to see
positive steps being made for the future of the Windows platform.

>
>> Also, where are the publicly accessible specifications and other technical
>> descriptions that MinGW-w64 would need to implement strong binary
>> compatibility with MSVC? As a random example, those for C++ name
>> mangling and the PDB file format would be very helpful.
>
> C++ name mangling is always an implementation detail and it changes from version to version. Luckily, CPython is entirely in C, so that doesn't matter. PDBs are another red herring - you can build a loadable PE file without PDBs.

Of course C++ can be called from C and that is done in some CPython
extensions, so it's not a red herring. If we want to talk about strong
binary compatibility I'd expect the aim would be to intermix freely
between compilers. We'd like people to be able to debug MinGW-w64 code
using CDB in Visual Studio if they want to, and on the flipside, to
have GDB able to read PDB files built by MSVC (actually there's a long
standing problem when debugging MinGW-w64 code in GDB that stack
unwinding out of MS built dlls is flaky at best) - so again this is
not really a red herring. I'm also led to believe that MSVC has a very
good optimizer so if some project wanted to build certain libraries or
objects with that for their performance critical paths then I can see
that as being useful to those projects and their users'.

>
> The full source code for the MSVCRT is available with any version of Visual Studio (including the free editions, last time I checked), so feel free to check whatever you need to ensure compatibility. I've suggested to the VC team that they could get in touch with the MinGW projects and offer to help them improve compatibility with MSVC, but unfortunately I don't think anyone will take me up on that. I'm happy to research what I can to answer specific questions, but there's very little that isn't already publicly available other than direct access to the devs.

Under what license? We'd rather have open specifications that
copyrighted, strictly licensed code that we can't look at for various
tainting reasons.

>
> Cheers,
> Steve
>
>>> Paul