Daniel Holth writes:

That is a concern, there seem to be a nearly unlimited number of ways that binaries can be incompatible with your platform, I believe some numpy stuff includes the version of a numeric library in their (non-wheel) system.

I designed that assuming PyPI would be expected to host binaries compatible with the official Windows release of Python, rather than including the Visual Studio version in the wheel tags themselves. Maybe you'd have a separate build server / binary package service for a popular but incompatible Windows Python.

The trouble is, mistakes happen, and people can upload stuff built with the "wrong" compiler without realising - say, corporates who build their own Pythons, perhaps for embedding scenarios. It may be that in some situations where there's a DLL mismatch, you'd get weird, hard-to-diagnose errors at run time. I assume that's why distutils is careful to catch the error, but with Wheel tags as they are now, there's no way of checking if a mistake has been made. We don't want people to not trust wheels as a foolproof binary distribution format. I hope some Windows-centric users weigh in on this; it would be good to tie up this loose end soon. Regards, Vinay Sajip

On 27 February 2013 18:49, Chris Lambacher wrote:

On Wed, Feb 27, 2013 at 1:01 PM, Vinay Sajip wrote:

...

The trouble is, mistakes happen, and people can upload stuff built with the "wrong" compiler without realising - say, corporates who build their own Pythons, perhaps for embedding scenarios. It may be that in some situations where there's a DLL mismatch, you'd get weird, hard-to-diagnose errors at run time.

It is pretty hard to do this. For one thing distutils will only use the version of visual studio it was built with so as long as people are using the official Windows binaries there is not problem. Someone would have to self build a version of Python with a different version of Visual Studio and the source does not come with build scripts for multiple versions of Visual Studio. So while it is possible to get modules built with a different version of Visual Studio, it is generally pretty hard to do and unlikely to happen "in the wild". I have heard of some companies creating their own internal build but then they are totally out of using PyPI at that point anyway. Active Python was, last I checked, compatible with the official build and I would expect other builds like Enthought to similarly be compatible or be excluded from using PyPI.

Agreed, while Vinay has a theoretical point, in any practical sense it's likely to be a non-issue. The key on Windows is the version of the MS C runtime. That's what the version of MSVC determines, in essence. Other than MSVC, only gcc (mingw) is supported for building extensions, and gcc builds binaries linking to the same version of the C runtime as the MSVC compiler uses. Certainly someone could build Python with a different (unsupported) version of MSVC, but if they did, I believe that they would not be able to build C extensions unless they also hacked distutils to change the allowed C compiler version. Again, of course, they could do this. But the possibility is pretty remote - and even then, before there would be an issue, they would need to upload generated binaries to PyPI by accident (we can assume that if they've got this far, they understand the issue enough not to do so deliberately!) It's certainly true that the ABI flags are incomplete (there's also the whole stable ABI to consider). But they are certainly good enough for the foreseeable future, and if & when we want to implement a more complete solution, automatically converting to a new scheme is trivial (as we can determine the MSVCRT version from the Python version for all existing cases). Paul.

On 28 February 2013 00:46, Oscar Benjamin wrote:

I apologise if I have misunderstood as I don't understand these things that well. But as someone who uses mingw on Windows (when I am on Windows) I have a question. What if I build with mingw on my computer and then upload to PyPI? Is it guaranteed that if my built version is compatible with standard MSVC-compiled Python x.y on my Windows machine then it also compatible with (standard) Python x.y on other Windows machines.

Yes. Unless you build your own Python *interpreter* with a different compiler, you should have no issues.

Perhaps not relevant but currently building extensions with mingw requires patching distutils (It's been a while since I actually did this because it's already done on all my computers but the bug for it is here http://bugs.python.org/issue12641).

Yes, that is relevant, to the extent that distutils is what sets the flags for mingw to ensure a compatible C runtime. If you patch distutils, you run the risk of breaking that (it's the -lmsvcrXX flag that matters). The patch you refer to will not cause issues, though. Paul

On 28 February 2013 06:07, Nick Coghlan wrote:

On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote:

...

It's certainly true that the ABI flags are incomplete (there's also the whole stable ABI to consider).

The stable ABI is covered in PEP 425: the stable ABI compatibility tag is "abi3", and you use the Python version tag to indicate the minimum required Python version. What part do you feel we missed?

You're confusing the tags here. cpXX is the "python" tag, which would be the same for both stable and normal ABI. The ABI tag for the normal ABI is "none". I'd missed the fact that the PEP specifies "abi3" for the stable ABI. The main issue is that the current tools (wheel and distlib) don't generate it on Windows (the PEP says "The CPython stable ABI is abi3 as in the shared library suffix", but imp.get_suffixes() doesn't show a .abi3 suffix on Windows - indeed, PEP 384 doesn't specify a naming convention for stable ABI compatible extensions on Windows. So a wheel with the API tag 'abi3' would never be accepted as compatible on Windows. For that matter, I don't believe it's even possible to tell if an extension only uses the stable ABI without looking at the source (at least on Windows). That's the main problem, as if I can't tell then how do I know that it's OK to use the extension in a version of Python it wasn't built for? Mainly, though, I just think that the tools need to properly generate a list of supported tags that includes the stable ABI on Windows. The rest is largely just about whether anyone is actually using the stable ABI, and how (as an end user) I can tell given the need to manually adjust tags until more sophisticated builders become available.

...

But they are certainly good enough for the foreseeable future, and if & when we want to implement a more complete solution, automatically converting to a new scheme is trivial (as we can determine the MSVCRT version from the Python version for all existing cases).

Indeed - the intent of PEP 425 is that all Windows wheel binaries using "cpXY" or "abi3" ABI tags *must* be compatible with the corresponding official CPython release from python.org. If they're not, then the tool doing the publishing is using the wrong tags (perhaps by misrepresenting itself at runtime as a normal CPython distribution).

As I said, at least on Windows, tools can't tell if the stable ABI was used, so they *have* to publish wheels as cpXY-none, which is tied to a specific minor version. An extension using the stable ABI would need to specify cpX-abi3 and neither distlib or wheel generates that combination when generating the list of "supported" tags for a Python implementation.

Any Python version linked against a different version of the MSVCRT is *not* considered the same implementation from an ABI perspective, and thus should *not* use the "cpXY" or "abi3" ABI tags. They can really use whatever other tags they like, though (for example "cp27_vc6").

I'm confused here, because you're mixing up tags. I agree that the ABI tag would have to be different. But cpXY is the *python* tag. Are you saying that extensions for Python compiled with a different MSVCRT should be considered as for a different implementation of Python? I can see an argument for that, but I'm not sure it's strong enough to justify changing that as well as the ABI tag. Paul.

On 28 February 2013 13:36, Daniel Holth wrote:

On Thu, Feb 28, 2013 at 4:02 AM, Paul Moore wrote:

...

On 28 February 2013 06:07, Nick Coghlan wrote:

...

On Thu, Feb 28, 2013 at 5:47 AM, Paul Moore wrote:

...

It's certainly true that the ABI flags are incomplete (there's also the whole stable ABI to consider).

The stable ABI is covered in PEP 425: the stable ABI compatibility tag is "abi3", and you use the Python version tag to indicate the minimum required Python version. What part do you feel we missed?

You're confusing the tags here. cpXX is the "python" tag, which would be the same for both stable and normal ABI. The ABI tag for the normal ABI is "none". I'd missed the fact that the PEP specifies "abi3" for the stable ABI.

Almost completely right. Technically the ABI for the normal ABI is also something like cp33d (d = "with debug"). The scheme is supposed to be similar to the filenames inside __pycache__ and of C extensions and is backported to Python 2.7. We don't generate these correctly at least for Python 2 in the wheel reference.

OK, we don't generate anything like that for Windows at the moment, then (where ABI is always "none"). Paul

...

OK, we don't generate anything like that for Windows at the moment, then (where ABI is always "none").

"none" means "this wheel has no dependency on the ABI at all". If a wheel includes C extensions and also has the ABI tag set to none, the tool creating those wheels is broken (although it shouldn't be too harmful in practice if it's also declaring an implementation specific Python dependency, for the reasons already discussed in this thread).

I don't entirely disagree. However, that does mean that at the moment, all existing wheel tools are broken on Windows, as "none" is the only ABI flag used on Windows. For practical purposes, they do work though... :-) PEP 425 is pretty vague on the contents of the ABI field, and tools (wheel and distlib) don't offer much help for a Windows user - they pick up config variable "SOABI" (which is not defined on Windows) and they then add the items in imp.get_suffixes() that start with .abi (again, none on Windows). So, maybe we should be more specific in the PEP. Some questions, that as a Windows-based writer of wheel tools, I don't know the answers to: 1. What are the valid abi values (at least for CPython)? It appears that the answer should be 'cpXY' (with suffixes such as 'm' and 'd' - can someone clarify which and what they mean?) plus 'abi3'. PEP 384 (the stable ABI) talks about new functions being added to the stable ABI, but offers no suggestion on how or if new versions would be specified in that case. Also, shouldn't the stable ABI also have debug and similar variations? This should definitely be stated explicitly in the PEP, IMO. 2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user. 3. Can we reasonably assume that all existing binary distributions (wininst and/or egg) use the 'cpXY' (no stable ABI, no debug, etc) ABI? (I think the answer is probably yes, although the "use" section mentioned above implies that "cp33m" is actually right - what's the "m"?). 4. Can we reasonably assume that all existing build toolchains (distutils and setuptools are the key ones) will generate the 'cpXY' ABI? (Again, I think the answer is yes). There's also the big overriding question, which is do we really need to worry right now? Personally, I feel that the current state of affairs ("none" as the ABI for Windows, and no real support for things like debug builds or the stable ABI) is useful as it stands, and we shouldn't let a quest for perfection block what is already a useful practical tool. (On the other hand, the "none on Windows" issue could give us a migration issue to deal with when we finally fix it, if we leave it as it is for now). Paul