PYD/SO platform tags (#22980)
Hi distutils-sig, There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows. There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :) Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h). By default, distutils' build_ext will use the first extension, and so builds will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag (for stable ABI) by customizing the build process - I don't think that's an unreasonable expectation, especially on Windows. I can see potential for making more generic installable packages with this change, though I'm not pushing for that. The big win is when you're building and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the output directories and search paths are organized. Thoughts/concerns? Cheers, Steve
On Dec 12, 2014, at 6:23 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Hi distutils-sig,
There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.
There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)
Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).
By default, distutils' build_ext will use the first extension, and so builds will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag (for stable ABI) by customizing the build process - I don't think that's an unreasonable expectation, especially on Windows.
I can see potential for making more generic installable packages with this change, though I'm not pushing for that. The big win is when you're building and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the output directories and search paths are organized.
Thoughts/concerns?
Well here’s the code that computes what tags pip will accept: https://github.com/pypa/pip/blob/develop/pip/pep425tags.py Obviously any additional information put into the tag will increase the number of builds someone has to do in order to get full coverage, however if all of the data in that are things that genuinely make things binary incompatible then that’s the right thing to do. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Fri, 12 Dec 2014 23:23:03 +0000 Steve Dower <Steve.Dower@microsoft.com> wrote:
Hi distutils-sig,
There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.
There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)
Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).
For the record, https://www.python.org/dev/peps/pep-3149/#pep-384 suggests "abi3" as the ABI tag for the Python 3 stable ABI. Regards Antoine.
Antoine Pitrou wrote:
Sent: Friday, December 12, 2014 1551 To: Distutils-Sig@Python.Org Subject: Re: [Distutils] PYD/SO platform tags (#22980)
On Fri, 12 Dec 2014 23:23:03 +0000 Steve Dower <Steve.Dower@microsoft.com> wrote:
Hi distutils-sig,
There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.
There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)
Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).
For the record, https://www.python.org/dev/peps/pep-3149/#pep-384 suggests "abi3" as the ABI tag for the Python 3 stable ABI.
My legalistic rationale for using cp3 is that it's actually the version tag, not the ABI tag. It seemed from my reading that you'd get tags like "cp35-abi3-win32", which is not helpful because you've still specified the minor version in the tag. Either it'll work in any CPython 3.x ("cp3") or it needs a specific minor version ("cp35"), so I didn't see the value in keeping the ABI tag. Of course, this only works because Windows doesn't use the ABI tag for anything else. Debug builds are encoded differently, and AFAICT the malloc and Unicode options are currently unnecessary and I'm not making any change to make them necessary. I could be swayed on this point fairly easily though, if say there are other interpreters out there not compatible enough with CPython to use the cp tag but that can still use stable ABI extensions that were build assuming CPython. Or I could just drop the second tag and only have "cp35-win32.pyd" or ".pyd". We've made it this long without a specifier, so there's no real need to go overboard :) Cheers, Steve
Regards
Antoine.
Donald Stufft wrote:
On Dec 12, 2014, at 6:23 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Hi distutils-sig,
There's a bit of discussion going on at http://bugs.python.org/issue22980 about extending SO tags to include bitness values, and I took the opportunity to look into adding platform tags for .pyd files on Windows.
There's a patch up there, but I'm interested in any thoughts or concerns people may have that I haven't considered. Distribution is likely to be most affected, and I figured distutils-sig is most likely to have people with strong opinions on the subject anyway, so I'm posting here :)
Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the platforms that have #ifdefs in pyconfig.h).
By default, distutils' build_ext will use the first extension, and so builds will get the ".cp35-win32.pyd" tag. You can only get the ".cp3-win32.pyd" tag (for stable ABI) by customizing the build process - I don't think that's an unreasonable expectation, especially on Windows.
I can see potential for making more generic installable packages with this change, though I'm not pushing for that. The big win is when you're building and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the output directories and search paths are organized.
Thoughts/concerns?
Well here’s the code that computes what tags pip will accept: https://github.com/pypa/pip/blob/develop/pip/pep425tags.py
Obviously any additional information put into the tag will increase the number of builds someone has to do in order to get full coverage, however if all of the data in that are things that genuinely make things binary incompatible then that’s the right thing to do.
It could reduce the number of builds that someone has to do, as you could have a single wheel (for Windows) that includes all the platform-specific extensions rather than needing separate wheels. Whether this is an advantage or not deserves to be debated, but there are other installers that could benefit from having incompatible pyd's side-by-side in a way that Python will load the correct one. Oh, it's also essential for user site-packages on Windows, since the same directory is used for both 32-bit and 64-bit versions of Python. That's actually the most compelling use-case for me, since I'm still keen to see about moving to a secure install for Python 3.5 and in that case we'll have more people using it. Cheers, Steve
Steve Dower wrote:
Antoine Pitrou wrote:
My legalistic rationale for using cp3 is that it's actually the version tag, not the ABI tag. It seemed from my reading that you'd get tags like "cp35-abi3-win32", which is not helpful because you've still specified the minor version in the tag. Either it'll work in any CPython 3.x ("cp3") or it needs a specific minor version ("cp35"), so I didn't see the value in keeping the ABI tag.
Apologies, I got this understanding from PEP 425, not 3149/384. Using "abi3-win32" rather than "cp3-win32" would also make sense. I'm not strongly for or against either variant. Cheers, Steve
participants (3)
-
Antoine Pitrou -
Donald Stufft -
Steve Dower