I've written up a long technical blog post about the compiler and CRT changes in Python 3.5, which will be of interest to those who build and distribute native extensions for Windows. http://stevedower.id.au/blog/building-for-python-3-5/ Hopefully it puts some of the changes we've made into a context where they don't just look like unnecessary pain. Feedback and discussion welcome, either on these lists or on the post itself. Cheers, Steve
On 8/25/2015 2:17 PM, Steve Dower wrote:
I've written up a long technical blog post about the compiler and CRT changes in Python 3.5, which will be of interest to those who build and distribute native extensions for Windows.
http://stevedower.id.au/blog/building-for-python-3-5/
Hopefully it puts some of the changes we've made into a context where they don't just look like unnecessary pain. Feedback and discussion welcome, either on these lists or on the post itself.
This is an excellent technical writeup. Can it be linked to from the devguide, or maybe the C-API docs, if they do not contain everything in the post? -- Terry Jan Reedy
On 25Aug2015 2153, Terry Reedy wrote:
On 8/25/2015 2:17 PM, Steve Dower wrote:
I've written up a long technical blog post about the compiler and CRT changes in Python 3.5, which will be of interest to those who build and distribute native extensions for Windows.
http://stevedower.id.au/blog/building-for-python-3-5/
Hopefully it puts some of the changes we've made into a context where they don't just look like unnecessary pain. Feedback and discussion welcome, either on these lists or on the post itself.
This is an excellent technical writeup. Can it be linked to from the devguide, or maybe the C-API docs, if they do not contain everything in the post?
I probably need to go through the "Building C and C++ Extensions on Windows" chapter (https://docs.python.org/3.5/extending/windows.html) and update it. Judging by the note near the top ("For example, if you are using Python 2.2.1") it's a little out of date :) Things I'd like to see on that page: * setup.py example to build simple extensions * command-line commands to build directly * VS walkthrough for setting up a project (like what is there now) * MinGW walkthrough for building extensions via distutils or directly (I'll need some help with this one) * deeper discussion on DLLs/static linking/distribution (like section 4.3 now, plus details from my post) On the VS walkthrough, my team at work already has a strong interest (and vague plans) to publish VS templates for building Python extensions, which naturally come with docs and maybe a video walkthrough (like https://youtu.be/D9RlT06a1EI, which I did for Python 3.4 without a template). If there's no opposition, it may be neater to link to that rather than walking through VS in Python's docs - then this section would just cover the command line invocations. I have no issues with linking to other IDE's templates/walkthroughs, but I don't know of any that exist yet. Cheers, Steve
On 26 August 2015 at 17:14, Steve Dower <steve.dower@python.org> wrote:
On 8/25/2015 2:17 PM, Steve Dower wrote:
I've written up a long technical blog post about the compiler and CRT changes in Python 3.5, which will be of interest to those who build and distribute native extensions for Windows.
* MinGW walkthrough for building extensions via distutils or directly (I'll need some help with this one)
Thanks for the detailed writeup Steve. Do you know how these changes to the python.org Windows binaries would impact on people building extension modules with MinGW? I think that some extension module authors use MinGW for their Windows binaries because then a single compiler installation can compile for multiple Python versions and link against whichever msvcrtXX.dll is needed. For example in the numpy release notes [1]: "The MinGW compilers used to build the official Numpy binary installers for 32-bit Python on Windows can be found in https://github.com/numpy/numpy-vendor." Perhaps it would be good to verify that the numpy MinGW build instructions still work? [1] https://github.com/numpy/numpy/blob/master/INSTALL.txt#L118 -- Oscar
On 01Sep2015 0747, Oscar Benjamin wrote:
Thanks for the detailed writeup Steve. Do you know how these changes to the python.org Windows binaries would impact on people building extension modules with MinGW?
Currently, no version of MinGW AFAIK will link against the UCRT, so they'll suffer from the same mixed-CRT issues as with any other arrangement. There is some work going towards making mingw-w64 work with UCRT, but I am not following it closely despite occasional contact with the dev(s) working on it.
I think that some extension module authors use MinGW for their Windows binaries because then a single compiler installation can compile for multiple Python versions and link against whichever msvcrtXX.dll is needed. For example in the numpy release notes [1]: "The MinGW compilers used to build the official Numpy binary installers for 32-bit Python on Windows can be found in https://github.com/numpy/numpy-vendor."
Perhaps it would be good to verify that the numpy MinGW build instructions still work?
I'd love it for someone to do that. My MinGW skills are so poor though that all I'd be verifying is whether I messed up or not :). Certainly most of numpy builds fine with MSVC - it's just the optimized Fortran pieces that require either Intel's compiler (to link against MSVC) or a complete switch to gcc. Cheers, Steve
[1] https://github.com/numpy/numpy/blob/master/INSTALL.txt#L118
-- Oscar
On 1 September 2015 at 17:15, Steve Dower <steve.dower@python.org> wrote:
On 01Sep2015 0747, Oscar Benjamin wrote:
Thanks for the detailed writeup Steve. Do you know how these changes to the python.org Windows binaries would impact on people building extension modules with MinGW?
Currently, no version of MinGW AFAIK will link against the UCRT, so they'll suffer from the same mixed-CRT issues as with any other arrangement. There is some work going towards making mingw-w64 work with UCRT, but I am not following it closely despite occasional contact with the dev(s) working on it.
One thing that I hit when trying to build vim with VS2015 is that it appears that even compiled object code isn't linkable with the ucrt. I don't know the correct terminology here, so let me explain in a bit more detail (I'm doing this from memory, so some symbol names might be wrong, but you'll get the idea, I hope). If source code refers to FILE* objects, then previously the compiled object file (compiled with mingw) would be linkable with any of the various C runtimes (msvcrt, msvcr10, etc). The object referred to a symbol __iob_base which is present in all the CRTs. If the actual code using the library doesn't actually use any of the functions that refer to stdio, then whether the actual use of that symbol is broken doesn't (seem to) matter in practice. That's likely to be luck, to a certain extent - there's no guarantee that the internals of the FILE* abstraction are compatible between CRTs, but it has been true thus far, and so use of object libraries built with mingw are typically OK, as long as you avoid using things that need the CRT types like FILE*. But with VS 2015 and the ucrt, there is no __iob_base symbol, and so linking to code that refers to stdio fails. The consequence of this (I believe) is that not only does mingw need to be able to link against the ucrt, but it also needs to *compile* with ucrt-compatible headers. In particular, when compiling for the ucrt, mingw needs to use a version of <stdio.h> that doesn't reference __iob_base. As the mingw developers maintain their own headers by some process of "clean room" reimplementation (to satisfy the needs of the GPL, as I understand it) updating the headers to cater for the ucrt could be a non-trivial issue for them. I have no idea what their plans are in this regard. The other side of this is that it means that all object libraries you use need to be recompiled to target ucrt, you can't simply use existing compiled libraries and relink. I hope this makes sense, and my interpretation is accurate - if I've misunderstood the implications of the switch to the ucrt, then please let me know. And if I'm wrong about not being able to use existing compiled libraries, I would definitely appreciate someone telling me what I'm doing wrong, because at the moment I'm unable to build my own copy of vim with VS 2015 because I can't rebuild the xpm library :-( Paul PS My ultimate goal with building Vim with VS 2015 is to create a build that uses the new embeddable distribution of Python 3.5 to create a self-contained copy of Vim with Python 3.5 support enabled. Something I think would be awesome :-)
Hi, that is https://bugs.python.org/msg248716 see also http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest Steve: is there more of that in the new universal runtimes? Carl 2015-09-02 12:16 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
On 01Sep2015 0747, Oscar Benjamin wrote:
Thanks for the detailed writeup Steve. Do you know how these changes to the python.org Windows binaries would impact on people building extension modules with MinGW?
Currently, no version of MinGW AFAIK will link against the UCRT, so
On 1 September 2015 at 17:15, Steve Dower <steve.dower@python.org> wrote: they'll
suffer from the same mixed-CRT issues as with any other arrangement. There is some work going towards making mingw-w64 work with UCRT, but I am not following it closely despite occasional contact with the dev(s) working on it.
One thing that I hit when trying to build vim with VS2015 is that it appears that even compiled object code isn't linkable with the ucrt. I don't know the correct terminology here, so let me explain in a bit more detail (I'm doing this from memory, so some symbol names might be wrong, but you'll get the idea, I hope).
If source code refers to FILE* objects, then previously the compiled object file (compiled with mingw) would be linkable with any of the various C runtimes (msvcrt, msvcr10, etc). The object referred to a symbol __iob_base which is present in all the CRTs. If the actual code using the library doesn't actually use any of the functions that refer to stdio, then whether the actual use of that symbol is broken doesn't (seem to) matter in practice.
That's likely to be luck, to a certain extent - there's no guarantee that the internals of the FILE* abstraction are compatible between CRTs, but it has been true thus far, and so use of object libraries built with mingw are typically OK, as long as you avoid using things that need the CRT types like FILE*.
But with VS 2015 and the ucrt, there is no __iob_base symbol, and so linking to code that refers to stdio fails.
The consequence of this (I believe) is that not only does mingw need to be able to link against the ucrt, but it also needs to *compile* with ucrt-compatible headers. In particular, when compiling for the ucrt, mingw needs to use a version of <stdio.h> that doesn't reference __iob_base. As the mingw developers maintain their own headers by some process of "clean room" reimplementation (to satisfy the needs of the GPL, as I understand it) updating the headers to cater for the ucrt could be a non-trivial issue for them. I have no idea what their plans are in this regard.
The other side of this is that it means that all object libraries you use need to be recompiled to target ucrt, you can't simply use existing compiled libraries and relink.
I hope this makes sense, and my interpretation is accurate - if I've misunderstood the implications of the switch to the ucrt, then please let me know. And if I'm wrong about not being able to use existing compiled libraries, I would definitely appreciate someone telling me what I'm doing wrong, because at the moment I'm unable to build my own copy of vim with VS 2015 because I can't rebuild the xpm library :-(
Paul
PS My ultimate goal with building Vim with VS 2015 is to create a build that uses the new embeddable distribution of Python 3.5 to create a self-contained copy of Vim with Python 3.5 support enabled. Something I think would be awesome :-) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
On 2 September 2015 at 13:01, Carl Kleffner <cmkleffner@gmail.com> wrote:
that is https://bugs.python.org/msg248716 see also http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest
Interesting. But it does confirm that object files have to be rebuilt, and there's no way to use existing object files. So if your source only builds with mingw, it needs mingw's headers updating before you can do that rebuild. FWIW, in the case I hit with xpm, the xpm library only uses FILE* internally - the external APIs are purely in terms of filenames. And it only uses FILE* via the standard C APIs for it. So it's not a simple case of "avoid using CRT types in external APIs". So unless I'm misunderstanding, *any* object code used in a project (whether directly or from a 3rd party library) has to be build using a compiler that supports the ucrt (which, given that mingw doesn't have that support yet, means VS 2015 only). Note: I understand why the move to the ucrt is good, and I view this as very much a short term transitional issue, but it is real and I think it's important people understand the implications. Paul
There are more semantic surprises as well. Some Math functions like powf, acosf and so on are available only with the help of the VS 2015 math.h include file (32 bit case). With exporting symbols names from the universal dll's and creating import libraries you are not done yet. Carl 2015-09-02 14:38 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
On 2 September 2015 at 13:01, Carl Kleffner <cmkleffner@gmail.com> wrote:
that is https://bugs.python.org/msg248716 see also http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest
Interesting. But it does confirm that object files have to be rebuilt, and there's no way to use existing object files. So if your source only builds with mingw, it needs mingw's headers updating before you can do that rebuild.
FWIW, in the case I hit with xpm, the xpm library only uses FILE* internally - the external APIs are purely in terms of filenames. And it only uses FILE* via the standard C APIs for it. So it's not a simple case of "avoid using CRT types in external APIs".
So unless I'm misunderstanding, *any* object code used in a project (whether directly or from a 3rd party library) has to be build using a compiler that supports the ucrt (which, given that mingw doesn't have that support yet, means VS 2015 only).
Note: I understand why the move to the ucrt is good, and I view this as very much a short term transitional issue, but it is real and I think it's important people understand the implications.
Paul
You can also build existing object or static libraries into their own DLL with the old compiler and dynamically link to them. It's not perfect, but it's no worse than trying to link them in directly. Otherwise, your analysis is correct. The OpenSSL issue was because they directly referred to __iob_func in their own code. It submitted a patch for it months ago and they fixed it, but possibly only in 1.0.2. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Paul Moore" <p.f.moore@gmail.com> Sent: 9/2/2015 5:38 To: "Carl Kleffner" <cmkleffner@gmail.com> Cc: "Steve Dower" <steve.dower@python.org>; "Python Dev" <python-dev@python.org> Subject: Re: [Python-Dev] Building Extensions for Python 3.5 on Windows On 2 September 2015 at 13:01, Carl Kleffner <cmkleffner@gmail.com> wrote:
that is https://bugs.python.org/msg248716 see also http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest
Interesting. But it does confirm that object files have to be rebuilt, and there's no way to use existing object files. So if your source only builds with mingw, it needs mingw's headers updating before you can do that rebuild. FWIW, in the case I hit with xpm, the xpm library only uses FILE* internally - the external APIs are purely in terms of filenames. And it only uses FILE* via the standard C APIs for it. So it's not a simple case of "avoid using CRT types in external APIs". So unless I'm misunderstanding, *any* object code used in a project (whether directly or from a 3rd party library) has to be build using a compiler that supports the ucrt (which, given that mingw doesn't have that support yet, means VS 2015 only). Note: I understand why the move to the ucrt is good, and I view this as very much a short term transitional issue, but it is real and I think it's important people understand the implications. Paul
On 2 September 2015 at 14:07, Steve Dower <steve.dower@python.org> wrote:
You can also build existing object or static libraries into their own DLL with the old compiler and dynamically link to them. It's not perfect, but it's no worse than trying to link them in directly.
Interesting approach. "gcc -shared -o xpm.dll xpm.a" does about what I want. Some thoughts: 1. This dynamically links to msvcrt.dll. Is that OK? I guess the answer is "it's no worse than using such a DLL with Python 3.4 would be" :-) 2. I presumably need an import lib. I can get gcc to generate a .a format one, I'll need to see if VC 2015 can handle those, or if there's a way to convert them (I'm sure there is, but it's been a long time since I had to do that...) Anyway, thanks for the response. Just one further question if I may - on going the other way. Is it acceptable to embed Python 3.5 (via fully runtime LoadLibrary/GetProcAddress calls to load python35.dll) in an application that uses an older CRT? My initial instinct is that it probably isn't, but I can't think it through - my head's definitely hurting by now :-) Paul
On 02Sep2015 0803, Paul Moore wrote:
On 2 September 2015 at 14:07, Steve Dower <steve.dower@python.org> wrote:
You can also build existing object or static libraries into their own DLL with the old compiler and dynamically link to them. It's not perfect, but it's no worse than trying to link them in directly.
Interesting approach. "gcc -shared -o xpm.dll xpm.a" does about what I want. Some thoughts:
1. This dynamically links to msvcrt.dll. Is that OK? I guess the answer is "it's no worse than using such a DLL with Python 3.4 would be" :-)
Yeah, "no worse" is the answer here.
2. I presumably need an import lib. I can get gcc to generate a .a format one, I'll need to see if VC 2015 can handle those, or if there's a way to convert them (I'm sure there is, but it's been a long time since I had to do that...)
I'm fairly sure there's a dumpbin.exe/lib.exe dance you can do, but it's been a long time since I've had to do it too.
Anyway, thanks for the response.
Just one further question if I may - on going the other way. Is it acceptable to embed Python 3.5 (via fully runtime LoadLibrary/GetProcAddress calls to load python35.dll) in an application that uses an older CRT? My initial instinct is that it probably isn't, but I can't think it through - my head's definitely hurting by now :-)
The answer is "yes, but", where the but depends on what you're sharing between the application and Python. As long as you're careful to let Python do its memory management and the app do its own and their paths never cross, you'll probably be okay. Cheers, Steve
Paul
participants (5)
-
Carl Kleffner -
Oscar Benjamin -
Paul Moore -
Steve Dower -
Terry Reedy