Mailman 3 Moving Python 3.5 on Windows to a new compiler - Python-Dev

Moving Python 3.5 on Windows to a new compiler

older
Issue #21205: add __qualname__ to...

Steve Dower

6 Jun 2014 6 Jun '14

3:41 p.m.

Hi all I would like to propose moving Python 3.5 to use Visual C++ 14.0 as the main compiler. The first CTP of Visual Studio "14" was released earlier this week: http://blogs.msdn.com/b/vcblog/archive/2014/06/03/visual-studio-14-ctp.aspx The major feature of interest in this version of MSVC is a new policy to maintain binary compatibility for the CRT into the future. (There will be a blog about this soon, but I didn't want to hold up getting the discussion started here.) What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later. Those who are aware of the current state of affairs where you need to use a matching compiler will hopefully see how big an improvement this will be. It is also likely that other compilers will have an easier time providing compatibility with this new CRT, making it simpler and more reliable to build extensions with LLVM or GCC against an MSVC CPython. The other major benefit is that both products are at points in their development where changes can be made. Being a Microsoft employee, I have the ability to test Python builds regularly against the daily MSVC builds and to file bugs directly to the VC team (crashes, incorrect code generation, incorrect linking, performance regressions, etc.). This is a great opportunity to make sure that our needs are covered by the compiler team - it's also a good chance to raise any particular missing features that would be beneficial. My internal testing shows that the core code is almost fully compatible and builds successfully with only trivial modifications (some CRT variables are now macros with a leading underscore). The project files need updating, but I am willing to do this as part of any migration. There may also be some work required for external dependencies, since I did not test these, but I am also willing to do that. Basically, what I am offering to do is: * Update the files in PCBuild to work with Visual Studio "14" * Make any code changes necessary to build with VC14 * Regularly test the latest Python source with the latest MSVC builds and report issues/suggestions to the MSVC team * Keep all changes in a separate (public) repo until early next year when we're getting close to the final VS "14" release What I am asking anyone else to do is: * Nothing Thoughts/comments/concerns? Cheers, Steve

Show replies by date

Chris Angelico

6 Jun 6 Jun

4:01 p.m.

On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote:

...

What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable. To what extent is this compatibility going to be maintained? Is there a guarantee that there'll be X versions (or X years) of cross-compilation support? ChrisA

Steve Dower

6:12 p.m.

Chris Angelico wrote:

...

On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote:

...
What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable.

Maybe, but I doubt it will ever be acceptable :)

...

To what extent is this compatibility going to be maintained? Is there a guarantee that there'll be X versions (or X years) of cross-compilation support?

There are a few breaking changes in this version that are designed to standardize on a function-call based ABI, which should effectively be a life-long guarantee. The only promise I can make is this: when cross-compilation support is eventually broken, it will be due to something that nobody has been able to predict up until now. (Hopefully that's better than promising that it will be broken in the very next release.)

...

ChrisA

Chris Angelico

6:19 p.m.

On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote:

...

Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote:

...
What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable.

Maybe, but I doubt it will ever be acceptable :)

Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved. ChrisA

Brian Curtin

6:25 p.m.

On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote:

...

On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote:

...
Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote:

...
What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable.

Maybe, but I doubt it will ever be acceptable :)

Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved.

We're going to have to change it at some point, otherwise we're going to have people in 2018 scrambling to find VS2008, which will be 35 versions too old by then. No matter what we do here, we're going to have a tough PR situation, but we have to make something workable. I'd rather cause a hassle than outright kill extensions. I would probably prefer we aim for VS 14 for 3.5, and then explore making the same change for the 2.7.x release that comes after 3.5.0 comes out. Lessons learned and all that.

M.-A. Lemburg

6:41 p.m.

On 06.06.2014 20:25, Brian Curtin wrote:

...

On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote:

...
Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote:

...
What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable.

Maybe, but I doubt it will ever be acceptable :)

Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved.

We're going to have to change it at some point, otherwise we're going to have people in 2018 scrambling to find VS2008, which will be 35 versions too old by then. No matter what we do here, we're going to have a tough PR situation, but we have to make something workable. I'd rather cause a hassle than outright kill extensions.

I would probably prefer we aim for VS 14 for 3.5, and then explore making the same change for the 2.7.x release that comes after 3.5.0 comes out. Lessons learned and all that.

Are you sure that's an option ? Changing the compiler the stock Python from python.org is built with will most likely render existing Python extensions built for 2.7.x with x < (release that comes after 3.5.0) broken, so users and installation tools will end up having to pay close attention to the patch level version of Python they are using... which is something we wanted to avoid after we ran into this situation with 1.5.1 and 1.5.2 a few years ago. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 06 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-05-28: Released mxODBC.Connect 2.1.0 ... http://egenix.com/go56 2014-07-02: Python Meeting Duesseldorf ... 26 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Brian Curtin

6:49 p.m.

On Fri, Jun 6, 2014 at 10:41 PM, M.-A. Lemburg wrote:

...

On 06.06.2014 20:25, Brian Curtin wrote:

...
On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote:

...
Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote:

...
What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable.

Maybe, but I doubt it will ever be acceptable :)

Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved.

We're going to have to change it at some point, otherwise we're going to have people in 2018 scrambling to find VS2008, which will be 35 versions too old by then. No matter what we do here, we're going to have a tough PR situation, but we have to make something workable. I'd rather cause a hassle than outright kill extensions.

I would probably prefer we aim for VS 14 for 3.5, and then explore making the same change for the 2.7.x release that comes after 3.5.0 comes out. Lessons learned and all that.

Are you sure that's an option ? Changing the compiler the stock Python from python.org is built with will most likely render existing Python extensions built for 2.7.x with x < (release that comes after 3.5.0) broken, so users and installation tools will end up having to pay close attention to the patch level version of Python they are using... which is something we wanted to avoid after we ran into this situation with 1.5.1 and 1.5.2 a few years ago.

None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old. Something less than awesome for everyone involved is going to have to happen to make that possible.

M.-A. Lemburg

6:52 p.m.

On 06.06.2014 20:49, Brian Curtin wrote:

...

On Fri, Jun 6, 2014 at 10:41 PM, M.-A. Lemburg wrote:

...
On 06.06.2014 20:25, Brian Curtin wrote:

...
On Fri, Jun 6, 2014 at 10:19 PM, Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 4:12 AM, Steve Dower wrote:

...
Chris Angelico wrote:

...
On Sat, Jun 7, 2014 at 1:41 AM, Steve Dower wrote: > What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later.

Oh, if only this had been available for 2.7!! Actually... this means that 14.0 would be a good target for a compiler change for 2.7.x, if such a change is ever acceptable.

Maybe, but I doubt it will ever be acceptable :)

Well, there were discussions. Since Python 2.7's support is far exceeding the Microsoft promise of support for the compiler it was built on, there's going to be a problem, one way or the other. I don't know how that's going to end up being resolved.

We're going to have to change it at some point, otherwise we're going to have people in 2018 scrambling to find VS2008, which will be 35 versions too old by then. No matter what we do here, we're going to have a tough PR situation, but we have to make something workable. I'd rather cause a hassle than outright kill extensions.

I would probably prefer we aim for VS 14 for 3.5, and then explore making the same change for the 2.7.x release that comes after 3.5.0 comes out. Lessons learned and all that.

Are you sure that's an option ? Changing the compiler the stock Python from python.org is built with will most likely render existing Python extensions built for 2.7.x with x < (release that comes after 3.5.0) broken, so users and installation tools will end up having to pay close attention to the patch level version of Python they are using... which is something we wanted to avoid after we ran into this situation with 1.5.1 and 1.5.2 a few years ago.

None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old. Something less than awesome for everyone involved is going to have to happen to make that possible.

Perhaps we could combine this with the breakage that a Python 2.7.10 would introduce due to the two digit patch level release version ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 06 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

dw+python-dev＠hmmz.org

6:56 p.m.

On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote:

...

None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old.

Surely that is infinitely less desirable than simply bumping the minor version? David

Brian Curtin

7:04 p.m.

On Fri, Jun 6, 2014 at 10:56 PM, wrote:

...

On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote:

...
None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old.

Surely that is infinitely less desirable than simply bumping the minor version?

It's definitely not desirable, but "simply" bumping the minor version is not A Thing.

Donald Stufft

7:08 p.m.

On Jun 6, 2014, at 3:04 PM, Brian Curtin wrote:

...

On Fri, Jun 6, 2014 at 10:56 PM, wrote:

...
On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote:

...
None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old.

Surely that is infinitely less desirable than simply bumping the minor version?

It's definitely not desirable, but "simply" bumping the minor version is not A Thing.

Why? I mean even if it’s the same thing as 2.7 just with an updated compiler that seems like a better answer than having to deal with 2.7.whatever suddenly breaking all C exts. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Brian Curtin

7:09 p.m.

On Fri, Jun 6, 2014 at 11:08 PM, Donald Stufft wrote:

...

On Jun 6, 2014, at 3:04 PM, Brian Curtin wrote:

...
On Fri, Jun 6, 2014 at 10:56 PM, wrote:

...
On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote:

...
None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old.

Surely that is infinitely less desirable than simply bumping the minor version?

It's definitely not desirable, but "simply" bumping the minor version is not A Thing.

Why? I mean even if it’s the same thing as 2.7 just with an updated compiler that seems like a better answer than having to deal with 2.7.whatever suddenly breaking all C exts.

Because then we have to maintain 2.8 at a time when no one even wants to maintain 2.7?

Donald Stufft

7:11 p.m.

On Jun 6, 2014, at 3:09 PM, Brian Curtin wrote:

...

On Fri, Jun 6, 2014 at 11:08 PM, Donald Stufft wrote:

...
On Jun 6, 2014, at 3:04 PM, Brian Curtin wrote:

...
On Fri, Jun 6, 2014 at 10:56 PM, wrote:

...
On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote:

...
None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old.

Surely that is infinitely less desirable than simply bumping the minor version?

It's definitely not desirable, but "simply" bumping the minor version is not A Thing.

Why? I mean even if it’s the same thing as 2.7 just with an updated compiler that seems like a better answer than having to deal with 2.7.whatever suddenly breaking all C exts.

Because then we have to maintain 2.8 at a time when no one even wants to maintain 2.7?

Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Chris Angelico

7:33 p.m.

On Sat, Jun 7, 2014 at 5:11 AM, Donald Stufft wrote:

...

Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out.

If the only difference between 2.7 and 2.8 is the compiler used on Windows, what happens on Linux and other platforms? A Python 2.8 would have to be materially different from Python 2.7, not just binarily incompatible on one platform. ChrisA

Donald Stufft

7:36 p.m.

On Jun 6, 2014, at 3:33 PM, Chris Angelico wrote:

...

On Sat, Jun 7, 2014 at 5:11 AM, Donald Stufft wrote:

...
Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out.

If the only difference between 2.7 and 2.8 is the compiler used on Windows, what happens on Linux and other platforms? A Python 2.8 would have to be materially different from Python 2.7, not just binarily incompatible on one platform.

ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Well it’d contain bug fixes and whatever other sorts of things you’d put into a 2.7.whatever release. So they’d still want to upgrade to 2.8 since that’ll have bug fixes. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Chris Angelico

7:46 p.m.

On Sat, Jun 7, 2014 at 5:36 AM, Donald Stufft wrote:

...

Well it’d contain bug fixes and whatever other sorts of things you’d put into a 2.7.whatever release. So they’d still want to upgrade to 2.8 since that’ll have bug fixes.

But it's not a potentially-breaking change. For example, on Debian Wheezy, there are a huge number of packages that depend on "python (<< 2.8)", because they expect Python 2.7 and *not* Python 2.8. A newer version 2.7 will satisfy that; a version 2.8 won't, because it's entirely possible that 2.8 will have something that's significantly different. That's what version numbers mean; Python follows the standard three-part convention, where you upgrade automatically only within the last part of the number. ChrisA

dw+python-dev＠hmmz.org

7:42 p.m.

On Sat, Jun 07, 2014 at 05:33:45AM +1000, Chris Angelico wrote:

...

...
Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out.

...

If the only difference between 2.7 and 2.8 is the compiler used on Windows, what happens on Linux and other platforms? A Python 2.8 would have to be materially different from Python 2.7, not just binarily incompatible on one platform.

Grrmph, that's fair. Perhaps a final alternative is simply continuing the 2.7 series with a stale compiler, as a kind of carrot on a stick to encourage users to upgrade? Gating 2.7 life on the natural decline of its supported compiler/related ecosystem seems somehow quite a gradual and natural demise.. :) David

Chris Angelico

8:23 p.m.

On Sat, Jun 7, 2014 at 5:42 AM, wrote:

...

Perhaps a final alternative is simply continuing the 2.7 series with a stale compiler, as a kind of carrot on a stick to encourage users to upgrade?

More likely, what would happen is that there'd be an alternate distribution of Python 2.7 (eg ActiveState), which would be language-compatible with python.org 2.7, but built with a different compiler, and therefore unable to use extensions built for python.org's 2.7. One way or another, pain will happen. ChrisA

Brian Curtin

8:28 p.m.

On Fri, Jun 6, 2014 at 11:42 PM, wrote:

...

On Sat, Jun 07, 2014 at 05:33:45AM +1000, Chris Angelico wrote:

...
...
Is it really any difference in maintenance if you just stop applying updates to 2.7 and switch to 2.8? If 2.8 is really just 2.7 with a new compiler then there should be no functional difference between doing that and doing a 2.7.whatever except all of the tooling that relies on the compiler not to change in micro releases won’t suddenly break and freak out.

...
If the only difference between 2.7 and 2.8 is the compiler used on Windows, what happens on Linux and other platforms? A Python 2.8 would have to be materially different from Python 2.7, not just binarily incompatible on one platform.

Grrmph, that's fair. Perhaps a final alternative is simply continuing the 2.7 series with a stale compiler, as a kind of carrot on a stick to encourage users to upgrade? Gating 2.7 life on the natural decline of its supported compiler/related ecosystem seems somehow quite a gradual and natural demise.. :)

Adding features into 3.x is already not enough of a carrot on the stick for many users. Intentionally leaving 2.7 on a dead compiler is like beating them with the stick.

Sturla Molden

11:01 p.m.

Brian Curtin wrote:

...

Adding features into 3.x is already not enough of a carrot on the stick for many users. Intentionally leaving 2.7 on a dead compiler is like beating them with the stick.

Those who want to build extensions on Windows will just use MinGW (currently GCC 2.8.2) instead. NumPy and SciPy are planning a switch to a GCC based toolchain with static linkage of the MinGW runtime on Windows. It is carefully configured to be binary compatible with VS2008 on Python 2.7. The major reason for this is to use gfortran also on Windows. But the result will be a GCC based toolchain that anyone can use to build extensions on Windows. Sturla

Brian Curtin

11:05 p.m.

On Jun 6, 2014 6:01 PM, "Sturla Molden" wrote:

...

Brian Curtin wrote:

...
Adding features into 3.x is already not enough of a carrot on the stick for many users. Intentionally leaving 2.7 on a dead compiler is like beating them with the stick.

Those who want to build extensions on Windows will just use MinGW (currently GCC 2.8.2) instead.

Well we're certainly not going to assume such a thing. I know people do that, but many don't (I never have).

Sturla Molden

11:32 p.m.

Brian Curtin wrote:

...

Well we're certainly not going to assume such a thing. I know people do that, but many don't (I never have).

If Python 2.7 users are left with a dead compiler on Windows, they will find a solution. For example, Enthought is already bundling their Python distribution with gcc 2.8.1 on Windows. Sturla

Eli Bendersky

11:45 p.m.

On Fri, Jun 6, 2014 at 4:32 PM, Sturla Molden wrote:

...

Brian Curtin wrote:

...
Well we're certainly not going to assume such a thing. I know people do that, but many don't (I never have).

If Python 2.7 users are left with a dead compiler on Windows, they will find a solution. For example, Enthought is already bundling their Python distribution with gcc 2.8.1 on Windows.

While we're at it, Clang in nearing a stage where it can compile C and C++ on Windows *with ABI-compatibility to MSVC* (yes, even C++) -- see http://clang.llvm.org/docs/MSVCCompatibility.html for more details. Could this help? Eli

Sturla Molden

7 Jun 7 Jun

12:05 a.m.

Eli Bendersky wrote:

...

While we're at it, Clang in nearing a stage where it can compile C and C++ on Windows *with ABI-compatibility to MSVC* (yes, even C++) -- see <a href="http://clang.llvm.org/docs/MSVCCompatibility.html">http://clang.llvm.org/docs/MSVCCompatibility.html</a> for more details. Could this help?

Possibly. "cl-clang" is exciting and I hope distutils will support it one day. Clang is not well known among Windows users as it is among users of "Unix" (Apple, Linux, FreeBSD, et al.) It would be even better if Python were bundled with Clang on Windows. The MinGW-based "SciPy toolchain" has ABI compatibility with MSVC only for C (and Fortran), not C++. Differences from vanilla MinGW is mainly static linkage of the MinGW runtime, different stack alignment (4 bytes instead of 16), and it links with msvcr91.dll instead of msvcrt.dll. Sturla

Brian Curtin

1:09 a.m.

On Jun 6, 2014 6:33 PM, "Sturla Molden" wrote:

...

Brian Curtin wrote:

...
Well we're certainly not going to assume such a thing. I know people do that, but many don't (I never have).

If Python 2.7 users are left with a dead compiler on Windows, they will find a solution. For example, Enthought is already bundling their Python distribution with gcc 2.8.1 on Windows.

Again, not something I think we should depend on. A lot of people use python.org installers.

Sturla Molden

1:35 a.m.

Brian Curtin wrote:

...

...
If Python 2.7 users are left with a dead compiler on Windows, they will find a solution. For example, Enthought is already bundling their Python distribution with gcc 2.8.1 on Windows.

Again, not something I think we should depend on. A lot of people use python.org installers.

I am not talking about changing the python.org installers. Let it remain on VS2008 for Python 2.7. I am only suggesting we make it easier to find a free C compiler compatible with the python.org installers. The NumPy/SciPy dev team have taken the burden to build a MinGW toolchain that is configured to be 100 % ABI compatible with the python.org installer. I am only suggesting a link to it or something like that, perhaps even host it as a separate download. (It is GPL, so anyone can do that.) That way it would be easy to find a compatible C compiler. We have to consider that VS2008 will be unobtainable abandonware long before the promised Python 2.7 support expires. When that happens, users of Python 2.7 will need to find another compiler to build C extensions. If Python.org makes this easier it would hurt less to have Python 2.7 remain on VS2008 forever. Sturla

Stephen J. Turnbull

6:09 a.m.

New subject: Moving Python 2.7 [was: 3.5] on Windows to a new compiler

Brian Curtin writes:

...

Adding features into 3.x is already not enough of a carrot on the stick for many users. Intentionally leaving 2.7 on a dead compiler is like beating them with the stick.

No, it's like a New Year's resolution to stop self-flagellating, and handing the whip to the users to use on themselves, or not, as they choose. Remember, the users *chose* to remain locked-in to 2.7, hoping that we would continue to provide support, maybe 2.8. They had alternatives: contributing resources (in full-time developer support units!) to the PSF earmarked for Python 2, porting their dependencies to Python 3, etc. All expensive, yes, but eventually they need to pay the price of support or switching. Staying with Python 2 was always a bet that switching would be cheaper in the future, or that they'd have more resources in the future, or both. Who knows about the private resources, but not only does Python 3 acquire more features steadily, but efforts in core by folks like Ethan, distutils, and Nick (just to name those I've followed personally), along with steadily and expanding ports of 3rd party libraries, are quickly making switching cheaper. Cheap *enough*? That's for the users themselves to decide. So I'm not arguing against support; this kind of support (*and* the people who argue that it's worth doing, and then *do* it!) is one reason I have *no* hesitation in recommending Python (3!) vs. any comparable language.[1] But whatever is decided here, we're doing it for pride or for our own use, not because we owe the users anything. Footnotes: [1] I don't know enough about languages like Ruby or Perl to say Python provides strictly better support. I just can't imagine that it gets better than this!

Brett Cannon

6 Jun 6 Jun

7:05 p.m.

On Fri Jun 06 2014 at 2:59:24 PM, wrote:

...

On Fri, Jun 06, 2014 at 10:49:24PM +0400, Brian Curtin wrote:

...
None of the options are particularly good, but yes, I think that's an option we have to consider. We're supporting 2.7.x for 6 more years on a compiler that is already 6 years old.

Surely that is infinitely less desirable than simply bumping the minor version?

Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language. -Brett

Sturla Molden

10:43 p.m.

Brett Cannon wrote:

...

Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway. Sturla

Nathaniel Smith

10:47 p.m.

On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden wrote:

...

Brett Cannon wrote:

...
Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

This would require recompiling all packages on OS X and Linux, even though nothing had changed. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

Terry Reedy

7 Jun 7 Jun

1:05 a.m.

On 6/6/2014 6:47 PM, Nathaniel Smith wrote:

...

On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden wrote:

...
Brett Cannon wrote:

...
Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

This would require recompiling all packages on OS X and Linux, even though nothing had changed.

If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree. Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different. -- Terry Jan Reedy

Donald Stufft

1:13 a.m.

On Jun 6, 2014, at 9:05 PM, Terry Reedy wrote:

...

On 6/6/2014 6:47 PM, Nathaniel Smith wrote:

...
On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden wrote:

...
Brett Cannon wrote:

...
Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

This would require recompiling all packages on OS X and Linux, even though nothing had changed.

If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree.

Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different.

-- Terry Jan Reedy

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

How are packaging tools supposed to cope with this? AFAIK there is nothing in most of them to deal with a X.Y.Z release suddenly dealing with a different compiler. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Terry Reedy

2:23 a.m.

On 6/6/2014 9:13 PM, Donald Stufft wrote:

...

On Jun 6, 2014, at 9:05 PM, Terry Reedy wrote:

...

...
If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree.

Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different.

...

How are packaging tools supposed to cope with this? AFAIK there is nothing in most of them to deal with a X.Y.Z release suddenly dealing with a different compiler.

For this option, packaging tools on Windows would have to gain a special rule to cope with a special, hopefully unique, not-to-be-repeated, series of releases. If VC2008 ceases to become available to those who do not already have it, and who machines do not break or get replaced, dealing with a different easily available compiler would be easier than dealing with having no compiler. -- Terry Jan Reedy

Mark Lawrence

7:57 a.m.

On 07/06/2014 02:13, Donald Stufft wrote:

...

On Jun 6, 2014, at 9:05 PM, Terry Reedy wrote:

...
On 6/6/2014 6:47 PM, Nathaniel Smith wrote:

...
On Fri, Jun 6, 2014 at 11:43 PM, Sturla Molden wrote:

...
Brett Cannon wrote:

...
Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

This would require recompiling all packages on OS X and Linux, even though nothing had changed.

If you are suggesting that a Windows compiler change should be invisible to non-Windows users, I agree.

Let us assume that /pcbuild remains for those who have vc2008 and that /pcbuild14 is added (and everything else remains as is). Then the only other thing that would change is the Windows installer released on Python.org. Call than 2.7.9W or whatever on the download site and interactive startup message to signal that something is different.

-- Terry Jan Reedy

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

How are packaging tools supposed to cope with this? AFAIK there is nothing in most of them to deal with a X.Y.Z release suddenly dealing with a different compiler.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Potentially completely stupid suggestion to get people thinking (or die laughing :) , but would it be possible to use hex digits, such that 2.7.A was the first release on Windows with the different compiler? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com

Chris Barker

4:01 a.m.

...

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

A reminder that this was brought up a few months ago, as a proposal by the stackless team, as they wanted to use a newer compiler for binaries. IIRC, there was a pretty resounding "don't do that" from this list. Makes sense to me -- we have how many different binaries of 2.7 on how many platforms, with how many compilers? Sure, python.org has been nicely consistent about what compiler (run time, really) they use to distribute Windows binaries, but the python version has NOTHING to do with what compiler is used. (for hat matter there is 32 bit and 64 bit 2.7 on Windows ...) I think, at the time, it was thought that pip, wheel, and the metadata standards should be extended to allow multiple binaries of the same version with different compilers to be in the wild. those projects have had bigger fish to fry, but maybe it's time to get ahead of the game with that, so we can accommodate this change. It's already getting hard to find VS2008 Express, and building 64 bit extensions is s serious pain. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Nick Coghlan

4:49 a.m.

On 7 June 2014 14:01, Chris Barker wrote:

...

...
Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

A reminder that this was brought up a few months ago, as a proposal by the stackless team, as they wanted to use a newer compiler for binaries. IIRC, there was a pretty resounding "don't do that" from this list. Makes sense to me -- we have how many different binaries of 2.7 on how many platforms, with how many compilers? Sure, python.org has been nicely consistent about what compiler (run time, really) they use to distribute Windows binaries, but the python version has NOTHING to do with what compiler is used. (for hat matter there is 32 bit and 64 bit 2.7 on Windows ...)

Supported by python-dev? We have two: 32-bit and 64-bit, both depending on the Microsoft C runtime, and both published as binary installers on python.org. That's it.

...

I think, at the time, it was thought that pip, wheel, and the metadata standards should be extended to allow multiple binaries of the same version with different compilers to be in the wild. those projects have had bigger fish to fry, but maybe it's time to get ahead of the game with that, so we can accommodate this change. It's already getting hard to find VS2008 Express, and building 64 bit extensions is s serious pain.

That was a largely independent discussion, noting that if we come up with a mechanism for dealing with Linux distro variances, it may also be useful for dealing with Windows C runtime variances. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

4:41 a.m.

On 7 June 2014 08:43, Sturla Molden wrote:

...

Brett Cannon wrote:

...
Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

It's honestly astonishing the number of people that tell us doing a new minor release of Python 2 is easy, and then refuse to believe us when we tell them it isn't. It's 2014 and Python *2.7*, which was released in *2010*, is STILL BEING ROLLED OUT. One part of the rollout that is near & dear to my own heart is the fact that Red Hat Enterprise Linux 7 and CentOS 7 are still in their respective release candidate phases, and it is the 6 -> 7 transition that finally upgrades their system Pythons from 2.6 to 2.7. Maya 2014 & MotionBuilder 2014 are also the first versions Autodesk are shipping that use 2.7 rather than 2.6 as the scripting engine (although my understanding is that Autodesk don't guarantee compatibility with Python C extensions that aren't built specifically for use with their products, so they already use a newer C runtime on Windows than we do). And once those two dominoes fall, then there'll be some additional follow on upgrade work in some parts of the developer community as the *users* that receive their Python through those channels rather than directly from upstream switch from 2.6 to 2.7 and stumble over the small compatibility breaks between those two releases. Words like "just", or "simple", or "easy" really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years. That said, there are definitely problems with toolchain availability on Windows for Python 2, and it isn't clear yet how that will be addressed in the long run. Steve is working on ensuring the official toolchain and C runtime binaries are more readily available from MS. Other folks are independently looking into ensuring that open source toolchains (like mingw) can be used effectively to at least build Python C extensions for Windows (and ironing out some of the glitches with that approach that others have mentioned). The Python Packaging Authority are continuing to work on the wheel based infrastructure to help avoid end users having to compile anything in the first place, and redistributors like ActiveState, Enthought & Continuum Analytics also make it possible for many end users to just ignore these upstream concerns. An extension compatibility break would be an absolutely last resort, pursued only if all other attempts at resolving the challenges have demonstrably failed - even at the best of times it can take months for C extension authors to start publishing compatible binaries for a new minor release, so we'd have to assume that time would be even longer for a Python 2.7 maintenance release, if they published updated binaries at all. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Donald Stufft

4:47 a.m.

On Jun 7, 2014, at 12:41 AM, Nick Coghlan wrote:

...

On 7 June 2014 08:43, Sturla Molden wrote:

...
Brett Cannon wrote:

...
Nope. A new minor release of Python is a massive undertaking which is why we have saved ourselves the hassle of doing a Python 2.8 or not giving a clear signal as to when Python 2.x will end as a language.

Why not just define Python 2.8 as Python 2.7 except with a newer compiler? I cannot see why that would be massive undertaking, if changing compiler for 2.7 is neccesary anyway.

It's honestly astonishing the number of people that tell us doing a new minor release of Python 2 is easy, and then refuse to believe us when we tell them it isn't.

It's 2014 and Python *2.7*, which was released in *2010*, is STILL BEING ROLLED OUT. One part of the rollout that is near & dear to my own heart is the fact that Red Hat Enterprise Linux 7 and CentOS 7 are still in their respective release candidate phases, and it is the 6 -> 7 transition that finally upgrades their system Pythons from 2.6 to 2.7. Maya 2014 & MotionBuilder 2014 are also the first versions Autodesk are shipping that use 2.7 rather than 2.6 as the scripting engine (although my understanding is that Autodesk don't guarantee compatibility with Python C extensions that aren't built specifically for use with their products, so they already use a newer C runtime on Windows than we do).

And once those two dominoes fall, then there'll be some additional follow on upgrade work in some parts of the developer community as the *users* that receive their Python through those channels rather than directly from upstream switch from 2.6 to 2.7 and stumble over the small compatibility breaks between those two releases.

Words like "just", or "simple", or "easy" really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years.

How much of that time exists because there were actual significant changes from 2.6 to 2.7 and how much of it would not need to exist if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it the *version* number that causes the slow upgrade, or is it the fact that there are enough changes that it can’t be safely applied automatically.

...

That said, there are definitely problems with toolchain availability on Windows for Python 2, and it isn't clear yet how that will be addressed in the long run. Steve is working on ensuring the official toolchain and C runtime binaries are more readily available from MS. Other folks are independently looking into ensuring that open source toolchains (like mingw) can be used effectively to at least build Python C extensions for Windows (and ironing out some of the glitches with that approach that others have mentioned). The Python Packaging Authority are continuing to work on the wheel based infrastructure to help avoid end users having to compile anything in the first place, and redistributors like ActiveState, Enthought & Continuum Analytics also make it possible for many end users to just ignore these upstream concerns. An extension compatibility break would be an absolutely last resort, pursued only if all other attempts at resolving the challenges have demonstrably failed - even at the best of times it can take months for C extension authors to start publishing compatible binaries for a new minor release, so we'd have to assume that time would be even longer for a Python 2.7 maintenance release, if they published updated binaries at all.

Regards, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Nick Coghlan

4:58 a.m.

On 7 June 2014 14:47, Donald Stufft wrote:

...

On Jun 7, 2014, at 12:41 AM, Nick Coghlan wrote:

...
Words like "just", or "simple", or "easy" really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years.

How much of that time exists because there were actual significant changes from 2.6 to 2.7 and how much of it would not need to exist if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it the *version* number that causes the slow upgrade, or is it the fact that there are enough changes that it can’t be safely applied automatically.

It's the version number change itself. Python 2.7 was covered by the language moratorium, so it consists almost entirely of standard library changes, and the porting notes are minimal: https://docs.python.org/2/whatsnew/2.7.html#porting-to-python-2-7 We didn't even switch compilers on Windows (both 2.6 and 2.7 use VS 2008). I can't think of a better demonstration than the slow pace of the Python 2.7 rollout that the challenges with doing a new minor release of Python really aren't technical ones at the language level - they're technical and administrative challenges in the way the language version number interacts with the broader Python ecosystem, especially the various redistribution channels. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Donald Stufft

5:05 a.m.

On Jun 7, 2014, at 12:58 AM, Nick Coghlan wrote:

...

On 7 June 2014 14:47, Donald Stufft wrote:

...
On Jun 7, 2014, at 12:41 AM, Nick Coghlan wrote:

...
Words like "just", or "simple", or "easy" really have no place being applied to a task where the time required to fully execute it with *no significant problems* is still measured in years.

How much of that time exists because there were actual significant changes from 2.6 to 2.7 and how much of it would not need to exist if 2.8 was literally 2.7.Z with a new compiler on Windows. IOW is it the *version* number that causes the slow upgrade, or is it the fact that there are enough changes that it can’t be safely applied automatically.

It's the version number change itself. Python 2.7 was covered by the language moratorium, so it consists almost entirely of standard library changes, and the porting notes are minimal: https://docs.python.org/2/whatsnew/2.7.html#porting-to-python-2-7

I’m not sure I agree, the porting docs only show a subset of changes, you also have a lot of new stuff like OrderedDict, dict comprehensions, set literals, argparse, dict views, memory views, etc. AFAIK stable releases don’t jump versions because all of these new features are risks, not because a number didn’t change. I don’t particularly care too much though, I just think that bumping the compiler in a 2.7.Z release is a really bad idea and that either of the other two options are massively better.

...

We didn't even switch compilers on Windows (both 2.6 and 2.7 use VS 2008).

I can't think of a better demonstration than the slow pace of the Python 2.7 rollout that the challenges with doing a new minor release of Python really aren't technical ones at the language level - they're technical and administrative challenges in the way the language version number interacts with the broader Python ecosystem, especially the various redistribution channels.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Nick Coghlan

5:18 a.m.

On 7 June 2014 15:05, Donald Stufft wrote:

...

I don’t particularly care too much though, I just think that bumping the compiler in a 2.7.Z release is a really bad idea and that either of the other two options are massively better.

It is *incredibly* unlikely that backwards compatibility with binary extensions will be broken within the Python 2.7 series - there's a reason we said "No" when the Stackless folks were asking about it a while back. Instead, the toolchain availability problem is currently being tackled by trying to make suitable build toolchains more readily available (both the official VS 2008 toolchain and alternative open source toolchains), and by reducing the reliance on building from source for end users. Both of those courses of action are likely to bear fruit. It's only in the case where those approaches *don't* solve the problem that we'll need to come back and revisit the question of a compatibility break for binary extensions - it is, as you say, a really bad idea, and hence not something we would pursue when there are better options available (I think a Python 2.8 release would be an *even worse* idea in terms of souring our relationships with redistributors, but fortunately, those aren't our only two choices). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nathaniel Smith

7:23 a.m.

Once 7 Jun 2014 06:19, "Nick Coghlan" wrote:

...

On 7 June 2014 15:05, Donald Stufft wrote:

...
I don’t particularly care too much though, I just think that bumping the compiler in a 2.7.Z release is a really bad idea and that either of the other two options are massively better.

It is *incredibly* unlikely that backwards compatibility with binary extensions will be broken within the Python 2.7 series - there's a reason we said "No" when the Stackless folks were asking about it a while back. Instead, the toolchain availability problem is currently being tackled by trying to make suitable build toolchains more readily available (both the official VS 2008 toolchain and alternative open source toolchains), and by reducing the reliance on building from source for end users.

A third piece of the puzzle could potentially be the availability of automated wheel-building services. (Personally I still haven't successfully managed to build windows wheels for my own packages, and envy my R-using colleagues whose PyPi equivalent does the building for them.) -n

Steve Dower

3:38 p.m.

One more possible concern that I just thought of is the availability of the build tools on Windows Vista and Windows 7 RTM (that is, without SP1). I'd have to check, but I don't believe anything after VS 2012 is supported on Vista and it's entirely possible that installation is blocked. This may be a non-issue. VC14 still has the "XP mode" that avoids using new APIs, so compiled Python will run fine, but it may be the case that the compiler doesn't (if we manage to get a separate, compiler-only package, that is. VS itself is definitely unusable). I assume gcc/clang will continue to support earlier OSs, so hopefully by the time 3.5 is getting early releases there will be an option for building extensions. I doubt anyone on this list is stuck on Vista or in a position where they can't keep Win7 updated, but do we know of any environments where this may be a problem?

"Martin v. Löwis"

10 Jun 10 Jun

10:15 p.m.

Am 07.06.14 17:38, schrieb Steve Dower:

...

One more possible concern that I just thought of is the availability of the build tools on Windows Vista and Windows 7 RTM (that is, without SP1). I'd have to check, but I don't believe anything after VS 2012 is supported on Vista and it's entirely possible that installation is blocked.

I wouldn't worry about that. People can be asked to update their build machines (within reason), as long as the resulting binaries should work on older systems still. There are testing issues, of course, but they show up even in other cases, like testing whether a 32-bit installer actually runs on a 32-bit system when the build system is a 64-bit system; such issues will always exist. Regards, Martin

Giampaolo Rodola'

7 Jun 7 Jun

9:41 a.m.

On Sat, Jun 7, 2014 at 7:05 AM, Donald Stufft wrote:

...

I don’t particularly care too much though, I just think that bumping the compiler in a 2.7.Z release is a really bad idea and that either of the other two options are massively better.

+1 -- Giampaolo - http://grodola.blogspot.com

"Martin v. Löwis"

6 Jun 6 Jun

7:37 p.m.

Am 06.06.14 20:25, schrieb Brian Curtin:

...

We're going to have to change it at some point, otherwise we're going to have people in 2018 scrambling to find VS2008, which will be 35 versions too old by then.

Not sure whether you picked 2018 deliberately: extended support for VS2008 Professional ends on April 10, 2018. In any case, the extension problem will occur regardless of what you do: - if you switch compilers within 2.7, applications may crash - if you switch compilers and declare it 2.8, you extensions might not be available precompiled for some time (in particular if the developers of some package have abandoned 2.7) - if you don't switch compilers, availability of the tool chain will be terrible. Regards, Martin

Guido van Rossum

7:46 p.m.

A reminder: https://lh5.googleusercontent.com/-d4rF0qJPskQ/U0qpNjP5GoI/AAAAAAAAPW0/4RF_7... -- --Guido van Rossum (python.org/~guido)

Jurko Gospodnetić

8:28 p.m.

Hi. On 6.6.2014. 21:46, Guido van Rossum wrote:

...

A reminder: https://lh5.googleusercontent.com/-d4rF0qJPskQ/U0qpNjP5GoI/AAAAAAAAPW0/4RF_7...

*ROFL* Subtle, ain't he? *gdr* Best regards, Jurko Gospodnetić

Donald Stufft

4:01 p.m.

On Jun 6, 2014, at 11:41 AM, Steve Dower wrote:

...

words

+1 from me. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Paul Moore

4:16 p.m.

On 6 June 2014 16:41, Steve Dower wrote:

...

Basically, what I am offering to do is:

* Update the files in PCBuild to work with Visual Studio "14" * Make any code changes necessary to build with VC14 * Regularly test the latest Python source with the latest MSVC builds and report issues/suggestions to the MSVC team * Keep all changes in a separate (public) repo until early next year when we're getting close to the final VS "14" release

What I am asking anyone else to do is:

* Nothing

+1 from me. Paul

Zachary Ware

4:22 p.m.

On Fri, Jun 6, 2014 at 10:41 AM, Steve Dower wrote:

...

Thoughts/comments/concerns?

My only concern is support for elderly versions of Windows, in particular: XP. I seem to recall the last "let's update our MSVC version" discussion dying off because of XP support. Even though MS has abandoned it, I'm not sure whether we can yet. If that's a non-issue, or if we can actually drop XP support, I'm all for it. -- Zach

Brian Curtin

5:31 p.m.

On Fri, Jun 6, 2014 at 8:22 PM, Zachary Ware wrote:

...

On Fri, Jun 6, 2014 at 10:41 AM, Steve Dower wrote:

...
Thoughts/comments/concerns?

My only concern is support for elderly versions of Windows, in particular: XP. I seem to recall the last "let's update our MSVC version" discussion dying off because of XP support. Even though MS has abandoned it, I'm not sure whether we can yet.

If that's a non-issue, or if we can actually drop XP support, I'm all for it.

Extended support ended in April of this year, so I think we should put XP as unsupported for 3.5 in PEP 11 - http://legacy.python.org/dev/peps/pep-0011/ I seem to remember that we were waiting for this anyway.

"Martin v. Löwis"

7:22 p.m.

Am 06.06.14 19:31, schrieb Brian Curtin:

...

...
If that's a non-issue, or if we can actually drop XP support, I'm all for it.

Extended support ended in April of this year, so I think we should put XP as unsupported for 3.5 in PEP 11 - http://legacy.python.org/dev/peps/pep-0011/

I seem to remember that we were waiting for this anyway.

We don't actually need to explicitly put XP there, as PEP 11 ties our support to the Microsoft product life cycle. XP is not supported by Python anymore. Regards, Martin

dw+python-dev＠hmmz.org

4:37 p.m.

On Fri, Jun 06, 2014 at 03:41:22PM +0000, Steve Dower wrote:

...

[snip]

Speaking as a third party who aims to provide binary distributions for recent Python releases on Windows, every new compiler introduces a licensing and configuration headache. So I guess the questions are: * Does the ABI stability address some historical real world problem with Python binary builds? (I guess possibly) * Is the existing solution of third parties building under e.g. Mingw as an option of last resort causing real world issues? It seems to work for a lot of people, although I personally avoid it. * Have other compiler vendors indicated they will change their ABI environment to match VS under this new stability guarantee? If not, then as yet there is no real world benefit here. * Has Python ever hit a showstopper release issue as a result of a bug in MSVC? (I guess probably not). * Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :) Sorry for dunking water on this, but I've recently spent a ton of time getting a Microsoft build environment running, and it seems possible a new compiler may not yet justify more effort if there is little tangible benefit. David

Stefan Krah

4:57 p.m.

dw+python-dev@hmmz.org wrote:

...

* Has Python ever hit a showstopper release issue as a result of a bug in MSVC? (I guess probably not).

Yes, a PGO issue: http://bugs.python.org/issue15993 To be fair, in that issue I did not look if there's some undefined behavior in longobject.c.

...

* Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :)

This is my only concern, too. Otherwise, +1 for the switch. Stefan Krah

Stefan Krah

5:24 p.m.

Stefan Krah wrote:

...

...
* Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :)

This is my only concern, too. Otherwise, +1 for the switch.

One more thing: Will the SDK 64-bit tools be available for the Express Versions? Stefan Krah

Steve Dower

5:42 p.m.

Stefan Krah wrote:

...

Stefan Krah wrote:

...
...
* Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :)

This is my only concern, too. Otherwise, +1 for the switch.

One more thing: Will the SDK 64-bit tools be available for the Express Versions?

They should be. If they're not, I'll certainly be making a noise about it (unless there's another, easier way to get the tools by then...)

R. David Murray

5:05 p.m.

On Fri, 06 Jun 2014 16:37:01 -0000, dw+python-dev@hmmz.org wrote:

...

On Fri, Jun 06, 2014 at 03:41:22PM +0000, Steve Dower wrote:

...
[snip]

Speaking as a third party who aims to provide binary distributions for recent Python releases on Windows, every new compiler introduces a licensing and configuration headache. So I guess the questions are:

* Does the ABI stability address some historical real world problem with Python binary builds? (I guess possibly)

* Is the existing solution of third parties building under e.g. Mingw as an option of last resort causing real world issues? It seems to work for a lot of people, although I personally avoid it.

* Have other compiler vendors indicated they will change their ABI environment to match VS under this new stability guarantee? If not, then as yet there is no real world benefit here.

* Has Python ever hit a showstopper release issue as a result of a bug in MSVC? (I guess probably not).

* Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :)

Sorry for dunking water on this, but I've recently spent a ton of time getting a Microsoft build environment running, and it seems possible a new compiler may not yet justify more effort if there is little tangible benefit.

If I understand correctly (but I may not as I'm not a windows dev) we're going to want to switch VS versions for 3.5 anyway, so switching to the cutting edge one but where Steve can be and is willing to be in a tight feedback loop with the developers sounds like a win to me. --David

Steve Dower

5:40 p.m.

dw+python-dev@hmmz.org wrote:

...

Speaking as a third party who aims to provide binary distributions for recent Python releases on Windows, every new compiler introduces a licensing and configuration headache. So I guess the questions are:

* Does the ABI stability address some historical real world problem with Python binary builds? (I guess possibly)

Yes. It's very hard to explain to users that even though they've gone out and paid for Visual Studio 2013 Ultimate, they don't really have a C compiler that works with Python. This stability will eventually get us to a place where it doesn't matter what version of the compiler you have, though for a while people will obviously need the latest. (Another thing I'm working on is making sure that it's really easy to get the latest... lots of pieces to this puzzle.)

...

* Is the existing solution of third parties building under e.g. Mingw as an option of last resort causing real world issues? It seems to work for a lot of people, although I personally avoid it.

I think it actually tends to solve more issues than it causes :( I want to fix that by making MSVC better for Python, rather than switching away to another toolset.

...

* Have other compiler vendors indicated they will change their ABI environment to match VS under this new stability guarantee? If not, then as yet there is no real world benefit here.

I have no idea, but I hope they do (eventually they almost certainly will). I've already mentioned to our team that they should reach out to the other projects and try to help them move it along, though I have no idea if they have the time or contacts to manage that. FWIW, the stability guarantee was only announced this week, so there's a good chance that the gcc/clang/etc. teams aren't even aware of it yet.

...

* Has Python ever hit a showstopper release issue as a result of a bug in MSVC? (I guess probably not).

Not to my knowledge, and I'm certainly hoping to avoid it by keeping the builds coming regularly. I can't do an official buildbot for it (and probably can't even reuse the infrastructure) since I'm going to work against the latest internal version as much as I can and we get new builds almost daily. More likely, building Python will reveal showstopper issues that actually get fixed (and it has done in the past, though that was never publicised :) )

...

* Will VS 14 be golden prior to Python 3.5's release? It would suck to rely on a beta compiler.. :)

I sure hope so. The current planning looks like it will (I'm assuming that Python 3.5 is going to be late next year, but I couldn't find a good reference). If things slip here, I'm going to be surrounded by very stressed people, which is not much fun. So I hope it'll be done! At worst, VS 14 RC (or whatever label it gets) will probably be released under a "go live" licence. If anything is dramatically broken at that point, we'll know and it should be fixed, or we know that it's going to be around for a while regardless and we can make the decision to either stick with VC10 or work around the issues.

...

Sorry for dunking water on this, but I've recently spent a ton of time getting a Microsoft build environment running, and it seems possible a new compiler may not yet justify more effort if there is little tangible benefit.

Not at all. I've spent far more time than I wanted to getting a build environment running for producing the Python 2.7 installers, and I spent just as long getting an environment for default going too. I'm personally a big fan of automating things like this, so you can also expect scripts (probably PowerShell) that will configure as much as possible. Cheers, Steve

...

David

"Martin v. Löwis"

7:20 p.m.

Am 06.06.14 17:41, schrieb Steve Dower:

...

Hi all

I would like to propose moving Python 3.5 to use Visual C++ 14.0 as the main compiler.

This is fine with me, but I'm worried about the precise timing of doing so. I assume that you would plan to do this moving before VC++ 14 is actually released. This worries me for three reasons: 1. what is the availability of the compiler during the testing phase, and what will it be immediately after the testing ends (where traditionally people would have to buy licenses, or wait for VS Express to be released)? 2. what is the risk of installing a beta compiler on what might otherwise be a "production" developer system? In particular, could it interfere with other VS installations, and could it require a complete system reinstall when the final release of VC 14 is available? 3. what is the chance of the final release being delayed beyond the planned release date of Python 3.5? Microsoft has a bad track record of meeting release dates (or the tradition of not announcing any for that reason); the blog says that it will be available "sometime in 2015". Now, Python 3.5 might appear November 2015, so what do we do if VS 2015 is not released by the time 3.5b1 is planned? Regards, Martin

"Martin v. Löwis"

8:12 p.m.

Am 06.06.14 21:20, schrieb "Martin v. Löwis":

...

2. what is the risk of installing a beta compiler on what might otherwise be a "production" developer system? In particular, could it interfere with other VS installations, and could it require a complete system reinstall when the final release of VC 14 is available?

I found an official answer here: http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs "Installing a CTP release will place a computer in an unsupported state. For that reason, we recommend only installing CTP releases in a virtual machine, or on a computer that is available for reformatting." So there is no promise that you will not need to reformat the system during the evolution of the compiler. Regards, Martin

Paul Moore

8:13 p.m.

On 6 June 2014 20:20, "Martin v. Löwis" wrote:

...

2. what is the risk of installing a beta compiler on what might otherwise be a "production" developer system? In particular, could it interfere with other VS installations, and could it require a complete system reinstall when the final release of VC 14 is available?

...

From http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs

""" Currently, Visual Studio "14" CTPs have known compatibility issues with previous releases of Visual Studio and should not be installed side-by-side on the same computer. """ It also states that installing the CTP on a PC puts that PC into "Unsupported" state. Paul

"Martin v. Löwis"

8:23 p.m.

Am 06.06.14 22:13, schrieb Paul Moore:

...

From http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs

""" Currently, Visual Studio "14" CTPs have known compatibility issues with previous releases of Visual Studio and should not be installed side-by-side on the same computer. """

I also found http://support.microsoft.com/kb/2967191 which is more specific about this issue: '''There are known issues when you install Visual Studio "14" CTP 14.0.21730.1 DP on the same computer as Visual Studio 2013. While we expect that an uninstallation of Visual Studio "14" and then a repair of Visual Studio 2013 should fix these issues, our safest recommendation is to install Visual Studio "14" in a VM, a VHD, a fresh computer, or another non-production test-only computer that does not have Visual Studio 2013 on it. All of these Visual Studio side-by-side issues are expected to be fixed soon. There is an installation block in this Visual Studio "14" CTP that will prevent installation on a computer where an earlier version of Visual Studio is already installed. To disable the block that will put the computer in an un-recommended state, add the value "BlockerOverride" to the registry: HKLM\SOFTWARE\Microsoft\DevDiv\vs\Servicing''' So it seems to me that switching to VS 14 at this point in time is not possible. Of course, Steve could certainly maintain a Mercurial clone in his hg.python.org sandbox that has all the necessary changes done, so people won't have to redo the porting over and over. Regards, Marti

Steve Dower

11:01 p.m.

Martin v. Löwis wrote:

...

Am 06.06.14 22:13, schrieb Paul Moore:

...
From http://www.visualstudio.com/en-us/downloads/visual-studio-14-ctp-vs

""" Currently, Visual Studio "14" CTPs have known compatibility issues with previous releases of Visual Studio and should not be installed side-by-side on the same computer. """

I also found

http://support.microsoft.com/kb/2967191

which is more specific about this issue:

'''There are known issues when you install Visual Studio "14" CTP 14.0.21730.1 DP on the same computer as Visual Studio 2013. While we expect that an uninstallation of Visual Studio "14" and then a repair of Visual Studio 2013 should fix these issues, our safest recommendation is to install Visual Studio "14" in a VM, a VHD, a fresh computer, or another non-production test-only computer that does not have Visual Studio 2013 on it. All of these Visual Studio side-by-side issues are expected to be fixed soon.

Somebody ran a test to see how well the install/uninstall/repair scenario works, and it isn't that great. There are a lot of teams who contribute to Visual Studio, and not all of them have updated their installers yet (my team included...). Unfortunately, it all happened too close to the release to fix it for this version, hence the recommendation. Eventually, VS 14 will be safe to install side-by-side with earlier versions. Chances are it is safe enough with VS 2010 or VS 2012 - it's the one-version-prior that's causing the most trouble.

...

There is an installation block in this Visual Studio "14" CTP that will prevent installation on a computer where an earlier version of Visual Studio is already installed. To disable the block that will put the computer in an un-recommended state, add the value "BlockerOverride" to the registry: HKLM\SOFTWARE\Microsoft\DevDiv\vs\Servicing'''

So it seems to me that switching to VS 14 at this point in time is not possible.

Of course, Steve could certainly maintain a Mercurial clone in his hg.python.org sandbox that has all the necessary changes done, so people won't have to redo the porting over and over.

That's what I had in mind. [Earlier post]

...

1. what is the availability of the compiler during the testing phase, and what will it be immediately after the testing ends (where traditionally people would have to buy licenses, or wait for VS Express to be released)?

It's freely available now as part of Visual Studio, and all the pre-release releases will include everything. The last release (RC or whatever they decide to call it this time) should have a go-live license, though it will also be time bombed. I believe Express will be released at the same time as the paid versions.

...

2. what is the risk of installing a beta compiler on what might otherwise be a "production" developer system? In particular, could it interfere with other VS installations, and could it require a complete system reinstall when the final release of VC 14 is available?

Answered above. It's as risky as it always is, though as I mentioned, VC 14 may well be fine against VC 10. Build-to-build upgrades may not be supported between pre-release versions, but typically RC to RTM upgrades are supported.

...

3. what is the chance of the final release being delayed beyond the planned release date of Python 3.5? Microsoft has a bad track record of meeting release dates (or the tradition of not announcing any for that reason); the blog says that it will be available "sometime in 2015". Now, Python 3.5 might appear November 2015, so what do we do if VS 2015 is not released by the time 3.5b1 is planned?

We keep the VS 2010 files around and make sure they keep working. This is the biggest risk of the whole plan, but I believe that there's enough of a gap between when VS 14 is planned to release (which I know, but can't share) and when Python 3.5 is planned (which I don't know, but have a semi-informed guess). Is Python 3.5b1 being built with VS 14 RC (hypothetically) a blocking issue? Do we need to resolve that now or can it wait until it happens?

...

Regards, Martin

Cheers, Steve

"Martin v. Löwis"

10 Jun 10 Jun

10:05 p.m.

Am 07.06.14 01:01, schrieb Steve Dower:

...

We keep the VS 2010 files around and make sure they keep working. This is the biggest risk of the whole plan, but I believe that there's enough of a gap between when VS 14 is planned to release (which I know, but can't share) and when Python 3.5 is planned (which I don't know, but have a semi-informed guess).

By "keep around", I'd be fine with "in a subdirectory of PC". PCbuild should either switch for sure, or not switch at all. People had proposed to come up with a "PCbuildN" directory (N=10, N=14, or whatever) to maintain two build environments simultaneously; I'd be -1 on such a plan. There needs to be one official toolset to build Python X.Y with, and it needs to be either VS 2010 or VS 2014, but not both.

...

Is Python 3.5b1 being built with VS 14 RC (hypothetically) a blocking issue? Do we need to resolve that now or can it wait until it happens?

It's up to the release manager, but I'd personally see it as a blocking issue: we shouldn't use a beta compiler for the final release, and we shouldn't switch compilers (back) after b1. The RM *could* opt to bet on VS 14 RTM appearing before 3.5rc1 is released (or otherwise blocking rc1 until VS 14 is released); I would consider this risy, but possibly worth it. We certainly don't need to resolve this now. We should discuss it again when the release schedule for 3.5 is proposed. Regards, Martin

Larry Hastings

13 Jun 13 Jun

12:55 p.m.

On 06/10/2014 03:05 PM, "Martin v. Löwis" wrote:

...

We certainly don't need to resolve this now. We should discuss it again when the release schedule for 3.5 is proposed.

I anticipate 3.5 should be released about 18 months after the release of 3.4, putting it mid-September 2015. //arry/

Nick Coghlan

7 Jun 7 Jun

5:28 a.m.

On 7 June 2014 01:41, Steve Dower wrote:

...

What this means for Python is that C extensions for Python 3.5 and later can be built using any version of MSVC from 14.0 and later. Those who are aware of the current state of affairs where you need to use a matching compiler will hopefully see how big an improvement this will be. It is also likely that other compilers will have an easier time providing compatibility with this new CRT, making it simpler and more reliable to build extensions with LLVM or GCC against an MSVC CPython.

\o/ That's great news. (I'm assuming that change in policy includes figuring out a solution to the file descriptor problem, since we determined during the Stackless 2.8 discussion that file descriptor mismatches were actually our biggest stumbling block when it came to mixing and matching different CRT versions in one process)

...

Basically, what I am offering to do is:

* Update the files in PCBuild to work with Visual Studio "14" * Make any code changes necessary to build with VC14 * Regularly test the latest Python source with the latest MSVC builds and report issues/suggestions to the MSVC team * Keep all changes in a separate (public) repo until early next year when we're getting close to the final VS "14" release

What I am asking anyone else to do is:

* Nothing

Thoughts/comments/concerns?

As long as we're also keeping the VS10 files up to date as a fallback option, which we will be, since the VS14 work will be in a separate repo, this sounds like a fine idea to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

3604

Age (days ago)

3611

Last active (days ago)

List overview

Download

66 comments

24 participants

participants (24)

"Martin v. Löwis"
Brett Cannon
Brian Curtin
Chris Angelico
Chris Barker
Donald Stufft
dw+python-dev＠hmmz.org
Eli Bendersky
Giampaolo Rodola'
Guido van Rossum
Jurko Gospodnetić
Larry Hastings
M.-A. Lemburg
Mark Lawrence
Nathaniel Smith
Nick Coghlan
Paul Moore
R. David Murray
Stefan Krah
Stephen J. Turnbull
Steve Dower
Sturla Molden
Terry Reedy
Zachary Ware

Moving Python 3.5 on Windows to a new compiler

dw+python-dev＠hmmz.org

dw+python-dev＠hmmz.org

Sturla Molden

Sturla Molden

Sturla Molden

Sturla Molden

Sturla Molden

Mark Lawrence

Jurko Gospodnetić

dw+python-dev＠hmmz.org

tags

participants (24)