Mailman 3 Python 2.7 patch levels turning two digit - Python-Dev

newer
Re: [Python-Dev] cpython (3.3):...

Python 2.7 patch levels turning two digit

older
subprocess shell=True on Windows...

M.-A. Lemburg

21 Jun 2014 21 Jun '14

10:27 a.m.

With PEP 466 and the constant flow of OpenSSL security fixes which are currently being handled via Python patch level releases, we will soon reach 2.7.10 and quickly go beyond that (also see http://bugs.python.org/issue21308). This opens up a potential backwards incompatibility with existing tools that assume the Python release version number to use the "x.y.z" single digit approach, e.g. code that uses sys.version[:5] for the Python version or relies on the lexicographic ordering of the version string (sys.version > '2.7.2'). Some questions we should probably ask ourselves (I've added my thoughts inline): * Is it a good strategy to ship to Python releases for every single OpenSSL security release or is there a better way to handle these 3rd party issues ? I think we should link to the OpenSSL libs dynamically rather than statically in Python 2.7 for Windows so that it's possible to provide drop-in updates for such issues. * Should we try to avoid two digit patch level release numbers by using some other mechanism such as e.g. a release date after 2.7.9 ? Grepping through our code, this will introduce some breakage, but not much. Most older code branches on minor versions, not patch levels. More recent code uses sys.python_info so is not affected. * Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ? I think this would be a good chance to update the compiler to whatever we use for Python 3 at the time. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 21 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Show replies by date

Nick Coghlan

21 Jun 21 Jun

10:51 a.m.

On 21 June 2014 20:27, M.-A. Lemburg wrote:

...

With PEP 466 and the constant flow of OpenSSL security fixes which are currently being handled via Python patch level releases, we will soon reach 2.7.10 and quickly go beyond that (also see http://bugs.python.org/issue21308).

This opens up a potential backwards incompatibility with existing tools that assume the Python release version number to use the "x.y.z" single digit approach, e.g. code that uses sys.version[:5] for the Python version or relies on the lexicographic ordering of the version string (sys.version > '2.7.2').

Such code has an easy fix available, though, as sys.version_info has existed since 2.0, and handles two digit micro releases just fine. The docs for sys.version also have this explicit disclaimer: "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module." Making it harder to tell whether or not someone's Python installation is affected by an OpenSSL CVE is also an undesirable outcome. On a Linux distro, folks will check the distro package database directly for the OpenSSL version, but on Windows, no such centralised audit mechanism is available by default. With OpenSSL statically linked, Python versions can just be mapped to OpenSSL versions (so, for example, 2.7.7 has 1.0.1g) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

M.-A. Lemburg

4:57 p.m.

On 21.06.2014 12:51, Nick Coghlan wrote:

...

On 21 June 2014 20:27, M.-A. Lemburg wrote:

...
With PEP 466 and the constant flow of OpenSSL security fixes which are currently being handled via Python patch level releases, we will soon reach 2.7.10 and quickly go beyond that (also see http://bugs.python.org/issue21308).

This opens up a potential backwards incompatibility with existing tools that assume the Python release version number to use the "x.y.z" single digit approach, e.g. code that uses sys.version[:5] for the Python version or relies on the lexicographic ordering of the version string (sys.version > '2.7.2').

Such code has an easy fix available, though, as sys.version_info has existed since 2.0, and handles two digit micro releases just fine. The docs for sys.version also have this explicit disclaimer: "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module."

I don't think that's a good argument. Of course, there are better ways to figure out the version number, but fact is, existing code, even in the stdlib, does use and parse the sys.version string version. During Python's lifetime, we've always avoided two digit version numbers, so people have been relying on this, even if it was never (AFAIK) documented anywhere.

...

Making it harder to tell whether or not someone's Python installation is affected by an OpenSSL CVE is also an undesirable outcome. On a Linux distro, folks will check the distro package database directly for the OpenSSL version, but on Windows, no such centralised audit mechanism is available by default. With OpenSSL statically linked, Python versions can just be mapped to OpenSSL versions (so, for example, 2.7.7 has 1.0.1g)

I have to disagree here as well :-) If people cannot upgrade to a higher patch level for whatever reason (say a patch level release introduced some other bugs), but still need to upgrade to the current OpenSSL version, they'd be stuck if we continue to bind the Python version number to some OpenSSL release version. We should definitely make it possible to address OpenSSL bugs without having to upgrade Python and it's not hard to do: just replace the static binding with dynamic binding and include the two OpenSSL DLLs with the Windows installer. People can then drop in new versions of those DLLs as needed, without having the core devs do a complete new release every time someone finds a new problem those libs. Security libs simply have a much higher release rate (if they are well maintained) than most other software. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 21 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Ned Deily

6:47 p.m.

In article <53A5B995.6040802@egenix.com>, "M.-A. Lemburg" wrote:

...

...
Making it harder to tell whether or not someone's Python installation is affected by an OpenSSL CVE is also an undesirable outcome. On a Linux distro, folks will check the distro package database directly for the OpenSSL version, but on Windows, no such centralised audit mechanism is available by default. With OpenSSL statically linked, Python versions can just be mapped to OpenSSL versions (so, for example, 2.7.7 has 1.0.1g)

I have to disagree here as well :-)

If people cannot upgrade to a higher patch level for whatever reason (say a patch level release introduced some other bugs), but still need to upgrade to the current OpenSSL version, they'd be stuck if we continue to bind the Python version number to some OpenSSL release version.

We should definitely make it possible to address OpenSSL bugs without having to upgrade Python and it's not hard to do: just replace the static binding with dynamic binding and include the two OpenSSL DLLs with the Windows installer.

People can then drop in new versions of those DLLs as needed, without having the core devs do a complete new release every time someone finds a new problem those libs. Security libs simply have a much higher release rate (if they are well maintained) than most other software.

I agree that with Nick and Barry that, due to the extended support period for 2.7, we have no choice but to bite the bullet and deal with micro levels exceeding 9. On the other hand, it would also be good to be better able to deal with third-party library revisions that only affect the Windows or OS X binary installers we supply. I don't know that we've ever had a process/policy in place to do that, certainly not recently. Even for statically linked libraries, we presumably could supply replacement re-linked files or even carefully repacked installers with the updated files. This might be something to discuss and add to PEP 101 or a new PEP. Up to now, this hasn't been a major concern since there have usually been other reasons to do full releases as well, e.g. source regressions. Given the still relatively high churn rate for changes going into 2.7 and the growing distance between the 2.7 and 3.x code bases (among other things, leading to more frequent inadvertent backporting errors), we'll probably need to keep making relatively frequent 2.7 releases unless we can further slow down the 2.7 change rate. -- Ned Deily, nad@acm.org

Chris Angelico

8:34 p.m.

On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg wrote:

...

On 21.06.2014 12:51, Nick Coghlan wrote:

...
Such code has an easy fix available, though, as sys.version_info has existed since 2.0, and handles two digit micro releases just fine. The docs for sys.version also have this explicit disclaimer: "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module."

I don't think that's a good argument. Of course, there are better ways to figure out the version number, but fact is, existing code, even in the stdlib, does use and parse the sys.version string version.

During Python's lifetime, we've always avoided two digit version numbers, so people have been relying on this, even if it was never (AFAIK) documented anywhere.

...

= "2.6"), and at least one place that checks sys.version[0] for the major version number, but I didn't find any that look at sys.version[:5] or equivalent. Everything that cares about the

It's going to be a broken-code-breaking change that's introduced in a point release, but since PEP 404 implicitly says that there won't be a 2.10.0, there's no way around that. Although actually, a glance at the stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than 2.7.10 would break - there are places where sys.version[:3] is used (or equivalents like "... %.3s ..." % sys.version), or a whole-string comparison is done against a two-part version string (eg: sys.version three-part version number seems to either look at sys.version.split()[0] or sys.version_info. Do you know where this problematic code is? I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy system, for what it's worth. ChrisA

Oleg Broytman

8:58 p.m.

On Sun, Jun 22, 2014 at 06:34:23AM +1000, Chris Angelico wrote:

...

Do you know where this problematic code is?

In many places: https://encrypted.google.com/search?q=%22sys.version[%3A3]%22 https://encrypted.google.com/search?q=%22sys.version[%3A5]%22 Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

M.-A. Lemburg

9:37 p.m.

On 21.06.2014 22:34, Chris Angelico wrote:

...

On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg wrote:

...
On 21.06.2014 12:51, Nick Coghlan wrote:

...
Such code has an easy fix available, though, as sys.version_info has existed since 2.0, and handles two digit micro releases just fine. The docs for sys.version also have this explicit disclaimer: "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module."

I don't think that's a good argument. Of course, there are better ways to figure out the version number, but fact is, existing code, even in the stdlib, does use and parse the sys.version string version.

During Python's lifetime, we've always avoided two digit version numbers, so people have been relying on this, even if it was never (AFAIK) documented anywhere.

...
= "2.6"), and at least one place that checks sys.version[0] for the major version number, but I didn't find any that look at sys.version[:5] or equivalent. Everything that cares about the

It's going to be a broken-code-breaking change that's introduced in a point release, but since PEP 404 implicitly says that there won't be a 2.10.0, there's no way around that. Although actually, a glance at the stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than 2.7.10 would break - there are places where sys.version[:3] is used (or equivalents like "... %.3s ..." % sys.version), or a whole-string comparison is done against a two-part version string (eg: sys.version three-part version number seems to either look at sys.version.split()[0] or sys.version_info. Do you know where this problematic code is?

I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy system, for what it's worth.

There are no places in the stdlib that parse sys.version in a way that would break wtih 2.7.10, AFAIK. I was just referring to the statement that Nick quoted. sys.version *is* used for parsing the Python version or using parts of it to build e.g. filenames and that's really no surprise. That said, and I also included this in my answers to the questions that Nick removed in his reply, I don't think that a lot of code would be affected by this. I do believe that we can use this potential breakage as a chance for improvement. See the last question (listed here again)... 1. Is it a good strategy to ship to Python releases for every single OpenSSL security release or is there a better way to handle these 3rd party issues ? 2. Should we try to avoid two digit patch level release numbers by using some other mechanism such as e.g. a release date after 2.7.9 ? 3. Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ? My answers to these are: 1. We should use dynamic linking instead and not let OpenSSL bugs trigger Python releases; 2. It's not a big problem; 3. Yes, please, since it is difficult for people to develop and debug their extensions with a 2008 compiler, when the rest of the world has long moved on. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 21 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Ethan Furman

9:48 p.m.

On 06/21/2014 02:37 PM, M.-A. Lemburg wrote:

...

My answers to these are: 1. We should use dynamic linking instead and not let OpenSSL bugs trigger Python releases; 2. It's not a big problem; 3. Yes, please, since it is difficult for people to develop and debug their extensions with a 2008 compiler, when the rest of the world has long moved on.

+1 (assuming not incredibly difficult and those that can are willing ;) -- ~Ethan~

Ethan Furman

23 Jun 23 Jun

9:19 p.m.

On 06/21/2014 02:48 PM, Ethan Furman wrote:

...

On 06/21/2014 02:37 PM, M.-A. Lemburg wrote:

...
My answers to these are: 1. We should use dynamic linking instead and not let OpenSSL bugs trigger Python releases; 2. It's not a big problem; 3. Yes, please, since it is difficult for people to develop and debug their extensions with a 2008 compiler, when the rest of the world has long moved on.

+1 (assuming not incredibly difficult and those that can are willing ;)

Revising this to: +1, -0, -1 It seems to me the intention of supporting 2.7 for so long was not to give ourselves additional nightmares, but to provide a basic level of support for those who are needing longer time before migrating. One of the reasons to migrate is to avoid future pain (pain is an excellent motivator -- it's why we don't go to the doctor when we're healthy, right? ;) If getting new or updated modules becomes more painful then that's motivation to upgrade -- not motivation for us to make both our lives (with the extra work) and everyone's else lives (why isn't this module working? oh, wrong compiler) more difficult. -- ~Ethan~

Phil Thompson

21 Jun 21 Jun

9:57 p.m.

On 21/06/2014 10:37 pm, M.-A. Lemburg wrote:

...

That said, and I also included this in my answers to the questions that Nick removed in his reply, I don't think that a lot of code would be affected by this. I do believe that we can use this potential breakage as a chance for improvement. See the last question (listed here again)...

1. Is it a good strategy to ship to Python releases for every single OpenSSL security release or is there a better way to handle these 3rd party issues ?

Isn't this only a packaging issue? There is no change to the Python API or implementation, so there is no need to change the version number. So just make new Windows packages. The precedent is to add a dash and a package number. I can't remember what version this was applied to before - but I got a +1 from Guido for suggesting it :) Phil

Steve Dower

10 p.m.

We can always lie about the version in sys.version. Existing code is unaffected and new code will have to use version_info (Windows developers will know that Windows pulls tricks like this every other version... doesn't make it a great idea, but it works). Changing compiler without changing at least the install directory and preventing in place upgrades is a really bad idea, and with those mitigations is only pretty bad. I'm torn here, because I know the current situation hurts, but it'd probably only move to VC10 which will hurt just as much in a few years... there are better tooling solutions (yes, I'm working on some behind the scenes). A separate distro of _ssl and _hashlib wouldn't be too hard and has the same effect as a dynamically linked OpenSSL. Maybe we can make these PyPI updateable? Top-posted from my Windows Phone ________________________________ From: M.-A. Lemburgmailto:mal@egenix.com Sent: ‎6/‎21/‎2014 14:38 To: Chris Angelicomailto:rosuav@gmail.com Cc: Python-Devmailto:python-dev@python.org Subject: Re: [Python-Dev] Python 2.7 patch levels turning two digit On 21.06.2014 22:34, Chris Angelico wrote:

...

On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg wrote:

...
On 21.06.2014 12:51, Nick Coghlan wrote:

...
Such code has an easy fix available, though, as sys.version_info has existed since 2.0, and handles two digit micro releases just fine. The docs for sys.version also have this explicit disclaimer: "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module."

I don't think that's a good argument. Of course, there are better ways to figure out the version number, but fact is, existing code, even in the stdlib, does use and parse the sys.version string version.

During Python's lifetime, we've always avoided two digit version numbers, so people have been relying on this, even if it was never (AFAIK) documented anywhere.

...
= "2.6"), and at least one place that checks sys.version[0] for the major version number, but I didn't find any that look at sys.version[:5] or equivalent. Everything that cares about the

It's going to be a broken-code-breaking change that's introduced in a point release, but since PEP 404 implicitly says that there won't be a 2.10.0, there's no way around that. Although actually, a glance at the stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than 2.7.10 would break - there are places where sys.version[:3] is used (or equivalents like "... %.3s ..." % sys.version), or a whole-string comparison is done against a two-part version string (eg: sys.version three-part version number seems to either look at sys.version.split()[0] or sys.version_info. Do you know where this problematic code is?

I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy system, for what it's worth.

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58 2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57 2014-07-02: Python Meeting Duesseldorf ... 11 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

Donald Stufft

10:18 p.m.

On Jun 21, 2014, at 6:00 PM, Steve Dower wrote:

...

We can always lie about the version in sys.version. Existing code is unaffected and new code will have to use version_info (Windows developers will know that Windows pulls tricks like this every other version... doesn't make it a great idea, but it works).

Changing compiler without changing at least the install directory and preventing in place upgrades is a really bad idea, and with those mitigations is only pretty bad. I'm torn here, because I know the current situation hurts, but it'd probably only move to VC10 which will hurt just as much in a few years... there are better tooling solutions (yes, I'm working on some behind the scenes).

A separate distro of _ssl and _hashlib wouldn't be too hard and has the same effect as a dynamically linked OpenSSL. Maybe we can make these PyPI updateable?

Stuff from PyPI installs later on in the sys.path than the stdlib. I wish it were different but it means without sys.path shenanigans you can’t replace the stdlib with something from PyPI.

...

Top-posted from my Windows Phone From: M.-A. Lemburg Sent: ‎6/‎21/‎2014 14:38 To: Chris Angelico Cc: Python-Dev Subject: Re: [Python-Dev] Python 2.7 patch levels turning two digit

On 21.06.2014 22:34, Chris Angelico wrote:

...
On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg wrote:

...
On 21.06.2014 12:51, Nick Coghlan wrote:

...
Such code has an easy fix available, though, as sys.version_info has existed since 2.0, and handles two digit micro releases just fine. The docs for sys.version also have this explicit disclaimer: "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module."

I don't think that's a good argument. Of course, there are better ways to figure out the version number, but fact is, existing code, even in the stdlib, does use and parse the sys.version string version.

During Python's lifetime, we've always avoided two digit version numbers, so people have been relying on this, even if it was never (AFAIK) documented anywhere.

...
= "2.6"), and at least one place that checks sys.version[0] for the major version number, but I didn't find any that look at sys.version[:5] or equivalent. Everything that cares about the

It's going to be a broken-code-breaking change that's introduced in a point release, but since PEP 404 implicitly says that there won't be a 2.10.0, there's no way around that. Although actually, a glance at the stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than 2.7.10 would break - there are places where sys.version[:3] is used (or equivalents like "... %.3s ..." % sys.version), or a whole-string comparison is done against a two-part version string (eg: sys.version three-part version number seems to either look at sys.version.split()[0] or sys.version_info. Do you know where this problematic code is?

I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy system, for what it's worth.

There are no places in the stdlib that parse sys.version in a way that would break wtih 2.7.10, AFAIK. I was just referring to the statement that Nick quoted. sys.version *is* used for parsing the Python version or using parts of it to build e.g. filenames and that's really no surprise.

That said, and I also included this in my answers to the questions that Nick removed in his reply, I don't think that a lot of code would be affected by this. I do believe that we can use this potential breakage as a chance for improvement. See the last question (listed here again)...

1. Is it a good strategy to ship to Python releases for every single OpenSSL security release or is there a better way to handle these 3rd party issues ?

2. Should we try to avoid two digit patch level release numbers by using some other mechanism such as e.g. a release date after 2.7.9 ?

3. Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

My answers to these are: 1. We should use dynamic linking instead and not let OpenSSL bugs trigger Python releases; 2. It's not a big problem; 3. Yes, please, since it is difficult for people to develop and debug their extensions with a 2008 compiler, when the rest of the world has long moved on.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 21 2014)

...
...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58 2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57 2014-07-02: Python Meeting Duesseldorf ... 11 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c... _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Chris Angelico

11:10 p.m.

On Sun, Jun 22, 2014 at 8:00 AM, Steve Dower wrote:

...

We can always lie about the version in sys.version. Existing code is unaffected and new code will have to use version_info (Windows developers will know that Windows pulls tricks like this every other version... doesn't make it a great idea, but it works).

I'd prefer a change of format to an outright lie. Something like "2.7._10" which will sort after "2.7.9". But ideally, nothing at all - just go smoothly to "2.7.10" and let broken code be broken. It'll think it's running on 2.7.1, and if anything needs to distinguish between 2.7.1 and 2.7.x, hopefully it's using version_info. ChrisA

Chris Angelico

11:11 p.m.

On Sun, Jun 22, 2014 at 7:37 AM, M.-A. Lemburg wrote:

...

There are no places in the stdlib that parse sys.version in a way that would break wtih 2.7.10, AFAIK. I was just referring to the statement that Nick quoted. sys.version *is* used for parsing the Python version or using parts of it to build e.g. filenames and that's really no surprise.

Right, good to know. I thought you were implying that stuff would break. Yes, stuff definitely does parse out the version number from sys.version, lots of that happens. ChrisA

Barry Warsaw

4:40 p.m.

On Jun 21, 2014, at 12:27 PM, M.-A. Lemburg wrote:

...

This opens up a potential backwards incompatibility with existing tools that assume the Python release version number to use the "x.y.z" single digit approach, e.g. code that uses sys.version[:5] for the Python version or relies on the lexicographic ordering of the version string (sys.version > '2.7.2').

Patient: Doctor, it hurts when I do this. Doctor: Don't do that!

...

* Should we try to avoid two digit patch level release numbers by using some other mechanism such as e.g. a release date after 2.7.9 ?

Grepping through our code, this will introduce some breakage, but not much. Most older code branches on minor versions, not patch levels. More recent code uses sys.python_info so is not affected.

s/sys.python_info/sys.version_info/ and yes the latter has been preferred for a long time now. Given that 2.7 is a long term support release, it's inevitable that we'll break the 2-digit micro release number barrier. So be it. A 2.7.10 isn't the end of the world. -Barry

"Martin v. Löwis"

23 Jun 23 Jun

6:09 a.m.

...

* Is it a good strategy to ship to Python releases for every single OpenSSL security release or is there a better way to handle these 3rd party issues ?

At least for Windows, a new release certainly needs to be made. It could be possible to produce MSI patch files, but this would still be a new release.

...

I think we should link to the OpenSSL libs dynamically rather than statically in Python 2.7 for Windows so that it's possible to provide drop-in updates for such issues.

It is possible to provide drop-in updates regardless of whether the OpenSSL libs are dynamically linked, as the _ssl module itself is a dynamic lib.

...

* Should we try to avoid two digit patch level release numbers by using some other mechanism such as e.g. a release date after 2.7.9 ?

If it was for me, then yes, certainly: the development of 2.7 should just stop :-)

...

* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers. Regards, Martin

Donald Stufft

4:09 p.m.

On Jun 23, 2014, at 2:09 AM, Martin v. Löwis wrote:

...

...
* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

M.-A. Lemburg

7:27 p.m.

On 23.06.2014 18:09, Donald Stufft wrote:

...

On Jun 23, 2014, at 2:09 AM, Martin v. Löwis wrote:

...
...
* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad.

Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-) But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves: What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ? Switching to a newer compiler will make things easier for everyone and we'd see more binary packages for Windows again. Given that Python 2.7 support was extended for another 5 years at the recent Python Language Summit to 2020, we have to face this breakage sooner or later anyway. Extended support for VS 2008 will end in 2018 (but then: Python developers usually don't have extended support contracts with MS). Service pack support has already ended in 2009. Depending on how you see it, using such an old compiler also poses security risks. The last security update for VS 2008 dates back to 2011 (http://support.microsoft.com/kb/2538243). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 23 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58 2014-07-02: Python Meeting Duesseldorf ... 9 days to go 2014-07-21: EuroPython 2014, Berlin, Germany ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Ned Deily

7:53 p.m.

In article <53A87FB3.2000100@egenix.com>, "M.-A. Lemburg" wrote: [...]

...

But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ?

Switching to a newer compiler will make things easier for everyone and we'd see more binary packages for Windows again.

It does seem like a conundrum. As I have no deep Windows experience to be able to have an appreciation of all of the technical issues involved, I ask out of ignorance: would it be possible and desirable to provide a transition period of n 2.7.x maintenance releases (where n is between 1 and, say, 3) where we would provide 2 sets of Windows installers, one set (32- and 64-bit) with the older compiler and CRT and one with the newer, and campaign to get users and packagers who provide binary extensions to migrate? Would that mitigate the pain, assuming that Steve (or someone else) would be willing to build the additional installers for the transition period? I've done something similar on a smaller scale with the OS X 32-bit installer for 2.7.x but that impact is much less as the audience for that installer is much smaller. -- Ned Deily, nad@acm.org

"Martin v. Löwis"

8:31 p.m.

Am 23.06.14 21:53, schrieb Ned Deily:

...

It does seem like a conundrum. As I have no deep Windows experience to be able to have an appreciation of all of the technical issues involved, I ask out of ignorance: would it be possible and desirable to provide a transition period of n 2.7.x maintenance releases (where n is between 1 and, say, 3) where we would provide 2 sets of Windows installers, one set (32- and 64-bit) with the older compiler and CRT and one with the newer, and campaign to get users and packagers who provide binary extensions to migrate?

The question is how exactly you implement the transition. I see two alternatives: 1. "Hijack" the 2.7 name space, in particular the name "python27.dll", along with registry keys, the .pyd extension, etc. Doing so would permit users to mix binaries from different compilers, and doing so would lead to crashes. Users would have to be careful to either install packages built for the old compiler or packages for the new compiler, and never mix. 2. "Sandbox" the 2.7 name space; come up with new names for everything. E.g. "python27vs13.dll", ".pydvs13" (or "_vs13.pyd"), "C:\Python27vs13", along with the corresponding changes to PyPI, pip, buildout, etc. which would need to learn to look for the right variant of a Python 2.7 package. This should work, but might take several years to implement: you need to find all the places in existing code that refer to the "old" names. If you do it right, you are done about the time when VS 13 becomes unavailable, so you'd then do another such transition to VS 2015, which promises forward-binary compatibility to future releases of Visual Studio.

...

Would that mitigate the pain, assuming that Steve (or someone else) would be willing to build the additional installers for the transition period? I've done something similar on a smaller scale with the OS X 32-bit installer for 2.7.x but that impact is much less as the audience for that installer is much smaller.

Well, the question really is whether precompiled extension modules available from PyPI would work on both compilers. I understand that for OSX, you typically don't have precompiled binaries for the extension modules, so installation compiles the modules from scratch. This is easier, as it can use the ABI of the Python which will be installed to. If you go the "parallel ABIs" route, extension authors have to provide two parallel sets of packages as well. Given 32-bit and 64-bit packages, this will make actually two additional packages - just as if they had to support another Python version. Regards, Martin

Donald Stufft

8:55 p.m.

On Jun 23, 2014, at 4:31 PM, Martin v. Löwis wrote:

...

...
Would that mitigate the pain, assuming that Steve (or someone else) would be willing to build the additional installers for the transition period? I've done something similar on a smaller scale with the OS X 32-bit installer for 2.7.x but that impact is much less as the audience for that installer is much smaller.

Well, the question really is whether precompiled extension modules available from PyPI would work on both compilers. I understand that for OSX, you typically don't have precompiled binaries for the extension modules, so installation compiles the modules from scratch. This is easier, as it can use the ABI of the Python which will be installed to.

If you go the "parallel ABIs" route, extension authors have to provide two parallel sets of packages as well. Given 32-bit and 64-bit packages, this will make actually two additional packages - just as if they had to support another Python version.

As far as I know, stuff on OSX is generally built for “X compiler or later” so binary compatibility is kept as long as you’re using an “or later” but I could be wrong about that. Using binary packages on OSX is a much less frequent thing I think though since getting a working compiler toolchain is easier there. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Ned Deily

9:14 p.m.

In article <14DE41E2-5314-4E49-BE93-85EEEDDDEEAD@stufft.io>, Donald Stufft wrote:

...

On Jun 23, 2014, at 4:31 PM, Martin v. Lowis wrote:

...
...
Would that mitigate the pain, assuming that Steve (or someone else) would be willing to build the additional installers for the transition period? I've done something similar on a smaller scale with the OS X 32-bit installer for 2.7.x but that impact is much less as the audience for that installer is much smaller.

Well, the question really is whether precompiled extension modules available from PyPI would work on both compilers. I understand that for OSX, you typically don't have precompiled binaries for the extension modules, so installation compiles the modules from scratch. This is easier, as it can use the ABI of the Python which will be installed to.

If you go the "parallel ABIs" route, extension authors have to provide two parallel sets of packages as well. Given 32-bit and 64-bit packages, this will make actually two additional packages - just as if they had to support another Python version.

As far as I know, stuff on OSX is generally built for "X compiler or later" so binary compatibility is kept as long as you're using an "or later" but I could be wrong about that. Using binary packages on OSX is a much less frequent thing I think though since getting a working compiler toolchain is easier there.

Both points are generally true on OS X so, yes, binary extensions are much less of an issue there. Where binary distributions on OS X are most used, I think, is when there are dependencies on third-party non-Python libraries that are not shipped by Apple with OS X. But, yes, if we were to go down the route of two sets of installers, that could mean two sets of third-party packages. I suppose there could potentially be some pip / wheel / possibly Distutils help by conditioning the platform name or other component used to generate the egg / wheel / and/or bdist file names on the CRT version (or compiler version), much as what we do today with OS X deployment target. Again, I'm speculating in ignorance here. If that were feasible, things built with the old toolchain could have unchanged names. And, clearly, we would want to keep that "n" number of releases with two sets of installers to be as small as possible, like 1. While there would be a certain amount of unavoidable disruption for some folks, others *might* welcome the opportunity to no longer have to keep around old versions of the tool chain, particularly if they now could use the same tool chain to produce binaries for both Py2 and Py3. -- Ned Deily, nad@acm.org

Antoine Pitrou

8:04 p.m.

Le 23/06/2014 15:27, M.-A. Lemburg a écrit :

...

Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

But without access to the VS 2008 compiler that is needed to compile those extensions,

It does seem to be available: http://www.microsoft.com/en-us/download/details.aspx?id=13276 What am I missing? Regards Antoine.

Ethan Furman

8:12 p.m.

On 06/23/2014 01:04 PM, Antoine Pitrou wrote:

...

Le 23/06/2014 15:27, M.-A. Lemburg a écrit :

...
Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

But without access to the VS 2008 compiler that is needed to compile those extensions,

It does seem to be available: http://www.microsoft.com/en-us/download/details.aspx?id=13276

What am I missing?

Is that VS 2008 /with/ the SP, or just the SP? -- ~Ethan~

Steve Dower

8:31 p.m.

...

Antoine Pitrou wrote: Le 23/06/2014 15:27, M.-A. Lemburg a écrit :

...
Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

But without access to the VS 2008 compiler that is needed to compile those extensions,

It does seem to be available: http://www.microsoft.com/en-us/download/details.aspx?id=13276

What am I missing?

That's the service pack, which will only install if you already have VS 2008 installed. The only official source for VS 2008 these days is through an MSDN subscription, though there's a link floating around that will get to an ISO of VC 2008 Express (but it could disappear or move at any time, which would break the link for good). It's also possible to get VC9 standalone through the Windows SDK for Windows 7 and .NET 3.5, but this installer has bugs, and distutils does not detect VC installs properly (it only detects Visual Studio and then assumes VC). This is fixable with a few extra files and registry keys, but not simple. The best answer here is making VC9 available in a long-term, unsupported manner (support is the main MSFT concern - simply throwing products out there and forgetting about them is very counter-cultural). I'm working on getting people to recognize the importance of keeping the old compilers available, but it's an uphill battle. Obviously I'll post here as soon as I have something I can officially share. :) Cheers, Steve

...

Regards

Antoine.

"Martin v. Löwis"

8:40 p.m.

Am 23.06.14 22:04, schrieb Antoine Pitrou:

...

Le 23/06/2014 15:27, M.-A. Lemburg a écrit :

...
Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

But without access to the VS 2008 compiler that is needed to compile those extensions,

It does seem to be available: http://www.microsoft.com/en-us/download/details.aspx?id=13276

What am I missing?

I believe (without testing) that this is just the service pack. Installing it requires a pre-existing installation of Visual Studio 2008, or else the installer will refuse to do anything. Note that it also won't install on top of Visual Studio Express: you need a licensed copy of Visual Studio to install the service pack. Visual Studio 2008 still *is* available to MSDN users. It's just not available through regular sales channels anymore. Regards, Martin

Donald Stufft

8:20 p.m.

On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg wrote:

...

On 23.06.2014 18:09, Donald Stufft wrote:

...
On Jun 23, 2014, at 2:09 AM, Martin v. Löwis wrote:

...
...
* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad.

Err, sorry that “We” was with my pip hat on.

...

Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

Nonetheless, changing the compiler will not only break pip, but every automated installer tool (easy_install, buildout) that i’m aware of. The blow back for binary installation is going to be huge I think.

...

But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ?

At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. It could even be a good place to move to the ssl backport changes too since they were the riskier set of changes in PEP466. But either way, if a compiler does change in a 2.7 release we’ll need to update a lot of tooling to cope with that, so any plan to do that should include that and a timeline for adoption of that.

...

Switching to a newer compiler will make things easier for everyone and we'd see more binary packages for Windows again.

Given that Python 2.7 support was extended for another 5 years at the recent Python Language Summit to 2020, we have to face this breakage sooner or later anyway. Extended support for VS 2008 will end in 2018 (but then: Python developers usually don't have extended support contracts with MS). Service pack support has already ended in 2009.

Depending on how you see it, using such an old compiler also poses security risks. The last security update for VS 2008 dates back to 2011 (http://support.microsoft.com/kb/2538243).

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 23 2014)

...
...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58 2014-07-02: Python Meeting Duesseldorf ... 9 days to go 2014-07-21: EuroPython 2014, Berlin, Germany ... 28 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Barry Warsaw

8:31 p.m.

On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:

...

At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes.

No. It's not going to happen, for all the reasons discussed previously. Python 2.8 is not a solution to anything. If a new, incompatible compiler suite is required, why can't there just be multiple Windows downloads on https://www.python.org/download/releases/2.7.7/ ? Well, on reason is that you'd have to convince MvL or someone else to take over the work that would require, but that's gotta be *much* lighter weight than releasing a Python 2.8. -Barry

"Martin v. Löwis"

8:42 p.m.

Am 23.06.14 22:31, schrieb Barry Warsaw:

...

On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:

...
At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes.

No. It's not going to happen, for all the reasons discussed previously. Python 2.8 is not a solution to anything.

If a new, incompatible compiler suite is required, why can't there just be multiple Windows downloads on https://www.python.org/download/releases/2.7.7/ ? Well, on reason is that you'd have to convince MvL or someone else to take over the work that would require, but that's gotta be *much* lighter weight than releasing a Python 2.8.

See my other message. It's actually heavier, since it requires changes to distutils, PyPI, pip, buildout etc., all which know how to deal with Python minor version numbers, but are unaware of the notion of competing ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit). Regards, Martin

Chris Angelico

9:48 p.m.

On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis" wrote:

...

See my other message. It's actually heavier, since it requires changes to distutils, PyPI, pip, buildout etc., all which know how to deal with Python minor version numbers, but are unaware of the notion of competing ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit).

Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of things to handle the different compilers? So, for instance, there might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new architectures, just as if someone came out with a 128-bit Windows and built Python 2.7 for it. Would packaging be able to handle that more easily than a compiler change within the same architecture? ChrisA

Donald Stufft

10:04 p.m.

On Jun 23, 2014, at 5:48 PM, Chris Angelico wrote:

...

On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis" wrote:

...
See my other message. It's actually heavier, since it requires changes to distutils, PyPI, pip, buildout etc., all which know how to deal with Python minor version numbers, but are unaware of the notion of competing ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit).

Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of things to handle the different compilers? So, for instance, there might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new architectures, just as if someone came out with a 128-bit Windows and built Python 2.7 for it. Would packaging be able to handle that more easily than a compiler change within the same architecture?

ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I’m not sure about this FWIW. I’d have to look at the implementations of stuff to see how they’d cope with a new thing like that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Donald Stufft

8:43 p.m.

On Jun 23, 2014, at 4:31 PM, Barry Warsaw wrote:

...

On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:

...
At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes.

No. It's not going to happen, for all the reasons discussed previously. Python 2.8 is not a solution to anything.

If a new, incompatible compiler suite is required, why can't there just be multiple Windows downloads on https://www.python.org/download/releases/2.7.7/ ? Well, on reason is that you'd have to convince MvL or someone else to take over the work that would require, but that's gotta be *much* lighter weight than releasing a Python 2.8.

As far as I am aware, a 2.7 with a different compiler, even if it’s just an option is an attractive nuisance. None of the tooling right now differentiates between binary compatibility by anything other than “CPython 2.7”. The end result of having a 2.7 which is built with the old compiler, and a 2.7 built with the new compiler is that you’ll end up with binary distributions which work sometimes if you’re lucky and the creator of the binary distribution and you happened to pick the same “variant” of 2.7. Most likely result is all the binary distributions will *mostly* still depend on using the old compiler because of the corpus of existing binary packages that depend on that. Which means that the 2.7 with new compiler will exist entirely to act as a footgun to anyone who picks it and also wants to use binary packages.

...

-Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

"Martin v. Löwis"

8:49 p.m.

Am 23.06.14 22:31, schrieb Barry Warsaw:

...

Well, on reason is that you'd have to convince MvL or someone else to take over the work that would require, but that's gotta be *much* lighter weight than releasing a Python 2.8.

Just to point this out in a separate message: it will have to be somebody else. I stepped down as the Windows release maintainer for 2.7 when I learned about the extended life of 2.7, much because I feared that exactly the thing would happen that we see happen now - and I didn't want to be the one who would have to deal with it. It is a mess, and it will get bigger the more time passes. Playing-the-role-of-Cassandra-ly y'rs, Martin

M.-A. Lemburg

9:07 p.m.

On 23.06.2014 22:20, Donald Stufft wrote:

...

On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg wrote:

...
On 23.06.2014 18:09, Donald Stufft wrote:

...
On Jun 23, 2014, at 2:09 AM, Martin v. Löwis wrote:

...
...
* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad.

Err, sorry that “We” was with my pip hat on.

...
Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

Nonetheless, changing the compiler will not only break pip, but every automated installer tool (easy_install, buildout) that i’m aware of. The blow back for binary installation is going to be huge I think.

...
But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ?

At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. It could even be a good place to move to the ssl backport changes too since they were the riskier set of changes in PEP466.

But either way, if a compiler does change in a 2.7 release we’ll need to update a lot of tooling to cope with that, so any plan to do that should include that and a timeline for adoption of that.

Sure, and we'd need to hash out possible solutions to minimize breakage, but first we'll have to see whether we want to consider this step or not. BTW: It's strange that I'm arguing for breaking things. I'm usually on the other side of such arguments :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source

...

...
...
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Chris Kaynor

9:14 p.m.

Not being a Python developer, I normally just lurk on Py-Dev, but I figured I'd throw this out there for this thread: Recent version of Maya embed Python 2.x, and the newer version of Maya (I believe 2012 was the first version) embeds a Python 2.7 compiled with VS 2010. From my experience, most C extensions work across compiler versions, however when they don't, it's generally a fairly difficult to debug issue - at least unless you know what to look for in the call stacks, and have access to the symbol files. Chris On Mon, Jun 23, 2014 at 2:07 PM, M.-A. Lemburg wrote:

...

On 23.06.2014 22:20, Donald Stufft wrote:

...
On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg wrote:

...
On 23.06.2014 18:09, Donald Stufft wrote:

...
On Jun 23, 2014, at 2:09 AM, Martin v. Löwis

...
...
...
...
...
* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the

wrote: point where

...
...
...
it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad.

Err, sorry that “We” was with my pip hat on.

...
Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

Nonetheless, changing the compiler will not only break pip, but every automated installer tool (easy_install, buildout) that i’m aware of. The blow back for binary installation is going to be huge I think.

...
But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ?

At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. It could even be a good place to move to the ssl backport changes too since they were the riskier set of changes in PEP466.

But either way, if a compiler does change in a 2.7 release we’ll need to update a lot of tooling to cope with that, so any plan to do that should include that and a timeline for adoption of that.

Sure, and we'd need to hash out possible solutions to minimize breakage, but first we'll have to see whether we want to consider this step or not.

BTW: It's strange that I'm arguing for breaking things. I'm usually on the other side of such arguments :-)

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source

...
...
...
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ckaynor%40zindagigames.co...

Donald Stufft

9:15 p.m.

On Jun 23, 2014, at 5:07 PM, M.-A. Lemburg wrote:

...

On 23.06.2014 22:20, Donald Stufft wrote:

...
On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg wrote:

...
On 23.06.2014 18:09, Donald Stufft wrote:

...
On Jun 23, 2014, at 2:09 AM, Martin v. Löwis wrote:

...
...
* Should we make use of the potential breakage with 2.7.10 to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries for 2.7, I think it would be a bad idea to switch compilers. It will cause severe breakage of 2.7 installations, much more problematic than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where it makes a lot of sense for binary distributions for Windows. Breaking all of them on 2.7 would be very bad.

Err, sorry that “We” was with my pip hat on.

...
Not sure what you mean. We've had binary wininst distributions for Windows for more than a decade, and egg and msi distributions for 8 years :-)

Nonetheless, changing the compiler will not only break pip, but every automated installer tool (easy_install, buildout) that i’m aware of. The blow back for binary installation is going to be huge I think.

...
But without access to the VS 2008 compiler that is needed to compile those extensions, it will become increasingly difficult for package authors to provide such binary packages, so we have to ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7 or not having updated and new Windows binaries for Python 2.7 at all in a few years ?

At the risk of getting Guido to post his slide again, I still think the solution to the old compiler is to just roll a 2.8 with minimal changes. It could even be a good place to move to the ssl backport changes too since they were the riskier set of changes in PEP466.

But either way, if a compiler does change in a 2.7 release we’ll need to update a lot of tooling to cope with that, so any plan to do that should include that and a timeline for adoption of that.

Sure, and we'd need to hash out possible solutions to minimize breakage, but first we'll have to see whether we want to consider this step or not.

BTW: It's strange that I'm arguing for breaking things. I'm usually on the other side of such arguments :-)

Ok. I’m just making sure that any proposal to do this includes how it plans to work around/minimize that. I agree with Martin (I think) that trying to fix the entire ecosystem to cope with that change is going to be far more work than folks realize and that it needs to be an explicit part of the discussion when deciding how to solve the problem. Normally when I see someone suggest that switching compilers in 2.7.x is likely to be less work than releasing a 2.8 It normally appears to me they haven’t looked at the impact on the packaging tooling.

...

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source

...
...
...
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Barry Warsaw

9:22 p.m.

On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:

...

Normally when I see someone suggest that switching compilers in 2.7.x is likely to be less work than releasing a 2.8 It normally appears to me they haven’t looked at the impact on the packaging tooling.

Just to be clear, releasing a Python 2.8 has enormous impact outside of just the amount of work to do it. It's an exceedingly bad idea. -Barry

Donald Stufft

9:28 p.m.

On Jun 23, 2014, at 5:22 PM, Barry Warsaw wrote:

...

On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:

...
Normally when I see someone suggest that switching compilers in 2.7.x is likely to be less work than releasing a 2.8 It normally appears to me they haven’t looked at the impact on the packaging tooling.

Just to be clear, releasing a Python 2.8 has enormous impact outside of just the amount of work to do it. It's an exceedingly bad idea.

Can you clarify? Also FWIW I’m not really married to the 2.8 thing, it’s mostly that, on Windows, the X.Y release prior to the ABI thing in 3.x _was_ the ABI so all the tooling builds on that. So you need to either 1) Stick with the old Compiler 2) Release 2.8 3) Do all the work to fix all the tooling to cope with the fact that X.Y isn’t the ABI on 2.x anymore I don’t think a reasonable option is: 4) Just switch compilers and leave it on someone else’s doorsteps to fix the entire packaging tool chain to cope. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Barry Warsaw

9:47 p.m.

On Jun 23, 2014, at 05:28 PM, Donald Stufft wrote:

...

Can you clarify?

What support guarantees will we make about Python 2.8? Will it be supported as long as Python 2.7? Longer? Will we now have two long-term support versions or change *years* of expectations that users should transition off of Python 2.7 onto Python 2.8? Will all the LTS policies for 2.7 (e.g. PEP 466) be retired for 2.7 and/or adopted completely into 2.8? What should Linux distros do? Should they support both 2.7 and 2.8 or begin the long and potentially arduous process of certifying and transitioning to 2.8? What about other operating systems and package managers, including commercial redistributors? Who is going to do the work to make sure patch are forward ported from 2.7 to 2.8? Who is going to be the 2.8 release manager? Will they be strong enough to reject any and all new features that wouldn't have already made it into 2.7 (due to the already approved, narrow exemptions)? Or will we open the flood gates to Just One More Little New Feature To Make It Easier To Port to Python 3? How will we manage the PR surrounding our backtracking on Python 2.8? How will we manage expectations that it's only released to support a new Windows compiler? Should non-Windows users just ignore it (much like the Python 1.6 releases were mostly ignored)? How do you know which tools, workflows, and processes that will break with a Python 2.8 release? What assumptions about 2.7 being EOL for Python 2 are baked into the ecosystems outside of core Python? I could probably go on, but I'm exhausted. -Barry

Nick Coghlan

11:42 p.m.

On 24 Jun 2014 07:29, "Donald Stufft" wrote:

...

On Jun 23, 2014, at 5:22 PM, Barry Warsaw wrote:

...
On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:

...
Normally when I see someone suggest that switching compilers in 2.7.x is likely to be less work than releasing a 2.8 It normally appears to me they haven’t looked at the impact on the packaging tooling.

Just to be clear, releasing a Python 2.8 has enormous impact outside of

...

...
the amount of work to do it. It's an exceedingly bad idea.

Can you clarify?

Also FWIW I’m not really married to the 2.8 thing, it’s mostly that, on Windows, the X.Y release prior to the ABI thing in 3.x _was_ the ABI so all the tooling builds on

just that. So you need to

...

either

1) Stick with the old Compiler

This is what we're going with. Steve is working on making that more manageable from the Visual Studio side, and there are some folks in the numeric/scientific community looking at improving the usability of the MinGW toolchain for the purpose of building Python 2.7 C extensions.

...

2) Release 2.8

Impractical for the various reasons Barry listed.

...

3) Do all the work to fix all the tooling to cope with the fact that X.Y isn’t the ABI on 2.x anymore

...

I don’t think a reasonable option is:

4) Just switch compilers and leave it on someone else’s doorsteps to fix

Impractical for the various reasons you listed. the entire packaging

...

tool chain to cope.

Agreed. We discussed this option in detail when the Stackless folks asked about it a while ago, and the conclusion was that the risk of obscure breakage was just too high. Cheers, Nick.

...

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372

DCFA

...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

...

3588

Age (days ago)

3590

Last active (days ago)

List overview

Download

39 comments

13 participants

participants (13)

"Martin v. Löwis"
Antoine Pitrou
Barry Warsaw
Chris Angelico
Chris Kaynor
Donald Stufft
Ethan Furman
M.-A. Lemburg
Ned Deily
Nick Coghlan
Oleg Broytman
Phil Thompson
Steve Dower

Python 2.7 patch levels turning two digit

Chris Kaynor

tags

participants (13)