Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build
On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
matthias.klose
http://hg.python.org/cpython/rev/8ee6d96a1019 changeset: 81859:8ee6d96a1019 branch: 2.7 parent: 81855:df9f8feb7444 user: doko@python.org date: Thu Jan 31 23:52:03 2013 +0100 summary: - Issue #17086: Backport the patches from the 3.3 branch to cross-build the package.
You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this? Regards Antoine.
On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou
On Thu, 31 Jan 2013 23:52:27 +0100 (CET) matthias.klose
wrote: http://hg.python.org/cpython/rev/8ee6d96a1019 changeset: 81859:8ee6d96a1019 branch: 2.7 parent: 81855:df9f8feb7444 user: doko@python.org date: Thu Jan 31 23:52:03 2013 +0100 summary: - Issue #17086: Backport the patches from the 3.3 branch to cross-build the package.
You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this?
One of the reasons for the long maintenance period on 2.7 is to keep it building as the underlying platforms change. With the rise of ARM systems, being able to reliably cross-build Python 2.7 for ARM from an x86_64 system is fairly important. That said, as a procedural point for build related new features in 2.7, I agree they should be proposed, with an explicit rationale, on python-dev before proceeding with the commit. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, 1 Feb 2013 11:00:24 +1000
Nick Coghlan
On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou
wrote: On Thu, 31 Jan 2013 23:52:27 +0100 (CET) matthias.klose
wrote: http://hg.python.org/cpython/rev/8ee6d96a1019 changeset: 81859:8ee6d96a1019 branch: 2.7 parent: 81855:df9f8feb7444 user: doko@python.org date: Thu Jan 31 23:52:03 2013 +0100 summary: - Issue #17086: Backport the patches from the 3.3 branch to cross-build the package.
You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this?
One of the reasons for the long maintenance period on 2.7 is to keep it building as the underlying platforms change. With the rise of ARM systems, being able to reliably cross-build Python 2.7 for ARM from an x86_64 system is fairly important.
I would like to see a better argument for this. The rise of ARM systems is the rise of ARM systems powerful enough to build Python without cross-compiling (which is why we *now* have ARM buildbots). The very small ARM systems which need cross-compiling have existed for decades.
That said, as a procedural point for build related new features in 2.7, I agree they should be proposed, with an explicit rationale, on python-dev before proceeding with the commit.
I think this huge changeset should be reverted. It's a complete failure in terms of procedure and following the rules. "Just because it can be useful" is not a good enough reason to violate our release model without even asking. Regards Antoine.
I personally agree that huge changesets aren't good. But I can also see the problem of cross compiling should be sooner or later addressed. I'm fairly interested in cross building python on android too. A point to support cross compiling (instead native build arm-on-arm) could be android doesn't provide (to my knowledge) a native toolchain: that's a quite large target. And assuming arm equals unix so there's always an installed or installable compiler on it is not always the case. If I can suggest a simple solution that'd be a simple makefile to compile python (I have one on my own): no "configure" magic involved more akin to the windows build style (yes, I've just said the W* word). I hope this helps, antonio
- Issue #17086: Backport the patches from the 3.3 branch to cross-build the package. You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this? One of the reasons for the long maintenance period on 2.7 is to keep it building as the underlying platforms change. With the rise of ARM systems, being able to reliably cross-build Python 2.7 for ARM from an x86_64 system is fairly important.
I would like to see a better argument for this. The rise of ARM systems is the rise of ARM systems powerful enough to build Python without cross-compiling (which is why we *now* have ARM buildbots). The very small ARM systems which need cross-compiling have existed for decades.
That said, as a procedural point for build related new features in 2.7, I agree they should be proposed, with an explicit rationale, on python-dev before proceeding with the commit.
I think this huge changeset should be reverted. It's a complete failure in terms of procedure and following the rules. "Just because it can be useful" is not a good enough reason to violate our release model without even asking.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu
On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou
On Fri, 1 Feb 2013 11:00:24 +1000 Nick Coghlan
wrote: On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou
wrote: On Thu, 31 Jan 2013 23:52:27 +0100 (CET) matthias.klose
wrote: http://hg.python.org/cpython/rev/8ee6d96a1019 changeset: 81859:8ee6d96a1019 branch: 2.7 parent: 81855:df9f8feb7444 user: doko@python.org date: Thu Jan 31 23:52:03 2013 +0100 summary: - Issue #17086: Backport the patches from the 3.3 branch to cross-build the package.
You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this?
One of the reasons for the long maintenance period on 2.7 is to keep it building as the underlying platforms change. With the rise of ARM systems, being able to reliably cross-build Python 2.7 for ARM from an x86_64 system is fairly important.
I would like to see a better argument for this. The rise of ARM systems is the rise of ARM systems powerful enough to build Python without cross-compiling (which is why we *now* have ARM buildbots). The very small ARM systems which need cross-compiling have existed for decades.
It is quite common for developers to build a single code base on a single workstation targeting a plethora of platforms all at once. Requiring native systems with self hosting tool-chains for builds is a non-starter as those often don't exist. Making Python 2.7's configure+makefiles easier to cross compile out of the box is a good thing. Side note: we really need a cross compiling build-bot host + target system or we'll inevitably break this.
That said, as a procedural point for build related new features in 2.7, I agree they should be proposed, with an explicit rationale, on python-dev before proceeding with the commit.
I think this huge changeset should be reverted. It's a complete failure in terms of procedure and following the rules. "Just because it can be useful" is not a good enough reason to violate our release model without even asking.
That's up to the 2.7 release manager. Yes, this could have been done better by asking first. But IMNSHO I'd prefer to see it stay in. -gps
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
Am 03.02.2013 22:20, schrieb Gregory P. Smith:
On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou
wrote: On Fri, 1 Feb 2013 11:00:24 +1000 Nick Coghlan
wrote: On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou
wrote: On Thu, 31 Jan 2013 23:52:27 +0100 (CET) matthias.klose
wrote: http://hg.python.org/cpython/rev/8ee6d96a1019 changeset: 81859:8ee6d96a1019 branch: 2.7 parent: 81855:df9f8feb7444 user: doko@python.org date: Thu Jan 31 23:52:03 2013 +0100 summary: - Issue #17086: Backport the patches from the 3.3 branch to cross-build the package.
You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this?
One of the reasons for the long maintenance period on 2.7 is to keep it building as the underlying platforms change. With the rise of ARM systems, being able to reliably cross-build Python 2.7 for ARM from an x86_64 system is fairly important.
I would like to see a better argument for this. The rise of ARM systems is the rise of ARM systems powerful enough to build Python without cross-compiling (which is why we *now* have ARM buildbots). The very small ARM systems which need cross-compiling have existed for decades.
It is quite common for developers to build a single code base on a single workstation targeting a plethora of platforms all at once. Requiring native systems with self hosting tool-chains for builds is a non-starter as those often don't exist. Making Python 2.7's configure+makefiles easier to cross compile out of the box is a good thing.
Side note: we really need a cross compiling build-bot host + target system or we'll inevitably break this.
That said, as a procedural point for build related new features in 2.7, I agree they should be proposed, with an explicit rationale, on python-dev before proceeding with the commit.
I think this huge changeset should be reverted. It's a complete failure in terms of procedure and following the rules. "Just because it can be useful" is not a good enough reason to violate our release model without even asking.
That's up to the 2.7 release manager. Yes, this could have been done better by asking first. But IMNSHO I'd prefer to see it stay in.
I filed Issue #17086, and got feedback from the release manager, which I maybe interpreted too easily as not objecting. And it looked like a nice opportunity to get this into a release (hoping not to listen to another PyCon talk how to hack cross-builds). Even for the trunk, getting feed-back for cross-build related issues is difficult. Maybe reviewers are turned off by impolite comments in some of the cross-build related issues in the bug tracker, but that doesn't explain that you don't get feedback for most of the cross-build issues. So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev? I'll see that I setup a cross buildd building in a cross-build environment and testing in a chroot with qemu installed. I hope that the buildd setup is able to support this. Talking about the release model, yes I think there are some issues: - the 2.7 branch is the only branch which doesn't have expected release dates on the calendar. And from a distributor/vendor point of view, I think yearly releases are too seldom. Such a release could end up only up to 24 months later in a (linux) distribution. - there were way too may regressions checked in on at least the 2.7 branch. Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk? Afaics python is the only project to have such an approach. Others have trunk first, maybe with immediate backport, maybe with later backport. Matthias PS: Antoine: Please CC the committer for such emails.
On 2/4/2013 3:04 PM, Matthias Klose wrote:
- the 2.7 branch is the only branch which doesn't have expected release dates on the calendar.
Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the Release Schecule at python.org.
And from a distributor/vendor point of view, I think yearly releases are too seldom. Such a release could end up only up to 24 months later in a (linux) distribution.
Some subset of 'we' proposed 5 instead of 2 years of bugfix support for 2.7, released 2010 July 4, but no particular interval, especially after the first 2 years.
- there were way too may regressions checked in on at least the 2.7 branch.
Regressions? That normally means adding a bug as part of a patch, especially one that was previously fixed. We continually add tests to try to prevent that. What period of time do you mean 'were' to refer to?
Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk?
Mercurial works best if merges are done in the forward direction. However, this is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The repository is a two-headed monster ;=).
Afaics python is the only project to have such an approach. Others have trunk first, maybe with immediate backport, maybe with later backport.
If a patch is first commited to tip (for 3.4) and then backported to 3.3, then when the backport is pushed, a null merge is needed to avoid making a third head, and the dag looks messier. (I believe 'null merge' is the right term, but I have never done this.) My impression is that most 3.x bugfix patches merge forward with little or no change. -- Terry Jan Reedy
Am 05.02.2013 07:13, schrieb Terry Reedy:
On 2/4/2013 3:04 PM, Matthias Klose wrote:
- the 2.7 branch is the only branch which doesn't have expected release dates on the calendar.
Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the Release Schecule at python.org.
maybe these are now updated. until recently 3.3.1 was targeted for Nov/Dec 2012, and 2.7.4 wasn't found on the calendar.
- there were way too may regressions checked in on at least the 2.7 branch.
Regressions? That normally means adding a bug as part of a patch, especially one that was previously fixed. We continually add tests to try to prevent that. What period of time do you mean 'were' to refer to?
- http://bugs.python.org/issue9374 - http://bugs.python.org/issue15906 - http://bugs.python.org/issue16012 - http://bugs.python.org/issue10182 - http://bugs.python.org/issue16828 not a complete list, all these are past the 2.7.3 release.
Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk?
Mercurial works best if merges are done in the forward direction. However, this is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The repository is a two-headed monster ;=).
so it should be possible to delay patches for 2.7 until they don't show issues in 3.2 or 3.3.
Afaics python is the only project to have such an approach. Others have trunk first, maybe with immediate backport, maybe with later backport.
If a patch is first commited to tip (for 3.4) and then backported to 3.3, then when the backport is pushed, a null merge is needed to avoid making a third head, and the dag looks messier. (I believe 'null merge' is the right term, but I have never done this.) My impression is that most 3.x bugfix patches merge forward with little or no change.
so is it only this technical limitation, why the bugfix process is defined this way?
Le Mon, 04 Feb 2013 21:04:39 +0100,
Matthias Klose
So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev?
Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches.
- there were way too may regressions checked in on at least the 2.7 branch. Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk? Afaics python is the only project to have such an approach. Others have trunk first, maybe with immediate backport, maybe with later backport.
I don't think this has anything to do with the merge model. Regressions can always happen for things that are not unit-tested. Also, looking at the regressions you listed in another message, none of them appeared serious enough. Regards Antoine.
On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
Le Mon, 04 Feb 2013 21:04:39 +0100, Matthias Klose
a écrit : So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev?
Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches.
Except that 2.7 is an exception to that since it's the last of the Python 2 series, and has a much longer lifespan than normal releases. I'm pretty sure we agreed that there would be some exceptions for issues like new platforms for 2.7. -Barry
On 2/6/2013 8:16 AM, Matthias Klose wrote:
Am 05.02.2013 07:13, schrieb Terry Reedy:
On 2/4/2013 3:04 PM, Matthias Klose wrote:
- there were way too may regressions checked in on at least the 2.7 branch.
I think you are using 'regression' too freely. But that aside, I think you are pointing to a legitimate issue, in the first and maybe second example of how to handle 2.7.
Regressions? That normally means adding a bug as part of a patch, especially one that was previously fixed. We continually add tests to try to prevent that. What period of time do you mean 'were' to refer to?
From what I read (but it was a long discussion), this patch fixed behavior that was agreed to be a bug. The disagreement was whether the bug was clear or obscure. Since the bug was long-standing, and since much code had accommodated to the bug, and even depended on it, the bugfix broke more code than usual. This is not a regression, but is a concern, and if I understand correctly, the fix was removed or revised for existing versions.
I did not understand the issue, except to note that it affected all versions, involved apparent ambiguity in the doc, and was caught by ubuntu testing.
This reported a clear regression, on all versions, not just 2.7, that was caught before release. As the person responsible for the regression said "Too bad it [the regressed behavior] wasn't tested for :(". We still need more patches to improve test coverage.
This fixed an all-versions bug in _sre. The initial patch was fine for 3.x but not sufficiently adjusted for 2.7. Py_BuildValue needed to be replaced in certain places with PyInt_FromSsize_t rather than PyLong_FromSsize_t to maintain the external API. This was not caught because by the stdlib test but was by a third party who tests their package with tip. Yay for them. The process worked.
4GB strings. Not a bad tradeoff, really, but not best. Again, this was not caught initially because there *was* no stdlib test for ''. But
Again, an all-versions issue, not 2.7 specific. http://bugs.python.org/issue14398 disabled compression of '' in the process of enabling compression of there was in a 2.7 ubuntu package.
not a complete list, all these are past the 2.7.3 release.
In only 1 of the 5 was there a 2.7-specific regression. I am not sure what specific change you would like. In the last three examples, would you want 2.7 left with the bug?
Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk?
Mercurial works best if merges are done in the forward direction. However, this is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The repository is a two-headed monster ;=).
so it should be possible to delay patches for 2.7 until they don't show issues in 3.2 or 3.3.
Yes, however 1. Later may mean never. 2. If the regression is 2.7 specific, as with #10182, then not showing an issue for 3.x is irrelevant. 3. If the regression is general, then *at present*, the 2.7 regression is the most likely to be caught by external testing (mostly by unbuntu). That says to me that bug fixes for 2.7 *should* go into tip along with the others. I could even argue that the 2.7 fix should go in first. 4. If bugfixes are not applied to 2.7, that increases the distance between 2.7 and 3.x and possibly increases the difficulty of writing 2&3 code. If we do leave a fixed-in-3.x bug in 2.7, then perhaps, if it is possible to detect that the bug is being depended upon, there should be a deprecation warning in the code. -- Terry Jan Reedy
Le Wed, 6 Feb 2013 16:08:39 -0500,
Barry Warsaw
On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
Le Mon, 04 Feb 2013 21:04:39 +0100, Matthias Klose
a écrit : So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev?
Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches.
Except that 2.7 is an exception to that since it's the last of the Python 2 series, and has a much longer lifespan than normal releases. I'm pretty sure we agreed that there would be some exceptions for issues like new platforms for 2.7.
Well, apparently we didn't make such an exception for MSVC :-) Regards Antoine.
On 7 Feb 2013 19:13, "Antoine Pitrou"
Le Wed, 6 Feb 2013 16:08:39 -0500, Barry Warsaw
a écrit : On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
Le Mon, 04 Feb 2013 21:04:39 +0100, Matthias Klose
a écrit : So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev?
Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches.
Except that 2.7 is an exception to that since it's the last of the Python 2 series, and has a much longer lifespan than normal releases. I'm pretty sure we agreed that there would be some exceptions for issues like new platforms for 2.7.
Well, apparently we didn't make such an exception for MSVC :-)
Python 2.7 still runs on Windows just fine, and changing the C runtime used is *not* purely a change to the build process (it effectively breaks the ABI). I'm not sure how you see that decision being remotely related to Benjamin's decision to allow the cross-compilation patches. Cheers, Nick.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Le Thu, 7 Feb 2013 20:02:39 +1000,
Nick Coghlan
On 7 Feb 2013 19:13, "Antoine Pitrou"
wrote: Le Wed, 6 Feb 2013 16:08:39 -0500, Barry Warsaw
a écrit : On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
Le Mon, 04 Feb 2013 21:04:39 +0100, Matthias Klose
a écrit : So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev?
Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches.
Except that 2.7 is an exception to that since it's the last of the Python 2 series, and has a much longer lifespan than normal releases. I'm pretty sure we agreed that there would be some exceptions for issues like new platforms for 2.7.
Well, apparently we didn't make such an exception for MSVC :-)
Python 2.7 still runs on Windows just fine, and changing the C runtime used is *not* purely a change to the build process (it effectively breaks the ABI).
There is a difference between supporting several MSVCs and changing the MSVC used for the official binary builds. People did ask us to support newer MSVCs.
I'm not sure how you see that decision being remotely related to Benjamin's decision to allow the cross-compilation patches.
I was merely replying to the idea that "we allow new build features in 2.7". Non-trivial build patches have a tendency to produce unexpected breakage (for the record, the initial cross-compiling commits on 3.x brought a lot of breakage at first). Our build system is complicated and *very* poorly documented, I am surprised you think it is a good idea to accept large changes in bugfix releases. I'm still of the advice that non-trivial build enhancements shouldn't be treated differently from any new feature. Patches for bugfix branches can be maintained externally, as they already are (e.g. by Linux distributions). (btw, nothing to do with this discussion, Nick, but it appears your RHEL buildbot is offline) Regards Antoine.
participants (7)
-
Antoine Pitrou
-
Antonio Cavallo
-
Barry Warsaw
-
Gregory P. Smith
-
Matthias Klose
-
Nick Coghlan
-
Terry Reedy