Draft PEP: Maintenance of Python Releases
This PEP attempts to formalize the existing practice, but goes beyond it in introducing security releases. The addition of security releases addresses various concerns I heard over the last year about Python being short-lived. Those concerns are typically raised by Linux distributors which see that they have to maintain Python releases much longer than python-dev does, and are now concerned about the manpower and Python expertise they need.
When looking in detail, they are primarily concerned with security fixes. They will not add new features to old releases, and they can ignore bug fixes, but they cannot ignore security fixes. So what I really think they want is some form of commitment that security fixes are still considered for a much longer period of time.
In discussions, people often consider "short-lived" as "shorter than five years". So the PEP proposes to produce security releases for five years after the initial release; this would mean that we are willing to make security releases for 2.3 (until July 2008) and 2.4 (until November 2009). To reduce the work-load, the PEP promises no more than one security release per branch per year (if no security fixes get contributed, no release needs to be made).
Notice that this setup significantly relies on *no* bug fixes (other than security fixes) being committed to a branch after the final bugfix release. Addition of bug fixes would require much more extensive testing, with release candidates and everything, so it is essential that there are very very few new patches in each security release.
Please let me know what you think.
Regards, Martin
PEP: XX Title: Maintenance of Python Releases Version: $Revision$ Last-Modified: $Date$ Author: Martin v. Löwis martin@v.loewis.de Discussions-To: python-dev@python.org Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2005
Abstract ========
This PEP defines the policy for the maintenance of release branches for Python.
Overview ========
The Python core developers maintain two major branches at any point in time: the trunk, which will eventually become the next major release, and the current release branch, from which bug-fix releases are made. As a special case, the number of branches may duplicate while Python 2.x and Python 3.x is developed in parallel.
Older releases of Python see no regular maintenance, however, security flaws will be fixed in older releases, eventually resulting in a security release.
Major Releases ==============
Major releases are numbered x.y (despite the .y being commonly named "minor" release in other projects). The release process is described in PEP 101. Major releases should be produced roughly every 18 months, but the actual release frequency may vary significantly.
With a major release x.y, this release becomes the current release. For the previous current release (x.(y-1)), a final bug fix release is produced shortly after the x.y release.
Bugfix Releases ===============
Bug fix releases (numbered x.y.z) are described in PEP 6. As stated there, they may only include bug fixes (no new features), and they are produced typically every six months.
Bug fixes should be committed to the release branch as an ongoing activity (e.g. along with fixing them in the trunk), rather than being backported just before the release.
After the final bug fix release for a branch has been made, the branch becomes unmaintained except for security fixes. No bug fix patches must be applied to the branch.
Security Releases =================
As users operate Python releases long after the release branch has become unmaintained, they still like to see security patches fixed.
The Python source repository can act as a repository for security fixes even after the branch has become unmaintained. Users wishing to contribute security patches must clearly indicate that a certain patch is a security patch, and to what branches they want to see it applied. Commit messages of security patches on unmaintained branches must indicate that the commit was done for security reasons; an explicit description of potential exploits should be avoided.
A security fix must not risk the releasability of the branch, i.e. the maintenance branch should be in a shape to produce a release out of it as-is at all times.
From time to time, security releases will be made from an unmaintained
branch. A security release will be a source-only release (i.e. no Windows or Macintosh binaries are provided); they will follow the numbering of minor releases (i.e. x.y.z).
Security releases should be made at most one year after a security patch has been committed to the branch; users wishing to deploy security patches earlier can safely export the maintenance branch, or otherwise incorporate all committed security fixes into their code base. Security releases should be made for a period of five years after the initial major release.
Copyright =========
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
Martin v. Löwis wrote:
Please let me know what you think.
This appears to be an accurate description of the way releases have been handled for the last few years (which appears to be working well), so +1 here.
Regards, Nick.
"Martin v. Löwis" writes:
A security fix must not risk the releasability of the branch, i.e. the maintenance branch should be in a shape to produce a release out of it as-is at all times.
[...]
Security releases should be made at most one year after a security patch has been committed to the branch; users wishing to deploy security patches earlier can safely export the maintenance branch, or otherwise incorporate all committed security fixes into their code base. Security releases should be made for a period of five years after the initial major release.
I don't understand the point of a "security release" made up to a year after commit, especially in view of the first quoted paragraph. A commit may not be made without confirming *immediate* releasability. Isn't that the painful part of a release? If so, shouldn't an immediate release should be made, and not be that much burden? (At least in my practice, all that's left is an announcement -- which is going to be about 2 lines of boilerplate, since detailed explanations are prohibited -- and rolling tarballs.)
If rolling tarballs etc is considered a burden, a "tag release" could be made. OS distributors are going to import into a vendor branch anyway, what they want is python-dev's certification that it's been checked and (as much as possible given the urgency of a security patch) is safe to apply to their package trees.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On May 12, 2007, at 9:02 AM, Stephen J. Turnbull wrote:
I don't understand the point of a "security release" made up to a year after commit, especially in view of the first quoted paragraph. A commit may not be made without confirming *immediate* releasability. Isn't that the painful part of a release? If so, shouldn't an immediate release should be made, and not be that much burden? (At least in my practice, all that's left is an announcement -- which is going to be about 2 lines of boilerplate, since detailed explanations are prohibited -- and rolling tarballs.)
Security releases should be coordinated with the Python Security Response Team (security at python dot org). There are legitimate reasons for wanting to coordinate security releases with this team, such as to ensure adequate and responsible reporting to vendors and other security organizations. Once a set of patches have been generated and (after an embargo period) committed to the public repository, I think we should indeed make a release fairly quickly.
If rolling tarballs etc is considered a burden, a "tag release" could be made. OS distributors are going to import into a vendor branch anyway, what they want is python-dev's certification that it's been checked and (as much as possible given the urgency of a security patch) is safe to apply to their package trees.
I don't think rolling out tarballs is all that much additional burden once everything else is said and done, so I think we should do it. I don't want to give Anthony more work than he wants to do, but I feel confident we can find volunteers to roll out the tarballs if necessary. I would certainly offer to do so.
- -Barry
I don't understand the point of a "security release" made up to a year after commit, especially in view of the first quoted paragraph.
The objective is to reduce load for the release manager. Any kind of release that is worth anything takes several hours to produce, in my experience (if it could be made completely automatic, it wouldn't be good, since glitches would not be detected).
I would like get Anthony's opinion on this aspect.
A commit may not be made without confirming *immediate* releasability. Isn't that the painful part of a release? If so, shouldn't an immediate release should be made, and not be that much burden? (At least in my practice, all that's left is an announcement -- which is going to be about 2 lines of boilerplate, since detailed explanations are prohibited -- and rolling tarballs.)
See PEP 101. A release involves many more steps, and it's not clear whether a release candidate could be skipped.
I think we would need to restrict the total number of releases made per year. The one-year limit may be debatable, and immediate releases might be possible, as long as there is some provision that releases are not made at a too-high rate.
If rolling tarballs etc is considered a burden, a "tag release" could be made. OS distributors are going to import into a vendor branch anyway, what they want is python-dev's certification that it's been checked and (as much as possible given the urgency of a security patch) is safe to apply to their package trees.
I think OS distributors typically *do* use official tar balls, even if they import them as a vendor branch somewhere. Also, creating the tar ball is not the only activity: creating a new web page on pydotorg, running the test suite, etc. all is still necessary.
In any case, the patch gets certified by being committed (with the indication that it is a security fix), so if they want certified patches, they can just import the maintenance branch.
Regards, Martin
""Martin v. Löwis"" martin@v.loewis.de wrote in message news:464764C9.8090707@v.loewis.de... |> I don't understand the point of a "security release" made up to a year | > after commit, especially in view of the first quoted paragraph.
A security release is presumably a response to a serious problem.
| I think we would need to restrict the total number of releases | made per year. The one-year limit may be debatable, and immediate | releases might be possible, as long as there is some provision | that releases are not made at a too-high rate.
I would agree, but... has there been more that the one security release that I know about?
tjr
| I think we would need to restrict the total number of releases | made per year. The one-year limit may be debatable, and immediate | releases might be possible, as long as there is some provision | that releases are not made at a too-high rate.
I would agree, but... has there been more that the one security release that I know about?
No, but I believe this partly due to the fact that Linux distributors don't provide their security patches, or that they get applied only to the current maintenance branch, and not to older branches. If this policy gets implemented, I believe there would be more security patches than those we had seen - this single one was only in response to some demanding users.
Regards, Martin
"Martin v. Löwis" writes:
The objective is to reduce load for the release manager. Any kind of release that is worth anything takes several hours to produce, in my experience (if it could be made completely automatic, it wouldn't be good, since glitches would not be detected).
I absolutely agree.
See PEP 101. A release involves many more steps, and it's not clear whether a release candidate could be skipped.
My point is that if those steps are required for a release, the branch is not "immediately releasable" by my standards if they're not done. Especially if a release candidate is required.
I guess you only meant that a security commit must meet the technical requirements of any other commit (plus being a security fix, of course), so that the release process can be started at any time?
In general, I recognize the burden on the release engineer, and obviously any burdensome policy needs his OK. But I think the policy should be *effective* too, and I just don't see that a policy that allows such long lags is a more effective security response than a policy that says "the tarballs are deprecated due to security fixes; get your Python by importing the branch, not by fetching a tarball."
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On May 14, 2007, at 11:32 AM, Stephen J. Turnbull wrote:
In general, I recognize the burden on the release engineer, and obviously any burdensome policy needs his OK. But I think the policy should be *effective* too, and I just don't see that a policy that allows such long lags is a more effective security response than a policy that says "the tarballs are deprecated due to security fixes; get your Python by importing the branch, not by fetching a tarball."
Like many other activities we do, if we find ourselves blocking because of resource constraints, we should recruit additional volunteers to reduce the load on any one person. Anthony does a masterful job as release manager, but maybe he would rather someone else perform security releases. (It's not a bad idea anyway so that others have experience doing releases too.)
We should decide what's right for security releases and then assess whether we need to recruit in order to perform that activity the way we want to.
- -Barry
We should decide what's right for security releases and then assess whether we need to recruit in order to perform that activity the way we want to.
I disagree. If you would like to see a certain policy implemented, you need to locate the volunteers *first*, and only then you can start setting a policy that these volunteers can agree to. When the volunteers then run away, or become inactive, the policy needs revisiting.
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On May 14, 2007, at 5:32 PM, Martin v. Löwis wrote:
We should decide what's right for security releases and then assess whether we need to recruit in order to perform that activity the way we want to.
I disagree. If you would like to see a certain policy implemented, you need to locate the volunteers *first*, and only then you can start setting a policy that these volunteers can agree to. When the volunteers then run away, or become inactive, the policy needs revisiting.
These are not mutually exclusive positions, but that's unimportant because in this specific case, I'm confident we can summon the necessary manpower.
Still, I'm in agreement with you that the repository holds the security patches and that the tarballs are a convenience. They are an important convenience though, so I would say that they should be released in a timely manner after the commit of the security patches. I don't think we need to be that exact about spelling out when that happens.
(I personally would like to see it within "weeks" of a security patch, not "months" or "years".)
Also, I would like to document explicit that it is the responsibility of the PSRT (or its designate) to commit security patches to revision control. The act of committing these patches is a public event and has an important impact on any embargoes agreed upon by the PSRT with other organizations.
- -Barry
Still, I'm in agreement with you that the repository holds the security patches and that the tarballs are a convenience. They are an important convenience though, so I would say that they should be released in a timely manner after the commit of the security patches. I don't think we need to be that exact about spelling out when that happens.
(I personally would like to see it within "weeks" of a security patch, not "months" or "years".)
Couldn't that lead to many more 2.x.y releases in the "security fixes" period than in the "active maintenance" period? For active maintenance, we don't do roll security fixes into releases more often than every six months.
I would dislike a quick succession of 2.3.7, 2.3.8, 2.3.9, etc. (also because these numbers should not grow above 10 :-)
Also, I would like to document explicit that it is the responsibility of the PSRT (or its designate) to commit security patches to revision control. The act of committing these patches is a public event and has an important impact on any embargoes agreed upon by the PSRT with other organizations.
I also disagree. Regular committers should continue to do that. I haven't seen a single activity from the PSRT (or, perhaps one), so for all I can tell, it doesn't exist.
If a security patch is reported to the Python bug tracker, it's as public as it can get. If PSRT members (who are they, anyway) also happen to be committers, they can commit these changes at the time the PSRT deems appropriate. If they are not committers, they need to post the patch to SF as anybody else.
(you can tell that I come from a country where people are quite skeptical about the secret service).
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On May 14, 2007, at 7:19 PM, Martin v. Löwis wrote:
Still, I'm in agreement with you that the repository holds the security patches and that the tarballs are a convenience. They are an important convenience though, so I would say that they should be released in a timely manner after the commit of the security patches. I don't think we need to be that exact about spelling out when that happens.
(I personally would like to see it within "weeks" of a security patch, not "months" or "years".)
Couldn't that lead to many more 2.x.y releases in the "security fixes" period than in the "active maintenance" period? For active maintenance, we don't do roll security fixes into releases more often than every six months.
I would dislike a quick succession of 2.3.7, 2.3.8, 2.3.9, etc. (also because these numbers should not grow above 10 :-)
I think we could reasonably batch security releases, if we know that there are a few critical issues in the pipeline. That ought to be part of the job of the PSRT. Personally, I don't care much if the micro release number goes above 10 although I know there are many here who don't like that. We should definitely try to reduce churn. It's a balancing act.
Also, I would like to document explicit that it is the responsibility of the PSRT (or its designate) to commit security patches to revision control. The act of committing these patches is a public event and has an important impact on any embargoes agreed upon by the PSRT with other organizations.
I also disagree. Regular committers should continue to do that. I haven't seen a single activity from the PSRT (or, perhaps one), so for all I can tell, it doesn't exist.
If a security patch is reported to the Python bug tracker, it's as public as it can get.
Right, but hopefully people know to report them to security at python dot org instead. Also hopefully, the new tracker will support private/security bug reports that won't be made public (I don't actually know if this is the case or not).
If PSRT members (who are they, anyway) also happen to be committers, they can commit these changes at the time the PSRT deems appropriate. If they are not committers, they need to post the patch to SF as anybody else.
(you can tell that I come from a country where people are quite skeptical about the secret service).
There's no secret police here, since almost anyone who's foolish enough to volunteer to do some work could easily infiltrate that most cloistered of organizations.
I believe there is value in having a PSRT that coordinates security reports, fixes, and disclosures. If the community disagrees, that's cool. But in that case there's not much point in a security at python dot org mailing list or a PSRT, so let's disband them.
- -Barry
My point is that if those steps are required for a release, the branch is not "immediately releasable" by my standards if they're not done. Especially if a release candidate is required.
But how does that help in practice? If you find after the release that the branch was not in a releasable state, will you fire your employee that caused the mess-up? Even though no problems are expected, you still have to *check* whether there are problems, and that is time-consuming. Better safe than sorry
(at least, this is what I understand Anthony Baxter's position on release engineering is - and I agree with that view).
I guess you only meant that a security commit must meet the technical requirements of any other commit (plus being a security fix, of course), so that the release process can be started at any time?
Exactly. I wouldn't require the release manager to actually commit all security patches - and requiring so would be the only way to guarantee that the branch is releasable (i.e. you have to release it to be sure).
In general, I recognize the burden on the release engineer, and obviously any burdensome policy needs his OK. But I think the policy should be *effective* too, and I just don't see that a policy that allows such long lags is a more effective security response than a policy that says "the tarballs are deprecated due to security fixes; get your Python by importing the branch, not by fetching a tarball."
In effect, this is what the PEP says. That's intentional (i.e. it is my intention - others may have different intentions). It's the repository that holds the security patches; the tarballs (and the version number bumps) are just a convenience.
Regards, Martin
"Martin v. Löwis" writes:
In general, I recognize the burden on the release engineer, and obviously any burdensome policy needs his OK. But I think the policy should be *effective* too, and I just don't see that a policy that allows such long lags is a more effective security response than a policy that says "the tarballs are deprecated due to security fixes; get your Python by importing the branch, not by fetching a tarball."
In effect, this is what the PEP says. That's intentional (i.e. it is my intention - others may have different intentions). It's the repository that holds the security patches; the tarballs (and the version number bumps) are just a convenience.
It's not the intentions of the Python developers that is my concern here. In effect, I can read this PEP as saying "we don't take security seriously enough to release in a timely fashion, why should you go to the effort of getting sources and applying patches?" and I fear that many users will do so. I think that the label of "release" is important.
In effect, this is what the PEP says. That's intentional (i.e. it is my intention - others may have different intentions). It's the repository that holds the security patches; the tarballs (and the version number bumps) are just a convenience.
It's not the intentions of the Python developers that is my concern here. In effect, I can read this PEP as saying "we don't take security seriously enough to release in a timely fashion, why should you go to the effort of getting sources and applying patches?" and I fear that many users will do so. I think that the label of "release" is important.
[Not sure who "you" is above: who should or should not go to the effort of getting sources, and what patches should they apply?]
I don't think I can be more plain than that: yes, I do not take security seriously enough to release security fixes for old Python versions more than once a year. As a user, it's easy to demand things, and people really have to learn that in open source, all things are done by volunteers, and that demanding gets you nowhere. To get a better service, somebody really has to volunteer and offer it.
Regards, Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On May 15, 2007, at 12:55 AM, Martin v. Löwis wrote:
I don't think I can be more plain than that: yes, I do not take security seriously enough to release security fixes for old Python versions more than once a year. As a user, it's easy to demand things, and people really have to learn that in open source, all things are done by volunteers, and that demanding gets you nowhere. To get a better service, somebody really has to volunteer and offer it.
I've volunteered, and I contend that this community is big enough that we can recruit more people if necessary. So the question really comes down to what is in the best interest of Python. If resources weren't an issue, would you still say that doing security releases once a year is enough? If so, and if that represents the consensus of the community, then that's what we'll do.
- -Barry
Martin v. Löwis wrote:
I don't understand the point of a "security release" made up to a year after commit, especially in view of the first quoted paragraph.
The objective is to reduce load for the release manager. Any kind of release that is worth anything takes several hours to produce, in
You can always can make a checkout of the security-manteined-only branch, if you're in a particular hurry (maybe the PEP should say something about this).
Regards,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On May 12, 2007, at 4:29 AM, Martin v. Löwis wrote:
This PEP attempts to formalize the existing practice, but goes beyond it in introducing security releases. The addition of security releases addresses various concerns I heard over the last year about Python being short-lived. Those concerns are typically raised by Linux distributors which see that they have to maintain Python releases much longer than python-dev does, and are now concerned about the manpower and Python expertise they need.
Martin,
I like this PEP; it addresses the issues I was trying to get at with my initial posting[1]. Stephen brings up some interesting points which I'll comment on in a follow up to his post.
Since one of the major focuses of this PEP is security releases, I wonder if we shouldn't mention that security issues should be reported to security at python dot org instead of public forums or trackers, so that the Python Security Response Team can take the appropriate and responsible actions?
- -Barry
[1] I still think we should craft some text for the website, but it can now be as simple as:
"For the policy on Python version maintenance and release, see PEP XXX."
participants (6)
-
"Martin v. Löwis" -
Barry Warsaw -
Facundo Batista -
Nick Coghlan -
Stephen J. Turnbull -
Terry Reedy