Security releases of CPython
Hello, I'm the primary maintainer of CPython packages in Gentoo. I would like to discuss possible improvement to the release process in order to accelerate releasing security fixes to users. I feel that vulnerability fixes do not make it to end users fast enough. For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month! Ironically, things look even worse for security-supported versions. Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind schedule (planned for Jan 15th), and they are known to be vulnerable since mid-October. In my opinion, this causes three issues: 1. Users using official releases are exposed to security vulnerabilities for prolonged periods of time. 2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes. 3. Effectively, it means that distribution developers need to track and backport security fixes themselves. In the end, this means a lot of duplicate work. I think that security fixes are important enough to justify not sticking to a strict release schedule. Therefore, I would like to propose that if vulnerability fixes are committed, new releases are made as frequently as necessary and as soon as possible (i.e. possibly giving some time for testing) rather than according to a strict schedule. Furthermore, I think that at least for branches that are in higher level of maintenance than security, it could make sense to actually make security releases (e.g. 3.9.1.x) that would include only security fixes without other changes. What do you think? -- Best regards, Michał Górny
On 2/11/2021 3:23 PM, Michał Górny wrote:
Hello,
I'm the primary maintainer of CPython packages in Gentoo. I would like to discuss possible improvement to the release process in order to accelerate releasing security fixes to users.
I feel that vulnerability fixes do not make it to end users fast enough.
When we introduce a bad regression in a release, including a severe-enough security vulnerability, and discover it soon enough, we have sometimes made immediately releases. Beyond that, your proposal to add several releases per Python version, perhaps double what we do now, runs up against the limits of volunteer time and effort. As it is now, becoming a release manager is a 7 year commitment, with at least one release about every other month for the first 4. There are also the 2 people who produce the Windows and macOS installers. I believe each has done it for at least 7 or 8 years already. Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month!
Ironically, things look even worse for security-supported versions. Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind schedule (planned for Jan 15th), and they are known to be vulnerable since mid-October.
In my opinion, this causes three issues:
1. Users using official releases are exposed to security vulnerabilities for prolonged periods of time.
2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes.
3. Effectively, it means that distribution developers need to track and backport security fixes themselves. In the end, this means a lot of duplicate work.
Perhaps people doing duplicate work could get together to eliminate some of the duplication. It should be possible to filter security commits from the python-checkins list by looking at the news entries and if need be, the bpo issues.
I think that security fixes are important enough to justify not sticking to a strict release schedule. Therefore, I would like to propose that if vulnerability fixes are committed, new releases are made as frequently as necessary and as soon as possible (i.e. possibly giving some time for testing) rather than according to a strict schedule.
Furthermore, I think that at least for branches that are in higher level of maintenance than security, it could make sense to actually make security releases (e.g. 3.9.1.x) that would include only security fixes without other changes.
-- Terry Jan Reedy
On Thu, Feb 11, 2021 at 8:29 PM Terry Reedy <tjreedy@udel.edu> wrote:
On 2/11/2021 3:23 PM, Michał Górny wrote:
Hello,
I'm the primary maintainer of CPython packages in Gentoo. I would like to discuss possible improvement to the release process in order to accelerate releasing security fixes to users.
I feel that vulnerability fixes do not make it to end users fast enough.
When we introduce a bad regression in a release, including a severe-enough security vulnerability, and discover it soon enough, we have sometimes made immediately releases.
Indeed. What Terry said. There is risk in doing that. It means the solution is more rushed and hasn't been vetted nearly as well. So without a compelling severe reason we're unlikely to. I would expect distros shipping their own python runtime packages to track security issues and their fixes and backport those (generally easy) to their own packages before an official release is ready anyways. The big distros definitely do this. How to deal with this is up to the individual policies of any given OS distro's package owners. -gps Beyond that, your proposal to add several releases per Python version,
perhaps double what we do now, runs up against the limits of volunteer time and effort. As it is now, becoming a release manager is a 7 year commitment, with at least one release about every other month for the first 4. There are also the 2 people who produce the Windows and macOS installers. I believe each has done it for at least 7 or 8 years already. Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month!
Ironically, things look even worse for security-supported versions. Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind schedule (planned for Jan 15th), and they are known to be vulnerable since mid-October.
In my opinion, this causes three issues:
1. Users using official releases are exposed to security vulnerabilities for prolonged periods of time.
2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes.
3. Effectively, it means that distribution developers need to track and backport security fixes themselves. In the end, this means a lot of duplicate work.
Perhaps people doing duplicate work could get together to eliminate some of the duplication. It should be possible to filter security commits from the python-checkins list by looking at the news entries and if need be, the bpo issues.
I think that security fixes are important enough to justify not sticking to a strict release schedule. Therefore, I would like to propose that if vulnerability fixes are committed, new releases are made as frequently as necessary and as soon as possible (i.e. possibly giving some time for testing) rather than according to a strict schedule.
Furthermore, I think that at least for branches that are in higher level of maintenance than security, it could make sense to actually make security releases (e.g. 3.9.1.x) that would include only security fixes without other changes.
-- Terry Jan Reedy
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4S5LOTVJ... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
On 2/11/2021 3:23 PM, Michał Górny wrote:
Hello,
I'm the primary maintainer of CPython packages in Gentoo. I would like to discuss possible improvement to the release process in order to accelerate releasing security fixes to users.
I feel that vulnerability fixes do not make it to end users fast enough.
When we introduce a bad regression in a release, including a severe-enough security vulnerability, and discover it soon enough, we have sometimes made immediately releases.
Beyond that, your proposal to add several releases per Python version, perhaps double what we do now, runs up against the limits of volunteer time and effort. As it is now, becoming a release manager is a 7 year commitment, with at least one release about every other month for the first 4. There are also the 2 people who produce the Windows and macOS installers. I believe each has done it for at least 7 or 8 years already. Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
While I understand your concerns and sympathize with them, I don't think it's fair to play the volunteer card here. After all, I am a volunteer too, and many distribution packagers are as well. We also have limited time and many other duties, and it just feels unfair for you to be dumping the security work on us. The thing is, CPython upstream has the opportunity to do the central work that benefits everyone. You get the first indication of the vulnerability, you have the first fix and backport to all branches. It is only the natural next step to make a release with it. While I realize it means more work, not doing that means more work for a number of other people who are impacted by your prior work. That said, if the regular release process is too much work, then maybe a process similar to security-supported branches would work? After all, I think it reasonable to assume that a large number of CPython users are using it via distribution packages and having Python 3.8.x.y source-only release will be already a big help to people making these packages.
For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month!
Ironically, things look even worse for security-supported versions. Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind schedule (planned for Jan 15th), and they are known to be vulnerable since mid-October.
In my opinion, this causes three issues:
1. Users using official releases are exposed to security vulnerabilities for prolonged periods of time.
2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes.
3. Effectively, it means that distribution developers need to track and backport security fixes themselves. In the end, this means a lot of duplicate work.
Perhaps people doing duplicate work could get together to eliminate some of the duplication. It should be possible to filter security commits from the python-checkins list by looking at the news entries and if need be, the bpo issues.
This makes me think of the 'systemd-stable' repository (where another maintainer is making bugfix releases to previous versions of systemd). I suppose this could work as a last resort but only if it was officially supported and endorsed by CPython. However, at this point it's not really clear why not go the one step further and make the official releases. -- Best regards, Michał Górny
On Fri, Feb 19, 2021 at 2:16 AM Michał Górny <mgorny@gentoo.org> wrote:
On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
On 2/11/2021 3:23 PM, Michał Górny wrote:
Hello,
I'm the primary maintainer of CPython packages in Gentoo. I would like to discuss possible improvement to the release process in order to accelerate releasing security fixes to users.
I feel that vulnerability fixes do not make it to end users fast enough.
When we introduce a bad regression in a release, including a severe-enough security vulnerability, and discover it soon enough, we have sometimes made immediately releases.
Beyond that, your proposal to add several releases per Python version, perhaps double what we do now, runs up against the limits of volunteer time and effort. As it is now, becoming a release manager is a 7 year commitment, with at least one release about every other month for the first 4. There are also the 2 people who produce the Windows and macOS installers. I believe each has done it for at least 7 or 8 years already. Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
While I understand your concerns and sympathize with them, I don't think it's fair to play the volunteer card here. After all, I am a volunteer too, and many distribution packagers are as well. We also have limited time and many other duties, and it just feels unfair for you to be dumping the security work on us.
I also don't think it's fair to say anyone is "dumping security work" on you. Volunteerism is about accepting that I can't ask anything of you while you can't ask anything of me. We all do our best here, but it's making choices with very finite resources and our release and security approach is the one we have made based on what we have to work with. If people would like to see more frequent releases then it will most likely require people volunteering to help with the security workload as well as with the release process, both of which are rather thankless jobs unfortunately. But simply because we have differing views on how to handle security doesn't mean anyone is actively "dumping" anything on anyone.
The thing is, CPython upstream has the opportunity to do the central work that benefits everyone. You get the first indication of the vulnerability, you have the first fix and backport to all branches. It is only the natural next step to make a release with it. While I realize it means more work, not doing that means more work for a number of other people who are impacted by your prior work.
That said, if the regular release process is too much work, then maybe a process similar to security-supported branches would work? After all, I think it reasonable to assume that a large number of CPython users are using it via distribution packages and having Python 3.8.x.y source-only release will be already a big help to people making these packages.
But a very large number of people get their releases from python.org, so if we do source-only releases without pushing out an install that then causes a schism between what users get on one platform versus another that goes beyond simply the delivery mechanism (and this isn't hypothetical; downstream packages by Linux distros have caused issues for users due to differing from what python.org pushes out, so we do need to consider this impact). -Brett
For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month!
Ironically, things look even worse for security-supported versions. Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind schedule (planned for Jan 15th), and they are known to be vulnerable since mid-October.
In my opinion, this causes three issues:
1. Users using official releases are exposed to security vulnerabilities for prolonged periods of time.
2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes.
3. Effectively, it means that distribution developers need to track and backport security fixes themselves. In the end, this means a lot of duplicate work.
Perhaps people doing duplicate work could get together to eliminate some of the duplication. It should be possible to filter security commits from the python-checkins list by looking at the news entries and if need be, the bpo issues.
This makes me think of the 'systemd-stable' repository (where another maintainer is making bugfix releases to previous versions of systemd). I suppose this could work as a last resort but only if it was officially supported and endorsed by CPython. However, at this point it's not really clear why not go the one step further and make the official releases.
-- Best regards, Michał Górny
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I52QGQVA... Code of Conduct: http://python.org/psf/codeofconduct/
On 2/19/2021 5:11 AM, Michał Górny wrote:
On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
While I understand your concerns and sympathize with them,
Your accusations of unfairness in response to my polite, volunteered response, suggest otherwise.
I don't think it's fair to play the volunteer card here. ... it just feels unfair for you to be dumping the security work on us.
Which I did not do. Bye. -- Terry Jan Reedy
The thing that stood out from this conversation, for me, is: Releases are too hard, and there’s a risk of not having enough volunteers as a result. How hard is it to fix that? Steve On Fri, 19 Feb 2021 at 22:05, Terry Reedy <tjreedy@udel.edu> wrote:
On 2/19/2021 5:11 AM, Michał Górny wrote:
On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
While I understand your concerns and sympathize with them,
Your accusations of unfairness in response to my polite, volunteered response, suggest otherwise.
I don't think it's fair to play the volunteer card here. ... it just feels unfair for you to be dumping the security work on us.
Which I did not do. Bye.
-- Terry Jan Reedy
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UOWWSWZ3... Code of Conduct: http://python.org/psf/codeofconduct/
On 19/02/2021 23.22, Stestagg wrote:
The thing that stood out from this conversation, for me, is: Releases are too hard, and there’s a risk of not having enough volunteers as a result.
How hard is it to fix that?
Actually it's easy to fix! The PSF needs needs sufficient money to hire a couple of people, so the PSF can turn release management and security maintenance from unpaid volunteers into paid fulltime jobs.
On Fri, 19 Feb 2021 at 22:50, Christian Heimes <christian@python.org> wrote:
On 19/02/2021 23.22, Stestagg wrote:
The thing that stood out from this conversation, for me, is: Releases are too hard, and there’s a risk of not having enough volunteers as a result.
How hard is it to fix that?
Actually it's easy to fix!
The PSF needs needs sufficient money to hire a couple of people, so the PSF can turn release management and security maintenance from unpaid volunteers into paid fulltime jobs.
That’s certainly one option. I’d be super surprised if that were the only one. What were the budget requirements for this? How far from the target is the PSF? Is there a plan to hit it? Are there no technical solutions that might help reduce the cost?
On Fri, Feb 19, 2021 at 3:10 PM Stestagg <stestagg@gmail.com> wrote:
On Fri, 19 Feb 2021 at 22:50, Christian Heimes <christian@python.org> wrote:
On 19/02/2021 23.22, Stestagg wrote:
The thing that stood out from this conversation, for me, is: Releases are too hard, and there’s a risk of not having enough volunteers as a result.
How hard is it to fix that?
Actually it's easy to fix!
The PSF needs needs sufficient money to hire a couple of people, so the PSF can turn release management and security maintenance from unpaid volunteers into paid fulltime jobs.
That’s certainly one option. I’d be super surprised if that were the only one.
What were the budget requirements for this? How far from the target is the PSF? Is there a plan to hit it?
There is no specific drive to hire someone to target security and/or release management at the moment. We just got enough funding for the first time to hire a dev-in-residence for Python itself to try to help tackle our 1.4K PR backlog in some fashion that they won't be dedicated to security or release management.
Are there no technical solutions that might help reduce the cost?
If you would like to help out, you can speak with the release managers to see if they could use help in some way; same goes for the security team. -Brett
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/55GFSMWJ... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
... Releases are not just a push of a button.
On 2021-02-19 15:05, Stestagg wrote:
> The thing that stood out from this conversation, for me, is: Releases > are too hard, and there’s a risk of not having enough volunteers as a > result. > > How hard is it to fix that?
Are there no technical solutions that might help reduce the cost?
Sounds like automating until it is "just a push of a button," should be a goal. According to Victor there has been progress, but always room for more. On 2021-02-19 18:36, Brett Cannon wrote:
There is no specific drive to hire someone to target security and/or release management at the moment. We just got enough funding for the first time to hire a dev-in-residence for Python itself to try to help tackle our 1.4K PR backlog in some fashion that they won't be dedicated to security or release management.
Looking at the PSF Annual report from a normal year it also looks like there is substantial revenue, income, and assets available. There are significant expenses as well. (And of course income down for ~two years, but it should return at some point.) I would argue that security releases are of higher importance than most, including sponsorship programs and grants, and that a mild change of priorities is in order. This is not to say (of course) that any other categories are not important, simply that having machines and networks pwned while being sponsored or educated is a devil's bargain. Such a position may not require a hundred-thousand a year salary either. A skilled contractor could improve automation, while a stipend might be enough to ensure releases get out within a ~week if they need to. -Mike
Mike Miller writes:
Sounds like automating until it is "just a push of a button," should be a goal. According to Victor there has been progress, but always room for more.
When XEmacs was releasing betas regularly, everything from tagging the local authoritative repo to pushing to the public repo to making tarballs to signing and authenticating them to Mad-Libbing the form for announcement was invocation of a single script. "Somehow" the typical release was a me-day's work and three calendar days (and unlike GNU Emacs with its "kitchen sink" standard library or Python with its batteries from CR-25s to Prius power replacements, XEmacs was just the Lisp interpreter and basic editor functions). What takes the time? 1. A courtesy pre-announcement to core devs that the release was coming (yes, it's on the schedule, no, they don't check their calendars). 2. Triaging incomplete tasks other folks labeled as "important" or above. 3. Writing news (somebody always forgets, and typically the choice was write it myself or delay the release). 4. Triaging last-minute "urgent" commits. (They rarely were.) 5. Triaging and where possible fixing test failures. 6. Rolling back premature commits. 7. Scheduling future work that didn't *need* to be in a beta release (we had a couple of developers who were trying but couldn't write documentation grammatically to save their commits; it just didn't make sense to force them to do work I could do in 1/10 the effort). 8. Coordination with the dev who built Windows installers. (A few Windows users could get pretty cranky if they didn't get an installer within a couple hours after the Unix-oriented tarball releases.) The Python process is more disciplined, and has a lot of automation to help with the QA stuff. But there's always last-minute "people work", which is what Brooks would call an "essential difficulty." It's pretty irreducible. I would imagine the same is true for Python RMs.
I would argue that security releases are of higher importance than most,
*Some* of them are, but this just isn't true most of the time. Take the very recent CVE-2021-3177, rated 9.8 ("critical", 9.8 out of 10, I believe) because it *might* lead to remote code execution (RCE) under (implausible, AFAICS) conditions in current Python. There is no RCE exploit for arbitrary Python webapps: there are a bunch of other conditions that must be satisfied. It might be possible to create a denial of service for some webapps, but that still depends on only somewhat more plausible conditions. Then, why 9.8? Well, "remote code execution". The critical rating is a *threat* assessment, based on *everything* going right for the attacker, not a *risk* assessment, based on likelihood and degree of damage. Most reported vulnerabilities have no known exploit and potential RCE is not the common case: risk is quite small. Such vulnerabilities must be fixed: as time goes on, risk increases. Not just more time = more time for the black hats, but the hazard rate increases too. The particular currently implausible kill chain may be generalized or others may arise in future versions of Python, or more webapps may incorporate the risky behavior, etc. But there's no reason to believe it will be a known exploit soon, and anybody with a million dollars at risk can (and should) keep a couple of hours of junior developer time on call to assess risk and apply the patch if needed. Not to mention that in 2025 there will still be a bunch of vulnerable 3.8 and 3.9 installations online, which are still whatever the patchlevel was when they were originally installed. I believe that had they let this one wait until the scheduled release, the odds are 1000:1 that *no harm whatsoever* would have been done in that window. Nevertheless, the relevant release managers (3.8, 3.9, and Windows) chose to make a quick release. I think "customer relations" (which are important, I'm not deprecating them!) had as much to do with the decision as the assessment of risk to Python users. I am not going to second-guess the actual decision, but I do think that this is very much a case by case issue not very different from any "urgent" defect, and in many cases "wait for scheduled release" is going to be a no-brainer, and in others that will be the outcome chosen. Some of the time, there will be an emergency release. That has been demonstrated to be possible and realistic, even in a case like this where the need is not open-and-shut. I don't see a need for anything except more love for the release managers. Of course IWBNI we had the resources to pay release managers. But we're doing OK. Steve
On Fri, Feb 19, 2021 at 10:53 PM Christian Heimes <christian@python.org> wrote:
On 19/02/2021 23.22, Stestagg wrote:
The thing that stood out from this conversation, for me, is: Releases are too hard, and there’s a risk of not having enough volunteers as a result.
How hard is it to fix that?
Actually it's easy to fix!
For some value of "easy" that involves raising a six-figure dollar sum annually ...
The PSF needs needs sufficient money to hire a couple of people, so the PSF can turn release management and security maintenance from unpaid volunteers into paid fulltime jobs.
Oh, is that all? Sustainability of the PSF, as has been shown over its 20 years of existence, is not an easy thing to achieve. It was hit by the financial crisis in 2008 and again by the coronavirus crisis last year, both things that affected all foundations. If you plan to bring this kind of money in and rely on it, you have to ensure it comes from sources that can't just be switched off when budgets tighten. It could be done, but "easy" sounds like exaggeration to me. Unless your suggestions were joking, but I saw no smiley ... Kind regards, Steve
On 2/19/21 11:55 PM, Steve Holden wrote:
The PSF needs needs sufficient money to hire a couple of people, so the PSF can turn release management and security maintenance from unpaid volunteers into paid fulltime jobs.
Oh, is that all? Sustainability of the PSF, as has been shown over its 20 years of existence, is not an easy thing to achieve. It was hit by the financial crisis in 2008 and again by the coronavirus crisis last year, both things that affected all foundations.
If you plan to bring this kind of money in and rely on it, you have to ensure it comes from sources that can't just be switched off when budgets tighten. It could be done, but "easy" sounds like exaggeration to me. Unless your suggestions were joking, but I saw no smiley ...
Steve's comments probably need no reinforcement, but I can speak as someone who's been funded by fees collected from motivated companies, and had the tap turn off. In my cases (more than one) they were consortia where the members committed a set fee yearly, and then one year: we've decided not to renew, from one or more... It's actually easier to raise funds for a one-time campaign than one that is committed to last for several years, in my limited experience on the fund-raising side.
On Fri, 2021-02-19 at 17:03 -0500, Terry Reedy wrote:
On 2/19/2021 5:11 AM, Michał Górny wrote:
On Thu, 2021-02-11 at 23:24 -0500, Terry Reedy wrote:
Releases are not just a push of a button. Make the release job too onerous, and there might not be any more volunteers.
While I understand your concerns and sympathize with them,
Your accusations of unfairness in response to my polite, volunteered response, suggest otherwise.
I don't think it's fair to play the volunteer card here. ... it just feels unfair for you to be dumping the security work on us.
Which I did not do. Bye.
I'm sorry, I did not mean to insult you. That's just how I felt after reading the response. -- Best regards, Michał Górny
Hi Michał, I created https://python-security.readthedocs.io/ website to track known Python vulnerabilities to help me checking if fixes are backported to all supported Python branches. I'm maintaing this list manually, it's far from being complete, and likely outdated. I also created https://github.com/vstinner/check_python_vuln project which implements runtime checks for a few Python vulnerabilities. It should help users and Linux packagers to check if their Python has known vulnerabilities or not. On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgorny@gentoo.org> wrote:
I feel that vulnerability fixes do not make it to end users fast enough. For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month!
We are doing our best to fix all known security vulnerabilities in all maintained Python versions (3.6, 3.7, 3.8, 3.9 and master: 5 branches). But we have no policy to define the severity of these vulnerabilities. IMO sometimes we are a little bit too conservative when marking some issues are "security" issues. For example, the https://bugs.python.org/issue41944 "CJK codecs tests call eval() on content received via HTTP" issue only impacts users running the Python test suite. https://python-security.readthedocs.io/vuln/cjk-codec-download-eval.html gives the timeline, extract: 2020-10-05 (+0 days): Reported (email sent to the PSRT list) (...) 2020-10-22 (+17 days): CVE-2020-27619 published 2020-12-07 (+63 days): Python 3.9.1 released 2020-12-21 (+77 days): Python 3.8.7 released IMO here the delay is reasonable (~2 months for 3.8 and 3.9 versions) for this kind of vulnerability. HTTP header and email header injections vulnerabilities are bad, but they can be prevented by applications. For example, good user inputs validation should prevent such vulnerability. Example of HTTP header injection timeline: https://python-security.readthedocs.io/vuln/http-header-injection-method.htm... Do you have a specific kind of vulnerability in mind that would require a faster release?
2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes.
We attempt to avoid incompatible changes in 3.x.y bug fix releases. If it happens, we can consider to revert it on a case by case basis. Usually, it is discussed before merging a change. Usually, the incompatible changes that we allow are justified... to fix security issues :-) Do you have a specific example of problematic incompatible change in mind? For me, the largest change was a Python 2.7 release which started to check TLS certificates on HTTP... It was to make Python more secure to protect users :-)
Furthermore, I think that at least for branches that are in higher level of maintenance than security, it could make sense to actually make security releases (e.g. 3.9.1.x) that would include only security fixes without other changes.
IMO that's too much work for everybody. developers, testers and packagers. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Tue, 2021-02-16 at 00:53 +0100, Victor Stinner wrote:
Hi Michał,
I created https://python-security.readthedocs.io/ website to track known Python vulnerabilities to help me checking if fixes are backported to all supported Python branches. I'm maintaing this list manually, it's far from being complete, and likely outdated.
I also created https://github.com/vstinner/check_python_vuln project which implements runtime checks for a few Python vulnerabilities. It should help users and Linux packagers to check if their Python has known vulnerabilities or not.
Thank you.
On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgorny@gentoo.org> wrote:
I feel that vulnerability fixes do not make it to end users fast enough. For example, according to the current release schedules for 3.9 and 3.8, the bugfix releases are planned two months apart. While the release is expected to happen in the next few days, both versions are known to be vulnerable for almost a month!
We are doing our best to fix all known security vulnerabilities in all maintained Python versions (3.6, 3.7, 3.8, 3.9 and master: 5 branches). But we have no policy to define the severity of these vulnerabilities. IMO sometimes we are a little bit too conservative when marking some issues are "security" issues.
I generally try to avoid making assumptions about severity of security bugs, and treat all of them the same.
For example, the https://bugs.python.org/issue41944 "CJK codecs tests call eval() on content received via HTTP" issue only impacts users running the Python test suite. https://python-security.readthedocs.io/vuln/cjk-codec-download-eval.html gives the timeline, extract:
2020-10-05 (+0 days): Reported (email sent to the PSRT list) (...) 2020-10-22 (+17 days): CVE-2020-27619 published 2020-12-07 (+63 days): Python 3.9.1 released 2020-12-21 (+77 days): Python 3.8.7 released
IMO here the delay is reasonable (~2 months for 3.8 and 3.9 versions) for this kind of vulnerability.
I have ignored this one as well, since we are running the test suite with network access disabled. I can't complain for this one but I don't know whether it doesn't impact other people's workflows. Please correct me if I'm missing something but if a distro packager uses a workflow like 'build, run tests, install' (this is what we do), I believe that this vulnerability could be used to inject malware into Python packages distributed with a lot of distributions.
HTTP header and email header injections vulnerabilities are bad, but they can be prevented by applications. For example, good user inputs validation should prevent such vulnerability. Example of HTTP header injection timeline: https://python-security.readthedocs.io/vuln/http-header-injection-method.htm...
Do you have a specific kind of vulnerability in mind that would require a faster release?
As I said above, I try not to make assumptions about severity. However, I personally wouldn't rely on people having their scripts fully protected or actually doing anything to workaround vulnerabilities in CPython. If we have a fix handy, it is our responsibility to deploy it.
2. When releases happen, security fixes are often combined with many other changes. This causes problems for distribution maintainers who, on one hand, would like to deploy the security fixes to production versions ASAP, and on the other, would prefer that the new version remained in testing for some time due to the other changes.
We attempt to avoid incompatible changes in 3.x.y bug fix releases. If it happens, we can consider to revert it on a case by case basis. Usually, it is discussed before merging a change.
Usually, the incompatible changes that we allow are justified... to fix security issues :-)
Do you have a specific example of problematic incompatible change in mind? For me, the largest change was a Python 2.7 release which started to check TLS certificates on HTTP... It was to make Python more secure to protect users :-)
I didn't mean intended incompatibilities. I meant the possibility of new bugs. I can't say I recall such a problem with CPython but that doesn't mean that there weren't sneaky issues in the past. We have a testing process in place to give braver users a chance to discover such problems.
Furthermore, I think that at least for branches that are in higher level of maintenance than security, it could make sense to actually make security releases (e.g. 3.9.1.x) that would include only security fixes without other changes.
IMO that's too much work for everybody. developers, testers and packagers.
I do realize that's extra work for people responsible for making the release but packagers are doing the work already (or should be doing it). -- Best regards, Michał Górny
On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgorny@gentoo.org> wrote:
I feel that vulnerability fixes do not make it to end users fast enough.
I think that it's time to put that into perspective with past vulnerabilities. Ok, let me look at the timeline of the discussed vulnerability, ctypes CVE-2021-3177: https://python-security.readthedocs.io/vuln/ctypes-buffer-overflow-pycarg_re... 2021-01-16: Python issue bpo-42938 reported by Jordy Zomer ... 2021-01-18 (+2 days): commit c347cbe (branch 3.9) 2021-01-18 (+2 days): commit ece5dfd (branch 3.8) 2021-01-19 (+3 days): CVE-2021-3177 published ... 2021-02-19 (+34 days): Python 3.8.8 released 2021-02-19 (+34 days): Python 3.9.2 released Ok. What about vulnerabilities fixes released last years? "HTTP header injection in urllib, urrlib2, httplib and http.client modules" https://python-security.readthedocs.io/vuln/http-header-injection.html 2017-09-19 (+1030 days): Python 3.3.7 released "CGI directory traversal" https://python-security.readthedocs.io/vuln/cgi-directory-traversal-is_cgi.h... 2011-05-09 (+1158 days): CVE-2011-1015 published 2013-04-07 (+1857 days): Python 3.2.4 released 2013-04-07 (+1857 days): Python 3.3.1 released "httplib unlimited read" https://python-security.readthedocs.io/vuln/httplib-unlimited-read.html 2011-06-11 (+652 days): Python 2.7.2 released 2011-06-11 (+652 days): Python 3.1.4 released "rgbimg and imageop overflows" https://python-security.readthedocs.io/vuln/rgbimg-imageop-overflows.html 2008-12-19 (+460 days): Python 2.5.3 released So the CVE-2021-3177 fix was delivery between 14x and 55x faster than the other listed fixes (I picked a few worst cases to put numbers in perspective). Congrats to the core developers for fixing the vulnerability in only *3* days and to release manager for releasing *4* (!) Python versions (3.6.13, 3.7.10, 3.8.8, 3.9.2) in only 34 days! I would like to highlight that exploiting a directory traversal or HTTP header injection is really trivial. Once you find a pattern to explore the filesystem / inject a HTTP header, the exploit is 100% reliable. On the other side, there is no known exploit for ctypes CVE-2021-3177 and ctypes is rarely used. I read that Django's GIS uses ctypes and floats, but so far nobody shows that PyCArg_repr() is called, and nobody published an exploit. To write a CVE-2021-3177 exploit, you must create a 64-bit floating point number (only 8 bytes!) which becomes valid machine code, and this code should allow to take control on the machine, once it's formatted as decimal. For example, PyCArg_repr(123.5) writes "123.5\0" string into the stack memory. but I don't think that it's valid x86-64 machine code. It is also hard to write a reliable exploit by injecting machine code in the stack memory. --- Nowadays it's way more difficult than 10 years ago to write an exploit using a stack overflow, C compilers provide multiple hardening layers: - FORTIFY_SOURCE, - Control flow Enforcement Technology (Intel CET), - Address Space Layout Randomization (ASLR), - stack protector, - Position Independent Executable (PIE), - etc. See https://wiki.debian.org/Hardening for example of C flags and linker flags for harderning. Did anyone notice that Red Hat Entreprise Linux 8 (RHEL) is *not* affected by the ctypes CVE-2021-3177 vulnerability thanks to hardening? "Red Hat Enterprise Linux 8: python36:3.6/python36: Not affected" and "This flaw could have had a higher Impact, however our packages are compiled with FORTIFY_SOURCE, which provides runtime protection to some memory and string functions and prevents this flaw from actually overwriting the buffer and potentially executing code." => https://access.redhat.com/security/cve/cve-2021-3177 I suggest you checking how your operating system built your Python executable, and libpython if Python is built in shared mode: hardening can prevent the ctypes vulnerabiliy, but also against *future* vulnerabilities! For example, Fedora 33 builds Python 3.9 with the following C flags: -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fcf-protection -fstack-clash-protection -fstack-protector-strong -fPIC -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 => add the -fPIE flag (...) And linker flags: -Wl,-z,now -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld => add the -pie flag (...) You can inspect hardening options on a ELF binary using hardening-check tool: $ hardening-check /usr/bin/python3.9 /usr/bin/python3.9: Position Independent Executable: yes Stack protected: no, not found! Fortify Source functions: unknown, no protectable libc functions used Read-only relocations: yes Immediate binding: yes Stack clash protection: unknown, no -fstack-clash-protection instructions found Control flow integrity: yes $ hardening-check /usr/lib64/libpython3.9.so.1.0 /usr/lib64/libpython3.9.so.1.0: Position Independent Executable: no, regular shared library (ignored) Stack protected: yes Fortify Source functions: yes (some protected functions found) Read-only relocations: yes Immediate binding: yes Stack clash protection: unknown, no -fstack-clash-protection instructions found Note: I don't know if this tool is reliable, but at least I can see that multiple hardening options are enabled ;-) On Fedora and RHEL 8, packages are built with the "annobin" extension which records C and linker flags (and much more information). So you check hardening using annocheck tool: $ annocheck /usr/lib64/libpython3.6m.so.1.0 annocheck: Version 9.50. Hardened: libpython3.6m.so.1.0: PASS. $ annocheck /usr/lib64/libpython3.6m.so.1.0 -v|grep -vE '(skip|warn|info):' annocheck: Version 9.50. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: One dynamic section/segment found. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: Stack not executable. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: DT_RPATH/DT_RUNPATH absent or rooted at /usr. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No RWX segments found. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No text relocations found. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No thread cancellation problems. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: GOT/PLT relocations are read only. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS. More information about annobin: https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-bi... Victor -- Night gathers, and now my watch begins. It shall not end until my death.
The default Make flags differ from platform to platform (and compiler to compiler, IIRC) as well. Thanks for this overview of RHEL/Fedora Python build security flags. ( Containers are the easiest way to get per- python interpreter SELinux contexts ( in order to limit the impact of exploitation of a vulnerability in CPython that an application exposes to users. FWIW, FWIU, OpenShift is the only k8s platform that does per-container contexts; and only RHEL/Fedora have a container-selinux? https://src.fedoraproject.org/rpms/container-selinux/blob/rawhide/f/containe... ) https://github.com/python/miss-islington is the backport bot. https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.9.2.... These are the patches necessary for conda-forge: https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml These are the only patches necessary on fedora: https://src.fedoraproject.org/rpms/python3.9/tree/rawhide On 2/20/21, Victor Stinner <vstinner@python.org> wrote:
On Thu, Feb 11, 2021 at 9:44 PM Michał Górny <mgorny@gentoo.org> wrote:
I feel that vulnerability fixes do not make it to end users fast enough.
I think that it's time to put that into perspective with past vulnerabilities.
Ok, let me look at the timeline of the discussed vulnerability, ctypes CVE-2021-3177: https://python-security.readthedocs.io/vuln/ctypes-buffer-overflow-pycarg_re...
2021-01-16: Python issue bpo-42938 reported by Jordy Zomer ... 2021-01-18 (+2 days): commit c347cbe (branch 3.9) 2021-01-18 (+2 days): commit ece5dfd (branch 3.8) 2021-01-19 (+3 days): CVE-2021-3177 published ... 2021-02-19 (+34 days): Python 3.8.8 released 2021-02-19 (+34 days): Python 3.9.2 released
Ok. What about vulnerabilities fixes released last years?
"HTTP header injection in urllib, urrlib2, httplib and http.client modules" https://python-security.readthedocs.io/vuln/http-header-injection.html 2017-09-19 (+1030 days): Python 3.3.7 released
"CGI directory traversal" https://python-security.readthedocs.io/vuln/cgi-directory-traversal-is_cgi.h... 2011-05-09 (+1158 days): CVE-2011-1015 published 2013-04-07 (+1857 days): Python 3.2.4 released 2013-04-07 (+1857 days): Python 3.3.1 released
"httplib unlimited read" https://python-security.readthedocs.io/vuln/httplib-unlimited-read.html 2011-06-11 (+652 days): Python 2.7.2 released 2011-06-11 (+652 days): Python 3.1.4 released
"rgbimg and imageop overflows" https://python-security.readthedocs.io/vuln/rgbimg-imageop-overflows.html 2008-12-19 (+460 days): Python 2.5.3 released
So the CVE-2021-3177 fix was delivery between 14x and 55x faster than the other listed fixes (I picked a few worst cases to put numbers in perspective).
Congrats to the core developers for fixing the vulnerability in only *3* days and to release manager for releasing *4* (!) Python versions (3.6.13, 3.7.10, 3.8.8, 3.9.2) in only 34 days!
I would like to highlight that exploiting a directory traversal or HTTP header injection is really trivial. Once you find a pattern to explore the filesystem / inject a HTTP header, the exploit is 100% reliable.
On the other side, there is no known exploit for ctypes CVE-2021-3177 and ctypes is rarely used. I read that Django's GIS uses ctypes and floats, but so far nobody shows that PyCArg_repr() is called, and nobody published an exploit.
To write a CVE-2021-3177 exploit, you must create a 64-bit floating point number (only 8 bytes!) which becomes valid machine code, and this code should allow to take control on the machine, once it's formatted as decimal. For example, PyCArg_repr(123.5) writes "123.5\0" string into the stack memory. but I don't think that it's valid x86-64 machine code. It is also hard to write a reliable exploit by injecting machine code in the stack memory.
---
Nowadays it's way more difficult than 10 years ago to write an exploit using a stack overflow, C compilers provide multiple hardening layers: - FORTIFY_SOURCE, - Control flow Enforcement Technology (Intel CET), - Address Space Layout Randomization (ASLR), - stack protector, - Position Independent Executable (PIE), - etc.
See https://wiki.debian.org/Hardening for example of C flags and linker flags for harderning.
Did anyone notice that Red Hat Entreprise Linux 8 (RHEL) is *not* affected by the ctypes CVE-2021-3177 vulnerability thanks to hardening?
"Red Hat Enterprise Linux 8: python36:3.6/python36: Not affected" and "This flaw could have had a higher Impact, however our packages are compiled with FORTIFY_SOURCE, which provides runtime protection to some memory and string functions and prevents this flaw from actually overwriting the buffer and potentially executing code." => https://access.redhat.com/security/cve/cve-2021-3177
I suggest you checking how your operating system built your Python executable, and libpython if Python is built in shared mode: hardening can prevent the ctypes vulnerabiliy, but also against *future* vulnerabilities!
For example, Fedora 33 builds Python 3.9 with the following C flags:
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fcf-protection -fstack-clash-protection -fstack-protector-strong -fPIC -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 => add the -fPIE flag (...)
And linker flags:
-Wl,-z,now -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld => add the -pie flag (...)
You can inspect hardening options on a ELF binary using hardening-check tool:
$ hardening-check /usr/bin/python3.9 /usr/bin/python3.9: Position Independent Executable: yes Stack protected: no, not found! Fortify Source functions: unknown, no protectable libc functions used Read-only relocations: yes Immediate binding: yes Stack clash protection: unknown, no -fstack-clash-protection instructions found Control flow integrity: yes
$ hardening-check /usr/lib64/libpython3.9.so.1.0 /usr/lib64/libpython3.9.so.1.0: Position Independent Executable: no, regular shared library (ignored) Stack protected: yes Fortify Source functions: yes (some protected functions found) Read-only relocations: yes Immediate binding: yes Stack clash protection: unknown, no -fstack-clash-protection instructions found
Note: I don't know if this tool is reliable, but at least I can see that multiple hardening options are enabled ;-)
On Fedora and RHEL 8, packages are built with the "annobin" extension which records C and linker flags (and much more information). So you check hardening using annocheck tool:
$ annocheck /usr/lib64/libpython3.6m.so.1.0 annocheck: Version 9.50. Hardened: libpython3.6m.so.1.0: PASS.
$ annocheck /usr/lib64/libpython3.6m.so.1.0 -v|grep -vE '(skip|warn|info):' annocheck: Version 9.50. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: One dynamic section/segment found. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: Stack not executable. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: DT_RPATH/DT_RUNPATH absent or rooted at /usr. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No RWX segments found. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No text relocations found. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: No thread cancellation problems. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS: GOT/PLT relocations are read only. Hardened: /usr/lib64/libpython3.6m.so.1.0: PASS.
More information about annobin: https://developers.redhat.com/blog/2018/02/20/annobin-storing-information-bi...
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XF33457K... Code of Conduct: http://python.org/psf/codeofconduct/
-- Wes Turner https://westurner.org https://wrdrd.com/docs/consulting/knowledge-engineering
Looking at the other replies, I'm wondering if you fully understand python's variant of version numbering. I suggest we change the announcement template from: "Python 3.9.2 is the newest major release of the Python programming language, and it contains many new features and optimizations." to: "Python 3.9 was the newest major release of the Python programming language, and it contains many new features and optimizations. Python 3.9.2 is a bugfix and security release; it has no new features relative to 3.9." 3.9.1 (and 3.9.27, if that ever happens) are supposed to the be the same as 3.9.0, except with bugs fixed. Because "a feature is just a bug with tenure", there comes a time when non-security bugs stop being fixed. There isn't a hard-and-fast rule on when that is, but within a year or two. Looking back, a .6 release was unlikely to contain much beyond security. Even before that time, CPython still tries to err on the overly-cautious side of "can this bug-fix be backported", because of an overly-optimistic assessment ~20 years ago. (A harmless feature was added in a backwards-compatible way, but for a while instructions had to specify a bugfix version as well.) In theory, someone could release 3.9.0s1, 3.9.0s2, ... 3.9.1s1 ... but what would be the point? 3.9.1s2 would have contained exactly the same changes as 3.9.2rc, which apparently didn't get picked up much. The difference between 3.9.2rc and 3.9.2 does include a non-security bugfix -- a part of the Windows API that was advertised as working will now actually work. Is that really adding much extra upgrade risk? Of course, to get these extra releases, someone will have to be more careful about deciding what counts as a security fix vs a regular bugfix, which is already sometimes fuzzy. And realistically, it is *only* the security fixes that are likely to be a problem for working code ... Even if the releases were trivial, how much value would that actually provide? -jJ
participants (13)
-
Brett Cannon -
Christian Heimes -
Gregory P. Smith -
Jim J. Jewett -
Mats Wichmann -
Michał Górny -
Mike Miller -
Stephen J. Turnbull -
Stestagg -
Steve Holden -
Terry Reedy -
Victor Stinner -
Wes Turner