
On Tue, 2020-09-08 at 23:15 +0300, Matti Picus wrote:
I have uploaded rc1 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) which should be mirrored soon to https://downloads.python.org/pypy/
The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download...
The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html
This release does include a 3.7 alpha.
Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them.
Any comments are welcome.
What's the vulnerability status of stdlib? I've tested pypy2.7 and pypy3.6 so far and neither seems to contain CVE- 2019-20907 fix (it was never backported to py2.7), the patch from [1] seems to apply cleanly to both. pypy3.6 seems to be missing bpo-39603, and the patch from [2] doesn't apply cleanly (does pypy3 contain outdated version or modified?). CVE-2020-14422 is also unresolved. Could you please either update stdlib of pypy3.6 or look through CPython changes and backport the security fixes? For pypy2.7, please backport [1] directly since upstream is no longer maintaining that branch. [1] https://github.com/python/cpython/commit/47a2955589bdb1a114d271496ff803ad73f... [2] https://github.com/python/cpython/commit/f02de961b9f19a5db0ead56305fe0057a78... -- Best regards, Michał Górny

On 9/9/20 9:55 AM, Michał Górny wrote:
On Tue, 2020-09-08 at 23:15 +0300, Matti Picus wrote:
I have uploaded rc1 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) which should be mirrored soon to https://downloads.python.org/pypy/
The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download...
The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html
This release does include a 3.7 alpha.
Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them.
Any comments are welcome.
What's the vulnerability status of stdlib?
I've tested pypy2.7 and pypy3.6 so far and neither seems to contain CVE- 2019-20907 fix (it was never backported to py2.7), the patch from [1] seems to apply cleanly to both.
pypy3.6 seems to be missing bpo-39603, and the patch from [2] doesn't apply cleanly (does pypy3 contain outdated version or modified?).
CVE-2020-14422 is also unresolved.
Could you please either update stdlib of pypy3.6 or look through CPython changes and backport the security fixes? For pypy2.7, please backport [1] directly since upstream is no longer maintaining that branch.
[1] https://github.com/python/cpython/commit/47a2955589bdb1a114d271496ff803ad73f... [2] https://github.com/python/cpython/commit/f02de961b9f19a5db0ead56305fe0057a78...
Thanks for looking at this.We ship stdlib 2.7.13, 3.6.9, 3.7.4 with some slight modifications, including backporting some fixes. I fixed CVE-2019-20907 for pypy2.7, pypy3,6, and CVE-2020-14422 for py3.6, 3.7 bpo-39603 is part of 3.6.12, 3.7.9 which were shipped 25 days ago, and that file has changed significantly since the versions we ship. Updating the stdlib is a large undertaking, help welcome for py2.7 and py3.7. I don't think it is worth the effort for py3.6. There are directions in lib-python/stdlib-upgrade.txt. Matti

On Wed, 2020-09-09 at 12:41 +0300, Matti Picus wrote:
On 9/9/20 9:55 AM, Michał Górny wrote:
On Tue, 2020-09-08 at 23:15 +0300, Matti Picus wrote:
I have uploaded rc1 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) which should be mirrored soon to https://downloads.python.org/pypy/
The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download...
The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html
This release does include a 3.7 alpha.
Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them.
Any comments are welcome.
What's the vulnerability status of stdlib?
I've tested pypy2.7 and pypy3.6 so far and neither seems to contain CVE- 2019-20907 fix (it was never backported to py2.7), the patch from [1] seems to apply cleanly to both.
pypy3.6 seems to be missing bpo-39603, and the patch from [2] doesn't apply cleanly (does pypy3 contain outdated version or modified?).
CVE-2020-14422 is also unresolved.
Could you please either update stdlib of pypy3.6 or look through CPython changes and backport the security fixes? For pypy2.7, please backport [1] directly since upstream is no longer maintaining that branch.
[1] https://github.com/python/cpython/commit/47a2955589bdb1a114d271496ff803ad73f... [2] https://github.com/python/cpython/commit/f02de961b9f19a5db0ead56305fe0057a78...
Thanks for looking at this.We ship stdlib 2.7.13, 3.6.9, 3.7.4 with some slight modifications, including backporting some fixes.
I fixed CVE-2019-20907 for pypy2.7, pypy3,6, and CVE-2020-14422 for py3.6, 3.7
bpo-39603 is part of 3.6.12, 3.7.9 which were shipped 25 days ago, and that file has changed significantly since the versions we ship.
Updating the stdlib is a large undertaking, help welcome for py2.7 and py3.7. I don't think it is worth the effort for py3.6.
Is this something to be done just before the release? Would you accept fixes rebased specifically on top of current pypy code? Unless I'm mistaken, bpo-39603 should be trivial to fix. I can submit a merge request if you want. However, it's so trivial it'd probably take you less time to fix it yourself than me to recall how to use mercurial again ;-). -- Best regards, Michał Górny

On 9/10/20 12:04 AM, Michał Górny wrote:
Is this something to be done just before the release? Would you accept fixes rebased specifically on top of current pypy code?
Unless I'm mistaken, bpo-39603 should be trivial to fix. I can submit a merge request if you want.
I am willing to accept small patches to stdlib in the next two days (unless something delays the release more). I don't have time to work on bpo-39603. but am willing to review a MR or even patch diff via email. Matti

On Thu, 2020-09-10 at 00:41 +0300, Matti Picus wrote:
On 9/10/20 12:04 AM, Michał Górny wrote:
Is this something to be done just before the release? Would you accept fixes rebased specifically on top of current pypy code?
Unless I'm mistaken, bpo-39603 should be trivial to fix. I can submit a merge request if you want.
I am willing to accept small patches to stdlib in the next two days (unless something delays the release more). I don't have time to work on bpo-39603. but am willing to review a MR or even patch diff via email.
I've filed two MRs for a start: https://foss.heptapod.net/pypy/pypy/-/merge_requests/752 https://foss.heptapod.net/pypy/pypy/-/merge_requests/753 Could you please let me know if that approach is ok? If so, I'll tackle the remaining vulnerabilities (around 6 patches FWICS). -- Best regards, Michał Górny

On 9/10/20 12:40 PM, Michał Górny wrote:
I've filed two MRs for a start: https://foss.heptapod.net/pypy/pypy/-/merge_requests/752 https://foss.heptapod.net/pypy/pypy/-/merge_requests/753
Could you please let me know if that approach is ok? If so, I'll tackle the remaining vulnerabilities (around 6 patches FWICS).
Thanks. Could you add comments in each patched chunk of each PR that will allow us to track how the code diverged? Something like "added manually to resolve bpo xxx". I am concerned this will make an eventual update to the stdlib difficult. MR753 is quite invasive. Are all the other patches against the same single file? Maybe a different strategy would be to adopt the file and tests as-is from the HEAD of the CPython branch, as long as no user-facing APIs change. It would be nice to make similar PRs against default (python2.7) in lib-python/2.7 on that branch. Matti

On Thu, 2020-09-10 at 13:37 +0300, Matti Picus wrote:
On 9/10/20 12:40 PM, Michał Górny wrote:
I've filed two MRs for a start: https://foss.heptapod.net/pypy/pypy/-/merge_requests/752 https://foss.heptapod.net/pypy/pypy/-/merge_requests/753
Could you please let me know if that approach is ok? If so, I'll tackle the remaining vulnerabilities (around 6 patches FWICS).
Thanks.
Could you add comments in each patched chunk of each PR that will allow us to track how the code diverged? Something like "added manually to resolve bpo xxx".
I don't see a problem with doing that.
I am concerned this will make an eventual update to the stdlib difficult. MR753 is quite invasive. Are all the other patches against the same single file? Maybe a different strategy would be to adopt the file and tests as-is from the HEAD of the CPython branch, as long as no user-facing APIs change.
I suppose I can try doing that. I presumed you'll want to keep the changes to bare minimum possible but it might turn out not much different once I iterate over all the CVEs.
It would be nice to make similar PRs against default (python2.7) in lib-python/2.7 on that branch.
The primary problem with that is that CPython 2.7 is abandoned and contrary to all the big talk about random people willing to maintain it post-EOL, I am yet to see anyone taking its security seriously. So far I and the Fedora maintainer were able to independently backport one vulnerability that clearly applied (the tarfile one) but we weren't able to get a clear match of any other Python 3.x fixes to 2.7 codebase. Well, until today when thanks to you I've noticed that http.request code has a vulnerable match in httplib. But this all is lots of work, and I'm really supposed to be doing something else right now. I'm trying my best but I'm not sure if I can manage to fix several months of negligence in two days. -- Best regards, Michał Górny

On 9/10/20 1:45 PM, Michał Górny wrote:
So far I and the Fedora maintainer were able to independently backport one vulnerability that clearly applied (the tarfile one) but we weren't able to get a clear match of any other Python 3.x fixes to 2.7 codebase. Well, until today when thanks to you I've noticed that http.request code has a vulnerable match in httplib.
But this all is lots of work, and I'm really supposed to be doing something else right now. I'm trying my best but I'm not sure if I can manage to fix several months of negligence in two days.
Thanks for all you are doing. The release deadline is only a motivator for now since we could do another much smaller release next month if needed. I want to move toward python3.7 as soon as possible since the scientific python stack's stated python version policy means 3.6 will no longer be expressly supported especially after 3.9 comes out. Matti

On Thu, 2020-09-10 at 14:00 +0300, Matti Picus wrote:
On 9/10/20 1:45 PM, Michał Górny wrote:
So far I and the Fedora maintainer were able to independently backport one vulnerability that clearly applied (the tarfile one) but we weren't able to get a clear match of any other Python 3.x fixes to 2.7 codebase. Well, until today when thanks to you I've noticed that http.request code has a vulnerable match in httplib.
But this all is lots of work, and I'm really supposed to be doing something else right now. I'm trying my best but I'm not sure if I can manage to fix several months of negligence in two days.
Thanks for all you are doing. The release deadline is only a motivator for now since we could do another much smaller release next month if needed.
I want to move toward python3.7 as soon as possible since the scientific python stack's stated python version policy means 3.6 will no longer be expressly supported especially after 3.9 comes out.
I wholeheartedly support that. Gentoo has switched to Python 3.7 by default already, and it sucks that the only version of PyPy3 is behind that. That said, have you considered going straight to 3.8? In my experience, the switch from 3.7 to 3.8 was far less painful than from 3.6 to 3.7, so it might be less work to skip 3.7 entirely. -- Best regards, Michał Górny

On 9/10/20 2:08 PM, Michał Górny wrote:
I wholeheartedly support that. Gentoo has switched to Python 3.7 by default already, and it sucks that the only version of PyPy3 is behind that. That said, have you considered going straight to 3.8? In my experience, the switch from 3.7 to 3.8 was far less painful than from 3.6 to 3.7, so it might be less work to skip 3.7 entirely.
We are on the cusp of releasing 3.7, I don't want to delay any more. Hopefully we will have some bandwidth to push forward soon. It is encouraging to hear that the transition may be easier than previous ones. Matti
participants (2)
-
Matti Picus
-
Michał Górny