Re: [SciPy-Dev] [Numpy-discussion] scipy 0.18 release candidate 1
On Tue, Jun 21, 2016 at 7:29 PM, Evgeni Burovski <evgeny.burovskiy@gmail.com
wrote:
On Tue, Jun 21, 2016 at 4:27 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Mon, Jun 20, 2016 at 6:31 AM, Evgeni Burovski <evgeny.burovskiy@gmail.com> wrote:
Hi,
I'm pleased to announce the availability of the first release candidate for scipy 0.18.0. Please try this release and report any issues on Github tracker, https://github.com/scipy/scipy, or scipy-dev mailing list. Source tarballs and release notes are available from Github releases, https://github.com/scipy/scipy/releases/tag/v0.18.0rc1
Please note that this is a source-only release. We do not provide Windows binaries for this release. OS X and Linux wheels will be provided for the final release.
The current release schedule is
27 June: rc2 (if necessary) 11 July: final release
Thanks to everyone who contributed to this release!
Cheers,
Evgeni
A part of the release notes follows:
========================== SciPy 0.18.0 Release Notes ==========================
.. note:: Scipy 0.18.0 is not released yet!
.. contents::
SciPy 0.18.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.19.x branch, and on adding new features on the master branch.
This release requires Python 2.7 or 3.4-3.5 and NumPy 1.7.1 or greater.
Thanks a lot for taking on the release.
I put the manylinux1 and OSX wheel building into a single repo to test 64- and 32-bit linux wheels. There's a test run with the 0.18.0rc1 code here:
https://travis-ci.org/MacPython/scipy-wheels/builds/139084454
For Python 3 I am getting these errors: https://github.com/scipy/scipy/issues/6292
For all 32-bit builds I am getting this error: https://github.com/scipy/scipy/issues/6093
For the Python 3 32-bit builds I am also getting this error: https://github.com/scipy/scipy/issues/6101
For the builds that succeeded without failure (all OSX and manylinux1 for 64 bit Python 2.7), you can test with:
python -m pip install -U pip pip install --trusted-host wheels.scipy.org -f https://wheels.scipy.org -U --pre scipy
Thanks again, sorry for the tiring news,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Thanks Matthew for testing and reporting these!
Indeed, thanks! Would it make sense to put these wheels on PyPI, so people can test them with ``pip install scipy --pre``? We do that with Numpy wheels as well now. Ralf
<snip>
Would it make sense to put these wheels on PyPI, so people can test them with ``pip install scipy --pre``? We do that with Numpy wheels as well now.
Is there some incantation at upload time to mark them as pre-release, so that `pip install scipy` does not pick them up?
It's keyed off the version number -- sdists or wheels whose version number contains alpha/beta/rc/dev tags are skipped by pip by default, unless --pre was passed. -n On Jun 21, 2016 12:39 PM, "Evgeni Burovski" <evgeny.burovskiy@gmail.com> wrote:
<snip>
Would it make sense to put these wheels on PyPI, so people can test them with ``pip install scipy --pre``? We do that with Numpy wheels as well now.
Is there some incantation at upload time to mark them as pre-release, so that `pip install scipy` does not pick them up? _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
On Tue, Jun 21, 2016 at 12:46 PM, Nathaniel Smith <njs@pobox.com> wrote:
It's keyed off the version number -- sdists or wheels whose version number contains alpha/beta/rc/dev tags are skipped by pip by default, unless --pre was passed.
-n
On Jun 21, 2016 12:39 PM, "Evgeni Burovski" <evgeny.burovskiy@gmail.com> wrote:
<snip>
Would it make sense to put these wheels on PyPI, so people can test them with ``pip install scipy --pre``? We do that with Numpy wheels as well now.
We don't have built manylinux wheels for Python 3 because of the test failures, but the incantation to upload the wheels that are built is (I think): git clone https://github.com/MacPython/terryfy.git python terryfy/wheel-uploader -r warehouse -v -s -t manylinux1 scipy 0.18.0rc1 python terryfy/wheel-uploader -r warehouse -v -s -t macosx scipy 0.18.0rc1 You'll also need a 'warehouse' section in your ~/.pypirc, as in: [distutils] index-servers = pypi warehouse [pypi] username: my_user password: my_password [warehouse] repository: https://upload.pypi.io/legacy/ username: my_warehouse_user password: my_warehouse_password and installations of "twine" and "beautifulsoup4". Cheers, Matthew
On Tue, Jun 21, 2016 at 9:16 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Tue, Jun 21, 2016 at 12:46 PM, Nathaniel Smith <njs@pobox.com> wrote:
It's keyed off the version number -- sdists or wheels whose version number contains alpha/beta/rc/dev tags are skipped by pip by default, unless --pre was passed.
-n
On Jun 21, 2016 12:39 PM, "Evgeni Burovski" <evgeny.burovskiy@gmail.com> wrote:
<snip>
Would it make sense to put these wheels on PyPI, so people can test them with ``pip install scipy --pre``? We do that with Numpy wheels as well now.
We don't have built manylinux wheels for Python 3 because of the test failures, but the incantation to upload the wheels that are built is (I think):
git clone https://github.com/MacPython/terryfy.git python terryfy/wheel-uploader -r warehouse -v -s -t manylinux1 scipy 0.18.0rc1 python terryfy/wheel-uploader -r warehouse -v -s -t macosx scipy 0.18.0rc1
You'll also need a 'warehouse' section in your ~/.pypirc, as in:
[distutils] index-servers = pypi warehouse
[pypi] username: my_user password: my_password
[warehouse] repository: https://upload.pypi.io/legacy/ username: my_warehouse_user password: my_warehouse_password
and installations of "twine" and "beautifulsoup4".
Cheers,
Matthew
Thanks! One question --- equally applicable to both pre-release and final releases: Security. If we download the wheels from the build farm and then upload to PyPI, how can a user check that what they download has not be tampered with? For source tarballs (and previously, Windows installers), we PGP sign the git tag and include checksums in the README file. This way they can at least verify the checksums.
On Jun 21, 2016 14:37, "Evgeni Burovski" <evgeny.burovskiy@gmail.com> wrote:
One question --- equally applicable to both pre-release and final releases: Security. If we download the wheels from the build farm and then upload to PyPI, how can a user check that what they download has not be tampered with?
For source tarballs (and previously, Windows installers), we PGP sign the git tag and include checksums in the README file. This way they can at least verify the checksums.
I'm dubious that this really accomplishes much: https://caremad.io/2013/07/packaging-signing-not-holy-grail/ But, if you want to include checksums in the README, you can do that by just downloading the build farm wheels and checksumming them. This doesn't protect against a compromised build farm, but neither does anything else. (Even PGP signing doesn't protect you if your release manager's laptop is compromised, and realistically any laptop that has write permissions on the repository could add a backdoor with no one noticing, just by pushing it directly to master with an innocuous commit message.) OTOH even this crude download and checksum approach does at least make life more difficult for anyone who tries to compromise the packages later after the checksum is made. -n
Tue, 21 Jun 2016 17:23:59 -0700, Nathaniel Smith kirjoitti:
On Jun 21, 2016 14:37, "Evgeni Burovski" <evgeny.burovskiy@gmail.com> wrote:
One question --- equally applicable to both pre-release and final releases: Security. If we download the wheels from the build farm and then upload to PyPI, how can a user check that what they download has not be tampered with?
For source tarballs (and previously, Windows installers), we PGP sign the git tag and include checksums in the README file. This way they can at least verify the checksums.
I'm dubious that this really accomplishes much: https://caremad.io/2013/07/packaging-signing-not-holy-grail/
Well, security is best done in depth, and signing source tarballs is little extra work. That article talks about package signing, but it is only from the point of view of a random user. If it later becomes necessary to try to find out whether some tarballs are compromised by someone replacing release files on Github (or sourceforge injecting adware) etc., this is possible for me to do. I have my own key, and the keys by Ralf and Evgeni that I know are with high likelihood valid (assuming their laptops are not compromised). -- Pauli Virtanen
On Jun 23, 2016 11:23 AM, "Pauli Virtanen" <pav@iki.fi> wrote:
Tue, 21 Jun 2016 17:23:59 -0700, Nathaniel Smith kirjoitti:
On Jun 21, 2016 14:37, "Evgeni Burovski" <evgeny.burovskiy@gmail.com> wrote:
One question --- equally applicable to both pre-release and final releases: Security. If we download the wheels from the build farm and then upload to PyPI, how can a user check that what they download has not be tampered with?
For source tarballs (and previously, Windows installers), we PGP sign the git tag and include checksums in the README file. This way they can at least verify the checksums.
I'm dubious that this really accomplishes much: https://caremad.io/2013/07/packaging-signing-not-holy-grail/
Well, security is best done in depth, and signing source tarballs is little extra work.
That article talks about package signing, but it is only from the point of view of a random user.
If it later becomes necessary to try to find out whether some tarballs are compromised by someone replacing release files on Github (or sourceforge injecting adware) etc., this is possible for me to do. I have my own key, and the keys by Ralf and Evgeni that I know are with high likelihood valid (assuming their laptops are not compromised).
I believe the question was specifically about wheels that aren't being built by any of those three people though? But anyway, yeah, that is the main situation where this kind of package signing might help, and which I addressed in the second half of the email :-). But note that it would also work just as well to, say, keep a text file in the scipy repo that has the sha256 of every file uploaded to pypi. (Maybe even better, because someone who attacked pypi could delete the PGP signatures to confuse matters, and do you have backups?) -n
Thu, 23 Jun 2016 11:47:37 -0700, Nathaniel Smith kirjoitti: [clip]
I believe the question was specifically about wheels that aren't being built by any of those three people though? But anyway, yeah, that is the main situation where this kind of package signing might help, and which I addressed in the second half of the email :-). But note that it would also work just as well to, say, keep a text file in the scipy repo that has the sha256 of every file uploaded to pypi. (Maybe even better, because someone who attacked pypi could delete the PGP signatures to confuse matters, and do you have backups?)
How do I know one of these people pushed the commit that changed the checksums to the Scipy repository? PGP signatures do add stronger guarantees than just trusting Github, provided they you know the people whose keys are in question.
On Thu, Jun 23, 2016 at 11:55 AM, Pauli Virtanen <pav@iki.fi> wrote:
Thu, 23 Jun 2016 11:47:37 -0700, Nathaniel Smith kirjoitti: [clip]
I believe the question was specifically about wheels that aren't being built by any of those three people though? But anyway, yeah, that is the main situation where this kind of package signing might help, and which I addressed in the second half of the email :-). But note that it would also work just as well to, say, keep a text file in the scipy repo that has the sha256 of every file uploaded to pypi. (Maybe even better, because someone who attacked pypi could delete the PGP signatures to confuse matters, and do you have backups?)
How do I know one of these people pushed the commit that changed the checksums to the Scipy repository?
Because it was pushed by their SSH key at the same time as they were doing the release, and then replicated to people's computers all over the world, making it impossible for anyone (even them, or even people who later steal their SSH/PGP keys) to alter later without creating incriminating entries in 'git log'. (This doesn't even require trusting github, really, so long as you trust that the people doing the release *did* intentionally look at this file and add some entries -- as long as you know that they did, and you know that only one entry was ever made to the file, then it must have been their entry.)
PGP signatures do add stronger guarantees than just trusting Github, provided they you know the people whose keys are in question.
Sure, in some situations. The main points I wanted to make are just (a) PGP-signing releases is less important than many people think, because it only helps in certain narrow situations (like trying to do forensics after an exploit is noticed), and (b) for those particular situations, there are several options that work fine, so we shouldn't stress out about PGP signatures -- if they work, great, if not, then there are other options that have slightly different behavior in edge cases but that still address the same threat model in an adequate way. -n -- Nathaniel J. Smith -- https://vorpus.org
OK, here's what I'm going to do: I'll download the wheels from Matthew's build farm, checksum them along with the source tarballs, and add the checksums to the README file which is clearsigned with my PGP signature. That file gets uploaded to PyPI, Github releases and sent along with the release announcement to a bunch of mailing lists. (like this, https://mail.scipy.org/pipermail/scipy-dev/2016-January/021189.html) AFAICS, this would cover the main vectors, apart from (i) the build farm producing malicious stuff, (ii) RM or RM's laptop doing what it shouldn't be doing, or (iii) someone patching the wheels en route from the build farm to RM's laptop. I don't see how to address two first points or whether we actually need to address those. The third one can be taken care of by checksumming the wheels on the build farm, so that RM can verify them on before uploading. This is probably not too hard to do with some tweaks to MacPython's build scripts and/or terryfy download machinery Matthew described upthread (I'm still to figure out how to use that machinery, but that's separate). On Thu, Jun 23, 2016 at 7:55 PM, Pauli Virtanen <pav@iki.fi> wrote:
Thu, 23 Jun 2016 11:47:37 -0700, Nathaniel Smith kirjoitti: [clip]
I believe the question was specifically about wheels that aren't being built by any of those three people though? But anyway, yeah, that is the main situation where this kind of package signing might help, and which I addressed in the second half of the email :-). But note that it would also work just as well to, say, keep a text file in the scipy repo that has the sha256 of every file uploaded to pypi. (Maybe even better, because someone who attacked pypi could delete the PGP signatures to confuse matters, and do you have backups?)
How do I know one of these people pushed the commit that changed the checksums to the Scipy repository?
PGP signatures do add stronger guarantees than just trusting Github, provided they you know the people whose keys are in question.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
On Thu, Jun 23, 2016 at 1:34 PM, Evgeni Burovski <evgeny.burovskiy@gmail.com
wrote:
OK, here's what I'm going to do: I'll download the wheels from Matthew's build farm, checksum them along with the source tarballs, and add the checksums to the README file which is clearsigned with my PGP signature. That file gets uploaded to PyPI, Github releases and sent along with the release announcement to a bunch of mailing lists. (like this, https://mail.scipy.org/pipermail/scipy-dev/2016-January/021189.html)
AFAICS, this would cover the main vectors, apart from (i) the build farm producing malicious stuff, (ii) RM or RM's laptop doing what it shouldn't be doing, or (iii) someone patching the wheels en route from the build farm to RM's laptop.
I don't see how to address two first points or whether we actually need to address those. The third one can be taken care of by checksumming the wheels on the build farm, so that RM can verify them on before uploading.
This is probably not too hard to do with some tweaks to MacPython's build scripts and/or terryfy download machinery Matthew described upthread (I'm still to figure out how to use that machinery, but that's separate).
I think there were problems with the terryfy machinery and signing, I asked Mathew about that before re NumPy. If you just download the built wheels, you can use twine to upload them with signatures, same with source files. <snip> Chuck
Hi, On Thu, Jun 23, 2016 at 3:36 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Thu, Jun 23, 2016 at 1:34 PM, Evgeni Burovski <evgeny.burovskiy@gmail.com> wrote:
OK, here's what I'm going to do: I'll download the wheels from Matthew's build farm, checksum them along with the source tarballs, and add the checksums to the README file which is clearsigned with my PGP signature. That file gets uploaded to PyPI, Github releases and sent along with the release announcement to a bunch of mailing lists. (like this, https://mail.scipy.org/pipermail/scipy-dev/2016-January/021189.html)
AFAICS, this would cover the main vectors, apart from (i) the build farm producing malicious stuff, (ii) RM or RM's laptop doing what it shouldn't be doing, or (iii) someone patching the wheels en route from the build farm to RM's laptop.
I don't see how to address two first points or whether we actually need to address those. The third one can be taken care of by checksumming the wheels on the build farm, so that RM can verify them on before uploading.
This is probably not too hard to do with some tweaks to MacPython's build scripts and/or terryfy download machinery Matthew described upthread (I'm still to figure out how to use that machinery, but that's separate).
I think there were problems with the terryfy machinery and signing, I asked Mathew about that before re NumPy. If you just download the built wheels, you can use twine to upload them with signatures, same with source files.
There were - I think that was to do with twine not accepting the flag as it should - but it appears to work now. For example, this command: python terryfy/wheel-uploader -r warehouse -v -s -t manylinux1 scipy 0.18.0rc1 has the `-s` flag which gets passed into twine, and so prompts you to sign each wheel. It would also be a good idea to prefer the https URL for the wheels, as in: python terryfy/wheel-uploader -u https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcd... -r warehouse -v -s -t manylinux1 scipy 0.18.0rc1 although the https directory listing and files appear to be updated later than then http directory. Cheers, Matthew
participants (6)
-
Charles R Harris -
Evgeni Burovski -
Matthew Brett -
Nathaniel Smith -
Pauli Virtanen -
Ralf Gommers