Mailman 3 March 2018 - Distutils-SIG

TLS support policy & PyPI communications
by Sumana Harihareswara 22 Mar '18

22 Mar '18

PSF blogged last year https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.ht… that > The more crucial deadline comes June 30, 2018. On that date all remaining python.org sites, including PyPI, will no longer support TSL 1.0 and 1.1. Older Python versions that do not implement TLSv1.2 will be prohibited from accessing PyPI. I asked Ernest W. Durbin III whether I ought to re-announce this to users in my PyPI announcements. He looked at our TLS trends/stats and told me we have a very very low proportion of traffic that will be affected when we shift over. Therefore, since it'll affect so few, I won't shout about TLS versions in my PyPI communications. Marking that here for the record. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

2 1

PyPI/Warehouse: infrastructure hardening & the CAPTCHA conundrum
by Sumana Harihareswara 21 Mar '18

21 Mar '18

So we aren't quite at beta yet, but we'll be shouting about pypi.org *really soon*. We have nearly all the Warehouse improvements we need for beta, and nearly all the infrastructure improvements we believe we'll need for the switchover. I'll tell you how you can help, then talk about the current state of things. * The big blocker keeping us from beta: China & CAPTCHAs. Help advise us.[1] * Comment on a "needs discussion" issue[2]. * Help us with large-scope JavaScript issues[3], like our frontend testing approach. * Please talk with Nicole about being a subject or interviewer for user tests[4]. * Tell me if you're planning to join us at sprints at PyCon or EuroPython[5]. * Check out our open good first Warehouse issues[6] (we usually have 10+ open) and get started[7]. If you follow https://status.python.org/ you saw we did some load testing last week and learned from it! We redirected some traffic, for a few periods, for `pip install`, from the old server to Warehouse, and learned from it. For instance, people running Ubuntu 14.04 LTS (long term service release)[8] are usually using a pretty old version of pip, and people on some versions of the Mac OS[9] have older versions of Python and old versions of security-related libraries that don't support the version of TLS that we want them to use. Ernest, Donald and Dustin did a bunch of work addressing this, including Donald putting out pip 9.0.2[10]. (A thing to understand about Ernest's continuing work on PyPI and distribution infrastructure is that it's in a lot of places. It's cabotage[11] & a test cabotage app[12], configuration with salt[13], conveyor[14], pip[15] & get-pip[16], and he filed a bug in Kubernetes[17] which I personally find particularly impressive. And it's in user-facing communication in IRC and GitHub comments and on our statuspage and Twitter, plus a lot of internal discussion with infrastructure colleagues. I have a harder time gathering links for Ernest's work for these emails than for my other teammates; regrets.) As usual, a summary of the past week's work is in our meeting notes[18]. We have new features like letting PyPI administrators add new trove classifiers easily[19], infrastructure improvements like this complexity reduction[20], ton of polish and bug fixing around layout, description content types (Markdown!), a FAQ restructuring[21], a more useful collaboration page[22], etc. And we reviewed and merged a lot of volunteers' pull requests! Thanks to our prolific volunteers: * pgadige making sure an error message reflects whether you're on PyPI or Test PyPI[23] * waseem18 providing an error message for the password reset[24] * cryvate fixing form requirements for password reset[25] * waseem18 fixing disabled button CSS[26] * yeraydiazdiaz fixing modal window behavior[27], then refixing[28] * berkerpeksag adding a "public profile" link to the user dropdown[29] * Mariatta sending notification email when a project collaborator's added[30] * berkerpeksag hiding the "view project" button for no-release-yet projects in maintainers' project lists[31] * alexwlchan renaming a CSS class for consistency[32] * jMuzsik improving documentation of owners' and maintainers' privileges[33] * yeraydiazdiaz adding JavaScript validation to show the user if "new password" and "confirm new password" don't match[34] * alexwlchan documenting all the modifiers in our SASS directory[35] * alanbato and yeraydiazdiaz adding a check to stop someone from uploading a file whose blake2 hash matches an already- uploaded file[36] * cryvate improving sorting of package versions in our /simple/ API[37] * jMuzsik improving how PyPI links look on Twitter, adding an image to our Twitter cards[38] * 9999years updating the Python Packaging User Guide[39] and sample project[40] for Markdown/PEP 566 And thanks to our many bug reporters, especially those who helped us learn from our load tests. Also, check out discussion on API key support/macaroons[41], supporting GitHub-flavored Markdown as Description-Content-Type[42], and project rating/ranking/stars[43]. And finally, we are ever closer to accepting PEP 541 (and planning followup tasks[44]) and are testing our PEP 566 compliance[45]. And I may start a PEP for a Python package index upload API specification[46]. More next week, as usual. *Thanks to Mozilla for their support[47] for the PyPI & Warehouse work[48]!* -- Sumana Harihareswara Warehouse project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://github.com/pypa/warehouse/issues/3174 2. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+sort%3Aupda… 3. https://github.com/pypa/warehouse/issues/1297 4. http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html 5. https://wiki.python.org/psf/PackagingSprints 6. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… 7. https://warehouse.readthedocs.io/development/getting-started/ 8. https://github.com/pypa/warehouse/issues/3280 9. https://github.com/pypa/warehouse/issues/3293 10. https://pypi.org/project/pip/9.0.2/ 11. https://github.com/cabotage/cabotage-app 12. https://github.com/cabotage/test-app/commits?author=ewdurbin&since=2018-03-… 13. https://github.com/python/pypi-salt/commit/1a20cd53ffce0fd3d018d989199d30e1… 14. https://github.com/pypa/conveyor/commits?author=ewdurbin&since=2018-03-13T0… 15. https://github.com/pypa/pip/pull/5076 16. https://github.com/pypa/get-pip/commits?author=ewdurbin&since=2018-03-01T05… 17. https://github.com/kubernetes/kubectl/issues/335 18. https://wiki.python.org/psf/PackagingWG/2018-03-19-Warehouse 19. https://github.com/pypa/warehouse/issues/2649 20. https://github.com/pypa/warehouse/pull/3289 21. https://github.com/pypa/warehouse/pull/3190 22. https://github.com/pypa/warehouse/pull/3047 23. https://github.com/pypa/warehouse/pull/3314 24. https://github.com/pypa/warehouse/pull/3220 25. https://github.com/pypa/warehouse/pull/3230 26. https://github.com/pypa/warehouse/pull/3254 27. https://github.com/pypa/warehouse/pull/3251 28. https://github.com/pypa/warehouse/pull/3291 29. https://github.com/pypa/warehouse/pull/3255 30. https://github.com/pypa/warehouse/pull/3155 31. https://github.com/pypa/warehouse/pull/3257 32. https://github.com/pypa/warehouse/pull/3261 33. https://github.com/pypa/warehouse/pull/3313 34. https://github.com/pypa/warehouse/pull/3219 35. https://github.com/pypa/warehouse/pull/3262 36. https://github.com/pypa/warehouse/pull/3310 37. https://github.com/pypa/warehouse/pull/2574 38. https://github.com/pypa/warehouse/pull/3304 39. https://github.com/pypa/python-packaging-user-guide/pull/457 40. https://github.com/pypa/sampleproject/pull/66 41. https://github.com/pypa/warehouse/issues/994 42. https://github.com/pypa/packaging-problems/issues/126 43. https://github.com/pypa/warehouse/issues/991#issuecomment-374665356 44. https://github.com/pypa/warehouse/issues/1506#issuecomment-374626455 45. https://github.com/pypa/warehouse/issues/3299 46. https://github.com/pypa/packaging-problems/issues/128 47. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 48. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html

1 0

prepping PEP 566 support in Twine for tomorrow
by Sumana Harihareswara 20 Mar '18

20 Mar '18

Per https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi , currently, Markdown support for a package long_description depends on a pre-release of Twine. I released Twine 1.11.0rc1 a few days ago. Today I'm fixing more bugs and putting out another release candidate, and then tomorrow I plan to release 1.11.0. Code review and testing is welcome, as is camaraderie in #pypa-dev on Freenode. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 1

timeline for 3.7 distribution testing
by Robin Becker 19 Mar '18

19 Mar '18

Anyone here know approximately when the manylinux docker or appveyor ci will be upgraded to supprt Python 3.7? -- Robin Becker

2 1

Cannot install pyopencl
by Tim Graham 19 Mar '18

19 Mar '18

Hello I wonder if you can help.... I am trying to install btcrecover which is no simple task. When I try and install pyopencl it keeps saying that it does not recognise it as a file name... Do you have any ideas? Thanks for your help. Best regards Tim Graham T: +61 8 9467 7664 M: +61 457 414 935 W: www.conveyorcn.com E: tgraham(a)minsep.com.au<mailto:tgraham@minsep.com.au> [cid:image003.jpg@01D3BD7C.2F1A3F40]

3 2

Fwd: Re: Pip 10 is nearing release
by Tim Golden 16 Mar '18

16 Mar '18

On 16/03/2018 09:08, Paul Moore wrote: > After much work, we're pleased to announce that pip 10 is finally > nearing release. We're planning on releasing a beta version over the > weekend of 31 Mar/1 Apr, with the final release to follow about a > fortnight after (assuming no major issues are found). > > Hopefully, as many people as possible will be able to test the new > version of pip during the beta period (or before - it's possible to > install pip from github if you don't want to wait!) Paul, am I right in thinking that this: https://github.com/pypa/pip is the place to look if we want an advanced preview? TJG

2 1

Pip 10 is nearing release
by Paul Moore 16 Mar '18

16 Mar '18

After much work, we're pleased to announce that pip 10 is finally nearing release. We're planning on releasing a beta version over the weekend of 31 Mar/1 Apr, with the final release to follow about a fortnight after (assuming no major issues are found). Hopefully, as many people as possible will be able to test the new version of pip during the beta period (or before - it's possible to install pip from github if you don't want to wait!) so that we can ensure that pip 10 is as robust as we can manage when it gets released. There are a number of new features in pip 10, as well as many months of bug fixes and minor improvements. Highlights include: 1. Support for PEP 518 (declaring build dependencies in `pyproject.toml`) 2. Fixes for Unicode handling in build tools (notably on Windows) 3. The previously announced reorganisation of pip's internals 4. A new `pip config` command 5. Default upgrade strategy becomes "only-if-needed" Thanks to everyone who has contributed to this version of pip - it's been a long time coming, and we apologise for the delay. The plan is to ensure that future releases aren't delayed as long as this one was! Paul

1 0

new stuff overview, beta next week, user tests, & other Warehouse updates
by Sumana Harihareswara 14 Mar '18

14 Mar '18

The new PyPI is still working towards our big public beta[1]. We have 7 open issues till we'll declare beta and make an outreach push (probably late this week or early next week), and then 19 more open issues till we can redirect/launch PyPI[2] probably in April (overview[3]). I've started preparing a draft overview of what's new in PyPI/packaging/distribution[4] to publicize along with the beta; it says "not to be publicized" but I'll let you in on the secret early. Maybe something in it is new to you as well! As usual, we had a Warehouse core developers' meeting on Monday[5]. The last week has seen a lot of polish and bugfixing and documentation for Warehouse. For instance, project deletion is cleaner[6], we more consistently indicate dangerous actions on a page[7], and there's now a migration guide for third-party services[8] which we told several projects about[9]. We've done some infrastructural work, like Datadog instrumentation[10], "Conveyor" (a shim for URL redirects)[11], and Cabotage improvements[12]. Here's an animated GIF demo of release phase commands (scale up, scale down).[13] And we improved other codebases as well, to fix Travis docs[14], get our HTTPS proxy service to deal with big embedded images[15], and deal better with parsing invalid URLs in READMEs[16]. Thanks to volunteers who got pull requests merged this week: * waseem[17]: we now send an email to primary email whenever primary email is changed * mds325[18]: clear input when the user closes the modal * dirn[19]: create a shortlink and redirect all requests for /p/<project>/ to /project/<project>/ * cryvate[20]: clarify project counter for searches with tons of results * Mariatta[21]: fix an email-sending issue And thanks to our many bug reporters, such as Andrew Nesbitt who noticed an RSS feed discrepancy[22]. Check out the current discussion[23] of API keys, a bearer token authentication scheme, and Macaroons in future PyPI. Want to help? * Talk with Nicole about being a subject or interviewer for user tests![24] She's been focusing on user tests and it's paid off, with a lot of bugs found and designs validated. * Got a good workaround for our CAPTCHA being blocked in China[25]? * Consider joining us at sprints[26] in the next few months. * We have 24 good first issues open[27], and a "getting started"[28] guide, and quick turnaround on code review. *Thanks to Mozilla Open Source Support[29] for their funding[30] for the PyPI & Warehouse work.* -- Sumana Harihareswara Warehouse project manager Changeset Consulting sh(a)changeset.nyc P.S. Usually I compose these weekly report emails in plain text; here I'm doing it in HTML with a plaintext fallback. Let me know if it's better, awful, etc. Also nearly no one *replies* to these emails so I'd also welcome your "hey this is useful to me!" offlist reply. Links: 1. https://github.com/pypa/warehouse/milestone/10 2. https://github.com/pypa/warehouse/milestone/1 3. https://github.com/pypa/warehouse/projects/1 4. https://wiki.python.org/psf/PackagingWG/PyPIBetaAnnouncement 5. https://wiki.python.org/psf/PackagingWG/2018-03-12-Warehouse 6. https://github.com/pypa/warehouse/pull/3212 7. https://github.com/pypa/warehouse/pull/3166 8. https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… 9. https://github.com/pypa/warehouse/issues/2935 10. https://github.com/pypa/warehouse/pull/3076 11. https://github.com/pypa/conveyor/commits?author=ewdurbin&since=2018-03-06T0… 12. https://github.com/cabotage/cabotage-app/commits?author=ewdurbin&since=2018… 13. https://ernest.ly/imgs/cabotage-release-scale-up-scale-down.gif 14. https://github.com/travis-ci/docs-travis-ci-com/pull/1726 15. https://github.com/pypa/warehouse-camo/pull/1 16. https://github.com/pypa/readme_renderer/pull/65 17. https://github.com/pypa/warehouse/pull/3158 18. https://github.com/pypa/warehouse/pull/3160 19. https://github.com/pypa/warehouse/pull/3165 20. https://github.com/pypa/warehouse/pull/3193 21. https://github.com/pypa/warehouse/pull/3214 22. https://github.com/pypa/warehouse/issues/3238 23. https://github.com/pypa/warehouse/issues/994 24. http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html 25. https://github.com/pypa/warehouse/issues/3174 26. https://wiki.python.org/psf/PackagingSprints 27. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… 28. https://warehouse.readthedocs.io/development/getting-started/ 29. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 30. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html

2 2

It should be possible to set a custom pypirc location
by Joel Croteau 11 Mar '18

11 Mar '18

Hardcoding ~/.pypirc as the only possible location for repo configs makes certain use cases, like deploying to a custom repo from Jenkins, difficult. It should be possible to set a custom pypirc location, either through an environment variable, or an argument to setup.

2 1

Twine 1.10.0rc1 on Test PyPI
by Sumana Harihareswara 07 Mar '18

07 Mar '18

(So it turns out I've taken on a volunteer gig, which is that I'm now one of the Twine maintainers. I may be wrong about how to do this - please feel free to comment on https://github.com/pypa/twine/pull/314 which is where I'm pulling together a new release checklist for myself.) https://test.pypi.org/manage/project/twine/release/1.10.0rc1/ This is a release candidate for Twine 1.10.0 which I'm planning to release early next week. This release improves project registration usage text (in some cases removing it where inapplicable), and updates `--repository[-url]` usage text, prints progress to `stdout` instead of `stderr`, improves the progressbar, and reorganizes and improves user and developer documentation. Please see the changelog https://twine.readthedocs.io/en/latest/changelog.html for detailed notes under "Next feature release". I believe this is how you test it out: pip install --upgrade --pre --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple twine Please check existing open issues at https://github.com/pypa/twine/issues and open new ones if you have problems. Thanks! -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

3 6