[Distutils] PyPI/Warehouse: infrastructure hardening & the CAPTCHA conundrum

Sumana Harihareswara sh at changeset.nyc
Tue Mar 20 22:12:48 EDT 2018


So we aren't quite at beta yet, but we'll be shouting about pypi.org
*really soon*. We have nearly all the Warehouse improvements we need for
beta, and nearly all the infrastructure improvements we believe we'll
need for the switchover.

I'll tell you how you can help, then talk about the current state
of things.
 * The big blocker keeping us from beta: China & CAPTCHAs. Help
   advise us.[1]
 * Comment on a "needs discussion" issue[2].
 * Help us with large-scope JavaScript issues[3], like our frontend
   testing approach.
 * Please talk with Nicole about being a subject or interviewer for
   user tests[4].
 * Tell me if you're planning to join us at sprints at PyCon or
   EuroPython[5].
 * Check out our open good first Warehouse issues[6] (we usually have
   10+ open) and get started[7].

If you follow https://status.python.org/  you saw we did some load
testing last week and learned from it! We redirected some traffic, for a
few periods, for `pip install`, from the old server to Warehouse, and
learned from it. For instance, people running Ubuntu 14.04 LTS (long
term service release)[8] are usually using a pretty old version of pip,
and people on some versions of the Mac OS[9] have older versions of
Python and old versions of security-related libraries that don't support
the version of TLS that we want them to use. Ernest, Donald and Dustin
did a bunch of work addressing this, including Donald putting out pip
9.0.2[10].

(A thing to understand about Ernest's continuing work on PyPI and
distribution infrastructure is that it's in a lot of places. It's
cabotage[11] & a test cabotage app[12], configuration with salt[13],
conveyor[14], pip[15] & get-pip[16], and he filed a bug in
Kubernetes[17] which I personally find particularly impressive. And it's
in user-facing communication in IRC and GitHub comments and on our
statuspage and Twitter, plus a lot of internal discussion with
infrastructure colleagues. I have a harder time gathering links for
Ernest's work for these emails than for my other teammates; regrets.)

As usual, a summary of the past week's work is in our meeting notes[18].
We have new features like letting PyPI administrators add new trove
classifiers easily[19], infrastructure improvements like this complexity
reduction[20],  ton of polish and bug fixing around layout, description
content types (Markdown!), a FAQ restructuring[21], a more useful
collaboration page[22], etc. And we reviewed and merged a lot of
volunteers' pull requests!

Thanks to our prolific volunteers:
 *  pgadige making sure an error message reflects whether you're on PyPI
    or Test PyPI[23] *  waseem18 providing an error message for the password reset[24]
 *  cryvate fixing form requirements for password reset[25]
 *  waseem18 fixing disabled button CSS[26]
 *  yeraydiazdiaz fixing modal window behavior[27], then refixing[28]
 *  berkerpeksag adding a "public profile" link to the user dropdown[29] *  Mariatta sending notification email when a project
    collaborator's added[30] *  berkerpeksag hiding the "view project" button for no-release-yet
    projects in maintainers' project lists[31] *  alexwlchan renaming a CSS class for consistency[32]
 *  jMuzsik improving documentation of owners' and maintainers'
    privileges[33] *  yeraydiazdiaz adding JavaScript validation to show the user if "new
    password" and "confirm new password" don't match[34] *  alexwlchan documenting all the modifiers in our SASS directory[35]
 *  alanbato and yeraydiazdiaz adding a check to stop someone
    from uploading a file whose blake2 hash matches an already-
    uploaded file[36] *  cryvate improving sorting of package versions in our /simple/
    API[37] *  jMuzsik improving how PyPI links look on Twitter, adding an image to
    our Twitter cards[38]
 * 9999years updating the Python Packaging User Guide[39] and sample
   project[40] for Markdown/PEP 566
And thanks to our many bug reporters, especially those who helped us
learn from our load tests.
Also, check out discussion on API key support/macaroons[41],  supporting
GitHub-flavored Markdown as Description-Content-Type[42],  and project
rating/ranking/stars[43].
And finally, we are ever closer to accepting PEP 541 (and planning
followup tasks[44])  and are testing our PEP 566 compliance[45]. And I
may start a PEP for a Python package index upload API specification[46].
More next week, as usual.

*Thanks to Mozilla for their support[47] for the PyPI & Warehouse
work[48]!*
--
Sumana Harihareswara
Warehouse project manager
Changeset Consulting
sh at changeset.nyc

Links:

   1. https://github.com/pypa/warehouse/issues/3174
   2. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22needs+discussion%22
   3. https://github.com/pypa/warehouse/issues/1297
   4. http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html
   5. https://wiki.python.org/psf/PackagingSprints
   6. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
   7. https://warehouse.readthedocs.io/development/getting-started/
   8. https://github.com/pypa/warehouse/issues/3280
   9. https://github.com/pypa/warehouse/issues/3293
  10. https://pypi.org/project/pip/9.0.2/
  11. https://github.com/cabotage/cabotage-app
  12. https://github.com/cabotage/test-app/commits?author=ewdurbin&since=2018-03-01T05:00:00Z&until=2018-03-21T04:00:00Z
  13. https://github.com/python/pypi-salt/commit/1a20cd53ffce0fd3d018d989199d30e11d35ad83
  14. https://github.com/pypa/conveyor/commits?author=ewdurbin&since=2018-03-13T05:00:00Z&until=2018-03-21T04:00:00Z
  15. https://github.com/pypa/pip/pull/5076
  16. https://github.com/pypa/get-pip/commits?author=ewdurbin&since=2018-03-01T05:00:00Z&until=2018-03-21T04:00:00Z
  17. https://github.com/kubernetes/kubectl/issues/335
  18. https://wiki.python.org/psf/PackagingWG/2018-03-19-Warehouse
  19. https://github.com/pypa/warehouse/issues/2649
  20. https://github.com/pypa/warehouse/pull/3289
  21. https://github.com/pypa/warehouse/pull/3190
  22. https://github.com/pypa/warehouse/pull/3047
  23. https://github.com/pypa/warehouse/pull/3314
  24. https://github.com/pypa/warehouse/pull/3220
  25. https://github.com/pypa/warehouse/pull/3230
  26. https://github.com/pypa/warehouse/pull/3254
  27. https://github.com/pypa/warehouse/pull/3251
  28. https://github.com/pypa/warehouse/pull/3291
  29. https://github.com/pypa/warehouse/pull/3255
  30. https://github.com/pypa/warehouse/pull/3155
  31. https://github.com/pypa/warehouse/pull/3257
  32. https://github.com/pypa/warehouse/pull/3261
  33. https://github.com/pypa/warehouse/pull/3313
  34. https://github.com/pypa/warehouse/pull/3219
  35. https://github.com/pypa/warehouse/pull/3262
  36. https://github.com/pypa/warehouse/pull/3310
  37. https://github.com/pypa/warehouse/pull/2574
  38. https://github.com/pypa/warehouse/pull/3304
  39. https://github.com/pypa/python-packaging-user-guide/pull/457
  40. https://github.com/pypa/sampleproject/pull/66
  41. https://github.com/pypa/warehouse/issues/994
  42. https://github.com/pypa/packaging-problems/issues/126
  43. https://github.com/pypa/warehouse/issues/991#issuecomment-374665356
  44. https://github.com/pypa/warehouse/issues/1506#issuecomment-374626455
  45. https://github.com/pypa/warehouse/issues/3299
  46. https://github.com/pypa/packaging-problems/issues/128
  47. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosystem/
  48. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20180320/0f75f4d9/attachment.html>


More information about the Distutils-SIG mailing list