Mailman 3 April 2018 - Distutils-SIG

Warehouse/PyPI update: launch, project wrapup approaching
by Sumana Harihareswara April 18, 2018

April 18, 2018

On Monday, we launched Warehouse and redirected[1] browser and API traffic so Warehouse is now the codebase, and http://pypi.org/ is the site, serving nearly everyone who requests files from PyPI. The old codebase is now up at http://legacy.pypi.org/ temporarily. We had a few hiccups (incident report)[2] and are now fixing up some search, indexing, caching, encoding, API compatibility, and UI issues. We're monitoring incoming bug reports and, so far, don't see anything new that absolutely needs fixing before we shut down the legacy site on Monday, April 30th. https://github.com/pypa/warehouse/projects/1 is the rollout board you can watch to see our progress, and our weekly meeting notes are up[3]. Please continue to report bugs -- if you know they're in PyPI, file them against Warehouse[4], and if you're not sure, file them in the "packaging problems" repository[5]. (The "nearly everyone" in that first sentence above is because of this User-Agent exclusion[6], and because I'm sure a few users are specifying legacy.pypi.org in their requests right now while working on forwards compatibility.) We've made a number of user-visible improvements in the past couple weeks. For instance, we added a "switch to desktop version" link in the mobile view[7], fixed dropdowns for accessibility[8], added user help[9] for folks affected by the TLS 1.0/1.1 deprecation (mea culpa[10], we should have done that sooner), and created a page thanking our sponsors[11]. And we've made many backend improvements to performance, API compatibility with legacy, sorting and indexing, instrumentation for metrics, and security -- and Donald implemented[12] email sending via SES. Thanks[13] to Noah Kantrowitz for reporting a privacy concern regarding Gravatar URLs and leaking users' email addresses (fixed[14]). It is not feasible for me to summarize all the work that volunteers put in, as testers, coders, code reviewers, writers, and user support helpers within the last two weeks. We have an embarrassment of riches here. Since April 3rd (two weeks ago) we've merged 98 PRs to Warehouse[15]; thanks to ymyzk, reaperhulk, glasnt, alex, RazerM, bskinn, saxenanurag, hugovk, waseem18, cheungnj, contrepoint, yeraydiazdiaz, jonparrott, jMuzsik, and aalmazan for those. And I'd also like to thank the many people who, on their own, provided and continue to provide help to affected users on IRC, Twitter, StackOverflow, and elsewhere. We don't have any virtual office hours coming up, but we have some other events planned. The Talk Python to Me podcast[16] just interviewed Dustin, Nicole, and Ernest for an upcoming episode. Dustin will be speaking on PyPI and packaging in general at PyCon NA in May[17] and at SciPy in July[18], and we're sprinting at PyCon and EuroPython[19] -- join us? And if you're in New York City, you can join Laura and me for a Warehouse sprint night[20] on Thursday, April 26th. And the project's starting to wrap up. Our MOSS funding (thanks to Mozilla[21] for their support[22] for the PyPI & Warehousework!) is nearly finished; after we shut down the legacy site on April 30th, the general pace of Warehouse development may slow down. (Warehouse has far more volunteer contributors than it did when we started MOSS-funded work in early December, but maintainer time available will diminish.) So we're seeking further funding, to speed up security and accessibility work, (potentially) localization, group/organization support[23], better notifications, better staging/testing workflow for project maintainers, GitHub signon, and more (see the "Post-Legacy shutdown" milestone[24] and "cool but not urgent" milestone[25]). We have submitted a few more grant proposals and are waiting to hear back. And donations to the Python Software Foundation’s Packaging Working Group, which works to sustain PyPI, pip, setuptools, and all other Python Packaging Ecosystem efforts, can now be made on a recurring basis! Please check out https://donate.pypi.org/ and consider pitching in or spreading the word. I'm working on writing up and sharing a more structured list explaining what we could do at various levels of funding. Thanks as always. Please keep the kind words, bug reports, and -- I hope -- funding coming! :)-- Sumana Harihareswara Warehouse/PyPI project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://github.com/python/pypi-infra/commits/master 2. https://status.python.org/incidents/mgjw1g5yjy5j 3. https://wiki.python.org/psf/PackagingWG/2018-04-17-Warehouse 4. https://github.com/pypa/warehouse/issues 5. https://github.com/pypa/packaging-problems/issues/ 6. https://github.com/pypa/warehouse/issues/3275 7. https://github.com/pypa/warehouse/pull/3602 8. https://github.com/pypa/warehouse/pull/3287 9. https://pypi.org/help/#tls-deprecation 10. https://groups.google.com/d/msg/pypa-dev/Oz6SGA7gefo/UXCu7jM6AQAJ 11. https://pypi.org/sponsors/ 12. https://github.com/pypa/warehouse/pull/3580 13. https://twitter.com/EWDurbin/status/984562893864816640 14. https://github.com/pypa/warehouse/pull/3648 15. https://github.com/pypa/warehouse/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Amerge… 16. https://talkpython.fm/ 17. https://us.pycon.org/2018/schedule/presentation/148/ 18. https://scipy2018.scipy.org/ehome/index.php?eventid=299527&tabid=712461&cid… 19. https://wiki.python.org/psf/PackagingSprints 20. https://www.meetup.com/nycpython/events/249722192/ 21. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 22. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html 23. https://github.com/pypa/warehouse/issues/201 24. https://github.com/pypa/warehouse/milestone/12 25. https://github.com/pypa/warehouse/milestone/11

1 0

ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI
by Donald Stufft April 17, 2018

April 17, 2018

Hello All! As most people are aware, there has been an effort under way to rewrite PyPI in order to solve a lot of long standing problems. For those who aren't aware, that is currently available at https://pypi.org/ and it uses the same database that "Legacy" PyPI does, so the two are essentially just different views over the same data. For a awhile now, Python, setuptools, and twine have all defaulted to using this new code base for uploading artifacts to PyPI. Now that we've gotten some testing of that code base, the infrastructure team and myself feel comfortable directing everyone to using the new endpoint and we're planning on shutting down uploads to Legacy PyPI. If you're using the latest versions of Python, setuptools, or twine and you have not put an explicit URL in your ~/.pypirc file, then there's nothing you should need to do. If you are not, then you should ideally upgrade to the latest version of whatever tool you're using to upload (the preferred tool is twine) and edit your ~/.pypirc so that it removes any explicit mention of an URL. Thus it should look something like: [distutils] index-servers = pypi [pypi] username:yourusername password:yourpassword If for some reason you're not able to update to the newest version of your upload tool, then you can configure it to upload to the new code base by switching the URL to use https://upload.pypi.org/legacy/ instead of https://pypi.python.org/pypi. Thus your ~/.pypirc would then become: [distutils] index-servers = pypi [pypi] repository:https://upload.pypi.org/legacy/ username:yourusername password:yourpassword For those of you who are using TestPyPI, that will also be affected, and the required URL for the new upload endpoint for TestPyPI is https://test.pypi.org/legacy/. We plan to disable uploads to legacy PyPI on July 3rd, 2017 so any configuration change will need to be made before that date. In addition, we plan to have a "brownout" on June 29th where we will shut the legacy endpoint down for that day. For TestPyPI the change to disable uploads to legacy will be made in the next couple of days, likely this weekend. As part of the error message that users will get when attempting to upload to legacy PyPI, we will include a link to a page that details how to ensure that they are using the new code base and not the legacy code base. Thanks everyone! — Donald Stufft

9 29

ANN: distlib 0.2.7 released on PyPI
by Vinay Sajip April 16, 2018

April 16, 2018

I've just released version 0.2.7 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Addressed #102: InstalledDistributions now have a modules attribute which is a list of top-level modules as read from top_level.txt, if that is in the distribution info. * Fixed #103: Now https downloads are preferred to those over http. Thanks to Saulius Žemaitaitis for the patch. * Fixed #104: Updated launcher binaries to properly handle interpreter paths with spaces. Thanks to Atsushi Odagiri for the diagnosis and fix. * Fixed #105: cache_from_source is now imported from importlib.util where available. * Added support for PEP 566 / Metadata 2.1. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.2.7/ [2] https://goo.gl/M3kQzR [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

New PyPI launched, legacy PyPI shutting down April 30
by Laura Hampton April 16, 2018

April 16, 2018

New PyPI launched, legacy PyPI shutting down April 30[1] Starting today, the canonical Python Package Index is at https://pypi.org and uses the new Warehouse codebase. We announced the https://pypi.org beta on March 26 and your feedback and test usage have helped us get it production-ready. Monday April 16 (2018-04-16): We launched the new PyPI, redirecting browser traffic and API calls (including "pip install") from pypi.python.org to the new site. The old codebase is still available at https://legacy.pypi.org for now. Monday April 30 (2018-04-30): We plan to shut down legacy PyPI https://legacy.pypi.org . The address pypi.python.org will continue to redirect to Warehouse. For more details, see our roadmap: https://wiki.python.org/psf/WarehouseRoadmap If your site/service links to or uses pypi.python.org, you should start using pypi.org instead: https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… Thank you. [1] https://blog.python.org/2018/04/new-pypi-launched-legacy-pypi-shutting.html Laura Hampton laura at laura-hampton dot com

3 2

Package metadata: which fields are optional
by Thomas Kluyver April 16, 2018

April 16, 2018

The core metadata specification (https://packaging.python.org/specifications/core-metadata/ ) notes whether each field is optional. However there are some discrepancies with my understanding: - Download-URL is not marked as optional, but in practice it's obsolete (since PEP 470) and not very helpful (there may be different places to download the same package). Flit has never set this, and I have had no bug reports about it, so in practice it definitely is optional. - Requires-Python is not marked as optional, though I'm pretty sure it also is in practice. - Only one of the multiple use fields is explicitly marked as optional, but my understanding is that 'multiple use' includes using them zero times, so they are all optional. I propose that we remove 'optional' from all the headings, and note at the top of the specification that the required fields are: - Metadata-Version - Name - Version - Summary (And we should check if Summary is really required) Thomas

3 4

Announcement: Pip 10 is coming, and will move all internal APIs
by Paul Moore April 15, 2018

April 15, 2018

We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications. Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-) So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released. And many thanks to anyone else who helps by testing out the new version, as well :-) Thanks, Paul

12 33

Pip 10.0 has been released
by Paul Moore April 15, 2018

April 15, 2018

On behalf of the PyPA, I am pleased to announce that pip 10.0 has just been released. This release has been the culmination of many months of work by the community. To install pip 10.0, you can run python -m pip install --upgrade pip or use get-pip, as described in https://pip.pypa.io/en/latest/installing. If you are using a version of pip supplied by your distribution vendor, vendor-supplied upgrades will be available in due course (or you can use pip 10 in a virtual environment). (One minor issue with using get-pip on Windows - when you download get-pip.py, rename it to something that doesn't include "pip" in the name, such as "gp.py", as the standard name triggers a check in pip that aborts the run - this is being tracked in https://github.com/pypa/pip/issues/5219). Highlights of the new release: * Python 2.6 is no longer supported - if you need pip on Python 2.6, you should stay on pip 9, which is the last version to support Python 2.6. * Support for PEP 518, which allows projects to specify what packages they require in order to build from source. (PEP 518 support is currently limited, with full support coming in future versions - see the documentation for details). * Significant improvements in Unicode handling for non-ASCII locales on Windows. * A new "pip config" command. * The default upgrade strategy has become "only-if-needed" * Many bug fixes and minor improvements. In addition, the previously announced reorganisation of pip's internals has now taken place. Unless you are the author of code that imports the pip module (or a user of such code), this change will not affect you. If you are affected, please report the issue to the author of the offending code (refer them to https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html for the details of the announcement). Thanks to everyone who put so much effort into the new release. Many of the contributions came from community members, whether in the form of code, participation in design discussions, or bug reports. The pip development team is extremely grateful to everyone in the community for their contributions. Thanks, Paul

4 3

PEP 571: Any further manylinux2010 concerns or questions?
by Nick Coghlan April 14, 2018

April 14, 2018

PEP link: https://www.python.org/dev/peps/pep-0571/ Thomas Kluyver has amended Mark Williams's PEP 571 to address the concerns & questions raised in the previous thread: * manylinux2 -> manylinux2010: https://github.com/python/peps/commit/70cbfda06534aedd6372f489090fdc8e1062d… * using glibc 2.12 as a compatibility marker: https://github.com/python/peps/commit/d43b984e021eddc11bdbc36863c5c285b473f… (We also dropped a potentially misleading aside that could be taken as implying inaccurate limitations on Anaconda platform compatibility) As I expect quite a few folks will be busy for the Easter weekend, I'm planning to accept the PEP a week from next Monday (i.e on April 9th) if no new concerns or objections are raised between now and then :) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 10

please mark good first issues in your projects
by Sumana Harihareswara April 13, 2018

April 13, 2018

Warehouse is attracting several newer contributors including people new to open source, which is great. As Warehouse matures, we have fewer and fewer easy small bugs *in the Python side* left. (So, we have more work for new frontend contributors, and less for Pythonists.) I'd love to refer these folks to other parts of the Python packaging and distribution ecosystem so we can improve the whole toolchain. Right now there are 29 open issues in PyPA projects on GitHub marked "good first issue", 11 in Warehouse and most of the rest in pip: https://github.com/issues?utf8=%E2%9C%93&q=user%3Apypa+is%3Aopen+label%3A%2… I'm totally fine with giving new volunteers teensy tiny doc fix tasks, "manually test this functionality" tasks, and "check whether this bug is still reproducible" tasks, in case you want to write up some of those. Here's a template we use to make good first issues in Warehouse, in case you want to emulate it: https://github.com/pypa/warehouse/issues/new?template=good-first-issue.md **Good First Issue**: This issue is good for first time contributors. If you've already contributed to Warehouse, please work on [another issue without this label](https://github.com/pypa/warehouse/issues?utf8=%E2%9C%93&q=is%3Aissue… instead. If there is not a corresponding pull request for this issue, it is up for grabs. For directions for getting set up, see our [Getting Started Guide](https://warehouse.pypa.io/development/getting-started/). If you are working on this issue and have questions, please feel free to ask them here, [`#pypa-dev` on Freenode](https://webchat.freenode.net/?channels=%23pypa-dev), or the [pypa-dev mailing list](https://groups.google.com/forum/#!forum/pypa-dev). If your project isn't under https://github.com/pypa , but you want to publicize your good first issues, reply to this thread? Thanks. -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

2 2

Pip upgrade
by Niharika Jakhar April 13, 2018

April 13, 2018

Hi I am having troubles upgrading my pip. terminal says: You are using pip version 9.0.1, however version 9.0.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command. I used the same command as mentioned but it keeps on repeating the same lines. python version: Python 2.7.12 pip version: pip 9.0.1 Kindly help me in this regard. Thanking you in advance Best Regards NIHARIKA

3 4