Summary of PyPI overhaul in new LWN article
Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots! If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/ This summary should help occasional Python programmers -- and frequent Pythonists who don't follow packaging/distro discussions closely -- understand why a new application is necessary, what's new, what features are going away, and what to expect in the near future. I also hope it catches the attention of downstreams that ought to migrate. -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc
On Wed, Apr 11, 2018 at 10:30 PM, Sumana Harihareswara <sh@changeset.nyc> wrote:
Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/
Thanks for the summary, and all your hard work, Sumana :) Happy to see this bit about TUF in future horizons: Warehouse's signature handling demonstrates a shift in Python's thinking
regarding key management and package signatures. Ideally, package users, software distributors, and package distribution tools would regularly use signatures to verify Python package integrity. For the most part, however, they don't, and there are major infrastructural barriers to them effectively doing so. Therefore, GPG/PGP signatures for packages are no longer visible in PyPI's web interface. Project maintainers can still attach signatures to their release uploads, and those signatures still appear in the Simple Project API as described in PEP 503. Stufft has made no secret of his opinion that "package signing is not the Holy Grail"; current discussion among packaging-tools developers leans toward removing signing features from another part of the Python packaging ecology (the wheel library) and working toward implementing The Update Framework instead. Relatedly, Warehouse, unlike legacy PyPI, does not provide an interface for users to manage GPG or SSH public keys.
We would love to help with this efforts any way we can. -- curl https://keybase.io/trishankdatadog/pgp_keys.asc | gpg --import
From "TUF, Warehouse, Pip, PyPA, ld-signatures, ed25519"
https://mail.python.org/pipermail/distutils-sig/2018-March/032081.html :
Are there pypa/warehouse github issues for implementing the TUF trust root support in warehouse; and client support in pip (or a module that pip and other tools can use)?
Read and review these PEPs: "PEP 458 -- Surviving a Compromise of PyPI" https://www.python.org/dev/peps/pep-0458/" "PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model" https://www.python.org/dev/peps/pep-0480/ On Thursday, April 12, 2018, Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
On Wed, Apr 11, 2018 at 10:30 PM, Sumana Harihareswara <sh@changeset.nyc> wrote:
Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/
Thanks for the summary, and all your hard work, Sumana :)
Happy to see this bit about TUF in future horizons:
Warehouse's signature handling demonstrates a shift in Python's thinking
regarding key management and package signatures. Ideally, package users, software distributors, and package distribution tools would regularly use signatures to verify Python package integrity. For the most part, however, they don't, and there are major infrastructural barriers to them effectively doing so. Therefore, GPG/PGP signatures for packages are no longer visible in PyPI's web interface. Project maintainers can still attach signatures to their release uploads, and those signatures still appear in the Simple Project API as described in PEP 503. Stufft has made no secret of his opinion that "package signing is not the Holy Grail"; current discussion among packaging-tools developers leans toward removing signing features from another part of the Python packaging ecology (the wheel library) and working toward implementing The Update Framework instead. Relatedly, Warehouse, unlike legacy PyPI, does not provide an interface for users to manage GPG or SSH public keys.
We would love to help with this efforts any way we can.
-- curl https://keybase.io/trishankdatadog/pgp_keys.asc | gpg --import
Hi Wes, On Thu, Apr 12, 2018 at 1:22 PM, Wes Turner <wes.turner@gmail.com> wrote:
Are there pypa/warehouse github issues for implementing the TUF trust root support in warehouse; and client support in pip (or a module that pip and other tools can use)?
For client support in pip, we are discussing this patch with Donald Stufft: https://github.com/pypa/pip/compare/release/9.0.3...trishankatdatadog:trisha... Happy to discuss more with anyone interested! -- curl https://keybase.io/trishankdatadog/pgp_keys.asc | gpg --import
On Thu, Apr 12, 2018 at 5:29 PM, Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
Hi Wes,
On Thu, Apr 12, 2018 at 1:22 PM, Wes Turner <wes.turner@gmail.com> wrote:
Are there pypa/warehouse github issues for implementing the TUF trust root support in warehouse; and client support in pip (or a module that pip and other tools can use)?
For client support in pip, we are discussing this patch with Donald Stufft:
https://github.com/pypa/pip/compare/release/9.0.3...trishank atdatadog:trishankatdatadog/9.0.3.tuf-in-toto
Happy to discuss more with anyone interested!
Sorry for the noise, but our repo has now moved here: https://github.com/pypa/pip/compare/9.0.3...DataDog: trishankatdatadog/9.0.3.tuf-in-toto -- https://keybase.io/trishankdatadog
Hey, what's the latest on this? - Python PEP458: - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf On Wed, Apr 18, 2018 at 5:43 PM Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0....
On Thu, Jul 26, 2018 at 1:37 PM Wes Turner <wes.turner@gmail.com> wrote:
Hey, what's the latest on this?
- Python PEP458: - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf
I'm not aware of what is happening on the Warehouse end, but I'm happy to report we have a version of pip + TUF Datadog is using in production: https://github.com/DataDog/pip/tree/trishankatdatadog/10.0.1.tuf I'm fairly confident we can use this same version on Warehouse + TUF :) -- https://keybase.io/trishankdatadog
On Thu, Jul 26, 2018 at 1:58 PM Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
I'm not aware of what is happening on the Warehouse end, but I'm happy to report we have a version of pip + TUF Datadog is using in production:
https://github.com/DataDog/pip/tree/trishankatdatadog/10.0.1.tuf
I'm fairly confident we can use this same version on Warehouse + TUF :)
When I get some downtime, I'll start writing some code for Warehouse + TUF... -- https://keybase.io/trishankdatadog
Hi Wes, Many members of the PyPA have been trying to obtain funding to try and work on this and other security features I believe. I don't believe, but could be wrong, anything has occurred in implementation here. But Ernest, Donald and Dustin would know best. Are you interested in helping? Cooper
On Jul 26, 2018, at 10:33 AM, Wes Turner <wes.turner@gmail.com> wrote:
Hey, what's the latest on this?
- Python PEP458: - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf <https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf>
On Wed, Apr 18, 2018 at 5:43 PM Trishank Kuppusamy <trishank.kuppusamy@datadoghq.com <mailto:trishank.kuppusamy@datadoghq.com>> wrote: https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0.... <https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0....>
-- https://keybase.io/trishankdatadog <https://keybase.io/trishankdatadog> -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2...
participants (4)
-
Cooper Ry Lees -
Sumana Harihareswara -
Trishank Kuppusamy -
Wes Turner