Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/
This summary should help occasional Python programmers -- and frequent Pythonists who don't follow packaging/distro discussions closely -- understand why a new application is necessary, what's new, what features are going away, and what to expect in the near future. I also hope it catches the attention of downstreams that ought to migrate.
On Wed, Apr 11, 2018 at 10:30 PM, Sumana Harihareswara sh@changeset.nyc wrote:
Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/
Thanks for the summary, and all your hard work, Sumana :)
Happy to see this bit about TUF in future horizons:
Warehouse's signature handling demonstrates a shift in Python's thinking
regarding key management and package signatures. Ideally, package users, software distributors, and package distribution tools would regularly use signatures to verify Python package integrity. For the most part, however, they don't, and there are major infrastructural barriers to them effectively doing so. Therefore, GPG/PGP signatures for packages are no longer visible in PyPI's web interface. Project maintainers can still attach signatures to their release uploads, and those signatures still appear in the Simple Project API as described in PEP 503. Stufft has made no secret of his opinion that "package signing is not the Holy Grail"; current discussion among packaging-tools developers leans toward removing signing features from another part of the Python packaging ecology (the wheel library) and working toward implementing The Update Framework instead. Relatedly, Warehouse, unlike legacy PyPI, does not provide an interface for users to manage GPG or SSH public keys.
We would love to help with this efforts any way we can.
From "TUF, Warehouse, Pip, PyPA, ld-signatures, ed25519"
https://mail.python.org/pipermail/distutils-sig/2018-March/032081.html :
Are there pypa/warehouse github issues for implementing the TUF trust
root support in warehouse; and client support in pip (or a module that pip and other tools can use)?
Read and review these PEPs:
"PEP 458 -- Surviving a Compromise of PyPI" https://www.python.org/dev/peps/pep-0458/"
"PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model" https://www.python.org/dev/peps/pep-0480/
On Thursday, April 12, 2018, Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
On Wed, Apr 11, 2018 at 10:30 PM, Sumana Harihareswara sh@changeset.nyc wrote:
Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/
Thanks for the summary, and all your hard work, Sumana :)
Happy to see this bit about TUF in future horizons:
Warehouse's signature handling demonstrates a shift in Python's thinking
regarding key management and package signatures. Ideally, package users, software distributors, and package distribution tools would regularly use signatures to verify Python package integrity. For the most part, however, they don't, and there are major infrastructural barriers to them effectively doing so. Therefore, GPG/PGP signatures for packages are no longer visible in PyPI's web interface. Project maintainers can still attach signatures to their release uploads, and those signatures still appear in the Simple Project API as described in PEP 503. Stufft has made no secret of his opinion that "package signing is not the Holy Grail"; current discussion among packaging-tools developers leans toward removing signing features from another part of the Python packaging ecology (the wheel library) and working toward implementing The Update Framework instead. Relatedly, Warehouse, unlike legacy PyPI, does not provide an interface for users to manage GPG or SSH public keys.
We would love to help with this efforts any way we can.
-- curl https://keybase.io/trishankdatadog/pgp_keys.asc | gpg --import
Hi Wes,
On Thu, Apr 12, 2018 at 1:22 PM, Wes Turner wes.turner@gmail.com wrote:
Are there pypa/warehouse github issues for implementing the TUF trust
root support in warehouse; and client support in pip (or a module that pip and other tools can use)?
For client support in pip, we are discussing this patch with Donald Stufft:
https://github.com/pypa/pip/compare/release/9.0.3...trishankatdatadog:trisha...
Happy to discuss more with anyone interested!
On Thu, Apr 12, 2018 at 5:29 PM, Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
Hi Wes,
On Thu, Apr 12, 2018 at 1:22 PM, Wes Turner wes.turner@gmail.com wrote:
Are there pypa/warehouse github issues for implementing the TUF trust
root support in warehouse; and client support in pip (or a module that pip and other tools can use)?
For client support in pip, we are discussing this patch with Donald Stufft:
https://github.com/pypa/pip/compare/release/9.0.3...trishank atdatadog:trishankatdatadog/9.0.3.tuf-in-toto
Happy to discuss more with anyone interested!
Sorry for the noise, but our repo has now moved here:
https://github.com/pypa/pip/compare/9.0.3...DataDog: trishankatdatadog/9.0.3.tuf-in-toto
Hey, what's the latest on this?
- Python PEP458: - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf
On Wed, Apr 18, 2018 at 5:43 PM Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0....
On Thu, Jul 26, 2018 at 1:37 PM Wes Turner wes.turner@gmail.com wrote:
Hey, what's the latest on this?
- Python PEP458:
I'm not aware of what is happening on the Warehouse end, but I'm happy to report we have a version of pip + TUF Datadog is using in production:
https://github.com/DataDog/pip/tree/trishankatdatadog/10.0.1.tuf
I'm fairly confident we can use this same version on Warehouse + TUF :)
On Thu, Jul 26, 2018 at 1:58 PM Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:
I'm not aware of what is happening on the Warehouse end, but I'm happy to report we have a version of pip + TUF Datadog is using in production:
https://github.com/DataDog/pip/tree/trishankatdatadog/10.0.1.tuf
I'm fairly confident we can use this same version on Warehouse + TUF :)
When I get some downtime, I'll start writing some code for Warehouse + TUF...
Hi Wes,
Many members of the PyPA have been trying to obtain funding to try and work on this and other security features I believe. I don't believe, but could be wrong, anything has occurred in implementation here. But Ernest, Donald and Dustin would know best.
Are you interested in helping?
Cooper
On Jul 26, 2018, at 10:33 AM, Wes Turner wes.turner@gmail.com wrote:
Hey, what's the latest on this?
- Python PEP458:
On Wed, Apr 18, 2018 at 5:43 PM Trishank Kuppusamy <trishank.kuppusamy@datadoghq.com mailto:trishank.kuppusamy@datadoghq.com> wrote: https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0.... https://github.com/pypa/pip/compare/10.0.0...DataDog:trishankatdatadog/10.0.0.tuf-in-toto
-- https://keybase.io/trishankdatadog https://keybase.io/trishankdatadog -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2...
-
Cooper Ry Lees -
Sumana Harihareswara -
Trishank Kuppusamy -
Wes Turner