Immutability of Release Metadata in Warehouse
For those who are not aware, legacy PyPI would allow you to run ``twine register`` on a release that had already been created in order to modify the metadata that PyPI had recorded for that release (keyed by version number). This wasn’t a super widely used feature, but it’s primary use case was when folks would mistakenly release a project that had a broken description that wouldn’t correctly render on PyPI. With the move from the legacy PyPI code base to Warehouse for handling uploads, this feature had been somewhat inadvertently lost. This issue was raised in https://github.com/pypa/warehouse/issues/2170 <https://github.com/pypa/warehouse/issues/2170>. This email is provide notice that after thinking about this issue for awhile, we’re *not* going to restore the ability to modify the metadata associated with a release after it has been uploaded.The eventual goal is that it should be ideally possible to treat the files that are on PyPI as the source of truth for all metadata, and that the data stored in the database is simply an optimization for accessing and presenting that data on PyPI. Obviously if we allow modifications to the metadata as stored in the PyPI database, that would allow this metadata to “drift” from what is actually stored in the files, which would prevent that goal from being realized. It is true that even with this change, there is not a guarantee that the metadata in the database does not match what is in the file(s) that have been uploaded to PyPI, even going into the future. Thus the decision to not restore this feature is not the only step on the way to being able to assert this guarantee, but it is *one* of them. The most common reason for wanting to modify any metadata after the fact is to fix typos etc that made it into the description prior to publication. It is our opinion that the best way to handle these is to either cut a new release (it can be a post release if that’s all that has changed) or to validate the description field prior to publication (which can currently be done using readme_renderer or restview). Longer term we also plan to introduce the ability to “stage” releases so that releases and files can be uploaded to a temporary location within Warehouse, and visible with a special URL to allow people to manually validate the actual bits that were uploaded before pushing a “finalize” or “publish” button that would flip it from being a mutable, hidden release to an immutable, publicly viewable release. If folks have other use cases where they’ve used the ability to modify release metadata after it had been released that they feel is an important enough use case that it needs to be supported in some way, please let us know either in this thread or by opening an issue on https://github.com/pypa/warehouse <https://github.com/pypa/warehouse> so we can figure out if it’s something we want to support, if one of the other mechanisms we’re planning on adding will support it, or if there is some new mechanism we can add that can support it better.
Couldn't Warehouse validate the description, and reject the upload (with an appropriate message) if it doesn't pass? This at least would eliminate those ugly project pages that failed to render...there are a lot of them on PyPI. Donald Stufft kirjoitti 19.12.2017 klo 21:43:
For those who are not aware, legacy PyPI would allow you to run ``twine register`` on a release that had already been created in order to modify the metadata that PyPI had recorded for that release (keyed by version number). This wasn’t a super widely used feature, but it’s primary use case was when folks would mistakenly release a project that had a broken description that wouldn’t correctly render on PyPI. With the move from the legacy PyPI code base to Warehouse for handling uploads, this feature had been somewhat inadvertently lost.
This issue was raised in https://github.com/pypa/warehouse/issues/2170. This email is provide notice that after thinking about this issue for awhile, we’re *not* going to restore the ability to modify the metadata associated with a release after it has been uploaded.The eventual goal is that it should be ideally possible to treat the files that are on PyPI as the source of truth for all metadata, and that the data stored in the database is simply an optimization for accessing and presenting that data on PyPI. Obviously if we allow modifications to the metadata as stored in the PyPI database, that would allow this metadata to “drift” from what is actually stored in the files, which would prevent that goal from being realized.
It is true that even with this change, there is not a guarantee that the metadata in the database does not match what is in the file(s) that have been uploaded to PyPI, even going into the future. Thus the decision to not restore this feature is not the only step on the way to being able to assert this guarantee, but it is *one* of them.
The most common reason for wanting to modify any metadata after the fact is to fix typos etc that made it into the description prior to publication. It is our opinion that the best way to handle these is to either cut a new release (it can be a post release if that’s all that has changed) or to validate the description field prior to publication (which can currently be done using readme_renderer or restview). Longer term we also plan to introduce the ability to “stage” releases so that releases and files can be uploaded to a temporary location within Warehouse, and visible with a special URL to allow people to manually validate the actual bits that were uploaded before pushing a “finalize” or “publish” button that would flip it from being a mutable, hidden release to an immutable, publicly viewable release.
If folks have other use cases where they’ve used the ability to modify release metadata after it had been released that they feel is an important enough use case that it needs to be supported in some way, please let us know either in this thread or by opening an issue on https://github.com/pypa/warehouse so we can figure out if it’s something we want to support, if one of the other mechanisms we’re planning on adding will support it, or if there is some new mechanism we can add that can support it better.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Dec 19, 2017, at 2:51 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
Couldn't Warehouse validate the description, and reject the upload (with an appropriate message) if it doesn't pass? This at least would eliminate those ugly project pages that failed to render...there are a lot of them on PyPI.
I forgot to mention that! So currently the answer is No, because some projects are legitimately not attempting to use rst descriptions and we have no way to know if a project wants a rst description or a plaintext description besides just attempting to render it (which unfortunately also means that projects who want plaintext but which accidentally have something that is valid rst will get rst forced on them). However! There is a desire to enable Markdown support for PyPI, and as part of that a new metadata field is being added that just indicates what format the description is in (rst, markdown, or plaintext is the supported ones once that field gets added). Once we get that field added and plumbed through, then we’ll have a mechanism by which we can determine if the field is supposed to render correctly, and the plan is to start rejecting uploads that include that field (the behavior if that field isn’t included would be the current behavior) where the description does not correctly render with the selected technology. So effectively, not right now, but at some point yes, but it’ll be opt-in by adding a new bit of metadata to your project.
On 12/19/2017 02:58 PM, Donald Stufft wrote:
On Dec 19, 2017, at 2:51 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
Couldn't Warehouse validate the description, and reject the upload (with an appropriate message) if it doesn't pass? This at least would eliminate those ugly project pages that failed to render...there are a lot of them on PyPI.
I forgot to mention that!
So currently the answer is No, because some projects are legitimately not attempting to use rst descriptions and we have no way to know if a project wants a rst description or a plaintext description besides just attempting to render it (which unfortunately also means that projects who want plaintext but which accidentally have something that is valid rst will get rst forced on them).
However! There is a desire to enable Markdown support for PyPI, and as part of that a new metadata field is being added that just indicates what format the description is in (rst, markdown, or plaintext is the supported ones once that field gets added). Once we get that field added and plumbed through, then we’ll have a mechanism by which we can determine if the field is supposed to render correctly, and the plan is to start rejecting uploads that include that field (the behavior if that field isn’t included would be the current behavior) where the description does not correctly render with the selected technology.
So effectively, not right now, but at some point yes, but it’ll be opt-in by adding a new bit of metadata to your project.
GitHub issue links for those who want to track them: * staged releases https://github.com/pypa/warehouse/issues/726 * supporting Markdown https://github.com/pypa/warehouse/issues/2206 * advising packagers to run `python setup.py check -r -s` https://github.com/pypa/python-packaging-user-guide/issues/210 -- Sumana Harihareswara Changeset Consulting https://changeset.nyc
On 12/19/2017 03:00 PM, Sumana Harihareswara wrote:
* supporting Markdown https://github.com/pypa/warehouse/issues/2206
Whoops: https://github.com/pypa/warehouse/issues/869 is the right link. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc
On 20 December 2017 at 05:58, Donald Stufft <donald@stufft.io> wrote:
However! There is a desire to enable Markdown support for PyPI, and as part of that a new metadata field is being added that just indicates what format the description is in (rst, markdown, or plaintext is the supported ones once that field gets added). Once we get that field added and plumbed through, then we’ll have a mechanism by which we can determine if the field is supposed to render correctly, and the plan is to start rejecting uploads that include that field (the behavior if that field isn’t included would be the current behavior) where the description does not correctly render with the selected technology.
The Description-Content-Type field has been added to the spec: https://packaging.python.org/specifications/core-metadata/#description-conte... (This was part of the motivation for Dustin's metadata 1.3 PEP, to help make it more readily apparent that we'd added that) So the missing step is the "plumbing through" part, such that the publishing tools support setting it, and Warehouse supports ensuring the description renders at upload time. However, there's still a difference between "renders without error" and "renders the way the developer intended", so your initial post will still stand for those kinds of problems (as well as typo fixes, etc). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
For those who want to track the relevant GitHub issues: * supporting Markdown https://github.com/pypa/warehouse/issues/869 * staged releases https://github.com/pypa/warehouse/issues/726 * advising packagers to run `python setup.py check -r -s` https://github.com/pypa/python-packaging-user-guide/issues/210 -- Sumana Harihareswara Changeset Consulting https://changeset.nyc
participants (4)
-
Alex Grönholm -
Donald Stufft -
Nick Coghlan -
Sumana Harihareswara