Migrating interoperability specs to packaging.python.org
Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now).
I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP.
I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time.
However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things:
1. Explicitly notes the addition of the two new fields 2. Describes the process change for packaging interoperability specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format
That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org as the canonical source of the specification, rather than being the reference documentation in its own right.
Cheers, Nick.
[1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388
P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral)
Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;)
Some people enjoy using metadata.json which tracked pep 426 but I have been meaning to take it out, and perhaps keep the key/value to json converter as a command.
On Mon, Sep 4, 2017, 09:33 Nick Coghlan ncoghlan@gmail.com wrote:
Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now).
I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP.
I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time.
However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things:
- Explicitly notes the addition of the two new fields
- Describes the process change for packaging interoperability
specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format
That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org as the canonical source of the specification, rather than being the reference documentation in its own right.
Cheers, Nick.
[1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388
P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral)
Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Yes, I see the inclusion of a metadata file which conforms to an unaccepted PEP as potentially dangerous.
Perhaps I should disable it in the next release?
Daniel Holth kirjoitti 04.09.2017 klo 17:03:
Some people enjoy using metadata.json which tracked pep 426 but I have been meaning to take it out, and perhaps keep the key/value to json converter as a command.
On Mon, Sep 4, 2017, 09:33 Nick Coghlan <ncoghlan@gmail.com mailto:ncoghlan@gmail.com> wrote:
Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org <http://packaging.python.org>, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now). I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP. I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time. However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things: 1. Explicitly notes the addition of the two new fields 2. Describes the process change for packaging interoperability specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org <http://packaging.python.org> as the canonical source of the specification, rather than being the reference documentation in its own right. Cheers, Nick. [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388 P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral) Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;) -- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <mailto:Distutils-SIG@python.org> https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Well, none of the metadata generated by bdist wheel conforms to an accepted pep. But if you rely on the json file then you won't be interoperable with wheels from any other generator.
On Mon, Sep 4, 2017, 10:06 Alex Grönholm alex.gronholm@nextday.fi wrote:
Yes, I see the inclusion of a metadata file which conforms to an unaccepted PEP as potentially dangerous.
Perhaps I should disable it in the next release?
Daniel Holth kirjoitti 04.09.2017 klo 17:03:
Some people enjoy using metadata.json which tracked pep 426 but I have been meaning to take it out, and perhaps keep the key/value to json converter as a command.
On Mon, Sep 4, 2017, 09:33 Nick Coghlan ncoghlan@gmail.com wrote:
Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now).
I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP.
I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time.
However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things:
- Explicitly notes the addition of the two new fields
- Describes the process change for packaging interoperability
specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format
That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org as the canonical source of the specification, rather than being the reference documentation in its own right.
Cheers, Nick.
[1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388
P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral)
Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Yeah, so we should maybe start using the upcoming v1.3 metadata when the related PEP is accepted...?
Daniel Holth kirjoitti 04.09.2017 klo 17:48:
Well, none of the metadata generated by bdist wheel conforms to an accepted pep. But if you rely on the json file then you won't be interoperable with wheels from any other generator.
On Mon, Sep 4, 2017, 10:06 Alex Grönholm <alex.gronholm@nextday.fi mailto:alex.gronholm@nextday.fi> wrote:
Yes, I see the inclusion of a metadata file which conforms to an unaccepted PEP as potentially dangerous. Perhaps I should disable it in the next release? Daniel Holth kirjoitti 04.09.2017 klo 17:03:Some people enjoy using metadata.json which tracked pep 426 but I have been meaning to take it out, and perhaps keep the key/value to json converter as a command. On Mon, Sep 4, 2017, 09:33 Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote: Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org <http://packaging.python.org>, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now). I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP. I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time. However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things: 1. Explicitly notes the addition of the two new fields 2. Describes the process change for packaging interoperability specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org <http://packaging.python.org> as the canonical source of the specification, rather than being the reference documentation in its own right. Cheers, Nick. [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388 P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral) Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;) -- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <mailto:Distutils-SIG@python.org> https://mail.python.org/mailman/listinfo/distutils-sig _______________________________________________ Distutils-SIG maillist -Distutils-SIG@python.org <mailto:Distutils-SIG@python.org> https://mail.python.org/mailman/listinfo/distutils-sig_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <mailto:Distutils-SIG@python.org> https://mail.python.org/mailman/listinfo/distutils-sig
On 4 September 2017 at 23:33, Nick Coghlan ncoghlan@gmail.com wrote:
Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now).
I'm happy to report that Dustin Ingram has started making some progress on this change, with https://packaging.python.org/specifications/ now broken out into a set of distinct subpages with appropriate URLs, and https://packaging.python.org/specifications/core-metadata/ now coversing *all* the fields that can appear in PKG-INFO and METADATA files, not just the ones that are post-1.2 additions.
Cheers, Nick.
On 4 September 2017 at 23:33, Nick Coghlan ncoghlan@gmail.com wrote:
However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things:
- Explicitly notes the addition of the two new fields
- Describes the process change for packaging interoperability specifications
- Defines a canonical transformation between the human-readable
key:value format and a more automation friendly JSON format
Discussing this idea further with Dustin, I now think it's conflating two different activities (one related to process management, one related to actually updating the metadata specification).
So I'd suggest that any metadata 1.3 PEP restrict itself to only covering the first point (more formally documenting the two fields added since metadata 1.2) and the last point (defining a standard translation from the Key:Value format to JSON and back).
For the process management question around improving traceability in our spec management processes, I'm wondering if it may make sense to write a meta-PEP called just "The Python Packaging Authority" that attempts to better articulate what that term actually encompasses:
* The published interoperability specifications at https://packaging.python.org/specifications/ * The process for updating those specifications at https://www.pypa.io/en/latest/specifications/ * Ecosystem & specification level design discussions on distutils-sig * Funding and resource allocation authority delegated from the PSF Board to the Python Packaging Working Group (overseen by the PSF's Director of Operations & Infrastructure Manager) * PyPI operating authority delegated from the PSF Board to the PSF Infrastructure team (overseen by the PSF's Director of Operations & Infrastructure Manager)
Such a meta-PEP could also clearly document the limits of Guido's original design delegation back at the 2013 Python Language Summit (i.e. as soon as we need or want to change anything in CPython or the standard library, then that's a change that needs to go through the regular PEP process, not the PyPA/distutils-sig specific variant of it)
Cheers, Nick.
participants (3)
-
Alex Grönholm -
Daniel Holth -
Nick Coghlan