[Distutils] RFC: PEP 566 - Metadata for Python Software Packages 1.3

[Nick Coghlan]

On 6 December 2017 at 07:02, Nathaniel Smith <njs at pobox.com> wrote:
> On Dec 5, 2017 08:42, "Dustin Ingram" <dustin.ingram at gmail.com> wrote:
>
>
> Provides-Extra (optional, multiple use)
> :::::::::::::::::::::::::::::::::::::::
>
> A string containing the name of an optional feature. Must be a valid Python
> identifier. May be used to make a dependency conditional on whether the
> optional feature has been requested.
>
> This introduction of this field allows packge installation tools (such as
> ``pip``) to determine which extras are provided by a given package, and so
> that
> package publication tools (such as ``twine``) can check for issues with
> environment markers which use extras.
>
>
> I haven't followed this so sorry if this is an annoying comment, but having
> read this description I still don't really understand what Provides-Extra is
> doing. Don't packages already include extra information? What problem
> motivated this?

The main problem it solves is a procedural one related to the fact
that I co-opted Daniel Holth's original metadata 1.3 PEP (i.e. PEP
426) as a more expansive "What if we were designing the metadata
system from scratch?" proposal, which then fell prey to second system
syndrome (https://en.wikipedia.org/wiki/Second-system_effect).

We'd subsequently mitigated that problem by allowing additional fields
to be defined directly in
https://packaging.python.org/specifications/core-metadata/ without
being defined in a PEP, but that turned out to create a new problem
with a lack of adequate transparency in and around the specification
update process.

So Dustin's PEP is aimed mainly at getting things back on track from a
procedural perspective, with
https://packaging.python.org/specifications/ becomes PyPA's equivalent
to the Python Language Reference at
https://docs.python.org/3/reference/, and we use Standards Track PEPs
to justify additions and changes to those specifications, rather than
using Informational PEPs directly as the reference documentation.

As such, PEP 566 doesn't actually *change* the current metadata
specification at the field level - it just takes the existing
post-PEP-345 changes, and formalises them as version 1.3 of the
metadata spec (which should then help with consistency of
implementation across various tools).

The one real change that PEP 566 does propose is a new approach to the
question of defining a JSON-compatible variant of the data format:
rather than designing that from scratch as I did in later versions of
PEP 426 (which would have required major changes for existing tools to
adopt), it instead defines it as a reversible algorithmic
transformation from the existing Key:Value based format, which will
allow new tools to adopt it as their preferred working format, while
still maintaining compatibility with existing tools at the level of
the on-disk file format. (Having that transformation defined in an
interoperability PEP will also allow both Warehouse and
https://packaging.pypa.io/ to implement it, which will be useful when
designing packaging related REST APIs and metadata caching formats).

Cheers,
Nick.

P.S. I'm seriously considering moving PEP 426 and 459 from Deferred to
Withdrawn - while there are still some worthwhile ideas in there, I
think we've learned through experience that the more practical path
forward is to identify specific problems that we can't solve with the
existing interoperability specifications, and then propose concrete
solutions to those problems. While entertaining to explore, "What
would we consider doing if we had unlimited development resources?"
flights of fancy aren't actually a practical approach to making
changes :)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia