[Distutils] distlib and wheel metadata

Wes Turner wes.turner at gmail.com
Tue Feb 14 15:28:41 EST 2017

On Tue, Feb 14, 2017 at 12:15 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Feb 14, 2017 at 10:10 AM, Vinay Sajip via Distutils-SIG
> <distutils-sig at python.org> wrote:
> >> humpty in term uses uses distlib which seems to mishandle wheel
> >> metadata. (For example, it chokes if there's extra distribution meta and
> >> makes it impossible for buildout to install python-dateutil from a
> wheel.)
> >
> > I looked into the "mishandling". It's that the other tools don't adhere
> to
> > [the current state of] PEP 426 as closely as distlib does. For example,
> > wheel writes JSON metadata to metadata.json in the .dist-info directory,
> > whereas PEP 426 calls for that data to be in pydist.json. The non-JSON
> > metadata in the wheel (the METADATA file) does not strictly adhere to
> any of
> > the metadata PEPs 241, 314, 345 or 426 (it has a mixture of incompatible
> > fields).
> >
> > I can change distlib to look for metadata.json, and relax the rules to be
> > more liberal regarding which fields to accept, but adhering to the PEP
> isn't
> > mishandling things, as I see it.
> I thought the current status was that it's called metadata.json
> exactly *because* it's not standardized, and you *shouldn't* look at
> it?
> It's too bad that the JSON thing didn't work out, but I think we're
> better off working on better specifying the one source of truth
> everything already uses (METADATA) instead of bringing in *new*
> partially-incompatible-and-poorly-specified formats.



PEP426 (Deferred)

Switching to a JSON compatible format

PEP 426: Define a JSON-LD context as part of the proposal

This doesn't work with JSON-LD 1.0:

releases = {
  "v0.0.1": {"url": ... },
  "v1.0.0": {"url": ...},

This does work with JSON-LD 1.0:

releases = [
 {"version": "v0.0.1", "url": ...},
 {"version": "v1.0.0", "url": ...},

... Then adding custom attributes could be as easy as defining a URI
namespace and additional attribute names;
because {distutils, setuptools, pip, pipenv(?)} only need to validate the
properties necessary for the relevant packaging operation.

Without any JSON-LD normalization, these aren't equal:

{"url": "#here"}
{"schema:url": "#here"}
{"http://schema.org/url", "#here"}

This is the JSON downstream tools currently have/want to consume (en masse,
for SAT solving, etc):

- It's a graph.
- JSON-LD is for graphs.
- There are normalizations and signatures for JSON-LD (ld-signatures != JWS)
- Downstream tools need not do anything with the @context. ("JSON-LD
- Downstream tools which generate pydist.jsonld should validate schema in

Downstream tools:
- https://github.com/pypa/pip/issues/988 "Pip needs a dependency resolver"
(-> JSON)
- https://github.com/pypa/warehouse/issues/1638 "API to get checksums" (->

Q: How do we get this (platform and architecture-specific) metadata to
warehouse, where it can be hosted?

A JSONLD entrypoint in warehouse (for each project, for every project, for

> I would accept a pull request to stop generating metadata.json in

What about a pull request to start generating metadata.jsonld or
pydist.jsonld instead?

- [ ] "@context": { },
- [ ] "@graph": { },   # optional


> -n
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170214/2fe33727/attachment.html>

More information about the Distutils-SIG mailing list