[Distutils] abstract build system PEP update

[Paul Moore]

On 17 February 2016 at 21:01, Nathaniel Smith <njs at vorpus.org> wrote:
> On Feb 17, 2016 4:44 AM, "Donald Stufft" <donald at stufft.io> wrote:
>>
> [...]
>> You could say that using twine to handle the uploading is a thing people
>> should
>> do (and I agree!) but that currently relies on having static metadata
>> inside of
>> the sdist that twine can parse, static metadata that isn't going to exist
>> if
>> you just simply tarball up a directory on disk.
>
> Ah-ha, this is useful. The reason this hasn't been considered, at least in
> my proposal, is that I think this is the first I've heard that there is
> anything that cares about what's in an sdist besides setup.py :-).

Yes, twine is a good, concrete example.

> Is there anything written up on what twine wants from an sdist? Would it
> make sense for you to write up a spec on what twine/pypi need and a better
> way to represent it than whatever it is distutils does now?

I think the key point here is that twine expects a *sdist*, that is,
an archive, containing a PKG-INFO file conforming to the Metadata 1.1
spec. See PEP 314. (Note, I only just read PEP 314 myself - it's as
much news to me that it defines how sdists have to contain metadata as
it probably is to everyone else...)

One of the problems here is that the older packaging PEPs got into
such a mess, that people have stopped believing in them at all. I just
looked at PEPs 241 and 314, and they actually do a pretty good job of
specifying how sdists must include metadata, and I think it's fair to
say that this should be our staring point.

> I think both Robert and my proposal basically see their scope as being
> strictly restricted to "here's how we want to replace pip-calling-setup.py
> with pip-calling-something-else", while keeping everything else the same or
> at least delegating any other changes to other PEPs. So we envision that
> build system authors will provide some way to package up source into an
> sdist, whatever that means; that could be a current-style sdist with
> metadata requirements reverse-engineered from twine and setuptools, or it
> could be done kind of new and improved sdist that is about to get its own
> PEP... either way, it's orthogonal to replacing setup.py.

Robert more or less did that, with the proviso that his PEP left open
the possibility that build systems can upload things that the PEP says
can be processed by pip, even though in practice they aren't sdists
(according to PEP 314). But given that no-one had really dug back into
the implications of the older PEPs, that's more of a loophole than a
deliberate choice. On the other hand, your proposal explicitly allows
a "version 1" sdist to not contain PEP 314 metadata, precisely because
it defines a new format of sdist *without* mandating that it must
conform to PEP 314 (no blame here - as I say, we'd all omitted to
check back at the older PEPs).

IMO, we should either bite the bullet and properly define a "new
sdist" format (which won't be an easy job!), or we should couch the
"build system abstraction" PEPs in terms of adding a new "build system
interface" metadata file to the existing sdist format. In the
interests of openness, I'd include comments pointing out that the
existing sdist format is defined as conforming to PEP 314, and that
even though pip itself (and the build system interface) doesn't use
the metadata mandated by PEP 314, other tools do.

The latter option is far easier (we're basically almost there) but it
will *not* suit build tools that can't (or won't) generate a sdist
that conforms to PEP 314 (i.e., contains PKG-INFO).

Paul