[Distutils] abstract build system PEP update

[Robert Collins]

On 18 February 2016 at 10:30, Paul Moore <p.f.moore at gmail.com> wrote:
> I think the key point here is that twine expects a *sdist*, that is,
> an archive, containing a PKG-INFO file conforming to the Metadata 1.1
> spec. See PEP 314. (Note, I only just read PEP 314 myself - it's as
> much news to me that it defines how sdists have to contain metadata as
> it probably is to everyone else...)
>
> One of the problems here is that the older packaging PEPs got into
> such a mess, that people have stopped believing in them at all. I just
> looked at PEPs 241 and 314, and they actually do a pretty good job of
> specifying how sdists must include metadata, and I think it's fair to
> say that this should be our staring point.

Hmm - I certainly didn't stop believing it.. its just that the
metadata *specified* by the PEP's doesn't include anything distutils
didn't know about: e.g. dependencies.

>> I think both Robert and my proposal basically see their scope as being
>> strictly restricted to "here's how we want to replace pip-calling-setup.py
>> with pip-calling-something-else", while keeping everything else the same or
>> at least delegating any other changes to other PEPs. So we envision that
>> build system authors will provide some way to package up source into an
>> sdist, whatever that means; that could be a current-style sdist with
>> metadata requirements reverse-engineered from twine and setuptools, or it
>> could be done kind of new and improved sdist that is about to get its own
>> PEP... either way, it's orthogonal to replacing setup.py.
>
> Robert more or less did that, with the proviso that his PEP left open
> the possibility that build systems can upload things that the PEP says
> can be processed by pip, even though in practice they aren't sdists
> (according to PEP 314). But given that no-one had really dug back into
> the implications of the older PEPs, that's more of a loophole than a
> deliberate choice. On the other hand, your proposal explicitly allows
> a "version 1" sdist to not contain PEP 314 metadata, precisely because
> it defines a new format of sdist *without* mandating that it must
> conform to PEP 314 (no blame here - as I say, we'd all omitted to
> check back at the older PEPs).

Well, I'm adamant that my PEP should not make anything worse! My
specific concern w.r.t. sdists is that its an orthogonal consideration
to 'not having a setup.py'. Not having a setup.py applies to any
source tree irrespective of how you obtained it. Sdists and their
constraints apply to well - sdists.

> IMO, we should either bite the bullet and properly define a "new
> sdist" format (which won't be an easy job!), or we should couch the
> "build system abstraction" PEPs in terms of adding a new "build system
> interface" metadata file to the existing sdist format. In the
> interests of openness, I'd include comments pointing out that the
> existing sdist format is defined as conforming to PEP 314, and that
> even though pip itself (and the build system interface) doesn't use
> the metadata mandated by PEP 314, other tools do.
>
> The latter option is far easier (we're basically almost there) but it
> will *not* suit build tools that can't (or won't) generate a sdist
> that conforms to PEP 314 (i.e., contains PKG-INFO).

I still don't understand how *making* an sdist is tied into this PEP?
AIUI flit's only concern with sdists was that they didn't want a
setup.py with all the stuff we've discussed about that. Perhaps a flit
author can weigh in - generating good PEP 314 PKG-INFO shouldn't be a
problem for flit, has no Python2/3 implication, and doesn't require a
setup.py.

PEP 241 and 314 do not require a setup.py to be present AFAICT.

I think all we need to do is make sure that its crystal clear that
we're not changing any specified part of the contract of sdists: you
should still make them, they still need to meet PEP 314.... setup.py
was never PEP'd as part of the contract.

We are changing the defacto contract, and so we should make that clear
in the prose.

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud