[Distutils] abstract build system PEP update

[Paul Moore]

On 17 February 2016 at 22:58, Robert Collins <robertc at robertcollins.net> wrote:
>> The latter option is far easier (we're basically almost there) but it
>> will *not* suit build tools that can't (or won't) generate a sdist
>> that conforms to PEP 314 (i.e., contains PKG-INFO).
>
> I still don't understand how *making* an sdist is tied into this PEP?
> AIUI flit's only concern with sdists was that they didn't want a
> setup.py with all the stuff we've discussed about that. Perhaps a flit
> author can weigh in - generating good PEP 314 PKG-INFO shouldn't be a
> problem for flit, has no Python2/3 implication, and doesn't require a
> setup.py.

That's why I didn't specifically refer to flit. In the context of this
point, if projects produce sdists (i.e., they contain PKG-INFO) then
we're find. AFAIK, there's no PEP saying a sdist has to contain a
setup.py.

Backward compatibility issues as noted in your PEP may imply that a
minimal "shim" setup.py should be included. I'd have to reread what
you said to confirm if you mandated a shim or not[1] but that's a side
issue for now.

> PEP 241 and 314 do not require a setup.py to be present AFAICT.

Correct (as far as *I* can tell). And that's fine.

> I think all we need to do is make sure that its crystal clear that
> we're not changing any specified part of the contract of sdists: you
> should still make them, they still need to meet PEP 314.... setup.py
> was never PEP'd as part of the contract.

Exactly. This was what I was proposing, so I agree 100%

> We are changing the defacto contract, and so we should make that clear
> in the prose.

And yet, this is where we have to be careful, as pip isn't the only
consumer of that contract (that's the point of Donald bringing up
twine). You say in another mail:

> How about:"""
> The existing
> PEP-314 [#pep314] requirements for sdists still apply, and distutils
> or setuptools
> users can use ``setup.py sdist`` to create an sdist. Other tools should create
> sdists compatible with PEP-314 [#pep314]. Note that pip itself does not require
> PEP-314 compatibility - it does not use any of the metadata from sdists - they
> are treated like source trees from disk or version control."""

While this is accurate as regards pip not using the PEP 314 metadata,
I think it's important to point out that in practical terms pip *does*
require PEP 314 compatible sdists, in that it *locates* source by
querying PyPI for a sdist, and so it inherits PyPI's rules for what is
hosted there. I don't know if PyPI checks for PEP 314 metadata, or if
Warehouse will in future, but that is something we need to take into
consideration. And even if they don't, we recommend using twine for
uploads, and twine *does* require PEP 314.

As worded, the note about pip will[2] increase pressure for a means to
host the sources for that tool on PyPI, which in turn will increase
pressure for PyPI/Warehouse to *not* enforce PEP 314, and in turn for
twine to not enforce that PEP. We aren't ready to offer an alternative
to PEP 314 yet, though.

What I don't understand is why it matters so much to you that the
proposal *doesn't* simply say "sdists should conform to PEP 314", and
you keep pushing to leave the door open for people to omit the
metadata. (That's in *sdists* - as you point out, pip doesn't use or
need metadata in the source tree, but that's already clear in the
proposal). All I can imagine is that it's because you want to allow
flit users the extra flexibility.

My questions for flit (and any other build tool that expects to use
these proposals):

1. Can you produce PEP 314 compatible sdists?
2. If not, what are your reasons (and does this proposal change
anything in that regard)?
3. Assuming you don't intend to produce PEP 314 compatible sdists,
what is your recommendation to your users on how they publish source,
should they wish to?
4. If your users were unable to publish source on PyPI because of your
lack of a sdist, would that change your view on whether you support
sdists?
5. What changes would you need to the sdist format to make it acceptable to you?

Unless those questions expose a significant roadblock that flit simply
cannot overcome with PEP 314 compatibility, it seems to me that
there's no harm in mandating PEP 314 for sdists without qualification,
and leaving further discussion about de facto behaviour, possible
changes, etc, to the "sdist 2.0" discussion that we're all trying to
defer.

By the way - I should point out that it's pretty straightforward for a
project author to *manually* create a PKG-INFO file in their flit
project, bundle it up with all of the rest of the sources, name the
archive correctly, and have a PEP 314 compatible sdist. Managing that
PKG-INFO file in parallel with flit's own copy of the metadata is
likely to be a PITA, but how is that not just a usability issue to be
dealt with between the users and authors of flit?

Paul

[1] One downsides of managing PEPs as PRs against the packaging specs
is that there is no easy quotable term to refer to them by - we could,
and probably should, use the PR number. And as a consequence, it's
hard to quickly find the text because there's no simple form for the
URL (again mitigated by using the PR number).
[2]  Once again, this is all on the assumption that tools will be
developed that use the build system abstraction and can't or won't
produce PEP 314 compatible sdists. Otherwise why are we even arguing?
If no-one has a problem following PEP 314, we should just note that
this proposal doesn't change the fact that it is a requirement for a
sdist and stop worrying.