[Distutils] abstract build system PEP update

[Donald Stufft]

> On Feb 17, 2016, at 8:06 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 17 February 2016 at 12:44, Donald Stufft <donald at stufft.io> wrote:
>> You could say that using twine to handle the uploading is a thing people should
>> do (and I agree!) but that currently relies on having static metadata inside of
>> the sdist that twine can parse, static metadata that isn't going to exist if
>> you just simply tarball up a directory on disk.
> 
> Thank you for confirming this. I'd mentioned it in my comments on
> Nathaniel's proposal but I wasn't sure.
> 
> So the situation we currently have with the "sdist format" (vaguely
> and unsatisfactorily defined as the current version is) is:
> 
> 1. It contains static metadata, added by distutils.
> 2. That metadata is not sufficiently reliable for pip to use so pip
> regenerates it.
> 3. Twine (and potentially other tools) needs to access that metadata,
> but isn't critically affected by the unreliability.
> 
> The two proposals address (2) by giving pip a formal interface to ask
> the build system for metadata. But in doing so, they drop totally the
> requirement for (1). As a result, tools like twine are broken by the
> proposals (to the extent that people adopt alternative build systems -
> and if no-one does, then the PEPs are useless anyway!).

Correct.

Through largely an accident of history, the static metadata inside of a sdist
does not contain what is likely the most contoversial thing to be "baked" into
a sdist-- namely the dependencies. It mostly contains things like name,
version, classifiers, long_description, author, etc. The unreliaiblity that
pip cares about are things like (or really, mainly just), projects that
dynamically adjust the install_requires based on attributes of the platform
it is being executed on (such as adding a dependency on argparse for py26). The
things that Twine cares about are generally not dynamic in that way and
typically are "static" within a specific sdist anyways (and if they aren't the
failure case is that PyPI displays stale metadata).

You can sort of get this information by interrogating the wheel metadata, but
that would be a regression for twine as well. One of the major use cases for
twine (in fact, the original use case!) was that Openstack wanted to be able
to create a sdist on one machine and upload from another machine. They wanted
this split because the first machine could be isolated, without access to the
PyPI credentials and thus it could safely execute untrusted code which is
required to build the sdist. They could then convey that sdist to another
machine and upload it, using twine, to PyPI without having to execute any
code from inside of the package to do so.

> 
> The proposals focus solely on the implications for pip. In the
> terminology introduced by Nathaniel's proposal, "build frontends" and
> "integration frontends". But they need to also discuss the
> implications for let's call them "source consumers" - tools that work
> with all forms of source that can be processed by a "build frontend",
> but which *don't* have any need to directly interact with the build
> system. Those tools *are* affected, as their focus is on "source
> distributions which can be handled by pip" and the set of such source
> distributions *is* changing.
> 
> Such "source consumers" would include twine and PyPI at a minimum. As
> well as adhoc tools users might write.
> 
> (Note that if a build tool doesn't want to provide a means to generate
> a sdist, then that's fine - but in terms of the above they will then
> explicitly be opting out of the formal "build system" infrastructure
> and be deliberately targeting the production of binary-only wheel
> distributions. Whether that's acceptable to their users isn't
> important to this discussion. But I think the fact that we're even
> talking about decoupling the build system from setuptools implies that
> some users do care about not focusing purely on binary wheel-only
> distribution)

In my opinion, the ability to create a sdist should be considered a standard
feature for whatever toolchain any particular project decides to use. I don't
much care if there is a seperate sdist building tool and a wheel building tool
or if they are rolled into one tool, but I think that a "standards compliant"
toolchain needs both and I don't think we can replace a toolchain that has both
with one that doesn't.

Now, we obviously can't stop tools like flit from saying they only want to
support wheels if that's their choice, but I don't think that we should go out
of our way to make it a possibility to do that either.

The flip side of that, is I do not think (as has been suggested in the past)
that mandating sdists on PyPI is the correct answer either. We're not trying to
*force* people to upload sdists, but what we do want is for it to be considered
standard to do so, and that if you don't upload one, you're doing something
"weird". My reasoning for this is:

* Not everything on PyPI needs to be open source, the only thing we require is
  a license to distribute, not a license to use or modify. It's perfectly valid
  to upload only a wheel to PyPI, but doing so has implications and should be
  something only done in edge cases.

* We can't even really force it for real, if someone doesn't want to produce a
  sdist and PyPI says they must, they can jsut upload an empty sdist.

Like many things in packaging (and life in general)! I think the way to handle
this is to pave cowpaths in the direction we want people to go. Make the
"right" (providing sdists on PyPI) thing easy and the "wrong" thing hard, but
possible. The current proposals pave cowpaths towards the "wrong" thing.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160217/2c36fd7e/attachment-0001.sig>