[Distutils] Provisionally accepting PEP 517's declarative build system interface

[Donald Stufft]

> On May 31, 2017, at 3:38 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 May 2017 at 20:20, Donald Stufft <donald at stufft.io> wrote:
>> The most likely outcome if PEP 517 is implemented as defined and people who
>> aren’t steeped in packaging lore hear about is, is they get excited about
>> being able to kill setup.py, they implement it, they find out some tool they
>> depend on doesn’t work and can’t work with it, they get discouraged and
>> start filling issues. Ideally those issues are filed on the tool that
>> implemented PEP 517, but most likely it will be filed on tox, Travis,
>> GemFury, etc.
>> 
>> I am struggling to figure out where there is opposition to simply exposing
>> something in a standard way, that you were already planning on implementing
>> anyways.
> 
> There's a lot of baggage associated with the term sdist.

I mean, PEP 517 explicitly redefines “sdist” to mean “a tarball that includes a pyproject.toml, setup.py not required”, so by accepting PEP 517 we’re accepting that a sdist is either the thing that we conventionally call a sdist, or a thing that is similar to it, but instead of a setup.py it contains a pyproject.toml. That’s from https://www.python.org/dev/peps/pep-0517/#source-distributions <https://www.python.org/dev/peps/pep-0517/#source-distributions> which states:

For now, we continue with the legacy sdist format which is mostly
undefined, but basically comes down to: a file named
``{NAME}-{VERSION}.{EXT}``, which unpacks into a buildable source tree
called ``{NAME}-{VERSION}/``. Traditionally these have always
contained ``setup.py``\-style source trees; we now allow them to also
contain ``pyproject.toml``\-style source trees.

Integration frontends require that an sdist named
``{NAME}-{VERSION}.{EXT}`` will generate a wheel named
``{NAME}-{VERSION}-{COMPAT-INFO}.whl``.

If we want to more rigorously define a sdist that’s fine, we can go down that rabbit hole, if we want to remove that and say for something to be a sdist they still have to have a setup.py that supports the expected commands and oh by the way you can use this new thing in PEP 517 to declare statically an alternative build tool that some setup_requires tool will use to replace bdist_wheel, then that’s fine too. Hell, If it wants to get rid of the sdist terminology completely and make a new format called a source wheel or a bagofiles or whatever, that’s fine too.

The PEP right now seems to want it both ways, it wants to declare this thing without the conventional interfaces is a sdist, while ignoring the fact that people are using those conventional interfaces. For me, if the PEP wants it’s new thing to be a sdist, then it needs to handle that case and anything else doesn’t sit right with me.

> 
> As a suggestion - if backends supplied a prepare_build_files hook,
> someone could write a pretty trivial tool that called that hook. Then
> call the build_wheel_metadata hook to get some details to put into
> PKG-INFO, zip the result up and call it a sdist. You could dump a
> setup.py replacement in there that used PEP 517 hooks to implement the
> setup.py interface, if you wanted.
> 
> Given how vaguely defined a sdist is, it would be hard to argue that
> the result is *not* a sdist. I'm not sure how much further you're
> going to insist on going. You no longer create a sdist using "setup.py
> sdist", sure. But at some point the tools have to deal with setup.py
> going away, so I don't see how that's a requirement forever.

That isn’t really the same though, prepare_build_files hook is presumably not going to be including things like the LICENSE file, documentation which are things you’d want in an sdist, but which would be non-obvious to include in the said prepare_build_files hook and is likely going to be vastly different than a sdist produced by ``myhypotheticalbuildtool sdist``, unless prepare_build_files hook is basically exactly the same as build_sdist in all but name.

I’m not stating that we need to support ``setup.py sdist`` forever, I’m saying we need to support a generic way to build a sdist to replace ``setup.py sdist``.

> 
> If you really think we need to cover these use cases solidly (and you
> have a point, I'm not saying they are irrelevant at all) then maybe we
> need to get input from the tox/travis/gemfury devs, to see what they
> actually think, rather than trying to guess their requirements?

I do think we need to cover them solidly yes.

I’m happy to try and bring them in, but as far as Travis/GemFury goes, I think their use case is pretty simple, given a python project using the hypothetical PEP 517, they want to produce a sdist and either serve it directly (GemFury) or publish it to PyPI (Travis). If we don’t add build_sdist, and every project implements their own mechanism for generating a sdist, then either they, or some wrapper tool has to know and understand every possible build tool they might use and how to get it to build a sdist (OR they have to start adding configruration for people to instruct them how to build a sdist for their specific project using whatever tool they’re using).

As far as Tox goes, I will poke them, but I’m pretty sure the answer is wanting to test the thing that is being installed as it would be installed from PyPI, to try and mitigate packaging errors where you forget a file in your MANIFEST.in or so.

> 
> Paul
> 
> PS None of this means I am in any way in favour of making it seem like
> we're OK with projects not providing sdists (in some form or other).
> We're an open source community, and I think publishing sources
> alongside your binaries, is key to that. A link to an external source
> code repository isn't sufficient, IMO (the repo could die, like Google
> code did).


Unfortunately, that is exactly how I think PEP 517 will end up, since it only requires building wheels, people are going to implement ones that only build wheels, and we have nothing to indicate to them that they generally shouldn’t do that (without a specific reason). On the contrary, the indications point to sdist not being important enough to even be given an option to build them at all, requiring tool authors to go out of their way to decide to add it.


—
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/f9939165/attachment.html>