[Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Wed, May 4, 2016 at 6:28 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 4 May 2016 at 23:00, Daniel Holth <dholth at gmail.com> wrote:
>> +1 It would be great to start with a real setup_requires and probably would
>> not interfere with later build system abstractions at all.
>
> If we're going to go down that path, perhaps it might make sense to
> just define a standard [setup_requires] section in setup.cfg?
>
> Quite a few projects already have one of those thanks to distutiils2,
> d2to1 and pbr, which means the pragmatic approach here might be to ask
> what needs to change so the qualifier can be removed from this current
> observation in the PBR docs: "The setup.cfg file is an ini-like file
> that can mostly replace the setup.py file."
>
> The build system abstraction config could then also just be another
> setup.cfg section.

I'm sympathetic to the general approach, but on net I think I prefer a
slightly different proposal.

Downsides to just standardizing [setup_requires]:

- if projects have existing ones, that's actually kinda good / kinda
bad -- pip has never been respecting these before, so if we suddenly
start doing that then existing stuff could break. I don't know how
likely this is, but experience suggests that *something* will break
and make someone angry. (I'm still blinking at getting angry
complaints arguing that uploading a wheel to pypi, where before there
was only an sdist, should be treated as a compatibility-breaking
change that requires a new version etc.)

- IMO an extremely valuable aspect of this new declarative
setup-requirements thing is that it gives us an opportunity to switch
to enforcing the accuracy of this metadata. Right now we have a swamp
we need to drain, where there's really no way to know what environment
any given setup.py needs to run. Sometimes there are setup_requires,
sometimes not; if there are setup_requires then sometimes they're
real, sometimes they're lies. We'd like to reach the point where for a
random package on pypi, either we can tell it's a legacy package, or
else we can be confident that the setup-requirements it declares are
actually accurate and sufficient to building the package. This is
important because it unblocks ecosystem improvements like automated
wheel builders, autoconfiguring CI systems, etc. And the only way
we'll get there, AFAICT, is if at some point we start enforcing build
isolation, so that by default a build *can't* import anything that
wasn't declared as a setup-requirement. And the only way I can see for
us to make this transition smoothly -- without any horrible flag days,
and with a nice carrot so projects feel like we're giving them a gift
instead of punishing them -- is if we make the rule be "projects that
use the declarative setup-requirements feature also get isolated build
environments". (Then the messaging is: "see, this helps you check that
you actually set it up right! if your new metadata works for you in
testing, it'll also work for your users!) But this again would mean we
can't reuse the existing [setup_requires] section.

- setup.cfg is kind of a terrible format for standardizing things
because the only definition of the format is "read the ConfigParser
source". You cannot parse setup.cfg from a non-Python language. And
the *only* benefit is that it already exists; teaching pip to read
pybuild.json or pybuild.toml instead would be completely trivial. So
if we aren't going to try and grandfather in existing [setup_requires]
sections, then we might as well switch to a better file format at the
same time -- this is not the hard part.

So I like the idea of splitting out the declarative setup-requirements
PEP from the new build system hook interface PEP, but I think that the
way we should do this is by defining a new pybuild.whatever file like
the ones in PEP 516 / PEP 517 (they're identical in this regard) that
*only* has schema-version and bootstrap-requirements, and then we can
add the build backend key as a second step.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org