[Distutils] A smaller step towards de-specializing setuptools/distutils

[Robert Collins]

On 30 October 2015 at 06:31, Donald Stufft <donald at stufft.io> wrote:

tl;dr:
 - requiring wheel before dependencies is a Big Deal IMO
 - I don't care about pypa.yaml vs setup.cfg - argue with the flit folk :)
 - I don't care about setup.py-as-interface vs described-by-metadata -
argue with the flit folk
 - I *do* want someone to pick choices on the 2 previous points so we
can move forward :)

> Hello!
>
> So I've been reading the various threads (well trying to, there's a lot going
> on in all of them) and I'm trying to reconile them all in my head and sorting
> out how things are going to look if we add them. I'm worried about adding more
> complication to an already underspecified format and making it a lot harder for
> tooling to work with these things.

Thats fair enough, and thanks for putting the effort in :).

> I'm wondering if there isn't a smaller step we can take towards allowing better
> interopability with diffrent build systems while also making the situation
> clearer when someone is still using distutils or setuptools. For the sake of
> argument, let's say we do something like this:

So - my proposal ended up where it did because:
 - the flit folk really didn't want to have to have setup.py in their
tree at all - they strongly objected to that.
 - setup.cfg had a similar (but weaker) objection
 - we obviously need a place for pip to key off of to drive things
 - building packages with non-trivial build times is something we
*really* need to keep out of the inner loop of resolving dependencies.
Building to wheel first is a *huge* potential cost, *especially* when
we haven't solved the wheels-for-linux problem.

I've previously (on this list, 6? 8? months backs) proposed using
setup.cfg directly to do build time requirements and use that as the
interface. I'm still happy doing that, so if thats the direction you
and Nick (or whatever the BDFL-equivalents here) want to take, I can
easily strip out the indirection involved in defining the interface to
the build tool and just assume setup.py with those setuptools-like
commands. The specific complexity in the interface was to avoid the
emergent need to update config in many trees when flit etc do change
things.

Similarly, if you want to add the dependency on defining 'develop' as
something pip does itself to this PEP, I can help write that PEP
(which I think we should do anywhich way), and pivot from asking for
develop to be done, and onto having a build inplace facility.

However, the big step you're proposing that I think is fundamentally
unsound is that of requiring a wheel be built before dependencies can
be queried. Wheels are *expensive* to build in enough cases that I
really believe we have a choice between being able to fix issue 988 or
utilising wheels as the interface to get at install metadata. Remember
too that the numpy ABI discussion is going to cause install
dependencies to depend on the version of numpy present at build-time,
so the resolver is going to have to re-evaluate the dependencies for
those distributions when it selects a new numpy version: so its not
even a one-time-per-package cost, its a
one-time-per-arc-of-the-N!-graph-paths cost.

What you're describing is in all other regards basically my PEP, so +1 sure.

-Rob