[Distutils] What metadata does pip actually need about sdists?

[Paul Moore]

On 11 October 2015 at 05:31, Nathaniel Smith <njs at pobox.com> wrote:
> Do the relevant pip maintainers
> even read this mailing list? :-)

Donald and I are pip maintainers (not the only ones, but probably the
ones who make the most noise on this list :-)). The comments I've made
that you quoted are based mostly on how I recall pip works, it's
unfortunately been some time since I looked into pip's dependency
resolution and install code (it's a bit of a pit of snakes) and I
haven't had time to check the details when commenting on this thread,
so I don't claim my comments are authoritative.

I think your description of the behaviour is right.

As regards what pip could do, technically you are of course correct
(it's possible, it just needs someone willing to make the code
changes). I don't know, however, if the approach you're proposing fits
with how we currently envisage pip developing. Specifically, my
personal goal [1] is that we get to a point where pip can do all of
the dependency resolution steps, getting to a point where it knows
exactly what it will install, *before* it starts downloading packages,
running build steps, etc. (And I don't see that sitting well with your
proposal). The reason I want that is so that I would like to be able
to see what pip plans on doing, and audit it before it begins (a "pip
install --dry-run" type of step). Typically, this lets me avoid
getting part-way through an install only to find that I need a project
that doesn't have a wheel and I can't build from the sdist. That's a
real problem I deal with a lot - at the moment I work with "pip list
--outdated", check the listing to see what provides wheels, and update
just those packages individually. But that's messy and manual, and not
always 100% effective - and I'd like to be able to do better.

I also don't think it's reasonable to focus solely on what *pip*
requires. Just as important (IMO) is allowing PyPI to provide rich
metadata. In particular, I write a lot of programs that query PyPI for
package information. There are a lot of useful scripts I would like to
be able to write that need to query PyPI for the dependency details of
a package, and that simply isn't possible at the moment. Most of these
are ad-hoc or personal-use scripts, and a lot of things I simply
haven't done because I can't. So it's hard to quantify the benefits.
But currently, the wonderful Metadata 2.0 ideal future promises me
static dependency data on PyPI. That will only happen if we get enough
people interested in implementing it, but I live in hope. If I
understand your proposals, they are closing off that option, and
explicitly stating that we'll never have dependency metadata on PyPI
for projects that don't publish wheels. Maybe you're right, and it'll
never be possible. But I don't want to give up that dream without
being sure it's necessary - which is why I push back so hard on the
idea.

Paul

[1] Which is not backed up by me having the time to write the code, so
the only relevance here is that I'm inclined to support proposals that
work towards that goal.