[Distutils] Improved dependency resolution in distlib

[Vinay Sajip]

Following some changes to distlib, there have been improvements in how it can
resolve dependencies. I ran a test on all source archives reachable through
PyPI, and initial results seem to show that out of over 25,000 projects and
112,000 source archives, we can extract dependency metadata from all but 2,540
archives relating to 916 projects.

The distlib locator code has been updated to access this metadata, and this
makes it possible to resolve dependency graphs reasonably quickly, without
downloading any archives.

The dependency finder command-line script is at

https://gist.github.com/3919026

The script, finddeps.py, when given a distribution name, will attempt to locate
it and its dependencies, identify requirements which couldn't be satisfied,
show a topological sort (advisory, given the existence of circular dependencies
on many PyPI projects) and a download ordering.

Example results are given in the same Gist for:

Flask (3 dists in all, resolved in ~1 second)
apycotbot (22 dists in all, resolved in ~5 seconds)
collective.megaphone (242 dists in all, resolved in ~55 seconds)

You can also specify constraints, e.g.

$ python finddeps.py "pyramid (> 1.0, < 1.3)"

I also included a separate script for testing the case described in

https://github.com/pypa/pip/issues/174

as mentioned by Carl. In this case, we get the expected result, with the
dependency tree looking like this:

A 1.0
    B 1.0 [B]
        D 0.9 [D (<= 0.9)]
    C 1.0 [C]
        D 0.9 [D (<= 1.1)]
D 0.9
C 1.0
    D 0.9 [D (<= 1.1)]
B 1.0
    D 0.9 [D (<= 0.9)]

If anyone has the time to try out distlib, I'd appreciate some feedback on the
dependency finder (or anything else for that matter) and any unexpected results
you get in your trials.

You should be able to run the dependency finder on most projects on PyPI, but
not ones which don't have a reachable download (for example, "Goose". But 
there, "pip install Goose" fails, too).

Thanks and regards,

Vinay Sajip