[Distutils] (How) do setuptools/distribute/pip handle circular dependencies?

[PJ Eby]

On Sat, Nov 10, 2012 at 6:51 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote:

> Carl Meyer <carl <at> oddbird.net> writes:
>
> > already satisfied. In pip this happens here:
> > https://github.com/pypa/pip/blob/develop/pip/req.py#L1091
> >
> > More generally, I wouldn't really recommend pip's dependency resolution
> > logic as a model for new Python code in this area. There are some not
> > uncommon cases that it handles poorly; see
> > https://github.com/pypa/pip/issues/174 and
> > http://bugs.python.org/issue8927. (To be fair to pip, these cases aren't
> > trivial when you have to download and unpack an sdist before you can
>
> Thanks for the pointers.
>
> > find out anything about its dependencies, but I'd hope that with the new
> > metadata PEPs Python packaging code could get a bit smarter in this
> area.)
>
> AFAICT, the proposed metadata PEP changes don't offer the same requirement
> granularity as setuptools / distribute (for example, 'Requires-Dist' as
> against
> 'install_requires', 'setup_requires', 'test_requires').
>
> Anyway, I'll take a look at the issue you mentioned and see how the
> dependency
> code in distlib stacks up. Currently, it keeps the requirements distinct
> for
> 'install', 'setup' and 'test'. The distinctions seem reasonable in theory,
> though I'm not sure how useful they are in practice.
>

Test dependencies allow you to depend on a test framework (e.g. nose)
without requiring it to be installed at runtime.  Setup dependencies let
you depend on tools like Pyrex or Cython in order to compile a binary
package, without requiring them to be available at runtime.


> In the case I was quoting, the circular dependency wasn't being treated as
> any
> kind of conflict - I just came across cycles when testing topological
> sorting of
> dependency graphs, and was curious about them.
>

Pkg_resources uses a "greedy" dependency resolver, so it ignores
circularity as a side-effect.  That is, as soon as the first requirement
for a project is found, a suitable version is grabbed and added to the
tentative new path.  Thus, if another reference to the same project occurs
elsewhere, the project has already been added to the path, so the
dependency is already satisfied and does not recurse.

Btw, there are other tricky corner cases for dependency resolution; with
distribute and sufficiently-old version of setuptools, circular setup-time
dependencies can lead to recursion, unless the circular package is
available in binary form.  This was fixed a few years ago in setuptools by
ensuring that nested source build invocations saved and restored
pkg_resources state, or something like that, but I don't think distribute
ever integrated those changes.

In any case, that's more of a practical issue for build tools than a
dependency resolver issue per se.

Btw, I did consider using a backtracking dependency resolver in
pkg_resources -- i.e. one that could handle the sort of conflicts linked
above -- but decided against it on the basis that

1. Backtracking resolution could be exponential time in worst-case
scenarios, and
2. Dependency information wasn't available from PyPI without actually
downloading and building things anyway, so there were limits to how much
the backtracking would help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20121112/4ab1e369/attachment.html>