[Distutils] GSoC 2017 - Plan of Action for dependency resolver

[Pradyun Gedam]

Hello Everyone!

Google released the list of accepted organizations for GSoC 2017 and PSF is
one of them. I guess this would a good time for me to seek feedback on the
approach I'm planning to take for my potential GSoC project. I hope this
mailing list is the right place to do so.

---

Here's my current plan of action along with reasoning for the choices made:

A separate PR will be made for each of these stages. Every stage does
depend on the previous ones being completed.

1. Refactor all dependency resolution responsibility in pip into a new,
separate module.

   This would allow any future changes/improvements in the dependency
resolution to be added without major changes in the rest of the code-base.

   As of today, the RequirementSet class within pip seems to be doing a lot
of work and dependency resolution is a responsibility that doesn't need to
given to it, especially when it's avoidable.

2. Implement dependency information caching.

   This would allow the resolver to not cause the re-computation of the
dependencies of a package, if they have already been computed, speeding up
the resolution.

3. Implement a backtracking resolver.

   A backtracking solver would be appropriate given that we don't have a
way to pre-compute the dependencies for *all* the packages or statically
determine the dependencies - a SAT solver would not be feasible.

4. (if time permits) Move any dependency resolution code out into a
separate library.

   This would make it possible for other projects (like buildout or a
future pip replacement) to reuse the dependency resolver.

By making each of the stages separate PRs, incremental improvements would
be made so that even if I leave this project midway, there will be some
work merged already if someone comes back to this problem later. That said,
I don't intend to leave this project midway.

I do intend to reuse some of the work done by Robert Collins in PR #2716 on
pip's GitHub repository.

Stages 2 and 3 are separate because I see them as distinctly different
tasks which touch very different portions of the code-base. There's is
strong coupling between them though.

I'm looking forward to the feedback. :)

Regards,
Pradyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170228/fa2863ab/attachment.html>