[Distutils] Buildout or distribute raise unnecessary version conflict?

[Jim Fulton]

On Fri, Dec 17, 2010 at 10:05 AM, Alan Franzoni <mailing at franzoni.eu> wrote:
> Hello,
>
> I've got a strange issue with version resolution when using
> zc.buildout; I don't know if it's zc.buildout specific or it's caused
> by something which is done by distribute.
>
> The example code can be fetched from there:
>
> hg clone https://Alan@bitbucket.org/Alan/buildoutversionissue
>
> End of output result:
>
> Installing test.
> While:
>  Installing test.
> Error: There is a version conflict.
> We already have: zope.testrunner 4.0.0
> but asd 0.1.0 requires 'zope.testrunner==4.0.0b1'.
>
> If you check within the downloaded eggs, you'll see there's no egg
> requiring zope.testrunner 4.0.0; there's just zc.recipe.testrunner
> requiring "zope.testrunner", without any version.
>
> if egg A requires "foolib" while egg B requires "foolib==1.0.0", and
> the latest available version of foolib is "1.0.1", then a version
> conflict error will arise; it's just as if a "naked" egg dependency is
> pinned to the latest available version of a package.
>
> Is that an intended behaviour?

Sort of.  buildout (and setuptools and distrtibute, I assume) will,
when fetching a distribution, always fetch the most recent that meets
the given requirements.  Buildout is a bit different in that it
checks for new versions any time it runs.

Now, if you have project A that requires B and C, which each
require D, then when installing B, buildout will retrieve the newest
distribution of D that satisfies B's requirements. Later, when it installs
C, if C requires a different version of D, then there is a conflict.
The same thing would happen afaik, using easy_install if none of the
projects are installed at the start.

It would be better if this situation was handled more cleanly.  In the past
I've considered backtracking schemes to handle situations like this,
which seemed scary. :)  It occurs to me writing this that a breadth first,
which wouldn't be hard to implement, might do much better than the current
depth first approach.

> Although specifying a dependency
> without a version shouldn't be a good practice,

Actually, it's best practice.

> it seems quite common
> - mostly because many people have got full control of the pypi section
> on their own repo and they can decide what's offered there, so they
> just specify "mydependency" on their other projects. I think it's
> quite a wide assumption, by the way.

It's best practice.  Requiring a specific version is extremely inflexible.
If everyone did it, you'd never be able to assemble anything of any size.

>
> Also, this is not the usualy way dependencies are resolved in most
> package managers. When using yum (and apt should be quite similar),
> something like that happens:
>
> Scenario:
> two versions of "mydep" are available: mydep 1.0.0 and 1.0.1
> "foo" depends just on mydep
> "bar" depends on mydep 1.0.0
>
> If I just want to install "foo", I'll get foo and mydep 1.0.1 on my
> system. But if I try installing both, I'll get foo, bar and mydep
> 1.0.0, because that will be resolved as the best way to handle all
> dependencies.
>
> Any comment?

Maybe system packing tools use better dependency-resolution
mechanisms.  It could happen! I have certainly been stymied by
version conflicts in system packaging systems, so I don't think their
algorithms are that great.

The way to work around this with buildout is to use a buildout versions
section:

  http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used

It would be interesting to see if a breadth first strategy would provide better
behavior.

Jim

-- 
Jim Fulton