[Distutils] Buildout or distribute raise unnecessary version conflict?

Jim Fulton jim at zope.com
Fri Dec 17 17:41:58 CET 2010


On Fri, Dec 17, 2010 at 11:05 AM, Alan Franzoni <mailing at franzoni.eu> wrote:
> On Fri, Dec 17, 2010 at 4:48 PM, Jim Fulton <jim at zope.com> wrote:
>>> Although specifying a dependency
>>> without a version shouldn't be a good practice,
>>
>> Actually, it's best practice.
>
> I don't think so. It's what people does, removing packages from pypi
> it's common, but it's awful.
>
> As a library mantainer, I can never break an API, not even if I
> release 2.0 after working on 1.0, because projects depending on my lib
> might just break.

Right. So you shouldn't break an API.

>
> Hence we get "project" "project2" "project3", e.g. versioning through
> project naming. Is that a good idea?

No.  I don't see how that follows.

In your original example, you had a distribution that depended on a
specific version of a project. That was a very bad idea.

In your example above, you have version 1 and version 2.  Let's assume
that version 2 ads some new API.

A client that works with version 1 should work just fine with version
2.  So it has no need to specify a version.  A client written against
version 2 might require at least version 2.  In that case it should
require a version >= 2.  It should not require version 2 (==2).

If all clients only require minimum versions, then you'll never get a
version conflict.  It's when clients specify maximum versions (of
which specifying a specific version is a case) that you potentially
get into trouble.  This might be needed in rare cases if a library
makes a breaking change, but that is an exception.

> Consider the Java world, where Maven is a de-facto standard. Once a
> package with a certain version is uploaded, it's usually never
> removed. You can decide to depend on the latest 3.x.x, on the latest
> 3.5.x, on version ranges, or on a very specific versions. This makes
> it possibile for developers to depend on the versions they really
> need.

Yeah, and buildout works the same way.

> Dependency problems may arise whatsoever and must be usually manually
> fixed, but in this case the Python way seems to say "hey, let's
> pretend the problem doesn't exist and the API never changes in a
> backwards incompatible way".

You said above that libraries shouldn't make breaking changes.  Now
you seem to be saying people should assume they do.  I'm not sure what
you're trying to say.

Of course, libraries do sometimes make breaking changes.  Any
responsible developer tries to avoid this.  A client that assumes a
library makes breaking changes should switch to an alternate library
if possible.

When dealing with breaking changes, it's generally easier to deal with
that at the level of application assembly, which is why buildout
provides a number of ways to control versions.

Being highly restrictive at the distribution level (unless the
distribution is, itself an application) is counter productive.


>>> it seems quite common
>>> - mostly because many people have got full control of the pypi section
>>> on their own repo and they can decide what's offered there, so they
>>> just specify "mydependency" on their other projects. I think it's
>>> quite a wide assumption, by the way.
>>
>> It's best practice.  Requiring a specific version is extremely inflexible.
>> If everyone did it, you'd never be able to assemble anything of any size.
>
> You might require a version range - I think I asked something about
> version specification, which is quite fuzzy in setuptools/distribute,
> some weeks ago.

A lower bound is fine. An upper bound is sometimes necessary, but you
want to be as nonrestrictive as possible,

> BTW if your require the API exposed by a project at version, let's
> say, 1.4 (or "at least 1.4 but less than 2.0") it won't do any good
> not to specify your version. You're effectively depending on the state
> of an external entity which is not under your control.
>
> Scenario:
> You project A today works 100% with a certain version of lib FOO,
> which is, let's say, 1.0.0, but you don't explicitly write that
> version requirement.
>
> In a year, you get back to project A, run the tests, and everything
> fails, because today's lib FOO 1.2.0 API changed.

That happens sometimes.  At that point, you should consider firing
project FOO.

How you approach this depends on whether project A is a top-level
application or a library module.  If it's an application, then nailing
all of the versions is a good idea.  This can be done via distribution
dependencies or via a buildout.  If it's a library, then nailing all
of the versions will cause a lot of pain for people trying to use your
library.

...

>> Maybe system packing tools use better dependency-resolution
>> mechanisms.  It could happen! I have certainly been stymied by
>> version conflicts in system packaging systems, so I don't think their
>> algorithms are that great.
>
> Version conflicts of course can arise. If there's no possible
> dependency path, nor yum neither apt will be able to do anything.

In all the cases where I've run into version conflicts, there have
been valid configurations, but the system couldn't find them without
me sorting it out manually.

>> The way to work around this with buildout is to use a buildout versions
>> section:
>>
>>  http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used
>>
>> It would be interesting to see if a breadth first strategy would provide better
>> behavior.
>
> I'll take a look at this and see what it happens. But I think we need
> to rethink the way versioning is handled.

This has been discussed at length in the past.  The consensus, AFAIK
is to be as nonrestrictive as possible in library distributions. Use
lower bounds when you depend on a new feature.  Use upper bounds if
you know you have to, but you probably don't know this at the time you
make the distribution.  Be really annoyed at the library that makes
you do this.  Applications with high availability requirements should
specify as many versions as possible.

There was a proposal floated to routinely set upper bounds to
dependency major versions. For example, if you use foo 1.2, you might
set an upper bound to foo <= 2dev, but almost no one has done this in
practice because it's too much of a pain.

There's no question that a better search algorithm would be
preferable.  It would be awesome if buildout could always find an
acceptable set of versions (assuming there is one). It would be
awesomer if it could find a best one. :) This could be rather hard.
Patches (to pkg_utils or a distribute equivalent)

Jim

--
Jim Fulton


More information about the Distutils-SIG mailing list