[Distutils] Buildout or distribute raise unnecessary version conflict?

[Alan Franzoni]

On Fri, Dec 17, 2010 at 5:41 PM, Jim Fulton <jim at zope.com> wrote:
>> As a library mantainer, I can never break an API, not even if I
>> release 2.0 after working on 1.0, because projects depending on my lib
>> might just break.
>
> Right. So you shouldn't break an API.
>
>>
>> Hence we get "project" "project2" "project3", e.g. versioning through
>> project naming. Is that a good idea?
>
> No.  I don't see how that follows.

Usually versions have meaning; a typical approach to versioning -
python itself works in a very similar way - is:

version = MAJOR.MINOR.REVISION

 in a new REVISION > no functional change should be done, only bugfixes.
in a new MINOR -> any backwards incompatible change should be
minimized, and there should be a "grace period" , e.g. if something
must be removed, if it works fine in MINOR, it should be deprecated in
MINOR + 1 and then can be removed in MINOR + 2. This
in a new MAJOR -> everything can be changed, APIs can be broken, etc. etc. etc.

The only way to currently achieve the MAJOR change effect with current
python tools and best practices is to create a new project name.
Otherwise old clients might break

> In your original example, you had a distribution that depended on a
> specific version of a project. That was a very bad idea.

Sure it was, and it was just an example in order to show the error to
the ML. This is not what I usually want to do.

> If all clients only require minimum versions, then you'll never get a
> version conflict.  It's when clients specify maximum versions (of
> which specifying a specific version is a case) that you potentially
> get into trouble.  This might be needed in rare cases if a library
> makes a breaking change, but that is an exception.

It's not so rare as you may expect. I encounter it every day, and
that's why I'm starting to think we might not be doing the best thing.

An example: a recent change in the minor in zope.testing (3.9 to 3.10)
recently broke collective.xmltestreport, since an API changed (a
subpackage was removed).

I find th

>> Consider the Java world, where Maven is a de-facto standard. Once a
>> package with a certain version is uploaded, it's usually never
>> removed. You can decide to depend on the latest 3.x.x, on the latest
>> 3.5.x, on version ranges, or on a very specific versions. This makes
>> it possibile for developers to depend on the versions they really
>> need.
>
> Yeah, and buildout works the same way.

No. Most mavenized projects specify a dependency, maybe a SNAPSHOT
one, but at least the major is specified, and often the minor is
specified as well. Peeking through 1.0-SNAPSHOT deps is not uncommon
while developing. Also, whenever doing a release, Maven freezes the
version numbers the library uses.

And it's possibile to add dependency ranges as well, but Maven won't
download the latest version whatsoever - it'll try downloading the
best suited dependency.

Don't think I don't like zc.buildout; I do. I think it's great, and
this issue is probably not zc.buildout's fault. But if there's a
problem we should start thinking about fixing it before it's too late.

> You said above that libraries shouldn't make breaking changes.  Now
> you seem to be saying people should assume they do.  I'm not sure what
> you're trying to say.
>
> Of course, libraries do sometimes make breaking changes.  Any
> responsible developer tries to avoid this.  A client that assumes a
> library makes breaking changes should switch to an alternate library
> if possible.

I think that there should a *policy* on breaking changes. Sometimes
you must do them - Python 3 is a good example of that. But if I can't
rely on the fact the clients will specify a dep at least to my MAJOR
then such policies are useless.

> Being highly restrictive at the distribution level (unless the
> distribution is, itself an application) is counter productive.

What does "restrictive" mean in this context? Dependencies are just
metadata, after all. I'd like to say "hey, my lib works with FOOLIB
from v1.2 till 1.4.3 and BARLIB from 2.0 till 2.2". If you are able to
pick a version that satisfies both my requirements and yours, then
you're just happy. Otherwise you can resolve deps in whatever way you
like, but you shouldn't shout at me saying that my code doesn't work;
you're on your own.

Inviting people not to specify versions as a "best practice" means
"less information is best". Versioning info could just be ignored or
skipped, but if you want to use it I've got it. Otherwise I might just
need to go crawling for a dependency version to see which one might
fit the app or lib I'm trying - not exactly an exciting activity.

> A lower bound is fine. An upper bound is sometimes necessary, but you
> want to be as nonrestrictive as possible,

If there's a policy, this could be done. If there's no policy

> That happens sometimes.  At that point, you should consider firing
> project FOO.

Or you could specify a auitable version number or range. This usually
happens in many Java/Maven projects.

> In all the cases where I've run into version conflicts, there have
> been valid configurations, but the system couldn't find them without
> me sorting it out manually.

Yes, that could happen. But I've seen times where it actually works.

> This has been discussed at length in the past.  The consensus, AFAIK
> is to be as nonrestrictive as possible in library distributions. Use
> lower bounds when you depend on a new feature.  Use upper bounds if
> you know you have to, but you probably don't know this at the time you
> make the distribution.  Be really annoyed at the library that makes
> you do this.  Applications with high availability requirements should
> specify as many versions as possible.

Unless there's a widely recognized best practice and a clear policy,
it's very hard to "be annoyed" with libraries breaking APIs.


-- 
Alan Franzoni
--
contact me at public@[mysurname].eu