[Distutils] Library instability on PyPI and impact on OpenStack

Mark McLoughlin markmc at redhat.com
Mon Mar 4 18:23:26 CET 2013


Hi Nick,

Thanks for the detailed reply. I'll stick to PEP426 related topics in
this mail.

Generally speaking, PEP426 looks like good progress, but the biggest
problem I see now is the lack of parallel installs for incompatible
versions.

On Mon, 2013-03-04 at 18:11 +1000, Nick Coghlan wrote:
> On Mon, Mar 4, 2013 at 1:54 AM, Mark McLoughlin <markmc at redhat.com> wrote:
> > I've tried to digest PEP426 and collected my thoughts below. Apologies
> > if you read through them and they offer you nothing useful.
> >
> > I'm trying to imagine the future we're driving towards here where
> > OpenStack is no longer suffering and how PEP426 helped get us there
> >
> > e.g. where
> >
> >   - Many libraries use semantic versioning and OpenStack specifies x.y
> >     compatible versions in their dependency lists. Incompatible changes
> >     only get made in x+N.y and OpenStack continues using the x.y+N.z
> >     versions.
> 
> Yep, PEP 426 pushes projects heavily in that direction by making it
> the path of least resistance. You *can* do something different if
> necessary, it's just less convenient than simply using the first 9
> clauses of semantic versioning.
> 
> >   - All the libraries which don't use semantic versioning use another
> >     clear versioning scheme that allows OpenStack to avoid incompatible
> >     updates while still using compatible updates.
> 
> Yes, the PEP will explicitly encourage projects to document how to
> define dependencies if the default "compatible release" clause isn't
> appropriate.

Ok, understood.

Is there a case for putting this information into the metadata?

e.g. if this was version 1.2.3 of a library and the project's policy is
to bump the major version if an incompatible API change is made, then
you can do:

  Requires-Dist: foo (1.2)

or:

  Requires-Dist: foo (1.2,>=1.2.3)

or:

  Requires-Dist: foo (>=1.2.3,<2)

but how about if foo-1.2.3 contained:

  Compatible-With: 1

and foo-2.0.0 contained:

  Compatible-With: 2

then this:

  Requires-Dist: foo (1.2)

could reject 2.0.0 as satisfying the compatible release specification.

> > == PEP426 ==
> >
> > === Date Based Versioning ===
> >
> > OpenStack uses date based versioning for its server components and
> > we've begun using it for the Oslo libraries described above. PEP426
> > says:
> >
> >   Date based release numbers are explicitly excluded from
> >   compatibility with this scheme, as they hinder automatic translation
> >   to other versioning schemes, as well as preventing the adoption
> >   of semantic versioning without changing the name of the
> >   project. Accordingly, a leading release component greater than or
> >   equal to 1980 is an error.
> >
> > This seems odd and the rationale seems week. It looks like an
> > abritrary half-way house between requiring semantic versioning and
> > allowing any scheme of increasing versions.
> 
> The assumption is that the version field will follow semantic
> versioning (that's not where the PEP started, it was originally
> non-committal on the matter like PEP 345 - however I've come to the
> conclusion that this assumption needs to be made explicit, but still
> need to update various parts of the PEP that are still wishy-washy
> about it). However, we can't actually enforce semantic versioning
> because we can't automatically detect if a project is adhering to
> those semantics. What we *can* enforce is the rejection of version
> numbers that obviously look like years rather than a semantic version.

I sympathise with wanting to do *something* here. Predictable
instability was the first thing I listed above and semantic versioning
would give that.

However, I also described the approach I'm taking to predictable
instability with the Oslo libraries. Any APIs we want to remove will be
marked as deprecated for a year, then removed.

We can encoded that as:

  oslo-config>=2013.1,<2014.1

if Compatible-With existed, we could do:

  Compatible-With: 2013.1,2013.2

If we used semantic versioning strictly, we'd bump the major release
every 6 months (since every release may see APIs which were deprecated a
year ago removed).

Basically, PEP426 desires to enforce some meaning around the major
number and API instability, while not enforcing a meaning of API
stability for the micro version.

I'm not sure it achieves a whole lot except making my life harder
without actually forcing me into adopting a more predictable versioning
scheme.

And to be clear - this approach of removing APIs after a year is because
we assume only other OpenStack projects are using Oslo libraries. If
that changed, we'd probably keep deprecated APIs around for a lot
longer.

> > === Compatible Release Version Specifier ===
> >
> > I know this is copied from Ruby's spermy operator, but it seems to be
> > to be fairly useless (overly pessimistic, at least) with semantic
> > versioning.
> >
> > i.e. assuming x.y.z releases, where an increase in X means that
> > incompatible changes have been made then:
> >
> >   ~>1.2.3
> >
> > unnecessarily limits you to the 1.2.y series of releases. Now, in an
> > ideal world, you would just say:
> >
> >   ~>1.2
> >
> > and you're good for the entire 1.x.y but often you know what your app
> > doesn't work with 1.2.2 and there's a fix in 1.2.3 that your app
> > absolutely needs.
> 
> Handling that kind of situation is why version specifiers allow
> multiple clauses that are ANDed together:
> 
>   mydependency (1.2, >= 1.2.3)

Thanks.

> > A random implementation detail I don't get here ... we use 'pip install
> > -r' a lot in OpenStack and so have files with e.g.
> >
> >   PasteDeploy>=1.5.0
> >
> > how will you specify compatible releases in these files? What's the
> > equivalent of
> >
> >   PasteDeploy~>1.5.0
> >
> > ?
> 
> Note that the pkg_resources specifier format (no parens allowed) is
> NOT the same as the PEP 345/426 format (parens required). The former
> is designed to work with arbitrary version schemes, while the latter
> is explicitly constrained to version schemes that abide by the
> constraints described in PEP 386. I expect the setuptools/distribute
> developers will choose a tilde-based operator to represent compatible
> release clauses in the legacy formats (likely Ruby's ~>, although it
> could be something like ~= or ~~ instead).

Ok, figures.

Cheers,
Mark.



More information about the Distutils-SIG mailing list