[Distutils] Library instability on PyPI and impact on OpenStack
Mark McLoughlin
markmc at redhat.com
Mon Mar 4 18:23:26 CET 2013
Hi Nick,
Thanks for the detailed reply. I'll stick to PEP426 related topics in
this mail.
Generally speaking, PEP426 looks like good progress, but the biggest
problem I see now is the lack of parallel installs for incompatible
versions.
On Mon, 2013-03-04 at 18:11 +1000, Nick Coghlan wrote:
> On Mon, Mar 4, 2013 at 1:54 AM, Mark McLoughlin <markmc at redhat.com> wrote:
> > I've tried to digest PEP426 and collected my thoughts below. Apologies
> > if you read through them and they offer you nothing useful.
> >
> > I'm trying to imagine the future we're driving towards here where
> > OpenStack is no longer suffering and how PEP426 helped get us there
> >
> > e.g. where
> >
> > - Many libraries use semantic versioning and OpenStack specifies x.y
> > compatible versions in their dependency lists. Incompatible changes
> > only get made in x+N.y and OpenStack continues using the x.y+N.z
> > versions.
>
> Yep, PEP 426 pushes projects heavily in that direction by making it
> the path of least resistance. You *can* do something different if
> necessary, it's just less convenient than simply using the first 9
> clauses of semantic versioning.
>
> > - All the libraries which don't use semantic versioning use another
> > clear versioning scheme that allows OpenStack to avoid incompatible
> > updates while still using compatible updates.
>
> Yes, the PEP will explicitly encourage projects to document how to
> define dependencies if the default "compatible release" clause isn't
> appropriate.
Ok, understood.
Is there a case for putting this information into the metadata?
e.g. if this was version 1.2.3 of a library and the project's policy is
to bump the major version if an incompatible API change is made, then
you can do:
Requires-Dist: foo (1.2)
or:
Requires-Dist: foo (1.2,>=1.2.3)
or:
Requires-Dist: foo (>=1.2.3,<2)
but how about if foo-1.2.3 contained:
Compatible-With: 1
and foo-2.0.0 contained:
Compatible-With: 2
then this:
Requires-Dist: foo (1.2)
could reject 2.0.0 as satisfying the compatible release specification.
> > == PEP426 ==
> >
> > === Date Based Versioning ===
> >
> > OpenStack uses date based versioning for its server components and
> > we've begun using it for the Oslo libraries described above. PEP426
> > says:
> >
> > Date based release numbers are explicitly excluded from
> > compatibility with this scheme, as they hinder automatic translation
> > to other versioning schemes, as well as preventing the adoption
> > of semantic versioning without changing the name of the
> > project. Accordingly, a leading release component greater than or
> > equal to 1980 is an error.
> >
> > This seems odd and the rationale seems week. It looks like an
> > abritrary half-way house between requiring semantic versioning and
> > allowing any scheme of increasing versions.
>
> The assumption is that the version field will follow semantic
> versioning (that's not where the PEP started, it was originally
> non-committal on the matter like PEP 345 - however I've come to the
> conclusion that this assumption needs to be made explicit, but still
> need to update various parts of the PEP that are still wishy-washy
> about it). However, we can't actually enforce semantic versioning
> because we can't automatically detect if a project is adhering to
> those semantics. What we *can* enforce is the rejection of version
> numbers that obviously look like years rather than a semantic version.
I sympathise with wanting to do *something* here. Predictable
instability was the first thing I listed above and semantic versioning
would give that.
However, I also described the approach I'm taking to predictable
instability with the Oslo libraries. Any APIs we want to remove will be
marked as deprecated for a year, then removed.
We can encoded that as:
oslo-config>=2013.1,<2014.1
if Compatible-With existed, we could do:
Compatible-With: 2013.1,2013.2
If we used semantic versioning strictly, we'd bump the major release
every 6 months (since every release may see APIs which were deprecated a
year ago removed).
Basically, PEP426 desires to enforce some meaning around the major
number and API instability, while not enforcing a meaning of API
stability for the micro version.
I'm not sure it achieves a whole lot except making my life harder
without actually forcing me into adopting a more predictable versioning
scheme.
And to be clear - this approach of removing APIs after a year is because
we assume only other OpenStack projects are using Oslo libraries. If
that changed, we'd probably keep deprecated APIs around for a lot
longer.
> > === Compatible Release Version Specifier ===
> >
> > I know this is copied from Ruby's spermy operator, but it seems to be
> > to be fairly useless (overly pessimistic, at least) with semantic
> > versioning.
> >
> > i.e. assuming x.y.z releases, where an increase in X means that
> > incompatible changes have been made then:
> >
> > ~>1.2.3
> >
> > unnecessarily limits you to the 1.2.y series of releases. Now, in an
> > ideal world, you would just say:
> >
> > ~>1.2
> >
> > and you're good for the entire 1.x.y but often you know what your app
> > doesn't work with 1.2.2 and there's a fix in 1.2.3 that your app
> > absolutely needs.
>
> Handling that kind of situation is why version specifiers allow
> multiple clauses that are ANDed together:
>
> mydependency (1.2, >= 1.2.3)
Thanks.
> > A random implementation detail I don't get here ... we use 'pip install
> > -r' a lot in OpenStack and so have files with e.g.
> >
> > PasteDeploy>=1.5.0
> >
> > how will you specify compatible releases in these files? What's the
> > equivalent of
> >
> > PasteDeploy~>1.5.0
> >
> > ?
>
> Note that the pkg_resources specifier format (no parens allowed) is
> NOT the same as the PEP 345/426 format (parens required). The former
> is designed to work with arbitrary version schemes, while the latter
> is explicitly constrained to version schemes that abide by the
> constraints described in PEP 386. I expect the setuptools/distribute
> developers will choose a tilde-based operator to represent compatible
> release clauses in the legacy formats (likely Ruby's ~>, although it
> could be something like ~= or ~~ instead).
Ok, figures.
Cheers,
Mark.
More information about the Distutils-SIG
mailing list