[Distutils] Library instability on PyPI and impact on OpenStack

[Mark McLoughlin]

Hey,

On Thu, 2013-02-28 at 15:39 +0000, Mark McLoughlin wrote:
...
> However, OpenStack is starting to get burned more often and some are
> advocating taking the second approach to managing our dependencies:
> 
>   http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6014
>   http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.html#6041
>   http://lists.openstack.org/pipermail/openstack-dev/2012-November/002075.html
> 
> It's probably not worthwhile for everyone to try and read the nuances of
> those threads. The tl;dr is we're hurting and hurting bad. Is this a
> problem the OpenStack and Python communities want to solve together? Or
> does the Python community fundamentally seem themselves as taking the
> same approach as the Ruby and Java communities?
...

I've tried to digest PEP426 and collected my thoughts below. Apologies
if you read through them and they offer you nothing useful.

I'm trying to imagine the future we're driving towards here where
OpenStack is no longer suffering and how PEP426 helped get us there

e.g. where

  - Many libraries use semantic versioning and OpenStack specifies x.y 
    compatible versions in their dependency lists. Incompatible changes 
    only get made in x+N.y and OpenStack continues using the x.y+N.z 
    versions.

  - All the libraries which don't use semantic versioning use another 
    clear versioning scheme that allows OpenStack to avoid incompatible 
    updates while still using compatible updates.

  - Incompatible versions of the same library are routinely installed 
    in parallel. Does PEP426 here, or is all the work to be done in 
    tools like PyPI, pip, setuptools, etc. Apps somehow specify which
    version they want to use since the incompatible versions of the 
    library use the same namespace.

There are fairly large gaps in my understanding of the plan here. How
quickly will we see libraries adopt PEP426? Will python 2.6/2.7 based
projects be able to consume these? How do parallel installs work in
practice? etc.

Thanks,
Mark.

== Basic Principles ==

What we (OpenStack) need from you (python library maintainers):

  1) API stability or, at least, predictable instability. We'd like to
     say "we require version N or any compatible later version"
     because just saying "we require version N" means our users can't
     use our application with later incompatible versions of your
     library.

  2) Support for installing multiple incompatible versions at
     once[*]. If there are two applications in the same
     system/distribution that require incompatible versions of the
     same library, you need to support having both installed at
     once.

[1] - throws me back to Havoc Pennington's way-back-when essay on
parallel installable incompatible versions of GNOME libraries:

      http://www106.pair.com/rhp/parallel.html

== Versioning ==

Semantic versioning is appealing here because, assuming all libraries
adopt it, it becomes very easy for us to predict which versions will
be incompatible.

For any API unstable library (0.x in semantic versioning), we need to
pin to a very specific version and require distributions to package
that exact version in order to run OpenStack. When moving to a newer
version of the library, we need to move all OpenStack projects at
once. Ideally, we'd just avoid such libraries.

Implied in semantic versioning, though, is that it's possible for
distributions to include both version X.y.z and X+N.y.z

== PEP426 ==

=== Date Based Versioning ===

OpenStack uses date based versioning for its server components and
we've begun using it for the Oslo libraries described above. PEP426
says:

  Date based release numbers are explicitly excluded from
  compatibility with this scheme, as they hinder automatic translation
  to other versioning schemes, as well as preventing the adoption
  of semantic versioning without changing the name of the
  project. Accordingly, a leading release component greater than or
  equal to 1980 is an error.

This seems odd and the rationale seems week. It looks like an
abritrary half-way house between requiring semantic versioning and
allowing any scheme of increasing versions.

I'm really not sure how we'd deal with this gracefully - assuming we
have releases out there which require e.g. oslo-config>=2013.1 and we
switch to:

  Version: 1.2
  Private-Version: 2013.2

then is the >=2013.1 requirement satisfied by the private version? If
not, should we just go ahead and take the pain now of releasing 2013.1
as 1.1?

=== Compatible Release Version Specifier ===

I know this is copied from Ruby's spermy operator, but it seems to be
to be fairly useless (overly pessimistic, at least) with semantic
versioning.

i.e. assuming x.y.z releases, where an increase in X means that
incompatible changes have been made then:

  ~>1.2.3

unnecessarily limits you to the 1.2.y series of releases. Now, in an
ideal world, you would just say:

  ~>1.2

and you're good for the entire 1.x.y but often you know what your app
doesn't work with 1.2.2 and there's a fix in 1.2.3 that your app
absolutely needs.

A random implementation detail I don't get here ... we use 'pip install
-r' a lot in OpenStack and so have files with e.g.

  PasteDeploy>=1.5.0

how will you specify compatible releases in these files? What's the
equivalent of

  PasteDeploy~>1.5.0

?

=== Multiple Release Streams ===

If version 1.1.0 and version 1.2.0 were already published to PyPI,
we'd like to be able to release a 1.1.1 bugfix update. I saw some
mention that PyPI would then treat 1.1.1 as the latest release, but I
can't find where I read that now.

Could someone confirm this won't be an issue in future. I guess I'm
unclear how the PEP relatest to problems with PyPI specifically.

=== Python 2 vs 3 ===

Apparently, part of our woes are down to library maintainers releasing
a version of their library that isn't supported on Python 2 (OpenStack
currently supports running on 2.6 and 2.7).

This is obviously an incompatible change from our perspective. Going
back the basic principles above, we'd like to be able to:

  1) specify that we "require version N or any later compatible
     version that supports python 2.6. and 2.7".

  2) have both the python 2 and python 3 compatible versions installed
     on the same system or included in the same distribution.

Environment markers in PEP426 sound tantalizingly close to (1) but
AFAICT the semantics of markers are "the set of things that must be
true for this field to be considered" i.e.

  Requires-Dist: PasteDeploy (>=1.5.0); python_version == '2.6' or python_version == '2.7'

means PasteDeploy is required on python 2.6/2.7 installs, but not
otherwise.

I really don't know what the story with (2) is.

== Oslo Libraries ==

The approach we had planned to take to versioning for the Oslo family
of libraries in OpenStack is:

  - APIs can be marked as deprecated in a release, kept around for
    another release and then removed.  For example, if we deprecated
    an API in the H release it would not be remove until 1 year later
    in the J release.

  - If we wanted to go down the route of a version with incompatible
    API changes with no deprecation period for the old APIs, we'd
    actually just introduce the new APIs under a new library name
    like oslo-config2. We could continue to support the old APIs as a
    separate project for a time.

This is an approach tailored to Oslo's use case as a library used by
server projects on a 6 month coordinated release cycle. It allows
servers from the H release to use Oslo libraries from the I release,
further allowing servers from the H and I release to be in the same
system/distribution.

To encode this in a requirements file, I guess we'd do:

  oslo-config>=2013.1,<2014.1

although that does mean predicting that 2014.1 changes which are
incompatible with 2013.1, even though it may not actually do so.

This approach doesn't work so well for "normal" Python libraries
because it's impossible to predict when it will be reasonable to
remove a deprecated API. Who's to say that no-one will have a 3 year
old application installed on their system that they expect to still
run fine even if they update to a newer version of your library?