[Distutils] Library instability on PyPI and impact on OpenStack

Mark McLoughlin markmc at redhat.com
Mon Mar 4 23:30:10 CET 2013


On Mon, 2013-03-04 at 13:07 -0500, Daniel Holth wrote:
> On Mon, Mar 4, 2013 at 12:23 PM, Mark McLoughlin <markmc at redhat.com> wrote:
> > Hi Nick,
> >
> > Thanks for the detailed reply. I'll stick to PEP426 related topics in
> > this mail.
> >
> > Generally speaking, PEP426 looks like good progress, but the biggest
> > problem I see now is the lack of parallel installs for incompatible
> > versions.
> 
> I don't know what to do with the "runtime linker" feature provided by
> setuptools' pkg_resources or Ruby's Gem; the approach taken is to
> treat it as a separate problem from the "project name, version, and
> list of requirements" that 426 tries to specify. PEP 376 (database of
> installed Python distributions; defines the .dist-info directory) has
> more to do with whether parallel installs happen. Oh, do you mean two
> incompatible versions loaded during the same invocation?

Not sure, I'm following you but what I mean is where you have foo-1.2.3
required by program A and foo-2.0 (which isn't backwards compatible with
foo-1.2.3) required by program B.

Both are installed in the standard python path and both programs
explicitly say which versions they need.

We seem to be pretty close to this with eggs and pkg_resources etc., but
Nick seems to be concerned that it's very error prone.

...
> > However, I also described the approach I'm taking to predictable
> > instability with the Oslo libraries. Any APIs we want to remove will be
> > marked as deprecated for a year, then removed.
> >
> > We can encoded that as:
> >
> >   oslo-config>=2013.1,<2014.1
> >
> > if Compatible-With existed, we could do:
> >
> >   Compatible-With: 2013.1,2013.2
> >
> > If we used semantic versioning strictly, we'd bump the major release
> > every 6 months (since every release may see APIs which were deprecated a
> > year ago removed).
> >
> > Basically, PEP426 desires to enforce some meaning around the major
> > number and API instability, while not enforcing a meaning of API
> > stability for the micro version.
> >
> > I'm not sure it achieves a whole lot except making my life harder
> > without actually forcing me into adopting a more predictable versioning
> > scheme.
> 
> We try :-)
> 
> I never dreamed it would be possible to force package producers to not
> break things with their new releases. I think you just have to know
> your publishers and do plenty of testing.

We do a lot of testing:

  http://status.openstack.org/zuul/

but we also depend on a fairly large bunch of libraries:

  https://github.com/markmc/requirements/blob/652644c/tools/pip-requires
  https://github.com/markmc/requirements/blob/652644c/tools/test-requires

The conclusion here is that we need to go through each of those
libraries, talk to upstream and figure how to cap our version ranges to
avoid unexpected incompatible updates. That's a lot of publishers to
talk to!

> The goal of the version specifiers section of PEP 426 is only to
> define a sort order and comparison operators. We threw in semantic
> versioning to recommend something more sane than the "mostly compat
> with pkg_resources" behavior that we are stuck with until the indices
> get smarter. I never hoped to be able to force packagers to actually
> do a good job of maintaining backwards compatibility.

Rejecting date based versions looks to be trying to force packagers into
a scheme whereby the major version doesn't increment needlessly. If
PEP426 isn't trying to force publishers to do better, then this looks
like a pretty random potshot to take :)

>  For example,
> zipimport._zip_directory_cache... it's there, and people started
> depending on it, and it's become part of the API even though it was
> never meant to be one. If I remove it, which version number or
> compatible-with tag do I have to specify?

IMHO, it'd be fair game to just remove - it's obviously intended to be
private.

> Take a look at https://crate.io/packages/apipkg/ ; it is a neat way to
> make it clear which parts of your program are the API.

Interesting, indeed.

Cheers,
Mark.



More information about the Distutils-SIG mailing list