[Distutils] Library instability on PyPI and impact on OpenStack

[Daniel Holth]

On Mon, Mar 4, 2013 at 12:23 PM, Mark McLoughlin <markmc at redhat.com> wrote:
> Hi Nick,
>
> Thanks for the detailed reply. I'll stick to PEP426 related topics in
> this mail.
>
> Generally speaking, PEP426 looks like good progress, but the biggest
> problem I see now is the lack of parallel installs for incompatible
> versions.

I don't know what to do with the "runtime linker" feature provided by
setuptools' pkg_resources or Ruby's Gem; the approach taken is to
treat it as a separate problem from the "project name, version, and
list of requirements" that 426 tries to specify. PEP 376 (database of
installed Python distributions; defines the .dist-info directory) has
more to do with whether parallel installs happen. Oh, do you mean two
incompatible versions loaded during the same invocation?

Due to the way JavaScript works (no global import namespace), npm
seems to allow dependencies to have their own private instances of
their sub-dependencies.

> On Mon, 2013-03-04 at 18:11 +1000, Nick Coghlan wrote:
>> On Mon, Mar 4, 2013 at 1:54 AM, Mark McLoughlin <markmc at redhat.com> wrote:
>> > I've tried to digest PEP426 and collected my thoughts below. Apologies
>> > if you read through them and they offer you nothing useful.
>> >
>> > I'm trying to imagine the future we're driving towards here where
>> > OpenStack is no longer suffering and how PEP426 helped get us there
>> >
>> > e.g. where
>> >
>> >   - Many libraries use semantic versioning and OpenStack specifies x.y
>> >     compatible versions in their dependency lists. Incompatible changes
>> >     only get made in x+N.y and OpenStack continues using the x.y+N.z
>> >     versions.
>>
>> Yep, PEP 426 pushes projects heavily in that direction by making it
>> the path of least resistance. You *can* do something different if
>> necessary, it's just less convenient than simply using the first 9
>> clauses of semantic versioning.
>>
>> >   - All the libraries which don't use semantic versioning use another
>> >     clear versioning scheme that allows OpenStack to avoid incompatible
>> >     updates while still using compatible updates.
>>
>> Yes, the PEP will explicitly encourage projects to document how to
>> define dependencies if the default "compatible release" clause isn't
>> appropriate.
>
> Ok, understood.
>
> Is there a case for putting this information into the metadata?
>
> e.g. if this was version 1.2.3 of a library and the project's policy is
> to bump the major version if an incompatible API change is made, then
> you can do:
>
>   Requires-Dist: foo (1.2)
>
> or:
>
>   Requires-Dist: foo (1.2,>=1.2.3)
>
> or:
>
>   Requires-Dist: foo (>=1.2.3,<2)
>
> but how about if foo-1.2.3 contained:
>
>   Compatible-With: 1
>
> and foo-2.0.0 contained:
>
>   Compatible-With: 2
>
> then this:
>
>   Requires-Dist: foo (1.2)
>
> could reject 2.0.0 as satisfying the compatible release specification.
>
>> > == PEP426 ==
>> >
>> > === Date Based Versioning ===
>> >
>> > OpenStack uses date based versioning for its server components and
>> > we've begun using it for the Oslo libraries described above. PEP426
>> > says:
>> >
>> >   Date based release numbers are explicitly excluded from
>> >   compatibility with this scheme, as they hinder automatic translation
>> >   to other versioning schemes, as well as preventing the adoption
>> >   of semantic versioning without changing the name of the
>> >   project. Accordingly, a leading release component greater than or
>> >   equal to 1980 is an error.
>> >
>> > This seems odd and the rationale seems week. It looks like an
>> > abritrary half-way house between requiring semantic versioning and
>> > allowing any scheme of increasing versions.
>>
>> The assumption is that the version field will follow semantic
>> versioning (that's not where the PEP started, it was originally
>> non-committal on the matter like PEP 345 - however I've come to the
>> conclusion that this assumption needs to be made explicit, but still
>> need to update various parts of the PEP that are still wishy-washy
>> about it). However, we can't actually enforce semantic versioning
>> because we can't automatically detect if a project is adhering to
>> those semantics. What we *can* enforce is the rejection of version
>> numbers that obviously look like years rather than a semantic version.
>
> I sympathise with wanting to do *something* here. Predictable
> instability was the first thing I listed above and semantic versioning
> would give that.

We should take it all back and derive the whole version number from a
sha hash, e.g. the Mercurial revision number. A Previous-Version: tag
(multiple-use of course) lets you assemble a DAG of the release
history. Just kidding :-)

> However, I also described the approach I'm taking to predictable
> instability with the Oslo libraries. Any APIs we want to remove will be
> marked as deprecated for a year, then removed.
>
> We can encoded that as:
>
>   oslo-config>=2013.1,<2014.1
>
> if Compatible-With existed, we could do:
>
>   Compatible-With: 2013.1,2013.2
>
> If we used semantic versioning strictly, we'd bump the major release
> every 6 months (since every release may see APIs which were deprecated a
> year ago removed).
>
> Basically, PEP426 desires to enforce some meaning around the major
> number and API instability, while not enforcing a meaning of API
> stability for the micro version.
>
> I'm not sure it achieves a whole lot except making my life harder
> without actually forcing me into adopting a more predictable versioning
> scheme.

We try :-)

I never dreamed it would be possible to force package producers to not
break things with their new releases. I think you just have to know
your publishers and do plenty of testing.

The goal of the version specifiers section of PEP 426 is only to
define a sort order and comparison operators. We threw in semantic
versioning to recommend something more sane than the "mostly compat
with pkg_resources" behavior that we are stuck with until the indices
get smarter. I never hoped to be able to force packagers to actually
do a good job of maintaining backwards compatibility. For example,
zipimport._zip_directory_cache... it's there, and people started
depending on it, and it's become part of the API even though it was
never meant to be one. If I remove it, which version number or
compatible-with tag do I have to specify?

Take a look at https://crate.io/packages/apipkg/ ; it is a neat way to
make it clear which parts of your program are the API.