[Distutils] Library instability on PyPI and impact on OpenStack

[Nick Coghlan]

On Tue, Mar 5, 2013 at 10:27 PM, Mark McLoughlin <markmc at redhat.com> wrote:
> Ok, there's a tonne of details there about pip, virtualenv and .pth
> files that are going over my head right now, but the general idea I'm
> taking away is:
>
>   - the system has multiple versions of somedep installed under /usr
>     somewhere
>
>   - the latest version (2.1) is what you get if you naively just do
>     'import somedep'
>
>   - most applications should instead somehow explicitly say they need
>     ~>1.3, ->1.6 or ~->2.0 or whatever
>
>   - distros would have multiple copies of the same library, but only
>     one copy for each incompatible stream rather than one copy for each
>     application
>
> That's definitely workable.

That's the general idea. There's a fair bit of work needed to get
there though, and the on-disk layouts for the distros would look
fairly different from the current approach of just dumping everything
in Python's site-packages directory. It also won't solve *all*
compatibility problems, so the distros may still be on the hook to
carry some compatibility patches in order to get a common dependency
that works for everyone.

(To be honest, I expect software collections will end up needing a
similar capability, for all the reasons you gave earlier in response
to the suggestion of just using virtualenv with the existing simple
bundling approach)

>> If any distros want that kind of thing to become a reality, though,
>> they're going to have to step up and contribute it. As noted above,
>> for the current tool development teams, the focus is on distributing,
>> maintaining and deploying cross-platform applications, not making it
>> easy to do security updates on a Linux distro. I believe it's possible
>> to satisfy both parties, but it's going to be up to the distros to
>> offer a viable plan for meeting their needs without disrupting
>> existing upstream practices.
>
> Point well taken.
>
> However, I am surprised the pendulum has swung such that Python platform
> maintainers only worry about app maintainers and not distro maintainers.

If you saw my rant on the Fedora python-devel list, you may have
guessed that a lot of the expressed frustration with the distro
approach and the lack of willingness to understand why bundling is
such an attractive option for cross-platform development comes from me
personally, rather than the Python community in general. However,
there's also a general dearth of distro people on the upstream
packaging lists to balance the web app developers - I'm one of the
ones that is *most sympathetic* to the distro point of view, and that
should worry you a bit.

> And also, Python embracing incompatible updates and bundling is either a
> new upstream practice or just that doesn't well understood on the distro
> side.

The popularity of bundling is primarily in devops and integrated
application stacks. You can't deploy on Windows without bundling, and
when you have to manage your own security updates for other platforms
anyway, the distros refusing to accommodate that model is just
frustrating. The distro packagers say "we don't trust the developers
of all of the apps we ship to provide timely security fixes" and the
app developers say "we don't trust the packagers of all of the distros
we support to provide timely security fixes" and everybody gets
annoyed with each other because their concerns and priorities are so
wildly divergent.

And we're definitely *not* embracing incompatible updates, we're
acknowledging their inevitable existence in volunteer driven projects,
and defining a way to at least communicate them clearly when they
happen.

>> I will note that making this kind of idea more feasible is one of the
>> reasons I am making "compatible release" the *default* in PEP 426
>> version specifiers, but it still needs people to actually figure out
>> the details and write the code.
>
> I still think that going down this road without the parallel installs
> issue solved is a dangerous move for Python. Leaving aside pain for
> distros for a moment, there was a perception (perhaps misguided) that
> Python was a more stable platform that e.g. Ruby or Java.
>
> If Python library maintainers will see PEP426 as a license to make
> incompatible changes more often so long as they bump their major number,
> then that perception will change.

This isn't a concern I had considered, but I'll review the wording in
the PEP with that in mind (there's a bunch of other stuff I need to
fix in the version scheme description anyway).


> By bundling, I mean that an app sees itself as in control of the
> versions of its dependencies. The app developer fundamentally thinks she
> is delivering a specific stack of dependencies and her application code
> on top rather than installing just their app and running it on a stable
> platform.

I'm actually trying to push *against* that by making the "compatible
release" specifier the default in PEP 426 rather than requiring a
separate operator (if you look at PEP 345, the predecessor that
defines metadata 1.2, it pinned dependencies by default). Actual
pinning (with "==") should be limited to devops type situations, where
an application really is explicitly controlling it's entire stack, but
that's not the kind of metadata anyone should be publishing on PyPI.

I can see how the perception of the current PEP might be different
without that background of knowing precisely what the previous version
said, though.

> Anyway, sounds like we have some ideas for parallel installs we can
> investigate.

Yes, I'm definitely not opposed to the idea of parallel installs - I'm
just opposed to the idea of parallel install systems that rely on
changes to PyPI packages in order for them to work properly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia