[Distutils] pbr issues (was: Where should I put tests when packaging python modules?)

[Robert Collins]

On 8 October 2015 at 04:32, Erik Bray <erik.m.bray at gmail.com> wrote:
> Starting a sub-thread since issues with pbr are orthogonal to the
> original disucssion.
>
> But one point I'd like to raise about this is that when I originally
> designed d2to1, on which a chunk of pbr is based, it was *explicitly*
> designed to never be installed in site-packages (with the exception of
> downstream packaging systems which can do what they want and are more
> controlled).  This is exactly because I knew different packages might
> have dependencies on different versions of d2to1 as features are
> added, and that if some version is installed in site-packages it can
> lead to VersionConflict issues (this is in part exacerbated by a
> bug/misfeature in setuptools--I fixed that bug a while ago but the fix
> had to be rolled back due to a regression [1]).

So yes - that principle makes a lot of sense. There are two factors for pbr.

a) as Donald mentions, there's a runtime API too - but we could indeed
split that out into a separate package, if it would help things.

b) more importantly, in OpenStack infra we can't use easy-install -
its inconsistency with pip and lack of updated handling for wheels,
HTTPS, separate configuration - consistently cause headaches every
single time it happens. So many years ago we put in place
pre-installation of all known build-time dependencies - we just
install them statically, because we find the effort required to make
them be compatible is less than the headaches from easy-install. As
such pbr has a hard API requirement: thou shalt be backwards
compatible.

Clearly b) can hit VersionConflicts if pbr (and any other build
dependencies like setuptools itself) are out of date, but that is
easily handled for automated environments (pip install -U pip
setuptools wheel pbr && echo YAY), and it errors cleanly enough for
hand-use by folk that its a decent enough tradeoff in our experience.

> I don't know what features pbr has grown that might make someone want
> it to be a runtime dependency (the only entry-points I noticed were
> for adding egg-info writers but that should only be needed at
> build-time too...), but maybe something like that should be split off
> as a separate module or something...

Its all about easy-install. This is why I put a proof of concept
static-setup-requires thing together for pip (and its on our teams
roadmap to deliver a production version of it via patches to all the
pypa projects, we're just not at that point yet - the resolver is
first, and we have to finish rolling out constraints within OpenStack
before that gets large timeslices).

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud