Re: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?)
On Tue, Oct 6, 2015 at 7:46 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins <robertc@robertcollins.net> wrote:
Hangon, there's clearly a *huge* gap in understanding here.
pbr does *not* modify *anyones* setup.py output unless its enabled.
Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr doing weird stuff to distributions by abusing said entrypoints.
For reference: https://bugs.launchpad.net/pbr/+bug/1483067
There's nothing special about pbr here. It's not like it's the first package doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like pdbpp, but you don't put that in production.
Starting a sub-thread since issues with pbr are orthogonal to the original disucssion. But one point I'd like to raise about this is that when I originally designed d2to1, on which a chunk of pbr is based, it was *explicitly* designed to never be installed in site-packages (with the exception of downstream packaging systems which can do what they want and are more controlled). This is exactly because I knew different packages might have dependencies on different versions of d2to1 as features are added, and that if some version is installed in site-packages it can lead to VersionConflict issues (this is in part exacerbated by a bug/misfeature in setuptools--I fixed that bug a while ago but the fix had to be rolled back due to a regression [1]). So TL;DR unless you know what you're doing, d2to1 should *never* be "installed"--it was only meant to be used with setup_requires, where the appropriate d2to1 used in building/installing a package is only temporarily enabled on sys.path via a temporary egg install. If some project is making it a *runtime* requirement that's a mistake. I don't know what features pbr has grown that might make someone want it to be a runtime dependency (the only entry-points I noticed were for adding egg-info writers but that should only be needed at build-time too...), but maybe something like that should be split off as a separate module or something... Best, Erik [1] https://bitbucket.org/tarek/distribute/pull-requests/20/fixes-and-adds-a-reg...
On October 7, 2015 at 11:32:32 AM, Erik Bray (erik.m.bray@gmail.com) wrote:
I don't know what features pbr has grown that might make someone want it to be a runtime dependency (the only entry-points I noticed were for adding egg-info writers but that should only be needed at build-time too...), but maybe something like that should be split off as a separate module or something…
It has runtime utilities for dealing with versions. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 8 October 2015 at 04:32, Erik Bray <erik.m.bray@gmail.com> wrote:
Starting a sub-thread since issues with pbr are orthogonal to the original disucssion.
But one point I'd like to raise about this is that when I originally designed d2to1, on which a chunk of pbr is based, it was *explicitly* designed to never be installed in site-packages (with the exception of downstream packaging systems which can do what they want and are more controlled). This is exactly because I knew different packages might have dependencies on different versions of d2to1 as features are added, and that if some version is installed in site-packages it can lead to VersionConflict issues (this is in part exacerbated by a bug/misfeature in setuptools--I fixed that bug a while ago but the fix had to be rolled back due to a regression [1]).
So yes - that principle makes a lot of sense. There are two factors for pbr. a) as Donald mentions, there's a runtime API too - but we could indeed split that out into a separate package, if it would help things. b) more importantly, in OpenStack infra we can't use easy-install - its inconsistency with pip and lack of updated handling for wheels, HTTPS, separate configuration - consistently cause headaches every single time it happens. So many years ago we put in place pre-installation of all known build-time dependencies - we just install them statically, because we find the effort required to make them be compatible is less than the headaches from easy-install. As such pbr has a hard API requirement: thou shalt be backwards compatible. Clearly b) can hit VersionConflicts if pbr (and any other build dependencies like setuptools itself) are out of date, but that is easily handled for automated environments (pip install -U pip setuptools wheel pbr && echo YAY), and it errors cleanly enough for hand-use by folk that its a decent enough tradeoff in our experience.
I don't know what features pbr has grown that might make someone want it to be a runtime dependency (the only entry-points I noticed were for adding egg-info writers but that should only be needed at build-time too...), but maybe something like that should be split off as a separate module or something...
Its all about easy-install. This is why I put a proof of concept static-setup-requires thing together for pip (and its on our teams roadmap to deliver a production version of it via patches to all the pypa projects, we're just not at that point yet - the resolver is first, and we have to finish rolling out constraints within OpenStack before that gets large timeslices). -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On Wed, Oct 7, 2015 at 3:39 PM, Robert Collins <robertc@robertcollins.net> wrote: [...]
Its all about easy-install. This is why I put a proof of concept static-setup-requires thing together for pip (and its on our teams roadmap to deliver a production version of it via patches to all the pypa projects, we're just not at that point yet - the resolver is first, and we have to finish rolling out constraints within OpenStack before that gets large timeslices).
I remembered Nick saying something about this at PyCon, but I couldn't find anything when I looked -- could you point me to the PoC? -n -- Nathaniel J. Smith -- http://vorpus.org
On 8 October 2015 at 17:47, Nathaniel Smith <njs@pobox.com> wrote:
On Wed, Oct 7, 2015 at 3:39 PM, Robert Collins <robertc@robertcollins.net> wrote: [...]
Its all about easy-install. This is why I put a proof of concept static-setup-requires thing together for pip (and its on our teams roadmap to deliver a production version of it via patches to all the pypa projects, we're just not at that point yet - the resolver is first, and we have to finish rolling out constraints within OpenStack before that gets large timeslices).
I remembered Nick saying something about this at PyCon, but I couldn't find anything when I looked -- could you point me to the PoC?
https://github.com/rbtcollins/pip/tree/issue-988 - resolver - its a little horked right now as I haven't fully addressed the constraints changes in the merge with it, but its readable. https://github.com/rbtcollins/pip/tree/declarative - declarative commits POC. Totally stale code but also worth reading (and not hard to update when the time comes) -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
participants (4)
-
Donald Stufft -
Erik Bray -
Nathaniel Smith -
Robert Collins