[Import-SIG] making it feasible to rely on loaders for reading intra-package data files

[Nick Coghlan]

On 7 February 2014 03:15, PJ Eby <pje at telecommunity.com> wrote:
> On Wed, Feb 5, 2014 at 5:58 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> And in the context of Chandler as an integrated
>> application and there being no "default" version of packages already on
>> sys.path, pkg_resources.WorkingSet works fine - problems only arise because
>> Fedora *does* bless one version as default, puts it directly on sys.path,
>> and then the first pkg_resources import in an application locks all of those
>> default versions in as the expected versions if you don't arrange to set
>> __main__.__requires__ first, which then doesn't play well with entry point
>> based script wrappers
>
> Huh?  Entry point script wrappers *set* __requires__ as the very first
> thing they do, followed by importing pkg_resources.  (Alternatively,
> if you build the scripts using buildout, they have paths hardcoded.
> Either way, no problems with default versions.)

If only a single layer of software is involved, you're correct, but
try it with something like nosetests, gunicorn or Sphinx: *they* use
entry point wrapper scripts themselves, so by the time execution gets
to the application that wants to set requirements for its
dependencies, it's far too late - the dependencies have all been
locked to their default versions simply by starting
nose/gunicorn/sphinx, and the actual application code doesn't get a
chance to change that.

> The only way I can see problems is if you *aren't* using entry point
> wrappers for your scripts, *and* you want non-default versions.

Correct - the problem is specifically with command line applications
and daemons that are themselves written to use setuptools wrapper
scripts, but then subsequently import application code into the same
process. If that application code needs non-default versions of
various dependencies, then it's necessary to find a way to bypass the
early import of pkg_resources so that you can set
__main__.__requires__ appropriately.

We currently hack around the limitation by using "python -c" to bypass
the normal script wrappers for the affected tools, but it's
tremendously ugly.

Most people would probably just give up on the idea and run inside a
virtual environment instead, but we actually want to get Beaker
accepted into Fedora itself eventually, so that means it needs to play
nice with the system Python environment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia