[Distutils] Importable wheels using distlib/distil

[PJ Eby]

On Fri, Mar 29, 2013 at 4:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Mar 30, 2013 at 6:42 AM, PJ Eby <pje at telecommunity.com> wrote:
>> On Fri, Mar 29, 2013 at 3:55 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> No, mutating sys.path for versioned imports is a broken design. You
>>> end up with two possibilities:
>>>
>>> * If you append, then you can't override modules that have a default
>>> version available on sys.path. This is not an acceptable restriction,
>>> which is why pkg_resources doesn't do it that way
>>> * If you prepend, then you have the existing pkg_resources failure
>>> mode where it can accidentally shadow more modules than intended. This
>>> is a nightmare to debug when it goes wrong (it took me months to
>>> realise this was why a system install of the main application I work
>>> on was shadowing the version in source checkout when running the test
>>> suite or building the documentation).
>>>
>>> The correct way to do it is with a path hook that processes a special
>>> "<versioned-packages>" marker label in sys.path (probably placed after
>>> the standard library but before site-packages by default). Any mounted
>>> directories would be tracked by that path hook, but never included
>>> directly in sys.path itself.
>>
>> How is that different from replacing "<versioned-packages>" with the
>> path of the versioned package being added?
>
> You don't lose the place where you want the inserts to happen. Without
> the marker, you end up having to come up with a heuristic for "make
> insertions here" and that gets messy as you modify the path
> (particularly since other code may also modify the path without your
> involvement).

But at least you can tell exactly what the order is by inspecting sys.path.

> Using a path hook to say "process these additional path
> entries here" cleans all that up and lets you precisely control the
> relative precedence of the additions as a group, without needing to
> care about their contents, or the other contents of sys.path.

Then can't you just bracket the entries with "<start-versioned>" and
"<end-versioned>"  then?  ;-)

(That would actually work right now without a path hook, since strings
that refer to non-existent paths are optimized away even in Python
2.5+.)

Also, btw, pkg_resources's sys.path munging is far more aggressive
than anything with wheels needs to be, because pkg_resources is
*always* working with individual eggs, not plain install directories.
If you are only using standalone wheels to handle alternate versions,
then you *definitely* want those standalone wheels to override other
things, so a strategy of always placing them immediately before their
containing directory is actually safe.  (In effect, treating the
containing directory as the insertion marker.)

So, if the standalone wheel is in site-packages, then activating it
would place it just ahead of site-packages -- i.e., the same place
you're saying it should go.

And as I believe I mentioned before, a single marker for insertion
points doesn't address the user site-packages, app directory or app
plugin directories, etc.  AFAICT your proposal only addresses the
needs of system packages, and punts on everything else.

At the least, may I suggest that instead of using a marker, if you
must use a path hook, simply install the path hook as a wrapper for a
specific directory (e.g. site-packages), and let it process insertions
for *that directory only*, rather than having a single global notion
of "all the versioned packages".