[Distutils] Parallel installation of incompatible versions

[Daniel Holth]

FWIW I always thought the long sys.path problem was a bug that could
be solved by improving sys.path.__repr__()

On Tue, Mar 19, 2013 at 2:06 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Mon, Mar 18, 2013 at 6:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> pkg_resources.requires() is our only current solution for parallel
>> installation of incompatible versions. This can be made to work and is
>> a lot better than the nothing we had before it was created, but also
>> has quite a few issues (and it can be a nightmare to debug when it
>> goes wrong).
>>
>> Based on the exchanges with Mark McLoughlin the other week, and
>> chatting to Matthias Klose here at the PyCon US sprints, I think I
>> have a design that will let us support parallel installs in a way that
>> builds on existing standards, while behaving more consistently in edge
>> cases and without making sys.path ridiculously long even in systems
>> with large numbers of potentially incompatible dependencies.
>>
>> The core of this proposal is to create an updated version of the
>> installation database format that defines semantics for *.pth files
>> inside .dist-info directories.
>>
>> Specifically, whereas *.pth files directly in site-packages are
>> processed automatically when Python starts up, those inside dist-info
>> directories would be processed only when explicitly requested
>> (probably through a new distlib API). The processing of the *.pth file
>> would insert it into the path immediately before the path entry
>> containing the .dist-info directory (this is to avoid an issue with
>> the pkg_resources insert-at-the-front-of-sys.path behaviour where
>> system packages can end up shadowing those from a local source
>> checkout, without running into the issue with
>> append-to-the-end-of-sys.path where a specifically requested version
>> is shadowed by a globally installed version)
>>
>> To use CherryPy2 and CherryPy3 on Fedora as an example, what this
>> would allow is for CherryPy3 to be installed normally (i.e. directly
>> in site-packages), while CherryPy2 would be installed as a split
>> install, with the .dist-info going into site-packages and the actual
>> package going somewhere else (more on that below). A cherrypy2.pth
>> file inside the dist-info directory would reference the external
>> location where cherrypy 2.x can be found.
>>
>> To use this at runtime, you would do something like:
>>
>>     distlib.some_new_requires_api("CherryPy (2.2)")
>>     import cherrypy
>>
>> The other part of this question is how to avoid the potential
>> explosion of one sys.path entry per dependency. The first part of that
>> is that for cases where there is no incompatible version installed,
>> there won't be a *.pth file, and hence no extra sys.path entry (the
>> module/package will just be installed directly into site-packages as
>> usual).
>>
>> The second part has to do with a possible way to organise the
>> versioned installs: group them by the initial fragment of the version
>> number according to semantic versioning. For example, define a
>> "versioned-packages" directory that sits adjacent to "site-packages".
>> When doing the parallel install of CherryPy2 the actual *code* would
>> be installed into "versioned-packages/2/", with the cherrypy2.pth file
>> pointing to that directory. For 0.x releases, there would be a
>> directory per minor version, while for higher releases, there would
>> only be a directory per major version.
>>
>> The nice thing though is that Python wouldn't actually care about the
>> actual layout of the installed versions, so long as the *.pth files in
>> the dist-info directories described the mapping correctly.
>
> Could you perhaps spell out why this is better than just dropping .whl
> files (or unpacked directories) into site-packages or equivalent?
>
> Also, one thing that actually confuses me about this proposal is that
> it sounds like you are saying you'd have two CherryPy.dist-info
> directories in site-packages, which sounds broken to me; the whole
> point of the existing protocol for .dist-info was that it allowed you
> to determine the importable versions from a single listdir().  Your
> approach would break that feature, because you'd have to:
>
> 1. Read each .dist-info directory to find .pth files
> 2. Open and read all the .pth files
> 3. Compare the .pth file contents with sys.path to find out what is
> actually *on* sys.path
>
> This is a lot more complexity and I/O overhead than PEP 376 and its
> antecedents in pkg_resources et al.
>
> In contrast, if you use .whl files or directories, you can both
> determine the available versions *and* the active versions from a
> single directory read.  And on everything but Windows, those could be
> symlinks to the target location rather than an actual file or
> directory, thus giving you the same kind of layout flexibility as what
> you've proposed.
>
> (Or, if you want a solution that works the same across platforms, just
> re-invent .egg-link files, which are basically a super-symlink
> anyway.)
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig