[Distutils] Some clarifications and/or corrections to PEP 376

[Paul Moore]

I've been looking at the details of PEP 376 (Database of Installed
Python Distributions) and there are a couple of aspects that I don't
think work properly alongside PEP 370 (Per-user site-packages
directory)..

1. The dist-info directory for a distribution is stated as being
"located in the site-packages directory".

It's not clear how this is intended to work in a PEP 370 world with
multiple site-packages. What I propose is that the description be
changed to be worded in terms of sysconfig-style locations: the
dist-info directory is located in whichever of purelib or platlib is
used by the distribution. When the distribution uses both, purelib is
preferred, when it uses neither (!) purelib is used. In nearly all
cases, this is the same as currently. The exceptions are posix_home
(where the directory name isn't "site-packages" but its function is
the same), posix_prefix (where purelib and platlib differ, and PEP 376
is currently ambiguous as to which is implied), and any custom schemes
that might be created (where PEP 376 is silent, and this proposal has
the benefit of at least being specific). I do not believe this changes
any actual practice - as far as I have been able to determine any code
using dist-info at the moment follows this proposal in the corner
cases where it differs from PEP 376.

2. File paths in RECORD are stated as being relative to the "base
location" as long as the files sit under that location, or under the
"install prefix" where that is above the base location. That's a messy
but precise way of saying you can go up a bit as long as you stay
within sys.prefix. (Other paths are platform format absolute paths,
which is an ugly discrepancy but the only really viable option).
Again, this doesn't work as I'd expect it should in a PEP 370 user
site-packages.

In this case, the answer isn't quite as clear. I would suggest,
looking at the schemes defined in sysconfig, allowing any files
located in one of the defined sysconfig paths (for the current scheme)
to be recorded as relative paths. The paths should be relative to
whichever of purelib and platlib was chosen in (1) above. The
exception is that where pureXXX and platXXX do not use the same base
(posix_prefix and possibly custom schemas) it should not be allowed to
have a relative path from a pureXXX to a platXXX or vice versa. That
equates to something slightly stricter than the current scheme
(although not in a way that's likely to be used in practice), but
using the appropriate one of {base} or {userbase} rather than
sys.prefix. As noted, the two exceptions are custom schemes (where PEP
376 is silent and would probably end up effectively mandating platform
absolute paths throughout) and posix_prefix, where
platlib/platinclude/platstdlib are located under {platbase} rather
than {base} and the current proposal would probably allow relative
paths to be used a little more often that PEP 376 allows. Current
practice that I've seen here is not as clear cut, but mostly in the
sense that code doesn't consider the corner cases at all, and probably
only follows PEP 376 by luck, if at all...

Comments would be appreciated. I'm pretty happy that my proposal for
(1) is an improvement. With (2) I'm concerned about the posix_prefix
case, particularly as I'm a Windows developer and my understanding of
Python's layout on POSIX is limited. But I honestly don't think it's
possible to do much better than my proposal, and I do think that PEP
376 as it stands now is distinctly worse because it doesn't consider
PEP 370 or the posix_prefix case.

Also, while these proposals are not 100% backward compatible, I don't
believe they change behaviour in the most common scenarios, and given
that PEP 376 currently has limited adoption (mainly in distutils2, I
believe) I'd suggest that the minor compatibility break now is better
than keeping the current system until it's more widely adopted and
there are more users to impact. Obviously I'd be looking for views
from Tarek and Éric in particular here, as it's their code in
distutils2 that would be directly impacted.

Paul.