[Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[Paul Moore]

On 22 March 2012 14:17, VanL <van.lindberg at gmail.com> wrote:
> As this has been brought up a couple times in this subthread, I figured that
> I would lay out the rationale here.

I'm repeating myself here after I promised not to. My apologies, but I
don't think this posting captures the debate completely. One reason I
suggested a PEP is to better ensure that the arguments both pro and
con were captured, as that is a key part of the PEP process.

> One place where Python is unnecessarily different, however, is in
> the layout and organization of the Python environment. This is most
> visible in the name of the directory for binaries on the Windows platform
> ("Scripts") versus the name of the directory for binaries on every other
> platform ("bin"),

First of all, this difference is almost entirely *invisible*. Apart
from possibly setting PATH (once!) users should not be digging around
in the Python installation directory. Certainly on Windows, it is a
very unusual application that expects users to even care how the
application is laid out internally. And I suspect that is also true on
Unix and Mac.

Secondly, the layouts are not as similar as you claim here, if I
understand things correctly. on Unix Python is installed in
/usr/local/bin so there isn't even a "Python installation directory"
there. And Macs use some sort of Mac-specific bundle technology as I
understand it. To be honest, I don't think that there's a lot of
similarity even with your proposed changes.

> but a full listing of the layouts shows
> substantial differences in layout and capitalization across platforms.

That's true, but largely incidental. And given that (a) Windows is
case insensitive and (b) the capitalisation, although inconsistent,
follows platform standards (Unix all lowercase, Windows capitalised)
it makes little practical difference.

> Sometimes the include is capitalized ("Include"), and sometimes not; and
> the python version may or may not be included in the path to the
> standard library or not.

Given that on Windows the Python library is usually in something like
C:\Python32\Lib whereas on Unix it's in /usr/lib/python3.2 (I think),
the difference is reasonable because the Python *base* location
(C:\Python32 on Windows vs /usr on Unix) is version specific in one
case but not the other. To keep the correspondence complete, you
should be suggesting installing in /python32 on Unix (which I doubt
would gain much support :-))

> This may seem like a harmless inconsistency, and if that were all it was, I
> wouldn't care. (That said, cross-platform consistency is its own good). But
> it becomes a real pain when combined with tools like virtualenv or the new
> pyvenv to create cross-platform development environments.

The issue with virtualenv and pyvenv may be more significant. But
you're only mentioning those incidentally. As a straw-man suggestion,
why can virtualenv not be changed to maintain a platform-neutral
layout in spite of what core Python does? This is a straw-man because
I know the answer, but can we get the point out in the open - it's
related to how distutils installs code, and that in turn hits you
straight up against the distutils freeze. If distutils' behaviour is
the issue here, then argue for more flexibility in packaging, and use
that extra flexibility to argue for changes to virtualenv and pyvenv
to maintain a standard cross-platform layout. Breaking the Python
installation layout isn't the only option here, and I'd like to see a
clear analysis of the tradeoffs. (I also get a sense of undue haste -
"we can change the Python layout for 3.3, but changing packaging and
virtualenv is a much longer process"...)

> In particular, I regularly do development on both Windows and a Mac, and
> then deploy on Linux. I do this in virtualenvs, so that I have a controlled
> and regular environment. I keep them in sync using source control.
>
> The problem comes when I have executable scripts that I want to include in
> my dvcs - I can't have it in the obvious place - the binaries directory -
> because *the name of the directory changes when you move between platforms.*
> More concretely, I can't hg add "Scripts/runner.py? on my windows
> environment (where it is put in the PATH by virtualenv) and thendo a pull on
> Mac or Linux and have it end up properly in "bin/runner.py" which is the
> correct PATH for those platforms.

This presupposes that your development workflow - developing in place
in the virtualenv itself - is "the obvious approach". From what I've
seen of tools like virtualenvwrapper, you're not alone in this. And
I'm pretty new to using virtualenv so I wouldn't like to claim to be
any expert. But can I respectfully suggest that other ways of working
wouldn't hit these issues? WhatI do is develop my project in a project
specific directory, just as I would if I were using the system Python.
And I have an activated virtualenv *located somewhere else* that I
install required 3rd party modules in, etc. I then do standard "pip
install" to install and test my project, etc etc. This adds an
"install to test" cycle (or you use something like setuptools'
"deploy" techniques, which I know nothing about myself) but in
exchange you are independent of the virtualenv layout (as pip install
puts your scripts in the appropriate Scripts/bin directory). Also, by
doing this you test your installation process, which you don't if you
develop in place.

Again, you may not like this way of working, and that's fine. But can
you acknowledge (and document) that "change your way of working" is
another alternative to "change Python".

> This applies anytime there are executable scripts that you want to manage
> using source control across platforms. Django projects regularly have these,
> and I suspect we will be seeing more of this with the new "project" support
> in virtualenvwrapper.

... if you develop inside the virtualenv.

> While a few people have wondered why I would want this -- hopefully answered
> above -- I have not heard any opposition to this part of the proposal.

See above. There has been opposition from a number of people. It's
relatively mild, simply because it's a niche area and doesn't cause
huge pain, but it's there. And you seem (based on the above analysis)
to be overstating the benefits, so the summary here is weighted
heavily in favour of change.

Also, you have made no attempt that I've seen to address the question
of why this is important enough to break backward compatibility. Maybe
it is - but why? Backward compatibility is a very strong rule, and
should be broken only with good justification. Consistency, and "it
makes my way of working easier" really shouldn't be sufficient.

Has anyone checked whether this will affect people like Enthought and
ActiveState who distribute their own versions of Python? Is
ActiveState's PPM tool affected?

> This first proposal is just to make the names of the directories match
> across platforms. There are six keys defined in the installer files
> (sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib',
> 'platlib', 'headers', 'scripts',  and 'data'.
>
> Currently on Windows, there are two different layouts defined:
>
>  'nt': {
>    'stdlib': '{base}/Lib',
>    'platstdlib': '{base}/Lib',
>    'purelib': '{base}/Lib/site-packages',
>    'platlib': '{base}/Lib/site-packages',
>    'include': '{base}/Include',
>    'platinclude': '{base}/Include',
>    'scripts': '{base}/Scripts',
>    'data'   : '{base}',
>    },
>
>  'nt_user': {
>    'stdlib': '{userbase}/Python{py_version_nodot}',
>    'platstdlib': '{userbase}/Python{py_version_nodot}',
>    'purelib': '{userbase}/Python{py_version_nodot}/site-packages',
>    'platlib': '{userbase}/Python{py_version_nodot}/site-packages',
>    'include': '{userbase}/Python{py_version_nodot}/Include',
>    'scripts': '{userbase}/Scripts',
>    'data'   : '{userbase}',
>    },
>
>
> The proposal is to make all the layouts change to:
>
>  'nt': {
>    'stdlib': '{base}/lib',
>    'platstdlib': '{base}/lib',
>    'purelib': '{base}/lib/site-packages',
>    'platlib': '{base}/lib/site-packages',
>    'include': '{base}/include',
>    'platinclude': '{base}/include',
>    'scripts': '{base}/bin',
>    'data'   : '{base}',
>    },
>
> The change here is that 'Scripts' will change to 'bin' and the
> capitalization will be removed. Also, "user installs" of Python will have
> the same internal layout as "system installs" of Python. This will also, not
> coincidentally, match the install layout for posix, at least with regard to
> the 'bin', 'lib', and 'include' directories.

Note - that is not "Regularizing the layout". You have not made any
changes to OS/2 (which matches Windows at the moment). And it doesn't
match Posix at all. I won't copy a large chunk of sysconfig.py in
here, but I'm tempted to, because your proposal as described really
doesn't match the reality of sysconfig._INSTALL_SCHEMES. I'd encourage
people to go and read sysconfig.py for details. There really isn't
much consistency at the moment, and fiddling with Windows but nothing
else doesn't regularise anything.

> Again, I have not heard *anyone* objecting to this part of the proposal as
> it is laid out here. (Paul had a concern with the lib directory earlier, but
> he said he was ok with the above).

That's somewhat odd, as I did hear a number of concerns. But it was
certainly not easy to tell which related to which part of the
proposal. And all of the objections were mild, mostly because it's not
a huge practical issue either way.

> Please let me know if you have any problems or concerns with this part 1.

Personally, my main concerns are around procedure and policy. The more
the discussion goes on, the more I feel that there should be a PEP to
capture the details of the debate clearly. Too much is getting lost in
the noise. And I think you should provide a clear statement of why
this issue is important enough to justify violating the backward
compatibility policies.

As Mark said (I think it was Mark...) if this had been proposed for
3.0, it would have been fine. Now we're at 3.2 with 3.3 close to
release, and it just seems too late to be worth the risk.

One plus point about your posting this separately. It's made me think
through the issue in a bit more detail, and I'm now a solid -1 on the
proposal.

Paul.