[Python-ideas] Draft PEP for the regularization of Python install layouts
VanL
van.lindberg at gmail.com
Fri Oct 28 23:53:10 CEST 2011
In part inspired by the virtualenv-in-the-stdlib PEP, I figured that it
might be a good time to draft up a PEP to fix one of my regular
annoyances: the ever-so-slightly different layouts for Python between
platforms. For someone who develops on Windows and a Mac and deploys on
Linux, this is a major pain in the rear - so much so that I always
change my system Python install to have matching environments.
I harbor no illusions that this is necessarily a thorn in anyone else's
side, or that this will go anywhere. However, I have written it up and I
would love any feedback.
Thanks,
Van
=====================================
PEP: XXX
Title: Consistent Python Environment Layout Across Platforms
Version: $Revision$
Last-Modified: $Date$
Author: Van Lindberg <van.lindberg at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-Oct-2011
Python-Version: 3.3
Post-History: 28-Oct-2011
Abstract
========
Python currently uses different environment layouts based upon the
underlying operating system and the interpreter implementation
language. This PEP proposes to regularize the directory layout for
Python environments.
Motivation
==========
One of Python's strengths is its cross-platform appeal. Carefully-
written Python programs are frequently portable between operating
systems and Python implementations with very few changes. Over the
years, substantial effort has been put into maintaining platform
parity and providing consistent interfaces to available functionality,
even when different underlying implementations are necessary (such
as with ntpath and posixpath).
One place where Python is unnecessarily different, however, is in
the layout and organization of the Python environment. This is most
visible in the name of the directory for binaries on the Windows platform
("Scripts") versus the name of the directory for binaries on every
other platform ("bin"), but a full listing of the layouts shows
substantial differences in layout and capitalization across platforms.
Sometimes the include is capitalized ("Include"), and sometimes not; and
the python version may or may not be included in the path to the
standard library or not.
The differences between platforms become particularly noticable when
attempting to do cross-platform development in an isolated environment
like a `virtualenv`_ or the proposed Python Virtual Environments.
Differences between platforms get hard-coded in or worked around so
that the same code will just work without having to artificially take
the platform into account. This information can also be made available
to Python tools (like virtualenv, `distribute`_, or `pip`_) so that
environment directories can be more easily found and changed.
Regularizing the Python environment layout across Python versions will
lower the differences between Python versions, making it more consistent
for all users, and enforcing a unified API for virtual environments and
installers rather than ad-hoc detection or hard-coded values.
.. _virtualenv: http://www.virtualenv.org
.. _distribute: http://packages.python.org/distribute/
.. _pip: http://www.pip-installer.org/
Specification
=============
When the Python binary is executed, it imports both sysconfig and
distutils.command.install. These two modules contain slightly different
versions of the environment layout. They are:
From sysconfig:
_INSTALL_SCHEMES = {
'posix_prefix': {
'stdlib': '{base}/lib/python{py_version_short}',
'platstdlib': '{platbase}/lib/python{py_version_short}',
'purelib': '{base}/lib/python{py_version_short}/site-packages',
'platlib': '{platbase}/lib/python{py_version_short}/site-packages',
'include': '{base}/include/python{py_version_short}',
'platinclude': '{platbase}/include/python{py_version_short}',
'scripts': '{base}/bin',
'data': '{base}',
},
'posix_home': {
'stdlib': '{base}/lib/python',
'platstdlib': '{base}/lib/python',
'purelib': '{base}/lib/python',
'platlib': '{base}/lib/python',
'include': '{base}/include/python',
'platinclude': '{base}/include/python',
'scripts': '{base}/bin',
'data' : '{base}',
},
'nt': {
'stdlib': '{base}/Lib',
'platstdlib': '{base}/Lib',
'purelib': '{base}/Lib/site-packages',
'platlib': '{base}/Lib/site-packages',
'include': '{base}/Include',
'platinclude': '{base}/Include',
'scripts': '{base}/Scripts',
'data' : '{base}',
},
'os2': {
'stdlib': '{base}/Lib',
'platstdlib': '{base}/Lib',
'purelib': '{base}/Lib/site-packages',
'platlib': '{base}/Lib/site-packages',
'include': '{base}/Include',
'platinclude': '{base}/Include',
'scripts': '{base}/Scripts',
'data' : '{base}',
},
'os2_home': {
'stdlib': '{userbase}/lib/python{py_version_short}',
'platstdlib': '{userbase}/lib/python{py_version_short}',
'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
'platlib': '{userbase}/lib/python{py_version_short}/site-packages',
'include': '{userbase}/include/python{py_version_short}',
'scripts': '{userbase}/bin',
'data' : '{userbase}',
},
'nt_user': {
'stdlib': '{userbase}/Python{py_version_nodot}',
'platstdlib': '{userbase}/Python{py_version_nodot}',
'purelib': '{userbase}/Python{py_version_nodot}/site-packages',
'platlib': '{userbase}/Python{py_version_nodot}/site-packages',
'include': '{userbase}/Python{py_version_nodot}/Include',
'scripts': '{userbase}/Scripts',
'data' : '{userbase}',
},
'posix_user': {
'stdlib': '{userbase}/lib/python{py_version_short}',
'platstdlib': '{userbase}/lib/python{py_version_short}',
'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
'platlib': '{userbase}/lib/python{py_version_short}/site-packages',
'include': '{userbase}/include/python{py_version_short}',
'scripts': '{userbase}/bin',
'data' : '{userbase}',
},
'osx_framework_user': {
'stdlib': '{userbase}/lib/python',
'platstdlib': '{userbase}/lib/python',
'purelib': '{userbase}/lib/python/site-packages',
'platlib': '{userbase}/lib/python/site-packages',
'include': '{userbase}/include',
'scripts': '{userbase}/bin',
'data' : '{userbase}',
},
}
From distutils.command.install:
if sys.version < "2.2":
WINDOWS_SCHEME = {
'purelib': '$base',
'platlib': '$base',
'headers': '$base/Include/$dist_name',
'scripts': '$base/Scripts',
'data' : '$base',
}
else:
WINDOWS_SCHEME = {
'purelib': '$base/Lib/site-packages',
'platlib': '$base/Lib/site-packages',
'headers': '$base/Include/$dist_name',
'scripts': '$base/Scripts',
'data' : '$base',
}
INSTALL_SCHEMES = {
'unix_prefix': {
'purelib': '$base/lib/python$py_version_short/site-packages',
'platlib': '$platbase/lib/python$py_version_short/site-packages',
'headers': '$base/include/python$py_version_short/$dist_name',
'scripts': '$base/bin',
'data' : '$base',
},
'unix_home': {
'purelib': '$base/lib/python',
'platlib': '$base/lib/python',
'headers': '$base/include/python/$dist_name',
'scripts': '$base/bin',
'data' : '$base',
},
'unix_user': {
'purelib': '$usersite',
'platlib': '$usersite',
'headers': '$userbase/include/python$py_version_short/$dist_name',
'scripts': '$userbase/bin',
'data' : '$userbase',
},
'nt': WINDOWS_SCHEME,
'nt_user': {
'purelib': '$usersite',
'platlib': '$usersite',
'headers': '$userbase/Python$py_version_nodot/Include/$dist_name',
'scripts': '$userbase/Scripts',
'data' : '$userbase',
},
'os2': {
'purelib': '$base/Lib/site-packages',
'platlib': '$base/Lib/site-packages',
'headers': '$base/Include/$dist_name',
'scripts': '$base/Scripts',
'data' : '$base',
},
'os2_home': {
'purelib': '$usersite',
'platlib': '$usersite',
'headers': '$userbase/include/python$py_version_short/$dist_name',
'scripts': '$userbase/bin',
'data' : '$userbase',
},
}
There is an API call (sysconfig.get_config_var) that is used to resolve
these various variables, but there is no way to set them other than
directly clobbering _INSTALL_SCHEMES/INSTALL_SCHEME dicts holding the
information (in both sysconfig and in distutils.command.install).
This PEP proposes to change the default installation layout as follows:
- Define a new config var, py_binaries, and a corresponding module-level
variable, _PY_BINARIES with the default value 'bin'
- Change sysconfig's _INSTALL_SCHEMES to INSTALL_SCHEMES, and make
it part of the public API. A default scheme equivalent to the current
posix_prefix would be added and made to be the default.
- To support virtual environments, a special key "current" would be
added to INSTALL_SCHEMES. The value associated with "current" would
be the name of the currently active scheme. During activation, a
virtual environment could place a custom layout in the
INSTALL_SCHEMES dict and set "current" to the custom scheme. This
could also be used to support legacy or custom layouts.
This would be done as follows::
from _collections import defaultdict
_DEFAULT_SCHEME = {
'stdlib': '{userbase}/lib/python{py_version_short}',
'platstdlib': '{userbase}/lib/python{py_version_short}',
'purelib':
'{userbase}/lib/python{py_version_short}/site-packages',
'platlib':
'{userbase}/lib/python{py_version_short}/site-packages',
'include': '{userbase}/include/python{py_version_short}',
'headers': '{userbase}/include/python{py_version_short}',
'scripts': '{userbase}/{py_binaries}',
'data' : '{userbase}',
}
INSTALL_SCHEMES = defaultdict(lambda: _DEFAULT_SCHEME.copy())
INSTALL_SCHEMES['install'] = _DEFAULT_SCHEME
INSTALL_SCHEMES['current'] = 'install'
- Change distutils.command.install to use information from INSTALL_SCHEMES
instead of embedding its own copy of the dict.
By using a defaultdict, prior users of the API could transparently query
the dict using the platform value and receive back a reasonable
response. The current virtual environment would be retrievable by
getting INSTALL_SCHEMES[INSTALL_SCHEMES['current']], and the installed
scheme retrievable by getting INSTALL_SCHEMES['install'].
Open Issues
==========
By default, OS X Framework installs use a format that is different than
the one specified above. Possible responses would be to change the
layout just for OS X, but special-casing as little as possible.
The double indirection to get the current install scheme is a little
ugly, but I am not sure what would be better.
This explicitly removes the compatibility code in distutils for 2.x
install layouts (particularly layouts pre-2.3). This is not considered
a major issue, as 2.2 code will largely not be compatible with 3.3, and
an individual user could add a custom layout if needed.
Based on actual use, some of the distinctions that appeared necessary
when distutils and sysconfig were first written (platlib, purelib,
include, headers) seem to be duplicative. As written above, the
distinctions are maintained for ease of moving to the standardized
layout, but it may be worth choosing just one name and sticking with it.
More information about the Python-ideas
mailing list