In part inspired by the virtualenv-in-the-stdlib PEP, I figured that it might be a good time to draft up a PEP to fix one of my regular annoyances: the ever-so-slightly different layouts for Python between platforms. For someone who develops on Windows and a Mac and deploys on Linux, this is a major pain in the rear - so much so that I always change my system Python install to have matching environments.

I harbor no illusions that this is necessarily a thorn in anyone else's side, or that this will go anywhere. However, I have written it up and I would love any feedback.

Thanks,

Van

===================================== PEP: XXX Title: Consistent Python Environment Layout Across Platforms Version: $Revision$ Last-Modified: $Date$ Author: Van Lindberg <van.lindberg at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Oct-2011 Python-Version: 3.3 Post-History: 28-Oct-2011

Abstract ========

Python currently uses different environment layouts based upon the underlying operating system and the interpreter implementation language. This PEP proposes to regularize the directory layout for Python environments.

Motivation ==========

One of Python's strengths is its cross-platform appeal. Carefully- written Python programs are frequently portable between operating systems and Python implementations with very few changes. Over the years, substantial effort has been put into maintaining platform parity and providing consistent interfaces to available functionality, even when different underlying implementations are necessary (such as with ntpath and posixpath).

One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the name of the directory for binaries on every other platform ("bin"), but a full listing of the layouts shows substantial differences in layout and capitalization across platforms. Sometimes the include is capitalized ("Include"), and sometimes not; and the python version may or may not be included in the path to the standard library or not.

The differences between platforms become particularly noticable when attempting to do cross-platform development in an isolated environment like a `virtualenv`_ or the proposed Python Virtual Environments. Differences between platforms get hard-coded in or worked around so that the same code will just work without having to artificially take the platform into account. This information can also be made available to Python tools (like virtualenv, `distribute`_, or `pip`_) so that environment directories can be more easily found and changed.

Regularizing the Python environment layout across Python versions will lower the differences between Python versions, making it more consistent for all users, and enforcing a unified API for virtual environments and installers rather than ad-hoc detection or hard-coded values.

.. _virtualenv: http://www.virtualenv.org

.. _distribute: http://packages.python.org/distribute/

.. _pip: http://www.pip-installer.org/

Specification =============

When the Python binary is executed, it imports both sysconfig and distutils.command.install. These two modules contain slightly different versions of the environment layout. They are:

From sysconfig: _INSTALL_SCHEMES = { 'posix_prefix': { 'stdlib': '{base}/lib/python{py_version_short}', 'platstdlib': '{platbase}/lib/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/site-packages', 'platlib': '{platbase}/lib/python{py_version_short}/site-packages', 'include': '{base}/include/python{py_version_short}', 'platinclude': '{platbase}/include/python{py_version_short}', 'scripts': '{base}/bin', 'data': '{base}', }, 'posix_home': { 'stdlib': '{base}/lib/python', 'platstdlib': '{base}/lib/python', 'purelib': '{base}/lib/python', 'platlib': '{base}/lib/python', 'include': '{base}/include/python', 'platinclude': '{base}/include/python', 'scripts': '{base}/bin', 'data' : '{base}', }, 'nt': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2_home': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'nt_user': { 'stdlib': '{userbase}/Python{py_version_nodot}', 'platstdlib': '{userbase}/Python{py_version_nodot}', 'purelib': '{userbase}/Python{py_version_nodot}/site-packages', 'platlib': '{userbase}/Python{py_version_nodot}/site-packages', 'include': '{userbase}/Python{py_version_nodot}/Include', 'scripts': '{userbase}/Scripts', 'data' : '{userbase}', }, 'posix_user': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'osx_framework_user': { 'stdlib': '{userbase}/lib/python', 'platstdlib': '{userbase}/lib/python', 'purelib': '{userbase}/lib/python/site-packages', 'platlib': '{userbase}/lib/python/site-packages', 'include': '{userbase}/include', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, }

From distutils.command.install:

if sys.version < "2.2": WINDOWS_SCHEME = { 'purelib': '$base', 'platlib': '$base', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } else: WINDOWS_SCHEME = { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', }

INSTALL_SCHEMES = { 'unix_prefix': { 'purelib': '$base/lib/python$py_version_short/site-packages', 'platlib': '$platbase/lib/python$py_version_short/site-packages', 'headers': '$base/include/python$py_version_short/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_home': { 'purelib': '$base/lib/python', 'platlib': '$base/lib/python', 'headers': '$base/include/python/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, 'nt': WINDOWS_SCHEME, 'nt_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/Python$py_version_nodot/Include/$dist_name', 'scripts': '$userbase/Scripts', 'data' : '$userbase', }, 'os2': { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', }, 'os2_home': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, }

There is an API call (sysconfig.get_config_var) that is used to resolve these various variables, but there is no way to set them other than directly clobbering _INSTALL_SCHEMES/INSTALL_SCHEME dicts holding the information (in both sysconfig and in distutils.command.install).

This PEP proposes to change the default installation layout as follows:

- Define a new config var, py_binaries, and a corresponding module-level variable, _PY_BINARIES with the default value 'bin'

- Change sysconfig's _INSTALL_SCHEMES to INSTALL_SCHEMES, and make it part of the public API. A default scheme equivalent to the current posix_prefix would be added and made to be the default.

- To support virtual environments, a special key "current" would be added to INSTALL_SCHEMES. The value associated with "current" would be the name of the currently active scheme. During activation, a virtual environment could place a custom layout in the INSTALL_SCHEMES dict and set "current" to the custom scheme. This could also be used to support legacy or custom layouts.

This would be done as follows::

from _collections import defaultdict

_DEFAULT_SCHEME = { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'headers': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/{py_binaries}', 'data' : '{userbase}', }

INSTALL_SCHEMES = defaultdict(lambda: _DEFAULT_SCHEME.copy()) INSTALL_SCHEMES['install'] = _DEFAULT_SCHEME INSTALL_SCHEMES['current'] = 'install'

- Change distutils.command.install to use information from INSTALL_SCHEMES instead of embedding its own copy of the dict.

By using a defaultdict, prior users of the API could transparently query the dict using the platform value and receive back a reasonable response. The current virtual environment would be retrievable by getting INSTALL_SCHEMES[INSTALL_SCHEMES['current']], and the installed scheme retrievable by getting INSTALL_SCHEMES['install'].

Open Issues ==========

By default, OS X Framework installs use a format that is different than the one specified above. Possible responses would be to change the layout just for OS X, but special-casing as little as possible.

The double indirection to get the current install scheme is a little ugly, but I am not sure what would be better.

This explicitly removes the compatibility code in distutils for 2.x install layouts (particularly layouts pre-2.3). This is not considered a major issue, as 2.2 code will largely not be compatible with 3.3, and an individual user could add a custom layout if needed.

Based on actual use, some of the distinctions that appeared necessary when distutils and sysconfig were first written (platlib, purelib, include, headers) seem to be duplicative. As written above, the distinctions are maintained for ease of moving to the standardized layout, but it may be worth choosing just one name and sticking with it.