Draft PEP for the regularization of Python install layouts
In part inspired by the virtualenv-in-the-stdlib PEP, I figured that it might be a good time to draft up a PEP to fix one of my regular annoyances: the ever-so-slightly different layouts for Python between platforms. For someone who develops on Windows and a Mac and deploys on Linux, this is a major pain in the rear - so much so that I always change my system Python install to have matching environments. I harbor no illusions that this is necessarily a thorn in anyone else's side, or that this will go anywhere. However, I have written it up and I would love any feedback. Thanks, Van ===================================== PEP: XXX Title: Consistent Python Environment Layout Across Platforms Version: $Revision$ Last-Modified: $Date$ Author: Van Lindberg <van.lindberg at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Oct-2011 Python-Version: 3.3 Post-History: 28-Oct-2011 Abstract ======== Python currently uses different environment layouts based upon the underlying operating system and the interpreter implementation language. This PEP proposes to regularize the directory layout for Python environments. Motivation ========== One of Python's strengths is its cross-platform appeal. Carefully- written Python programs are frequently portable between operating systems and Python implementations with very few changes. Over the years, substantial effort has been put into maintaining platform parity and providing consistent interfaces to available functionality, even when different underlying implementations are necessary (such as with ntpath and posixpath). One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform ("Scripts") versus the name of the directory for binaries on every other platform ("bin"), but a full listing of the layouts shows substantial differences in layout and capitalization across platforms. Sometimes the include is capitalized ("Include"), and sometimes not; and the python version may or may not be included in the path to the standard library or not. The differences between platforms become particularly noticable when attempting to do cross-platform development in an isolated environment like a `virtualenv`_ or the proposed Python Virtual Environments. Differences between platforms get hard-coded in or worked around so that the same code will just work without having to artificially take the platform into account. This information can also be made available to Python tools (like virtualenv, `distribute`_, or `pip`_) so that environment directories can be more easily found and changed. Regularizing the Python environment layout across Python versions will lower the differences between Python versions, making it more consistent for all users, and enforcing a unified API for virtual environments and installers rather than ad-hoc detection or hard-coded values. .. _virtualenv: http://www.virtualenv.org .. _distribute: http://packages.python.org/distribute/ .. _pip: http://www.pip-installer.org/ Specification ============= When the Python binary is executed, it imports both sysconfig and distutils.command.install. These two modules contain slightly different versions of the environment layout. They are: From sysconfig: _INSTALL_SCHEMES = { 'posix_prefix': { 'stdlib': '{base}/lib/python{py_version_short}', 'platstdlib': '{platbase}/lib/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/site-packages', 'platlib': '{platbase}/lib/python{py_version_short}/site-packages', 'include': '{base}/include/python{py_version_short}', 'platinclude': '{platbase}/include/python{py_version_short}', 'scripts': '{base}/bin', 'data': '{base}', }, 'posix_home': { 'stdlib': '{base}/lib/python', 'platstdlib': '{base}/lib/python', 'purelib': '{base}/lib/python', 'platlib': '{base}/lib/python', 'include': '{base}/include/python', 'platinclude': '{base}/include/python', 'scripts': '{base}/bin', 'data' : '{base}', }, 'nt': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2': { 'stdlib': '{base}/Lib', 'platstdlib': '{base}/Lib', 'purelib': '{base}/Lib/site-packages', 'platlib': '{base}/Lib/site-packages', 'include': '{base}/Include', 'platinclude': '{base}/Include', 'scripts': '{base}/Scripts', 'data' : '{base}', }, 'os2_home': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'nt_user': { 'stdlib': '{userbase}/Python{py_version_nodot}', 'platstdlib': '{userbase}/Python{py_version_nodot}', 'purelib': '{userbase}/Python{py_version_nodot}/site-packages', 'platlib': '{userbase}/Python{py_version_nodot}/site-packages', 'include': '{userbase}/Python{py_version_nodot}/Include', 'scripts': '{userbase}/Scripts', 'data' : '{userbase}', }, 'posix_user': { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, 'osx_framework_user': { 'stdlib': '{userbase}/lib/python', 'platstdlib': '{userbase}/lib/python', 'purelib': '{userbase}/lib/python/site-packages', 'platlib': '{userbase}/lib/python/site-packages', 'include': '{userbase}/include', 'scripts': '{userbase}/bin', 'data' : '{userbase}', }, } From distutils.command.install: if sys.version < "2.2": WINDOWS_SCHEME = { 'purelib': '$base', 'platlib': '$base', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } else: WINDOWS_SCHEME = { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', } INSTALL_SCHEMES = { 'unix_prefix': { 'purelib': '$base/lib/python$py_version_short/site-packages', 'platlib': '$platbase/lib/python$py_version_short/site-packages', 'headers': '$base/include/python$py_version_short/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_home': { 'purelib': '$base/lib/python', 'platlib': '$base/lib/python', 'headers': '$base/include/python/$dist_name', 'scripts': '$base/bin', 'data' : '$base', }, 'unix_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, 'nt': WINDOWS_SCHEME, 'nt_user': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/Python$py_version_nodot/Include/$dist_name', 'scripts': '$userbase/Scripts', 'data' : '$userbase', }, 'os2': { 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'headers': '$base/Include/$dist_name', 'scripts': '$base/Scripts', 'data' : '$base', }, 'os2_home': { 'purelib': '$usersite', 'platlib': '$usersite', 'headers': '$userbase/include/python$py_version_short/$dist_name', 'scripts': '$userbase/bin', 'data' : '$userbase', }, } There is an API call (sysconfig.get_config_var) that is used to resolve these various variables, but there is no way to set them other than directly clobbering _INSTALL_SCHEMES/INSTALL_SCHEME dicts holding the information (in both sysconfig and in distutils.command.install). This PEP proposes to change the default installation layout as follows: - Define a new config var, py_binaries, and a corresponding module-level variable, _PY_BINARIES with the default value 'bin' - Change sysconfig's _INSTALL_SCHEMES to INSTALL_SCHEMES, and make it part of the public API. A default scheme equivalent to the current posix_prefix would be added and made to be the default. - To support virtual environments, a special key "current" would be added to INSTALL_SCHEMES. The value associated with "current" would be the name of the currently active scheme. During activation, a virtual environment could place a custom layout in the INSTALL_SCHEMES dict and set "current" to the custom scheme. This could also be used to support legacy or custom layouts. This would be done as follows:: from _collections import defaultdict _DEFAULT_SCHEME = { 'stdlib': '{userbase}/lib/python{py_version_short}', 'platstdlib': '{userbase}/lib/python{py_version_short}', 'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 'platlib': '{userbase}/lib/python{py_version_short}/site-packages', 'include': '{userbase}/include/python{py_version_short}', 'headers': '{userbase}/include/python{py_version_short}', 'scripts': '{userbase}/{py_binaries}', 'data' : '{userbase}', } INSTALL_SCHEMES = defaultdict(lambda: _DEFAULT_SCHEME.copy()) INSTALL_SCHEMES['install'] = _DEFAULT_SCHEME INSTALL_SCHEMES['current'] = 'install' - Change distutils.command.install to use information from INSTALL_SCHEMES instead of embedding its own copy of the dict. By using a defaultdict, prior users of the API could transparently query the dict using the platform value and receive back a reasonable response. The current virtual environment would be retrievable by getting INSTALL_SCHEMES[INSTALL_SCHEMES['current']], and the installed scheme retrievable by getting INSTALL_SCHEMES['install']. Open Issues ========== By default, OS X Framework installs use a format that is different than the one specified above. Possible responses would be to change the layout just for OS X, but special-casing as little as possible. The double indirection to get the current install scheme is a little ugly, but I am not sure what would be better. This explicitly removes the compatibility code in distutils for 2.x install layouts (particularly layouts pre-2.3). This is not considered a major issue, as 2.2 code will largely not be compatible with 3.3, and an individual user could add a custom layout if needed. Based on actual use, some of the distinctions that appeared necessary when distutils and sysconfig were first written (platlib, purelib, include, headers) seem to be duplicative. As written above, the distinctions are maintained for ease of moving to the standardized layout, but it may be worth choosing just one name and sticking with it.
On Fri, Oct 28, 2011 at 5:53 PM, VanL <van.lindberg@gmail.com> wrote:
In part inspired by the virtualenv-in-the-stdlib PEP, I figured that it might be a good time to draft up a PEP to fix one of my regular annoyances: the ever-so-slightly different layouts for Python between platforms.
Is this something that Python even *can* reasonably control, particularly on the various Linux distributions? What might be helpful would be a few more symbolics (if any are actually missing) and a few test environments that use something unexpected for each value, so that you *will* notice if you have hardcoded assumptions specific to your own setup. -jJ
Hi Jim, On Oct 30, 2011 5:58 PM, "Jim Jewett" <jimjjewett@gmail.com> wrote:
Is this something that Python even *can* reasonably control, particularly on the various Linux distributions?
In my experience, the location for the Python environment changes a bit, but the internal layout is general consistent with what is set out in sysconfig and distutils.command.install. The most unique layout I have seen is for Ubuntu, which adds a vendor-packages directory. I would love to be corrected in this regard.
What might be helpful would be a few more symbolics (if any are actually missing) and a few test environments that use something unexpected for each value, so that you *will* notice if you have hardcoded assumptions specific to your own setup.
The suggested values are taken from the stdlib, not from my own setup, so if someone is generating or using different values, they are not coming from the stdlib. Thanks, Van
On Tue, Nov 01, 2011 at 12:08:22PM -0500, VanL wrote:
On Oct 30, 2011 5:58 PM, "Jim Jewett" <jimjjewett@gmail.com> wrote:
Is this something that Python even *can* reasonably control, particularly on the various Linux distributions?
In my experience, the location for the Python environment changes a bit, but the internal layout is general consistent with what is set out in sysconfig and distutils.command.install. The most unique layout I have seen is for Ubuntu, which adds a vendor-packages directory.
Debian 6 "squeeze" added dist-packages for 3rd-party modules installed via apt/dpkg. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
participants (3)
-
Jim Jewett -
Oleg Broytman -
VanL