[Python-ideas] PEP 4XX: Adding sys.implementation

Eric Snow ericsnowcurrently at gmail.com
Sat Apr 28 08:06:29 CEST 2012


Here's an update to the PEP.  Though I have indirect or old feedback
already, I'd love to hear from the other main Python implementations,
particularly regarding the version variable.  Thanks.

-eric

-------------------------------------------------------------


PEP: 421
Title: Adding sys.implementation
Version: $Revision$
Last-Modified: $Date$
Author: Eric Snow <ericsnowcurrently at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 26-April-2012
Post-History: 26-April-2012


Abstract
========

This PEP introduces a new variable for the ``sys`` module:
``sys.implementation``.  The variable holds consolidated information
about the implementation of the running interpreter.  Thus
``sys.implementation`` is the source to which the standard library may
look for implementation-specific information.

The proposal in this PEP is in line with a broader emphasis on making
Python friendlier to alternate implementations.  It describes the new
variable and the constraints on what that variable contains.  The PEP
also explains some immediate use cases for ``sys.implementation``.


Motivation
==========

For a number of years now, the distinction between Python-the-language
and CPython (the reference implementation) has been growing.  Most of
this change is due to the emergence of Jython, IronPython, and PyPy as
viable alternate implementations of Python.

Consider, however, the nearly two decades of CPython-centric Python
(i.e. most of its existance).  That focus had understandably contributed
to quite a few CPython-specific artifacts both in the standard library
and exposed in the interpreter.  Though the core developers have made an
effort in recent years to address this, quite a few of the artifacts
remain.

Part of the solution is presented in this PEP:  a single namespace on
which to consolidate implementation specifics.  This will help focus
efforts to differentiate the implementation specifics from the language.
Additionally, it will foster a multiple-implementation mindset.


Proposal
========

We will add ``sys.implementation``, in the ``sys`` module, as a namespace
to contain implementation-specific information.

The contents of this namespace will remain fixed during interpreter
execution and through the course of an implementation version.  This
ensures behaviors don't change between versions which depend on variables
in ``sys.implementation``.

``sys.implementation`` will be a dictionary, as opposed to any form of
"named" tuple (a la ``sys.version_info``).  This is partly because it
doesn't have meaning as a sequence, and partly because it's a potentially
more variable data structure.

The namespace will contain at least the variables described in the
`Required Variables`_ section below.  However, implementations are free
to add other implementation information there.  Some possible extra
variables are described in the `Other Possible Variables`_ section.

This proposal takes a conservative approach in requiring only two
variables.  As more become appropriate, they may be added with discretion.


Required Variables
--------------------

These are variables in ``sys.implementation`` on which the standard
library would rely, meaning implementors must define them:

name
   the name of the implementation (case sensitive).

version
   the version of the implementation, as opposed to the version of the
   language it implements.  This would use a standard format, similar to
   ``sys.version_info`` (see `Version Format`_).


Other Possible Variables
------------------------

These variables could be useful, but don't necessarily have a clear use
case presently:

cache_tag
   a string used for the PEP 3147 cache tag (e.g. 'cpython33' for
   CPython 3.3).  The name and version from above could be used to
   compose this, though an implementation may want something else.
   However, module caching is not a requirement of implementations, nor
   is the use of cache tags.

repository
   the implementation's repository URL.

repository_revision
   the revision identifier for the implementation.

build_toolchain
   identifies the tools used to build the interpreter.

url (or website)
   the URL of the implementation's site.

site_prefix
   the preferred site prefix for this implementation.

runtime
   the run-time environment in which the interpreter is running.

gc_type
   the type of garbage collection used.


Version Format
--------------

A main point of ``sys.implementation`` is to contain information that
will be used in the standard library.  In order to facilitate the
usefulness of a version variable, its value should be in a consistent
format across implementations.

XXX Subject to feedback

As such, the format of ``sys.implementation['version']`` must follow that
of ``sys.version_info``, which is effectively a named tuple.  It is a
familiar format and generally consistent with normal version format
conventions.


Rationale
=========

The status quo for implementation-specific information gives us that
information in a more fragile, harder to maintain way.  It's spread out
over different modules or inferred from other information, as we see with
``platform.python_implementation()``.

This PEP is the main alternative to that approach.  It consolidates the
implementation-specific information into a single namespace and makes
explicit that which was implicit.  The ``sys`` module should old the new
namespace because ``sys`` is the depot for interpreter-centric variables
and functions.

With the single-namespace-under-sys so straightforward, no alternatives
have been considered for this PEP.


Discussion
==========

The topic of ``sys.implementation`` came up on the python-ideas list in
2009, where the reception was broadly positive [1]_.  I revived the
discussion recently while working on a pure-python ``imp.get_tag()`` [2]_.
The messages in `issue #14673`_ are also relevant.


Use-cases
=========

``platform.python_implementation()``
------------------------------------

"explicit is better than implicit"

The ``platform`` module guesses the python implementation by looking for
clues in a couple different ``sys`` variables [3]_.  However, this
approach is fragile.  Beyond that, it's limited to those implementations
that core developers have blessed by special-casing them in the
``platform`` module.

With ``sys.implementation`` the various implementations would
*explicitly* set the values in their own version of the ``sys`` module.

Aside from the guessing, another concern is that the ``platform`` module
is part of the stdlib, which ideally would minimize implementation details
such as would be moved to ``sys.implementation``.

Any overlap between ``sys.implementation`` and the ``platform`` module
would simply defer to ``sys.implementation`` (with the same interface in
``platform`` wrapping it).


Cache Tag Generation in Frozen Importlib
----------------------------------------

PEP 3147 defined the use of a module cache and cache tags for file names.
The importlib bootstrap code, frozen into the Python binary as of 3.3,
uses the cache tags during the import process.  Part of the project to
bootstrap importlib has been to clean out of `Python/import.c` any code
that did not need to be there.

The cache tag defined in `Python/import.c` was hard-coded to
``"cpython" MAJOR MINOR`` [4]_.  For importlib the options are either
hard-coding it in the same way, or guessing the implementation in the
same way as does ``platform.python_implementation()``.

As long as the hard-coded tag is limited to CPython-specific code, it's
livable.  However, inasmuch as other Python implementations use the
importlib code to work with the module cache, a hard-coded tag would
become a problem..

Directly using the ``platform`` module in this case is a non-starter.  Any
module used in the importlib bootstrap must be built-in or frozen,
neither of which apply to the ``platform`` module.  This is the point that
led to the recent interest in ``sys.implementation``.

Regardless of the outcome for the implementation name used, another
problem relates to the version used in the cache tag.  That version is
likely to be the implementation version rather than the language version.
However, the implementation version is not readily identified anywhere in
the standard library.


Implementation-Specific Tests
-----------------------------

Currently there are a number of implementation-specific tests in the test
suite under ``Lib/test``.  The test support module (`Lib/test/support.py`_)
provides some functionality for dealing with these tests.  However, like
the ``platform`` module, ``test.support`` must do some guessing that
``sys.implementation`` would render unnecessary.


Jython's ``os.name`` Hack
-------------------------

XXX

http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512


Feedback From Other Python Implementators
=========================================

IronPython
----------

XXX

Jython
------

XXX

PyPy
----

XXX


Past Efforts
============

PEP 3139
--------

This PEP from 2008 recommended a clean-up of the ``sys`` module in part
by extracting implementation-specific variables and functions into a
separate module.  PEP 421 is a much lighter version of that idea.
While PEP 3139 was rejected, its goals are reflected in PEP 421 to a
large extent, though with a much lighter approach.


PEP 399
-------

This informational PEP dictates policy regarding the standard library,
helping to make it friendlier to alternate implementations.  PEP 421 is
proposed in that same spirit.


Open Issues
===========

* What are the long-term objectives for ``sys.implementation``?

  - possibly pull in implementation details from the main ``sys`` namespace
    and elsewhere (PEP 3137 lite).

* Alternatives to the approach dictated by this PEP?

* ``sys.implementation`` as a proper namespace rather than a dict.  It
  would be it's own module or an instance of a concrete class.


Implementation
==============

The implementatation of this PEP is covered in `issue #14673`_.


References
==========

.. [1] http://mail.python.org/pipermail/python-dev/2009-October/092893.html

.. [2] http://mail.python.org/pipermail/python-ideas/2012-April/014878.html

.. [3] http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247

.. [4] http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c#l121

.. [5] Examples of implementation-specific handling in test.support:

| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509
| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246
| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252
| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275

.. _issue #14673: http://bugs.python.org/issue14673

.. _Lib/test/support.py:
http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py

Copyright
=========

    This document has been placed in the public domain.



More information about the Python-ideas mailing list