[Python-Dev] PEP 408 -- Standard library preview package

Fri Jan 27 14:21:33 CET 2012

Hello,

Following an earlier discussion on python-ideas [1], we would like to
propose the following PEP for review. Discussion is welcome. The PEP
can also be viewed in HTML form at
http://www.python.org/dev/peps/pep-0408/

[1] http://mail.python.org/pipermail/python-ideas/2012-January/013246.html

Eli

---------------------------

PEP: 408
Title: Standard library __preview__ package
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan at gmail.com>,
        Eli Bendersky <eliben at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2012-01-07
Python-Version: 3.3
Post-History: 2012-01-27

Abstract
========

The process of including a new module into the Python standard library is
hindered by the API lock-in and promise of backward compatibility implied by
a module being formally part of Python.  This PEP proposes a transitional
state for modules - inclusion in a special ``__preview__`` package for the
duration of a minor release (roughly 18 months) prior to full acceptance into
the standard library.  On one hand, this state provides the module with the
benefits of being formally part of the Python distribution.  On the other hand,
the core development team explicitly states that no promises are made with
regards to the module's eventual full inclusion into the standard library,
or to the stability of its API, which may change for the next release.

Proposal - the __preview__ package
==================================

Whenever the Python core development team decides that a new module should be
included into the standard library, but isn't entirely sure about whether the
module's API is optimal, the module can be placed in a special package named
``__preview__`` for a single minor release.

In the next minor release, the module may either be "graduated" into the
standard library (and occupy its natural place within its namespace, leaving the
``__preview__`` package), or be rejected and removed entirely from the Python
source tree.  If the module ends up graduating into the standard library after
spending a minor release in ``__preview__``, its API may be changed according
to accumulated feedback.  The core development team explicitly makes no
guarantees about API stability and backward compatibility of modules in
``__preview__``.

Entry into the ``__preview__`` package marks the start of a transition of the
module into the standard library.  It means that the core development team
assumes responsibility of the module, similarly to any other module in the
standard library.

Which modules should go through ``__preview__``
-----------------------------------------------

We expect most modules proposed for addition into the Python standard library
to go through a minor release in ``__preview__``. There may, however, be some
exceptions, such as modules that use a pre-defined API (for example ``lzma``,
which generally follows the API of the existing ``bz2`` module), or modules
with an API that has wide acceptance in the Python development community.

In any case, modules that are proposed to be added to the standard library,
whether via ``__preview__`` or directly, must fulfill the acceptance conditions
set by PEP 2.

It is important to stress that the aim of of this proposal is not to make the
process of adding new modules to the standard library more difficult.  On the
contrary, it tries to provide a means to add *more* useful libraries.  Modules
which are obvious candidates for entry can be added as before.  Modules which
due to uncertainties about the API could be stalled for a long time now have
a means to still be distributed with Python, via an incubation period in the
``__preview__`` package.

Criteria for "graduation"
-------------------------

In principle, most modules in the ``__preview__`` package should eventually
graduate to the stable standard library.  Some reasons for not graduating are:

* The module may prove to be unstable or fragile, without sufficient developer
  support to maintain it.
* A much better alternative module may be found during the preview release

Essentially, the decision will be made by the core developers on a per-case
basis.  The point to emphasize here is that a module's appearance in the
``__preview__`` package in some release does not guarantee it will continue
being part of Python in the next release.

Example
-------

Suppose the ``example`` module is a candidate for inclusion in the standard
library, but some Python developers aren't convinced that it presents the best
API for the problem it intends to solve.  The module can then be added to the
``__preview__`` package in release ``3.X``, importable via::

    from __preview__ import example

Assuming the module is then promoted to the the standard library proper in
release ``3.X+1``, it will be moved to a permanent location in the library::

    import example

And importing it from ``__preview__`` will no longer work.

Rationale
=========

Benefits for the core development team
--------------------------------------

Currently, the core developers are really reluctant to add new interfaces to
the standard library.  This is because as soon as they're published in a
release, API design mistakes get locked in due to backward compatibility
concerns.

By gating all major API additions through some kind of a preview mechanism
for a full release, we get one full release cycle of community feedback
before we lock in the APIs with our standard backward compatibility guarantee.

We can also start integrating preview modules with the rest of the standard
library early, so long as we make it clear to packagers that the preview
modules should not be considered optional.  The only difference between preview
APIs and the rest of the standard library is that preview APIs are explicitly
exempted from the usual backward compatibility guarantees.

Essentially, the ``__preview__`` package is intended to lower the risk of
locking in minor API design mistakes for extended periods of time.  Currently,
this concern can block new additions, even when the core development team
consensus is that a particular addition is a good idea in principle.

Benefits for end users
----------------------

For future end users, the broadest benefit lies in a better "out-of-the-box"
experience - rather than being told "oh, the standard library tools for task X
are horrible, download this 3rd party library instead", those superior tools
are more likely to be just be an import away.

For environments where developers are required to conduct due diligence on
their upstream dependencies (severely harming the cost-effectiveness of, or
even ruling out entirely, much of the material on PyPI), the key benefit lies
in ensuring that anything in the ``__preview__`` package is clearly under
python-dev's aegis from at least the following perspectives:

* Licensing:  Redistributed by the PSF under a Contributor Licensing Agreement.
* Documentation: The documentation of the module is published and organized via
  the standard Python documentation tools (i.e. ReST source, output generated
  with Sphinx and published on http://docs.python.org).
* Testing: The module test suites are run on the python.org buildbot fleet
  and results published via http://www.python.org/dev/buildbot.
* Issue management: Bugs and feature requests are handled on
  http://bugs.python.org
* Source control: The master repository for the software is published
  on http://hg.python.org.

Candidates for inclusion into __preview__
=========================================

For Python 3.3, there are a number of clear current candidates:

* ``regex`` (http://pypi.python.org/pypi/regex)
* ``daemon`` (PEP 3143)
* ``ipaddr`` (PEP 3144)

Other possible future use cases include:

* Improved HTTP modules (e.g. ``requests``)
* HTML 5 parsing support (e.g. ``html5lib``)
* Improved URL/URI/IRI parsing
* A standard image API (PEP 368)
* Encapsulation of the import state (PEP 368)
* Standard event loop API (PEP 3153)
* A binary version of WSGI for Python 3 (e.g. PEP 444)
* Generic function support (e.g. ``simplegeneric``)

Relationship with PEP 407
=========================

PEP 407 proposes a change to the core Python release cycle to permit interim
releases every 6 months (perhaps limited to standard library updates). If
such a change to the release cycle is made, the following policy for the
``__preview__`` namespace is suggested:

* For long term support releases, the ``__preview__`` namespace would always
  be empty.
* New modules would be accepted into the ``__preview__`` namespace only in
  interim releases that immediately follow a long term support release.
* All modules added will either be migrated to their final location in the
  standard library or dropped entirely prior to the next long term support
  release.

Rejected alternatives and variations
====================================

Using ``__future__``
--------------------

Python already has a "forward-looking" namespace in the form of the
``__future__`` module, so it's reasonable to ask why that can't be re-used for
this new purpose.

There are two reasons why doing so not appropriate:

1. The ``__future__`` module is actually linked to a separate compiler
directives feature that can actually change the way the Python interpreter
compiles a module.  We don't want that for the preview package - we just want
an ordinary Python package.

2. The ``__future__`` module comes with an express promise that names will be
maintained in perpetuity, long after the associated features have become the
compiler's default behaviour.  Again, this is precisely the opposite of what is
intended for the preview package - it is almost certain that all names added to
the preview will be removed at some point, most likely due to their being moved
to a permanent home in the standard library, but also potentially due to their
being reverted to third party package status (if community feedback suggests the
proposed addition is irredeemably broken).

Versioning the package
----------------------

One proposed alternative [1]_ was to add explicit versioning to the
``__preview__`` package, i.e. ``__preview34__``.  We think that it's better to
simply define that a module being in ``__preview__`` in Python 3.X will either
graduate to the normal standard library namespace in Python 3.X+1 or will
disappear from the Python source tree altogether.  Versioning the ``_preview__``
package complicates the process and does not align well with the main intent of
this proposal.

Using a package name without leading and trailing underscores
-------------------------------------------------------------

It was proposed [1]_ to use a package name like ``preview`` or ``exp``, instead
of ``__preview__``.  This was rejected in the discussion due to the special
meaning a "dunder" package name (that is, a name *with* leading and
trailing double-underscores) conveys in Python.  Besides, a non-dunder name
would suggest normal standard library API stability guarantees, which is not
the intention of the ``__preview__`` package.

Preserving pickle compatibility
-------------------------------

A pickled class instance based on a module in ``__preview__`` in release 3.X
won't be unpickle-able in release 3.X+1, where the module won't be in
``__preview__``.  Special code may be added to make this work, but this goes
against the intent of this proposal, since it implies backward compatibility.
Therefore, this PEP does not propose to preserve pickle compatibility.

Credits
=======

Dj Gilcrease initially proposed the idea of having a ``__preview__`` package
in Python [2]_.  Although his original proposal uses the name
``__experimental__``, we feel that ``__preview__`` conveys the meaning of this
package in a better way.

References
==========

.. [#] Discussed in this thread:
       http://mail.python.org/pipermail/python-ideas/2012-January/013246.html

.. [#] http://mail.python.org/pipermail/python-ideas/2011-August/011278.html

Copyright
=========

This document has been placed in the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

[Python-Dev] PEP 408 -- Standard library __preview__ package

[Python-Dev] PEP 408 -- Standard library preview package