[Python-Dev] PEP 411: Provisional packages in the Python standard library

[Eli Bendersky]

Hi all,

Following the intensive and fruitful discussion of the (now rejected)
PEP 408 (http://mail.python.org/pipermail/python-dev/2012-January/115850.html),
we've drafted PEP 411 to summarize the conclusions with regards to the
process of marking packages provisional. Note that this is an
informational PEP, and that for the sake of completeness it duplicates
some of the contents of PEP 408.

It is pasted below, as well as online at
http://www.python.org/dev/peps/pep-0411/.

Comments are welcome.

Eli

------------------------------------------------


PEP: 411
Title: Provisional packages in the Python standard library
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan at gmail.com>,
        Eli Bendersky <eliben at gmail.com>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 2012-02-10
Python-Version: 3.3
Post-History: 2012-02-10


Abstract
========

The process of including a new package into the Python standard library is
hindered by the API lock-in and promise of backward compatibility implied by
a package being formally part of Python.  This PEP describes a methodology
for marking a standard library package "provisional" for the period of a single
minor release.  A provisional package may have its API modified prior to
"graduating" into a "stable" state.  On one hand, this state provides the
package with the benefits of being formally part of the Python distribution.
On the other hand, the core development team explicitly states that no promises
are made with regards to the the stability of the package's API, which may
change for the next release.  While it is considered an unlikely outcome,
such packages may even be removed from the standard library without a
deprecation period if the concerns regarding their API or maintenante prove
well-founded.


Proposal - a documented provisional state
=========================================

Whenever the Python core development team decides that a new package should be
included into the standard library, but isn't entirely sure about whether the
package's API is optimal, the package can be included and marked as
"provisional".

In the next minor release, the package may either be "graduated" into a normal
"stable" state in the standard library, or be rejected and removed entirely
from the Python source tree.  If the package ends up graduating into the
stable state after being provisional for a minor release, its API may be
changed according to accumulated feedback.  The core development team
explicitly makes no guarantees about API stability and backward compatibility
of provisional packages.


Marking a package provisional
-----------------------------

A package will be marked provisional by including the following paragraph as
a note at the top of its documentation page:

    The <X> package has been included in the standard library on a
    provisional basis. While major changes are not anticipated, as long as
    this notice remains in place, backwards incompatible changes are
    permitted if deemed necessary by the standard library developers. Such
    changes will not be made gratuitously - they will occur only if
    serious API flaws are uncovered that were missed prior to inclusion of
    the package.

Moving a package from the provisional to the stable state simply implies
removing this note from its documentation page.


Which packages should go through the provisional state
------------------------------------------------------

We expect most packages proposed for addition into the Python standard library
to go through a minor release in the provisional state. There may, however,
be some exceptions, such as packages that use a pre-defined API (for example
``lzma``, which generally follows the API of the existing ``bz2`` package),
or packages with an API that has wide acceptance in the Python development
community.

In any case, packages that are proposed to be added to the standard library,
whether via the provisional state or directly, must fulfill the acceptance
conditions set by PEP 2.

Criteria for "graduation"
-------------------------

In principle, most provisional packages should eventually graduate to the
stable standard library.  Some reasons for not graduating are:

* The package may prove to be unstable or fragile, without sufficient developer
  support to maintain it.
* A much better alternative package may be found during the preview release.

Essentially, the decision will be made by the core developers on a per-case
basis.  The point to emphasize here is that a packages's inclusion in the
standard library as "provisional" in some release does not guarantee it will
continue being part of Python in the next release.


Rationale
=========

Benefits for the core development team
--------------------------------------

Currently, the core developers are really reluctant to add new interfaces to
the standard library.  This is because as soon as they're published in a
release, API design mistakes get locked in due to backward compatibility
concerns.

By gating all major API additions through some kind of a provisional mechanism
for a full release, we get one full release cycle of community feedback
before we lock in the APIs with our standard backward compatibility guarantee.

We can also start integrating provisional packages with the rest of the standard
library early, so long as we make it clear to packagers that the provisional
packages should not be considered optional.  The only difference between
provisional APIs and the rest of the standard library is that provisional APIs
are explicitly exempted from the usual backward compatibility guarantees.

Benefits for end users
----------------------

For future end users, the broadest benefit lies in a better "out-of-the-box"
experience - rather than being told "oh, the standard library tools for task X
are horrible, download this 3rd party library instead", those superior tools
are more likely to be just be an import away.

For environments where developers are required to conduct due diligence on
their upstream dependencies (severely harming the cost-effectiveness of, or
even ruling out entirely, much of the material on PyPI), the key benefit lies
in ensuring that all packages in the provisional state are clearly under
python-dev's aegis from at least the following perspectives:

* Licensing:  Redistributed by the PSF under a Contributor Licensing Agreement.
* Documentation: The documentation of the package is published and organized via
  the standard Python documentation tools (i.e. ReST source, output generated
  with Sphinx and published on http://docs.python.org).
* Testing: The package test suites are run on the python.org buildbot fleet
  and results published via http://www.python.org/dev/buildbot.
* Issue management: Bugs and feature requests are handled on
  http://bugs.python.org
* Source control: The master repository for the software is published
  on http://hg.python.org.


Candidates for provisional inclusion into the standard library
==============================================================

For Python 3.3, there are a number of clear current candidates:

* ``regex`` (http://pypi.python.org/pypi/regex) - approved by Guido [#]_.
* ``daemon`` (PEP 3143)
* ``ipaddr`` (PEP 3144)

Other possible future use cases include:

* Improved HTTP modules (e.g. ``requests``)
* HTML 5 parsing support (e.g. ``html5lib``)
* Improved URL/URI/IRI parsing
* A standard image API (PEP 368)
* Encapsulation of the import state (PEP 368)
* Standard event loop API (PEP 3153)
* A binary version of WSGI for Python 3 (e.g. PEP 444)
* Generic function support (e.g. ``simplegeneric``)


Rejected alternatives and variations
====================================

See PEP 408.


References
==========

.. [#] http://mail.python.org/pipermail/python-dev/2012-January/115962.html

Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: