<div>While at a mini-PyPy sprint w/ Alex Gaynor of PyPy and Phil Jenvey of Jython, I decided to finally put the time in to update this PEP yet again.</div><div><br></div><div>The biggest changes is that the 100% branch coverage requirement has been replaced with "comprehensive" coverage from the tests. I think we are all enough grown-ups to not have to specify anything tighter than this. I also added a paragraph in the Details section about using the abstract C APIs (e.g., PyObject_GetItem) over type-specific ones (e.g., PyList_GetItem) in order to be more supportive of duck typing from the Python code. I figure the "be API compatible" assumes this, but mentioning it doesn't hurt (and should help make Raymond less angry =).</div>
<div><br></div><div><br></div><div>PEP: 399</div><div>Title: Pure Python/C Accelerator Module Compatibility Requirements</div><div>Version: $Revision: 88219 $</div><div>Last-Modified: $Date: 2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011) $</div>
<div>Author: Brett Cannon <<a href="mailto:brett@python.org">brett@python.org</a>></div><div>Status: Draft</div><div>Type: Informational</div><div>Content-Type: text/x-rst</div><div>Created: 04-Apr-2011</div><div>Python-Version: 3.3</div>
<div>Post-History: 04-Apr-2011, 12-Apr-2011, 17-Jul-2011</div><div><br></div><div>Abstract</div><div>========</div><div><br></div><div>The Python standard library under CPython contains various instances</div><div>of modules implemented in both pure Python and C (either entirely or</div>
<div>partially). This PEP requires that in these instances that the</div><div>C code *must* pass the test suite used for the pure Python code</div><div>so as to act as much as a drop-in replacement as reasonably possible</div>
<div>(C- and VM-specific tests are exempt). It is also required that new</div><div>C-based modules lacking a pure Python equivalent implementation get</div><div>special permission to be added to the standard library.</div>
<div><br></div><div><br></div><div>Rationale</div><div>=========</div><div><br></div><div>Python has grown beyond the CPython virtual machine (VM). IronPython_,</div><div>Jython_, and PyPy_ are all currently viable alternatives to the</div>
<div>CPython VM. The VM ecosystem that has sprung up around the Python</div><div>programming language has led to Python being used in many different</div><div>areas where CPython cannot be used, e.g., Jython allowing Python to be</div>
<div>used in Java applications.</div><div><br></div><div>A problem all of the VMs other than CPython face is handling modules</div><div>from the standard library that are implemented (to some extent) in C.</div><div>Since other VMs do not typically support the entire `C API of CPython`_</div>
<div>they are unable to use the code used to create the module. Often times</div><div>this leads these other VMs to either re-implement the modules in pure</div><div>Python or in the programming language used to implement the VM itself</div>
<div>(e.g., in C# for IronPython). This duplication of effort between</div><div>CPython, PyPy, Jython, and IronPython is extremely unfortunate as</div><div>implementing a module *at least* in pure Python would help mitigate</div>
<div>this duplicate effort.</div><div><br></div><div>The purpose of this PEP is to minimize this duplicate effort by</div><div>mandating that all new modules added to Python's standard library</div><div>*must* have a pure Python implementation _unless_ special dispensation</div>
<div>is given. This makes sure that a module in the stdlib is available to</div><div>all VMs and not just to CPython (pre-existing modules that do not meet</div><div>this requirement are exempt, although there is nothing preventing</div>
<div>someone from adding in a pure Python implementation retroactively).</div><div><br></div><div>Re-implementing parts (or all) of a module in C (in the case</div><div>of CPython) is still allowed for performance reasons, but any such</div>
<div>accelerated code must pass the same test suite (sans VM- or C-specific</div><div>tests) to verify semantics and prevent divergence. To accomplish this,</div><div>the test suite for the module must have comprehensive coverage of the</div>
<div>pure Python implementation before the acceleration code may be added.</div><div><br></div><div><br></div><div>Details</div><div>=======</div><div><br></div><div>Starting in Python 3.3, any modules added to the standard library must</div>
<div>have a pure Python implementation. This rule can only be ignored if</div><div>the Python development team grants a special exemption for the module.</div><div>Typically the exemption will be granted only when a module wraps a</div>
<div>specific C-based library (e.g., sqlite3_). In granting an exemption it</div><div>will be recognized that the module will be considered exclusive to</div><div>CPython and not part of Python's standard library that other VMs are</div>
<div>expected to support. Usage of ``ctypes`` to provide an</div><div>API for a C library will continue to be frowned upon as ``ctypes``</div><div>lacks compiler guarantees that C code typically relies upon to prevent</div>
<div>certain errors from occurring (e.g., API changes).</div><div><br></div><div>Even though a pure Python implementation is mandated by this PEP, it</div><div>does not preclude the use of a companion acceleration module. If an</div>
<div>acceleration module is provided it is to be named the same as the</div><div>module it is accelerating with an underscore attached as a prefix,</div><div>e.g., ``_warnings`` for ``warnings``. The common pattern to access</div>
<div>the accelerated code from the pure Python implementation is to import</div><div>it with an ``import *``, e.g., ``from _warnings import *``. This is</div><div>typically done at the end of the module to allow it to overwrite</div>
<div>specific Python objects with their accelerated equivalents. This kind</div><div>of import can also be done before the end of the module when needed,</div><div>e.g., an accelerated base class is provided but is then subclassed by</div>
<div>Python code. This PEP does not mandate that pre-existing modules in</div><div>the stdlib that lack a pure Python equivalent gain such a module. But</div><div>if people do volunteer to provide and maintain a pure Python</div>
<div>equivalent (e.g., the PyPy team volunteering their pure Python</div><div>implementation of the ``csv`` module and maintaining it) then such</div><div>code will be accepted. In those instances the C version is considered</div>
<div>the reference implementation in terms of expected semantics.</div><div><br></div><div>Any new accelerated code must act as a drop-in replacement as close</div><div>to the pure Python implementation as reasonable. Technical details of</div>
<div>the VM providing the accelerated code are allowed to differ as</div><div>necessary, e.g., a class being a ``type`` when implemented in C. To</div><div>verify that the Python and equivalent C code operate as similarly as</div>
<div>possible, both code bases must be tested using the same tests which</div><div>apply to the pure Python code (tests specific to the C code or any VM</div><div>do not follow under this requirement). The test suite is expected to</div>
<div>be extensive in order to verify expected semantics.</div><div><br></div><div>Acting as a drop-in replacement also dictates that no public API be</div><div>provided in accelerated code that does not exist in the pure Python</div>
<div>code. Without this requirement people could accidentally come to rely</div><div>on a detail in the accelerated code which is not made available to</div><div>other VMs that use the pure Python implementation. To help verify</div>
<div>that the contract of semantic equivalence is being met, a module must</div><div>be tested both with and without its accelerated code as thoroughly as</div><div>possible.</div><div><br></div><div>As an example, to write tests which exercise both the pure Python and</div>
<div>C accelerated versions of a module, a basic idiom can be followed::</div><div><br></div><div> import collections.abc</div><div> from test.support import import_fresh_module, run_unittest</div><div> import unittest</div>
<div><br></div><div> c_heapq = import_fresh_module('heapq', fresh=['_heapq'])</div><div> py_heapq = import_fresh_module('heapq', blocked=['_heapq'])</div><div><br></div><div><br></div>
<div> class ExampleTest(unittest.TestCase):</div><div><br></div><div> def test_heappop_exc_for_non_MutableSequence(self):</div><div> # Raise TypeError when heap is not a</div><div> # collections.abc.MutableSequence.</div>
<div> class Spam:</div><div> """Test class lacking many ABC-required methods</div><div> (e.g., pop())."""</div><div> def __len__(self):</div>
<div> return 0</div><div><br></div><div> heap = Spam()</div><div> self.assertFalse(isinstance(heap,</div><div> collections.abc.MutableSequence))</div>
<div> with self.assertRaises(TypeError):</div><div> self.heapq.heappop(heap)</div><div><br></div><div><br></div><div> class AcceleratedExampleTest(ExampleTest):</div><div><br></div><div> """Test using the accelerated code."""</div>
<div><br></div><div> heapq = c_heapq</div><div><br></div><div><br></div><div> class PyExampleTest(ExampleTest):</div><div><br></div><div> """Test with just the pure Python code."""</div>
<div><br></div><div> heapq = py_heapq</div><div><br></div><div><br></div><div> def test_main():</div><div> run_unittest(AcceleratedExampleTest, PyExampleTest)</div><div><br></div><div><br></div><div> if __name__ == '__main__':</div>
<div> test_main()</div><div><br></div><div><br></div><div>If this test were to provide extensive coverage for</div><div>``heapq.heappop()`` in the pure Python implementation then the</div><div>accelerated C code would be allowed to be added to CPython's standard</div>
<div>library. If it did not, then the test suite would need to be updated</div><div>until proper coverage was provided before the accelerated C code</div><div>could be added.</div><div><br></div><div>To also help with compatibility, C code should use abstract APIs on</div>
<div>objects to prevent accidental dependence on specific types. For</div><div>instance, if a function accepts a sequence then the C code should</div><div>default to using `PyObject_GetItem()` instead of something like</div>
<div>`PyList_GetItem()`. C code is allowed to have a fast path if the</div><div>proper `PyList_Check()` is used, but otherwise APIs should work with</div><div>any object that duck types to the proper interface instead of a</div>
<div>specific type.</div><div><br></div><div><br></div><div>Copyright</div><div>=========</div><div><br></div><div>This document has been placed in the public domain.</div><div><br></div><div><br></div><div>.. _IronPython: <a href="http://ironpython.net/">http://ironpython.net/</a></div>
<div>.. _Jython: <a href="http://www.jython.org/">http://www.jython.org/</a></div><div>.. _PyPy: <a href="http://pypy.org/">http://pypy.org/</a></div><div>.. _C API of CPython: <a href="http://docs.python.org/py3k/c-api/index.html">http://docs.python.org/py3k/c-api/index.html</a></div>
<div>.. _sqlite3: <a href="http://docs.python.org/py3k/library/sqlite3.html">http://docs.python.org/py3k/library/sqlite3.html</a></div><div><br></div>