[pypy-dev] PyPy support for Python 3 -- request for comments

[Antonio Cuni]

Hello pypy-dev,

in the past weeks, I and the other core developers have talked a lot about
supporting Python 3 in PyPy.  The task is huge and it's unlikely that it will
be completed shortly based only on volunteer work, so we came up with the
following proposal, which splits the work into several steps and sub-steps,
togheter with an estimate of how much money is needed to complete each one.

The plan is to publish the proposal on the blog, and publicly ask for
donations, with the hope to collect enough to cover the whole work.  However,
before putting it on the blog we would like to ask for your comments/thoughts
about it.  Any feedback will be appreciated!

Thank you,
Antonio (on the behalf of the PyPy team)


Python 3 on PyPy
=================

The release of Python 3 has been a major undertaking for the Python community,
both technically and socially.  So far PyPy implements only the version 2 of
the Python language, which creates a very dangerous potential for a community
split.  We believe that by supporting both versions of the language we will
help to fill the gap.

This project should help both the part of the Python community which is
reluctant to use PyPy because it does not support Python 3, and the part which
is reluctant to move to Python 3 because they are already PyPy users.

However, porting PyPy to Python 3 requires a lot of work, and it will take a
long time before we can complete it by relying only on volunteer work.  Thus,
we are asking the community to help with funding the necessary work, to make
it happen faster.

High level description
-----------------------

The goal of this project is to write an interpreter that interprets version
3 of Python language. To be precise we would aim at having Python 3.2
interpreter together in the same codebase as python 2.7 one.

At the end of the project, it will be possible to decide at translation time
whether to build an interpreter which supports Python 2.7 or Python 3.2 and
both versions will be nightly tested and available from nightly builds.

The focus of this project is on compatibility, not performance.  In
particular, it might be possible that the resulting Python 3 interpreter will
be slower than the Python 2 one.  If needed, optimizing and making it more JIT
friendly will be the scope of a separate project.

Step 1: core language
----------------------

In this step, we implement all the changes to the core language,
i.e. everything which is not in the extension modules.  This includes, but it
is not necessarily limited to the following items, which are split into two
big areas:

* **Sub-step 1.1**: string vs unicode and I/O:

     - adapt the existing testing infrastructure to support running Python 3 code

     - string vs bytes: the interpreter uses unicode strings everywhere.

     - the ``print`` function

     - ``open`` is now an alias for ``io.open``, removal of the old file type.

     - string formatting (for the part which is not already implemented in Python
       2.7)

     - the _io module (for the part which is not already implemented in Python
       2.7)

     - syntactic changes to make ``io.py`` importable (in particular:
       ``metaclass=...`` in class declarations)

     - **Estimate cost**: 35.000 $

* **Sub-step 1.2**: other syntactic changes, builtin types and functions,
     exceptions:

     - views and iterators instead of lists (e.g., ``dict.items()``, ``map``,
       ``range`` & co.)

     - new rules for ordering comparisons

     - removal of old-style classes

     - int/long unification

     - function annotations

     - smaller syntax changes, such as keyword-only arguments, ``nonlocal``,
       extended iterable unpacking, set literals, dict and set comprehension, etc.

     - changes to exceptions: ``__traceback__`` attribute, chained exceptions,
       ``del e`` at the end of the except block, etc.

     - changes to builtins: ``super``, ``input``, ``next()``, etc.

     - improved ``with`` statement

     - **Estimate cost**: 25.000 $


Note that the distinction between sub-steps 1.1 and 1.2 is blur, and it might be
possible that during the development we will decide to move items between the
two sub-steps, as needed.

For more information, look at the various "What's new" documents:

- http://docs.python.org/py3k/whatsnew/3.0.html

- http://docs.python.org/py3k/whatsnew/3.1.html

- http://docs.python.org/py3k/whatsnew/3.2.html

**Total estimate cost**: 60.000 $


Step 2: extension modules
--------------------------

In this step, we implement all the changes to the extension modules which are
written in C in CPython.  This includes, but it is not necessarily limited to:

- ``collections``, ``gzip``, ``bz2``, ``decimal``, ``itertools``, ``re``,
   ``functools``, ``pickle``, ``_elementtree``, ``math``, etc.

**Estimate cost**: this is hard to do at this point, we will be able to give a
more precise estimate as soon as Step 1 is completed.  As a reference, it
should be possible to complete it with 35.000 $

Step 3: cpyext
--------------

The ``cpyext`` module allows to load CPython C extensions in PyPy.  Since the
C API changed a lot between Python 2.7 and Python 3.2, ``cpyext`` will not
work out of the box in the Python 3 PyPy interpreter.  In this step, we will
adapt it to work with Python 3 as well.

Note that, even for Python 2, ``cpyext`` is still in a beta state.  In
particular, not all extension modules compile and load correctly.  As a
consequence, the same will be true for Python 3 as well.  As a general rule,
we expect that if a Python 2 module works with ``cpyext``, the corresponding
Python 3 version will also work when this step is completed, although the
details might vary depending on the exact C extension module.

**Estimate cost**: 10.000 $