[pypy-dev] Porting PyPy/rpython to Python 3

[VanL]

Hi Maciej,

On Thu, Apr 16, 2015 at 3:48 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hi Van.
>
> First of all I'm really sorry if we ever gave an impression that
> working on porting RPython to Python 3 would not be welcomed and I
> would like to strongly disagree with that.
>
> What we did say (or wanted to say) is that we're unlikely to put a
> significant effort into doing the porting ourselves, however reviewing
> the pull requests is definitely within the scope of the work someone
> will do (if they languish, feel free to poke me personally).
>

That is good to hear, and consistent with the conversations I had at PyCon.
I wasn't discouraged from doing the work (as you can see from this thread),
but I just figured I needed to do it in a private fork that wouldn't be
accepted upstream. With this encouragement, you will start to see PRs from
me.

So it might be useful to explain what I am about, as that will give context
for what I am doing.

I am creating a language which I call spy, for "sub-python," that is a
strict semantic and syntactic subset of Python 3 that is AOT-compilable.
You can think of this as an evolved version of rpython, but in Python 3,
fully specified, and with a slightly different compilation model.

Why? I have lots of reasons, but the initial seed of the idea came out of
following a number of the "compile Python" projects over the years. What I
determined was that, regardless of the approach used, there was a ceiling
that each project reached *at almost exactly the same place.* For example,
the subset of Python that can be handled in rpython, shedskin, and
starkiller is almost feature-for-feature identical, despite the fact that
each of these projects used a different approach. In conversations with the
Numba folks, they are approaching this same boundary. Thus, there is a
natural AOT subset of Python... that isn't too far from full Python. What's
more, you all - the PyPy project - implemented full Python in this subset.

A few Q&As:

Why this project:  For fun. But I also have professional interest in a
couple things that I think this would fix.

First, deployment. There are some applications that it would *really* help
me to be able to drop a binary on a server and go. AOT compilation gives me
that, if I am willing to restrict myself to the spy subset.

Second, libraries loadable in either cPython, PyPy, or even Jython (all via
cffi).

Third, I have an interest in mobile, where I believe that this approach
would work better (it is similar to what Unity does, for example).

Why Py3?:  I like Py3 better.  I want to use function annotations to
provide information to the inferencing process. The annotations would
provide new roots for a the type inferencing process - and importantly,
would allow me to stop that process at module boundaries efficiently.

Type inferencing? The types in typing.py don't work for us:  Yes, but we
don't need to be restricted to those types only. There is no reason not to
declare the types that we need - for example to have UInt32 as a possible
type in a function annotation. This allows us to get rid of a fair amount
of "noise" is the rpython implementation (or at least to sequester it
better).

What do you mean fully specified?  Well, I want to have a spec as to what
is within this AOT subset. I think I can even detect unsupported language
constructs on import and throw a SyntaxError (or whichever error is
appropriate) if a non-AOT feature is used. If someone figures out how to
AOT compile a new feature, then it is added to the subset. I want to allow
for  different implementations.

What changes in the compilation model?   One big one: Be able to
effectively do type inferencing on a smaller piece of the program than "the
whole program." I would like to stop/start at either function boundaries or
at module boundaries. Declaring appropriate type information would let me
do that.

As for modules, I would just require that anything in __all__ be annotated.
Only functions in __all__ would be exported, and it would be an error to
access anything else. From a pypy/rpython perspective, this would allow
builtin modules (like sre) to be separately compiled and not have to
recompile the world when a change was made somewhere else.

Why bother with rpython at all?   It seemed the fastest way to get to where
I wanted to go:

1. Port rpython to Py3.
2. Reuse the PyPy3 target, but strip out everything that can't be AOT
compiled.
3. Evolve the rpython syntax to py3/spy, make it nice, doc it, start work
on new compilation model, etc.

Thanks,
Van
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150416/5dd3e196/attachment-0001.html>