[pypy-dev] Porting PyPy/rpython to Python 3

Ronan Lamy ronan.lamy at gmail.com
Thu Apr 16 18:39:37 CEST 2015


Le 16/04/15 14:55, VanL a écrit :
> Hi Maciej,
>
> On Thu, Apr 16, 2015 at 3:48 AM, Maciej Fijalkowski <fijall at gmail.com
> <mailto:fijall at gmail.com>> wrote:
>
>     Hi Van.
>
>     First of all I'm really sorry if we ever gave an impression that
>     working on porting RPython to Python 3 would not be welcomed and I
>     would like to strongly disagree with that.
>
>     What we did say (or wanted to say) is that we're unlikely to put a
>     significant effort into doing the porting ourselves, however reviewing
>     the pull requests is definitely within the scope of the work someone
>     will do (if they languish, feel free to poke me personally).
>
>
> That is good to hear, and consistent with the conversations I had at
> PyCon. I wasn't discouraged from doing the work (as you can see from
> this thread), but I just figured I needed to do it in a private fork
> that wouldn't be accepted upstream. With this encouragement, you will
> start to see PRs from me.
>
> So it might be useful to explain what I am about, as that will give
> context for what I am doing.
>
> I am creating a language which I call spy, for "sub-python," that is a
> strict semantic and syntactic subset of Python 3 that is AOT-compilable.
> You can think of this as an evolved version of rpython, but in Python 3,
> fully specified, and with a slightly different compilation model.
>
> Why? I have lots of reasons, but the initial seed of the idea came out
> of following a number of the "compile Python" projects over the years.
> What I determined was that, regardless of the approach used, there was a
> ceiling that each project reached *at almost exactly the same place.*
> For example, the subset of Python that can be handled in rpython,
> shedskin, and starkiller is almost feature-for-feature identical,
> despite the fact that each of these projects used a different approach.
> In conversations with the Numba folks, they are approaching this same
> boundary. Thus, there is a natural AOT subset of Python... that isn't
> too far from full Python. What's more, you all - the PyPy project -
> implemented full Python in this subset.

That's an interesting goal. Using RPython to create a standardised AOT 
subset of Python is something I've had in the back of my mind for quite 
a while.

> A few Q&As:
>
> Why this project:  For fun. But I also have professional interest in a
> couple things that I think this would fix.
>
> First, deployment. There are some applications that it would *really*
> help me to be able to drop a binary on a server and go. AOT compilation
> gives me that, if I am willing to restrict myself to the spy subset.

You can basically do that today with RPython - provided you don't care 
too much about the size of the binary.
>
> Second, libraries loadable in either cPython, PyPy, or even Jython (all
> via cffi).

RPython can almost do that already. All that's needed is a nice way of 
specifying the interface and a bit of tooling to generate cffi bindings 
from that.

> Third, I have an interest in mobile, where I believe that this approach
> would work better (it is similar to what Unity does, for example).
>
> Why Py3?:  I like Py3 better.  I want to use function annotations to
> provide information to the inferencing process. The annotations would
> provide new roots for a the type inferencing process - and importantly,
> would allow me to stop that process at module boundaries efficiently.

Syntax aside, it's already in RPython, cf. 
rpython.rlib.objectmodel.enforceargs and rpython.rlib.signature (which 
sadly are incompatible with each other).

> Type inferencing? The types in typing.py don't work for us:  Yes, but we
> don't need to be restricted to those types only. There is no reason not
> to declare the types that we need - for example to have UInt32 as a
> possible type in a function annotation. This allows us to get rid of a
> fair amount of "noise" is the rpython implementation (or at least to
> sequester it better).

Unless you want to get rid of type inferencing altogether (or strictly 
restrict it to function locals, perhaps), that noise isn't going away.

> What do you mean fully specified?  Well, I want to have a spec as to
> what is within this AOT subset. I think I can even detect unsupported
> language constructs on import and throw a SyntaxError (or whichever
> error is appropriate) if a non-AOT feature is used. If someone figures
> out how to AOT compile a new feature, then it is added to the subset. I
> want to allow for  different implementations.

Deciding whether a program is valid RPython requires full typing 
information. It's always going to be rather more complicated than 
validating syntax, but I guess it goes with the AOT territory.
Anyway, if you wanted to create a spec for RPython as it exists now, you 
could take a note of which bytecodes are unsupported and grep for 
AnnotatorError.
>
> What changes in the compilation model?   One big one: Be able to
> effectively do type inferencing on a smaller piece of the program than
> "the whole program." I would like to stop/start at either function
> boundaries or at module boundaries. Declaring appropriate type
> information would let me do that.

Removing whole-program type inferencing would change the character of 
the language a lot. In addition to function signatures, you'd also need 
to declare types for global constants, class and instance attributes, ...

> As for modules, I would just require that anything in __all__ be
> annotated. Only functions in __all__ would be exported, and it would be
> an error to access anything else. From a pypy/rpython perspective, this
> would allow builtin modules (like sre) to be separately compiled and not
> have to recompile the world when a change was made somewhere else.

Well, separate compilation has long been a wanted feature for RPython. 
Whole-program inferencing is a major roadblock, so you might have more 
luck implementing this, but it's not the only obstacle.

> Why bother with rpython at all?   It seemed the fastest way to get to
> where I wanted to go:
>
> 1. Port rpython to Py3.
> 2. Reuse the PyPy3 target, but strip out everything that can't be AOT
> compiled.

I don't understand how this fits with the rest of your plans. By 
definition, PyPy3 will be able to run spy, so why do you need your own 
interpreter?

> 3. Evolve the rpython syntax to py3/spy, make it nice, doc it, start
> work on new compilation model, etc.
>
> Thanks,
> Van
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



More information about the pypy-dev mailing list