[pypy-dev] needed 18 month plan of what we would do in developing PyPy if we got funding.

[Armin Rigo]

Hello Samuele,

Let me try to fill some bits in your list in an attempt to make it more nicely
linear...

The resulting document is in the svn repository at doc/EU_funding/plan.txt. 
I'm attaching a copy to this e-mail. Wwow, I somehow managed to make it look
very nicely structured with clear categories :-)

Everything in there is wide open for discussion of course! That's also quite
my own point of view (as I guess you'll see if only by the fact that
"infrastructure" is the last of the 6 categories ;-) but it should maybe be
put much earlier, if the categories are supposed to be ordered by starting
date).


A few Psyco-specific questions from Samuele...

> - integration of Psyco technology (rewrite to translated Python plus 
> automatic code generation of all what can be deduced from our PyPy codebase 
> or ???)

For both Psyco and Stackless I expect that we will divide them in two parts, 
the one that we tediously maintained from the Python source, and the one that 
is really unique to the project. For the first part we will try as much as 
possible to replace it by hooking into the translator. For the second part, it 
depends, I guess it can also be divided into fairly generic stuff that can be 
written in Python (interp- or even app-level) and stuff that really depends on 
the low-level.

More specifically (this only goes as far as I understand Stackless):


+ in PyPy's ---------- Psyco ------------------- Stackless -------

translator     translate the PyPy           translate regular functions
               interpreter into             into Continuation-Passing
               an abstract interpreter      Style (CPS) (Stackless 1.0)

pure Python    binding logic,               high-level tasklet aspects,
               profiler heuristics          scheduling
               
run-time       machine code emission,       C stack switching
               run-time support for         (Stackless 2.0)
               the abstact interpreter

In both case it might be possible to reduce the "run-time" part drastically,
thus making the tool essentially independent from a low-level run-time -- it
would work in Java too :-)  For Psyco this would be done using a generic 
virtual machine as the one I just finished (yup, Psyco works on PowerPC now, 
and probably on any 32-bit machine :-)  -- it would be done by writing the 
virtual machine's implementation itself in interpreter-level Python. For 
Stackless you'd have full continuations using the translator-to-CPS, which 
means essentially that you don't need C stack switching if you don't want it 
(you'd get a full Stackless 1.0).

Oh yes, BTW, thread pickling is another example of what the translator can
help make possible :-) It would be possible because to pickle a thread state
cleanly you really need all information you can get about the data structures
and where they are on the stack. A translator could emit special C code that
keeps track of that in a way that would be extremely tedious to do manually.

> - need to improve profiling approach to find hotspots for Psyco ?  (I don't 
> know how Psyco is doing about this now, Armin?)

The profiler is already reasonable, I think, but more testing would be cool.

> - optimizations for lookup/globals while
> supporting dynamic changes (things AFAIK Psyco does not address completely 
> ? is that correct)

Yes. Global lookups are cached but still work correctly in presence of dynamic
changes, but what doesn't at the moment is lookups of methods in new-style
classes.


A bientôt,

Armin.