[Python-Dev] Psyco requests and patches

[Armin Rigo]

Hello everybody,

As I have been putting more work into Psyco to make sure it will work
well with Python 2.3, I came up with a few support requests.

Non-technical stuff first.  People tend to think about Psyco as an
obscure and complex piece of software for gurus -- should they know
about it at all.  Of course I would argue instead that it is exactly the
piece of software that Python users should not have to worry about,
because it should be perfectly transparent.  As you can guess I would
even say that the first widely-used dynamic programming language to
natively include this kind of techniques would see a huge widening of
its application domain.  For Psyco to be a first step in that direction,
here are a few goals that I would like you to comment.

Publicity: well, I could do with a little more advertising, e.g. on
www.python.org or better yet in the docs and tutorials, at places where
performance is discussed.  Right now I am not sure that you can learn
about Psyco's existence unless you read python-dev or comp.lang.python.

Ease of use: currently the psyco.bind() interface is easy to use, but it
could be trivial instead of just easy.  It should actually be fully
transparent.  (That's what psyco.jit() promised to do, but right now it
badly sucks; keeping it around may have been an error, as people is
actually trying it and getting bad results.)

Portability: the easy part! Indeed, an empty module named 'psyco'
portably provides exactly the same functionality as Psyco everywhere... 
On non-supported platforms you just get nothing better than the current
interpreter performance, but nothing worse either.  (Besides, emitting
portable pseudo-code could still be a future option.  Of course,
emitting a platform's machine code is not that easy at all, but this
fact doesn't compromise portability.)

Ease of maintenance: the hard part.  Psyco is not currently ready for
that.  The implementation is way too obscure.  You have no guarantee if
I tell -- and I do -- that I plan to upgrade Psyco to all newer Python
versions for at least as long as the 2.x series is alive.  We need more
than one person involved.  (Note that upgrades are not as time-consuming
as, say, in the case of Christian Tismer's first Stackless
implementation that required a complete rewrite of the interpreter.)
Let's defer this point to the bottom of this e-mail.

Inclusion in the Standard Library or the interpreter core itself: not
right now! This is a guideline for the goals discussed above.  I might
be inclined to think that completion of all the previous goals would be
enough for consideration...?


-=- Python patches -=-

Psyco requires no change in the interpreter core -- however, there are a
couple of things that needed obfuscated workarounds or just don't
completely work at all.  A few minor changes would make it all much
cleaner.  I submitted three patches to SF
(http://www.python.org/sf/617309, http://www.python.org/sf/617311,
http://www.python.org/sf/617312).  The patches should be clear enough,
not introducing obscure "features" or measurable performance loss.  The
diffs are currently against the 2.2 branch -- it would greatly help
Psyco if the patches made their way into Python 2.2.2, which I
understood to be the candidate Python-in-a-tie.  I will submit 2.3
versions of the patches as well, of course.


-=- Maintenance of Psyco -=-

Psyco is currently highly obscure.  I want to rewrite it in Python. 
Still, however, there will always be the problem that for any feature
supported by Psyco someone must track the changes in the corresponding
feature in the interpreter, and port them.  (This doesn't concern the
features not supported by Psyco: for these it always calls their
up-to-date C implementation.) This process might be semi-automated with
a tool that would be able to dig into C code and analyse it, but this
would be sort of a hack.

We might do some thinking about this synchronization problem when the
discussion about Python 3 is raised again: how hard would it be e.g. to
write some parts of the interpreter directly in Python -- or at least in
some specific dialect that can be turned into C (a la Pyrex) for ease of
compilation? It may actually help make the interpreter more
understandable.  This looks a bit far-fetched, but I will never give up
my claims that high-level languages are just better than low-level ones
for most purposes, and writing an interpreter is no exception!

For example, there was a December 2000 proposal to rewrite the "cmath"
module in Python (http://www.python.org/sf/210838).  It never actually
made its way into the Standard library, which is too bad for Psyco,
because Psyco-with-cmath-in-Python would be faster than
Psyco-with-cmath-in-C.

Practically, the whole point of having a single high-level source code
is that you can do more than just one thing with it.  With C code, it is
a horrible hack to do anything else than compile and execute it.  With
Python code it's a different matter.  You can use this single source to
produce both the equivalent C code and to produce Psyco itself.  In each
case there is still an important part written directly in C, of course;
I am not (yet;-) suggesting that Python should be written *only* in
Python! Very roughly, the separation would be between the built-in
objects + abstract protocols in 'Objects/' (which could be written in
Python) and the core in 'Python/' (which could not).

The framework required to do such a thing looks complicated.  I would
argue that it probably isn't (and in any case it might be worth being
carefully considered); this extra complexity might be favorably balanced
if only just because writing Python in Python is easier than in C  :-)


A bientot,

Armin

PS: I just obtained my math Ph.D. -- it's now Psyco time again!