[pypy-dev] Gothenburg sprint report

[Carl Friedrich Bolz]

Hi PyPy-dev!

Finally we have found some time to write a sprint report on what is
day three or day five of the sprint depending on how you count.

The reason for the uncertainty is that on Monday us lifers^Wfulltimers
gathered for two days of EU-report writing ahead of our review meeting
in January.  We'll spare you the details, but as most of the technical
reports are also part of the documention on our website, it's worth
mentioning that two documents:

     http://codespeak.net/pypy/dist/pypy/doc/low-level-encapsulation.html
     http://codespeak.net/pypy/dist/pypy/doc/translation-aspects.html

received significant updates in these days.  The former is a more
friendly read, start with that one.  Consensus opinion is that two
days of proof-reading and generally attempting to write nice prose is
even more exhausting than coding.

It was thus with some relief that on Wednesday morning we met to plan
our programming tasks.  The task that most combined enormity and
urgency was starting work on the the Just-In-Time compiler.  Samuele,
Armin, Eric, Carl and Arre all worked on this.  Eric and Arre
implemented yet another interpreter for a very simple toy language as
we needed something that would be simpler to apply a JIT to than the
PyPy behemoth and then moved on to looking at pypy-c's performance.

Carl, Armin and Samuele began writing a partial specializer for
low-level graphs, which is to say a way to rewrite a graph in the
knowledge that certain values are constants.  This is related to the
JIT effort because a JIT will partially specialize the interpreter
main-loop for a particular code object.  The combination of the two
sub-tasks will allow us to experiment with ways of applying these
specialization techniques.

Anders L and Nik repeatedly bashed their heads on the brick wall of
issues surrounding a working socket module, and are making good
progress although running a useful program is still a way off.  The
usual platform dependent horrors are, predictably, horrific.

Michael and the sole sprint newcomer this time, Johan, implemented
support in the translation toolchain for C's "long long" type which
involved taking Johan on a whirlwind tour of the annotator, rtyper and
C backend.  By Thursday lunchtime they had a translated pypy that
supported returning a long long value to app-level from exactly one
external function (os.lseek) and declared success.

Richard and Christian worked on exposing the raw stackless facilities
that have existed for some time in a useful way.  At this point they
have written a 'tasklet' module that is usable by RPython which is
probably best considered as an experiment on ways to write a module
that can expose stackless features to application level.

Adrien and Ludovic worked on increasing the flexibility of our parser
and compiler, in particular aiming to allow modification of the syntax
at runtime.  Their first success was to allow syntax to be *removed*
from PyPy: a valuable tool for making web frameworks harder to write::

    Clearly, the only way to cut down on the number of Web frameworks
    is to make it much harder to write them. If Guido were really going
    to help resolve this Web frameworks mess, he'd make Python a much
    uglier and less powerful language. I can't help but feel that
    that's the wrong way to go ;).

    (from Titus Brown's Advogato diary of 7 Dec 2005:
     http://www.advogato.org/person/titus/diary.html?start=134)

But they have also now allowed the definition of rules that can modify
the syntax tree on the fly.

On Friday morning a task group reshuffle gave Michael a unique
opportunity to work with Armin on the 'JIT' and Carl and Johan became
a new '__del__' taskforce.  By the end of the day, '__del__' was
supported in the genc backend when using the reference counting
garbage collector.  This also involved changing details at all levels
of the translation process, so by now Johan has seen a very little bit
of very large amounts of PyPy...

Arre and Eric (with some crucial hints from Richard) had a
successful hunt for performance problems: they changed
about 5 lines of code affecting the way we use the Boehm GC
which resulted in a remarkable 30-40% speed up in pystone after
translation.  Now if they can change 10 lines to get an 80%
improvement we _will_ be impressed :)

That's all for now.  We'll write a report on the last two days, err,
sometime. :)

Cheers,

Michael & Carl Friedrich