[pypy-dev] day 6: breakthrough!

[holger krekel]

hi folks, 

two and a half years later ... in the very same room where the
first PyPy sprint happened ... we got to the point where PyPy
basically runs on its own, after whole-program type inference and
translation to C! 

The sixth day was a very productive day in various respects
and we still had a long and nice lunch break.  If you want to 
see the visual flow of events then revisit the pictures i took: 

    http://codespeak.net/~hpk/hildesheim2-sprint-www/

Here is a more detailed overview of what happended on the
sixth day: 

- Carl, Samuele and holger finished the Heidelberg sprint 
  announcement, updated the website and the announcement 
  was sent out. See the home page for more details and links. 

- Christian with initial help from Samuele went off to work 
  on the interpreter level marshal module which we will need 
  for better bootstrapping. Unfortunately, Christian had
  a bad stomach and missed much of the fun of day 6. 

- Armin and Holger as well as others continously restarted 
  the whole translation process to fix small bugs and issues
  here and there. In between 

  - they extended testing to allow applevel tests to easily
    access values setup_*'ed from interpreter level. 

  - they fixed the posix module which was completely broken 
    with respect to exception handling (you cannot let 
    escape interpreter-level exception to pypy application
    level without carefully wrapping them in an OperationError).
    Samuele helped with making os.strerror available 
    as an external (c-backend supported) function. 

  - analysed and fixed with Samuele and Carl segmentation faults ... 
    but note that GCC was  behaving very fine and it turned
    out that our c-code generator currently consumes more RAM 
    than GCC compiling the result at the moment :-) 
  
  - they also started refactoring the translation driving 
    interface (unfinished) 

  - Carl Friedrich and Samuele used and extended/fixed 
    the ll-interpreter which was sometimes able to 
    discover possible segfaults and problems earlier 
    (and clearer) than just running the C-backend. 

- Carl Friedrich started working on GC in PyPy
  in constant consultation with Samuele and did first working
  drafts in pypy/rpython/memory. Basically the whole memory
  simulator is working and gives nices errors if you do
  something strange like accessing freed or uninitialized.
  Furthermore they implemented annotation for memory addresses.

- Richard communicated with Eric van Riet Paap (who has made
  good progress on the llvm-backend off-sprint!) and
  refactored, extended LLVM a bit driven by test suggestions 
  from Armin. 
 
- the group discussed that there likely is a deeper exception
  handling problem lurking: implicitely exceptions are
  attached to the last (high level) operation in a block but
  when the RTyper converts this operation into possibly many
  low-level operations then it's unlikely that we can preserve
  the implicit assumption that only the last (low-level)
  operation possibly raises an exception. 

  solution: unknown at this point. 

  We should probably write down a few scenarios regarding
  exception handling describing the interactions/processing 
  of exceptions in flowobjspace/annotation/rtyping 

- Armin communicated with Niklaus Heidimann who meanwhile 
  has successfully implemented the _sre module as part of his 
  SoC project.  We'd like to  first integrate this as an 
  application level module but Niklaus wants to transform 
  it to an interpreter-level one soon. 

- After we got the initial breakthrough run (goal/app_example.py 
  running on top of translated non-faking PyPy) we continued 
  a bit with various example and fixes.  Running example code
  and modules showed very good results (and no segfaults). 

  After having drunk champagne Samuele quickly mapped the good time
  we were having into a better 'time' module for PyPy.  With that 
  we could easily run the unmodified pystone.py on top of PyPy
  and reached some 337 pystones (as opposed to 9 pystones
  with PyPy on top of CPython).  We are happy with this
  because translated PyPy seems to run stable and we are not using 
  any optimizations yet.  We quickly agreed that we only want to 
  tackle optimizations at this point that result in at least
  factors-of-2 speedups rather than 5-10% speedups. Again please
  note that PyPy - as opposed to other python implementations -
  generally tackles compliancy, proper implementation and
  tests first before diving into optimizations. 

ok, so much so far.  Today - on the last day - we are probably 
going to discuss on how to continue from here and also tell 
how you can actually run the translation and the translated PyPy 
yourselves (the current pypy-dist is able to but it currently
requires a bit of manual tweaks at the very invocation-end,
maybe you can already figure it out looking at the pictures
and the entry_point code :-). 

cheers, 

    holger