[pypy-dev] Python parsers...

holger krekel hpk at trillke.net
Fri Aug 1 13:03:50 CEST 2003

Hi Jonathan,

[Jonathan David Riehl Wed, Jul 30, 2003 at 06:20:23PM -0500]
> On Thu, 31 Jul 2003, holger krekel wrote:
> > [Jeremy Hylton Wed, Jul 30, 2003 at 06:09:41PM -0400]
> > >
> > > Are you interested in integrating it into the compiler package?  The Python
> > > CVS trunk is now wide open for development, and the compiler package could
> > > definitely use some tidying up.
> ... 
> > note, that we mirror the Python-CVS trunk into our svn-repository.
> > This makes it easy to have a compiler-package within pypy and merge
> > back and fro with CPython's compiler package.
> Heh, this would make three repositories for my code.  I already have it
> integrated into my Basil CVS tree. :)  I guess the more the merrier!

For starters, just mail me your prefered username and password and i
set up an account and help you through occuring problems.  I have
repeatedly merged an experimental CPython-fork from Python-2.2 
to Python-2.3 from our svn-CPython-mirror without any problems. 
> > However, i am not sure how many design goals a pypy-compiler and
> > cpython-compiler will share in the longer run.  E.g. we probably
> > want the pypy-compiler to be configurable at run time, e.g.
> >
> >     from pypy import compiler
> >     compiler.set_minimal()  # only compile very basic constructs
> >     compiler.allow_listcomprehension()
> >     compiler.allow_classes()
> >
> >     # set a different way to compile try/except/finally constructs [*]
> >     compiler.set_exception_compiler(MyExcCompiler)
> >
> >     compiler.compile(testsource) # ...
> >
> > and CPython's compiler package probably doesn't have such
> > design goals.  But then again, it wouldn't hurt if CPython's
> > compiler package could do such things :-)
> >
> This is cool stuff Holger.  Could you give me pointers as to where I
> should head with this stuff?  I've had pretty firm ideas in the past, but
> I'm not sure how they'd be received.  

After reading about your GRAD/Path and Basil i think that you know
much more about compiler & parser technology than me :-) 

Nevertheless, here are my *wishes* to a *compiler* package as i currently
see it fit for PyPy. 

- separate "aspects" of compiling the python language out 
  into classes so that they can be easily subclassed/replaced. 

- completly dynamic approach to bytecodes.  Each "aspect"
  adds/references bytecodes including their implementation. 
  Actually the aspect classes should generate the bytecode
  as a sequence of arbitrary Bytecode instances, i guess. 

- the list of bytecode instances might be converted into 
  a Std-CPython code object or into an enriched PyPy-one
  (or into Jython/Java-code?)

- i wonder if the enriched PyPy code objects could just be a pickle of 
  of the 'internal compiler result' so that the PyPy-interpreter at runtime calls
  the 'code.create_frame' function passing in an ObjSpace and wrapped args/kwargs. 
  and then the frame.eval-method execute the code object.  This model
  might also fit with gatewaying to C or other languages. 

- there seems to be a chicken-and-egg problem with the implementation
  of a Bytecode instance 'B': it needs to be compiled, too.  The closure
  of all Bytecode instances needed for this specific 'B'
  implementation should obviously not contain itself. Maybe bytecode
  implementations should be compiled to a different (restricted) set of 
  Bytecodes alltogether? 

I haven't thought about how parser & Grammars kick into this picture 
but ... 

> For example, using the parse tree of
> the input grammar, I either have or could easily create a tool that would
> build a base tree walker class and then developers could derive custom
> behavior from there.

... this sounds like an excellent starting point.  From an 
experimentator's/implementor's POV it would probably be nice to 
put all aspects of 

    providing a grammar 
    providing a compile for the constructs in the grammar
    providing the implementations of the compiled bytecodes 

into one simple unit you can pass to the parser/compiler-package.
I am unsure how decomposable a grammar like CPython's is.  

> Perhaps the compiler would not be stateful (in your
> example you use several method/function calls to set compiler state) so
> much as a module that exposes multiple tree walker classes. 

Sure, i didn't mean to suggest an actual API but just the functionality. 

> A lot of this technology is key to the Basil project as I was going to 
> build a control flow and data flow analysis toolset for C/C++/Python.

Here Armin or others need to chime in.  The whole abstract
interpretation/annoated object space thing is still somewhat opaque to me. 
> FYI, as I stated above, the code (or most of it, anyway) is the Basil SF
> CVS repository (basil.sf.net) if anyone wants a sneak peak (see the
> basil.lang.python & basil.parsing packages.)  The actual Python parser is
> still hidden in a test of the PyPgen stuff and is built at run time, but
> I should get around to pretty printing it and packaging it soon.

"The SourceForge.net Website is currently down for maintenance." 



P.S.: If i used (PyPy) terminology you don't know or deem dubious 
      please feel free to ask/correct me.  So far we have carried pretty
      ok through all the vagueness but questioning and clarifying
      concepts can't hurt.  

More information about the Pypy-dev mailing list