[pypy-dev] Python parsers...
hpk at trillke.net
Fri Aug 1 13:03:50 CEST 2003
[Jonathan David Riehl Wed, Jul 30, 2003 at 06:20:23PM -0500]
> On Thu, 31 Jul 2003, holger krekel wrote:
> > [Jeremy Hylton Wed, Jul 30, 2003 at 06:09:41PM -0400]
> > >
> > > Are you interested in integrating it into the compiler package? The Python
> > > CVS trunk is now wide open for development, and the compiler package could
> > > definitely use some tidying up.
> > note, that we mirror the Python-CVS trunk into our svn-repository.
> > This makes it easy to have a compiler-package within pypy and merge
> > back and fro with CPython's compiler package.
> Heh, this would make three repositories for my code. I already have it
> integrated into my Basil CVS tree. :) I guess the more the merrier!
For starters, just mail me your prefered username and password and i
set up an account and help you through occuring problems. I have
repeatedly merged an experimental CPython-fork from Python-2.2
to Python-2.3 from our svn-CPython-mirror without any problems.
> > However, i am not sure how many design goals a pypy-compiler and
> > cpython-compiler will share in the longer run. E.g. we probably
> > want the pypy-compiler to be configurable at run time, e.g.
> > from pypy import compiler
> > compiler.set_minimal() # only compile very basic constructs
> > compiler.allow_listcomprehension()
> > compiler.allow_classes()
> > # set a different way to compile try/except/finally constructs [*]
> > compiler.set_exception_compiler(MyExcCompiler)
> > compiler.compile(testsource) # ...
> > and CPython's compiler package probably doesn't have such
> > design goals. But then again, it wouldn't hurt if CPython's
> > compiler package could do such things :-)
> This is cool stuff Holger. Could you give me pointers as to where I
> should head with this stuff? I've had pretty firm ideas in the past, but
> I'm not sure how they'd be received.
After reading about your GRAD/Path and Basil i think that you know
much more about compiler & parser technology than me :-)
Nevertheless, here are my *wishes* to a *compiler* package as i currently
see it fit for PyPy.
- separate "aspects" of compiling the python language out
into classes so that they can be easily subclassed/replaced.
- completly dynamic approach to bytecodes. Each "aspect"
adds/references bytecodes including their implementation.
Actually the aspect classes should generate the bytecode
as a sequence of arbitrary Bytecode instances, i guess.
- the list of bytecode instances might be converted into
a Std-CPython code object or into an enriched PyPy-one
(or into Jython/Java-code?)
- i wonder if the enriched PyPy code objects could just be a pickle of
of the 'internal compiler result' so that the PyPy-interpreter at runtime calls
the 'code.create_frame' function passing in an ObjSpace and wrapped args/kwargs.
and then the frame.eval-method execute the code object. This model
might also fit with gatewaying to C or other languages.
- there seems to be a chicken-and-egg problem with the implementation
of a Bytecode instance 'B': it needs to be compiled, too. The closure
of all Bytecode instances needed for this specific 'B'
implementation should obviously not contain itself. Maybe bytecode
implementations should be compiled to a different (restricted) set of
I haven't thought about how parser & Grammars kick into this picture
> For example, using the parse tree of
> the input grammar, I either have or could easily create a tool that would
> build a base tree walker class and then developers could derive custom
> behavior from there.
... this sounds like an excellent starting point. From an
experimentator's/implementor's POV it would probably be nice to
put all aspects of
providing a grammar
providing a compile for the constructs in the grammar
providing the implementations of the compiled bytecodes
into one simple unit you can pass to the parser/compiler-package.
I am unsure how decomposable a grammar like CPython's is.
> Perhaps the compiler would not be stateful (in your
> example you use several method/function calls to set compiler state) so
> much as a module that exposes multiple tree walker classes.
Sure, i didn't mean to suggest an actual API but just the functionality.
> A lot of this technology is key to the Basil project as I was going to
> build a control flow and data flow analysis toolset for C/C++/Python.
Here Armin or others need to chime in. The whole abstract
interpretation/annoated object space thing is still somewhat opaque to me.
> FYI, as I stated above, the code (or most of it, anyway) is the Basil SF
> CVS repository (basil.sf.net) if anyone wants a sneak peak (see the
> basil.lang.python & basil.parsing packages.) The actual Python parser is
> still hidden in a test of the PyPgen stuff and is built at run time, but
> I should get around to pretty printing it and packaging it soon.
"The SourceForge.net Website is currently down for maintenance."
P.S.: If i used (PyPy) terminology you don't know or deem dubious
please feel free to ask/correct me. So far we have carried pretty
ok through all the vagueness but questioning and clarifying
concepts can't hurt.
More information about the Pypy-dev