[pypy-svn] r16694 - pypy/release/0.7.x/pypy/doc
ludal at codespeak.net
ludal at codespeak.net
Fri Aug 26 22:06:41 CEST 2005
Date: Fri Aug 26 22:06:40 2005
New Revision: 16694
wrote the full readme hopfully explaining the current status and reasons
for having 4 compiler options
--- pypy/release/0.7.x/pypy/doc/README.compiling (original)
+++ pypy/release/0.7.x/pypy/doc/README.compiling Fri Aug 26 22:06:40 2005
@@ -2,7 +2,103 @@
-stable: use interpreter/stablecompiler at interp-level
-_stable: use lib/_stablecompiler at applevel.
-ast: use in-development translatable interplevel compiling.
-cpython: use cpython builtin c-level compiler.
+PyPy includes four compiling approaches which are reflecting the
+states of different part of its compiling pipeline.
+You can select one of this approach by use of the --compiler option.
+The options are:
+- stable: use interpreter/stablecompiler at interp-level
+- _stable: use lib/_stablecompiler at applevel.
+- ast: use in-development translatable interplevel compiling.
+- cpython: use cpython builtin c-level compiler.
+The following is a more detailed explanation of each option.
+The compiling pipeline comprises three parts:
+- the tokenizer turns source code into tokens
+- the parser analyzes a sequence of tokens and if it's grammatically
+ correct it turns it into an internal representation called a parse
+ tree or a syntax tree
+- the compiler turns the internal representation into code objects containing
+the --compiler option of pypy selects a full compiling pipeline
+including the tokenizer, parser, and actual compiler.
+The existence of those different parts is explained by several needs:
+to fully translate pypy, all code running at interpreter level needs to be
+If some part is not translatable we need to interpret it at application level
+The compiler module from c-python is entirely written in python which is good
+but it is far from being translatable
+We keep c-python implementations for testing purposes (because its much faster)
+For the same reasons we keep some code at interpreter level which is not translatable.
+there is only two tokenizer in pypy. ones is c-python's own tokenizer
+and the other one is an automata based recognizer
+as for the tokenizer there is two (three) parsers one from c-python and one home-made
+The pypy parser can build different kinds of representations with the help
+of a 'builder' object and pypy can use two builders:
+- tuplebuilder produces a representation similar to that of the c-python parser module
+ the building of the internal representation is fully annotatable, unfortunately we
+ need to transform this representation into a tree of tuples which is an operation
+ that cannot be annotatable
+- astbuilder is a builder that directly produces the Abstract Syntax Tree needed by
+ the pypy compiler to produce code objects. The astbuilder is fully annotatable and
+ so can run at interpreter level
+again we have the choice between c-python's own compiler and pypy compiler
+The pypy compiler is based on the compiler package from c-python.
+It contains two main parts : the transformer living in transformer.py and
+the ast to bytecode compiler implemented mainly in pycodegen.py and pyassem.py
+The transformer turns tuple into an AST (Abstract Syntax Tree).
+The AST to bytecode transforms the AST into bytecode.
+- when we use the tuplebuilder from the parser we feed its output to
+ the transformer first and then feed the AST to the AST to bytecode compiler
+- when we use the astbuilder from the parser we can feed its output directly to
+ the AST to bytecode compiler
+The compiler package has been modified and splitted into three different versions.
+interpreter/stablecompiler: this version mostly contains bug fixes and is the closer
+ to the original c-python implementation
+lib/_stablecompiler: this version is very similar to stable but needs to run at
+ application-level (that is interpreted by pypy itself)
+ which means that it cannot use things like 'eval' which
+ would recursively call itself.
+interpreter/astcompiler: this version intends to be run at interpreter level
+ and be translatable. it is heavily modified and doesn't
+ use the transformer since we provide directly the ast
+ tree produced by the astbuilder
+So in summary the four options correspond to the following:
+stable: uses the pypy parser with the tuplebuilder feeding tuples to the transformer
+ and compiler from the interpreter/stablecompiler
+ this version is not translatable because the compiler and transformer are not.
+_stable: uses the pypy parser with the tuplebuilder feeding tuples to
+ the application-level transformer and compiler from lib/_stablecompiler
+ this version is fully translatable at the cost of running the transformer
+ and compiler at application level
+ast: uses the pypy parser with the astbuilder feeding the AST tree to
+ the compiler from interpreter/astcompiler
+ this version is intended to be fully translatable. This is a work in progress.
+ it can be used instead of stable except some corner case compliance test won't
+ pass yet.
+cpython: uses the c-python builtin `compile` function to compile the source code
+ this version uses a c-builtin so it's not translatable but much faster
+ (note that the speed gain is only during compilation, exec and eval)
More information about the Pypy-commit