[pypy-svn] r10801 - pypy/dist/pypy/documentation

arigo at codespeak.net arigo at codespeak.net
Sun Apr 17 23:41:35 CEST 2005

Author: arigo
Date: Sun Apr 17 23:41:34 2005
New Revision: 10801

      - copied, changed from r10793, pypy/dist/pypy/documentation/annotation.txt
The last big txt file is translation.txt.  Draft for now.

Deleted: /pypy/dist/pypy/documentation/annotation.txt
--- /pypy/dist/pypy/documentation/annotation.txt	Sun Apr 17 23:41:34 2005
+++ (empty file)
@@ -1,306 +0,0 @@
-The annotation pass
-We describe below how a control flow graph can be "annotated" to 
-discover the types of the objects.  This annotation pass is a form of 
-type inference.  It is done after control flow graphs are built by the 
-FlowObjSpace, but before these graphs are translated into low-level code 
-(e.g. C/Lisp/Pyrex).
-The major goal of the annotator is to "annotate" each variable that 
-appears in a flow graph.  An "annotation" describes all the possible 
-Python objects that this variable could contain at run-time, based on a 
-whole-program analysis of all the flow graphs --- one per function.
-An "annotation" is an instance of ``SomeObject``.  There are subclasses 
-that are meant to represent specific families of objects.  Note that 
-these classes are all meant to be instantiated; the classes ``SomeXxx`` 
-themselves are not the annotations.
-Here is an overview (see ``pypy.annotation.model``):
-* ``SomeObject`` is the base class.  An instance ``SomeObject()`` 
-  represents any Python object.  It is used for the case where we don't 
-  have enough information to be more precise.  In practice, the presence 
-  of ``SomeObject()`` means that we have to make the annotated source code 
-  simpler or the annotator smarter.
-* ``SomeInteger()`` represents any integer.  
-  ``SomeInteger(nonneg=True)`` represent a non-negative integer (``>=0``).
-* ``SomeString()`` represents any string; ``SomeChar()`` a string of 
-  length 1.
-* ``SomeTuple([s1,s2,..,sn])`` represents a tuple of length ``n``.  The 
-  elements in this tuple are themselves constrained by the given list of 
-  annotations.  For example, ``SomeTuple([SomeInteger(), SomeString()])`` 
-  represents a tuple with two items: an integer and a string.
-There are more complex subclasses of ``SomeObject`` that we describe in 
-more details below.
-All the ``SomeXxx`` instances can optionally have a ``const`` attribute, 
-which means that we know exactly which Python object the Variable will 
-All the ``SomeXxx`` instances are supposed to be immutable.  The 
-annotator manages a dictionary mapping Variables (which appear in flow 
-graphs) to ``SomeXxx`` instances; if it needs to revise its belief about 
-what a Variable can contain, it does so by updating this dictionary, not 
-the ``SomeXxx`` instance.
-The annotator itself (``pypy.translator.annrpython``) works by 
-propagating the annotations forward in the flow graphs, starting at some 
-entry point function, possibly with explicitely provided annotations 
-about the entry point's input arguments.  It considers each operation in 
-the flow graph in turn.  Each operation takes a few input arguments 
-(Variables and Constants) and produce a single result (a Variable).  
-Depending on the input argument's annotations, an annotation about the 
-operation result is produced.  The exact rules to do this are provided 
-by the whole ``pypy.annotation`` subdirectory, which defines all the 
-cases in detail according to the R-Python semantics.  For example, if 
-the operation is 'v3=add(v1,v2)' and the Variables v1 and v2 are 
-annotated with ``SomeInteger()``, then v3 also receives the annotation 
-``SomeInteger()``.  So for example the function::
-    def f(n):
-        return n+1
-corresponds to the flow graph::
-    start ----------.
-                    |
-                    V 
-           +-------------------+
-           |  v2 = add(v1, 1)  |
-           +-------------------+
-                    |
-                    `---> return block
-If the annotator is told that v1 is ``SomeInteger()``, then it will 
-deduce that v2 (and hence the function's return value) is 
-This step-by-step annotation phase proceeds through all the operations 
-in a block, and then along the links between the blocks of the flow 
-graph.  If there are loops in the flow graph, then the links will close 
-back to already-seen blocks, as in::
-    def g(n):
-        i = 0
-        while n:
-            i = i + n
-            n = n - 1
-whose flow graph is::
-    start -----.           ,-----------------.
-               | n1 0      | m3 j3           |
-               V           v                 |
-           +-------------------+             |
-           |   input: n2 i2    |             |
-           |  v2 = is_true(n2) |             |
-           +-------------------+             |
-               |             |               |
-               |ifFalse      |ifTrue         |
-    return <---'             | n2 i2         |
-                             V               |
-                    +--------------------+   |
-                    |   input: n3 i3     |   |
-                    |  j3 = add(i3, n3)  |   |
-                    |  m3 = sub(n3, 1)   |---'
-                    +--------------------+
-Be sure to follow the variable renaming that occurs systematically 
-across each link in a flow graph.  In the above example the Variables 
-have been given names similar to the name of the original variables in 
-the source code (the FlowObjSpace tries to do this too) but keep in mind 
-that all Variables are different: n1, n2, i2, v2, n3, i3, j3, m3.
-Assume that we call the annotator with an input annotation of 
-``SomeInteger()`` for n1.  Following the links from the start, the 
-annotator will first believe that the Variable i2, whose value comes 
-from the constant 0 of the first link, must always be zero.  It will 
-thus use the annotation ``SomeInteger(const=0)`` for i2.  Then it will 
-propagate the annotations through both blocks, and find that v2 is 
-``SomeBool()`` and all other variables are ``SomeInteger()``.  In 
-particular, the annotation of j3 is different from the annotation of the 
-Variable i2 into which it is copied (via the back-link).  More 
-precisely, j3 is ``SomeInteger()`` but i2 is the more specific 
-``SomeInteger(const=0)``.  This means that the assumption that i2 must 
-always be zero is found to be wrong.  At this point, the annotation of 
-i2 is *generalized* to include both the existing and the new annotation.  
-(This is the purpose of ``pypy.annotation.model.unionof()``).  Then 
-these more general annotations must again be propagated forward.
-This process of successive generalizations continues until the 
-annotations stabilize.  In the above example, it is sufficient to 
-re-analyse the first block once, but in general it can take several 
-iterations to reach a fixpoint.  Annotations may also be propagated from 
-one flow graph to another and back repeatedly, across ``call`` 
-operations.  The overall model should ensure that this process 
-eventually terminates under reasonable conditions.  Note that as long as 
-the process is not finished, the annotations given to the Variables are 
-wrong, in the sense that they are too specific; at run-time, the 
-Variables will possibly contain Python objects outside the set defined 
-by the annotation, and the annotator doesn't know it yet.
-Description of the available types
-The reference and the details for the annotation model is found in the 
-module ``pypy.annotation.model``.  We describe below the issues related 
-to the various kinds of annotations.
-Simple Types
-``SomeInteger``, ``SomeBool``, ``SomeString``, ``SomeChar`` all stands 
-for the obvious corresponding set of immutable Python objects.
-``SomeTuple`` only considers tuples of known length.  We don't try to 
-handle tuples of varying length (the program should use lists instead).
-Lists and Dictionaries
-``SomeList`` stands for a list of homogenous type (i.e. all the elements 
-of the list are represented by a single common ``SomeXxx`` annotation).
-``SomeDict`` stands for a homogenous dictionary (i.e. all keys have the 
-same ``SomeXxx`` annotation, and so have all values).
-These types are mutable, which requires special support for the 
-annotator.  The problem is that in code like::
-   lst = [42]
-   update_list(lst)
-   value = lst[0]
-the annotation given to ``value`` depends on the order in which the 
-annotator progresses.  As ``lst`` is originally considered as a list of 
-``SomeInteger(const=42)``, it is possible that ``value`` becomes 
-``SomeInteger(const=42)`` as well if the analysis of ``update_list()`` 
-is not completed by the time the third operation is first considered.  
-To solve this problem, each ``SomeList`` or ``SomeDict`` is linked to a 
-set of so-called *factories*.  Each creation point, i.e. each 'newlist' 
-or 'newdict' operation, gets its associated factory.  The factory 
-remembers what kind of object it really needs to build.  For example, in 
-code like::
-   lst = [42]
-   lst.append(43)
-the factory associated with the first line originally builds a list 
-whose items are all constants equal to 42; when the ``append(43)`` call 
-is then found, the factory is updated to build a more general list of 
-integers, and the annotator restarts its analysis from the factory 
-position.  Our model is not sensitive to timing: it doesn't know that 
-the same list object may contain different items at different times.  It 
-only computes how general the items in the list must be to cover all 
-For initially empty lists, as created by ``lst = []``, we build a list 
-whose items have the annotation ``SomeImpossibleValue``.  This is an 
-annotation that denotes that no Python object at all can possibly appear 
-here at run-time.  It is the least general annotation.  The rationale is 
-   lst = []
-   oups = lst[0]
-will give the variable ``oups`` the annotation ``SomeImpossibleValue``, 
-which is reasonable given that no concrete Python object can ever be put 
-in ``oups`` at run-time.  In a more usual example::
-   lst = []
-   lst.append(42)
-the list is first built with ``SomeImpossibleValue`` items, and then the 
-factory is generalized to produce a list of ``SomeInteger(const=42)``.  
-With this "impossible object" trick we don't have to do anything special 
-about empty lists.
-User-defined Classes and Instances
-``SomeInstance`` stands for an instance of the given class or any 
-subclass of it.  For each user-defined class seen by the annotator, we 
-maintain a ClassDef (``pypy.annotation.classdef``) describing the 
-attributes of the instances of the class; essentially, a ClassDef gives 
-the set of all class-level and instance-level attributes, and for each 
-one, a corresponding ``SomeXxx`` annotation.
-Instance-level attributes are discovered progressively as the annotation 
-progresses.  Assignments like::
-   inst.attr = value
-update the ClassDef of the given instance to record that the given 
-attribute exists and can be as general as the given value.
-For every attribute, the ClassDef also records all the positions where 
-the attribute is *read*.  If, at some later time, we discover an 
-assignment that forces the annotation about the attribute to be 
-generalized, then all the places that read the attribute so far are 
-marked as invalid and the annotator will have to restart its analysis 
-from there.
-The distinction between instance-level and class-level attributes is 
-thin; class-level attributes are essentially considered as initial 
-values for instance-level attributes.  Methods are not special in this 
-respect, expect that they are bound to the instance (i.e. ``self = 
-SomeInstance(cls)``) when considered as the initial value for the 
-The inheritance rules are as follows: the union of two ``SomeInstance`` 
-annotations is the ``SomeInstance`` of the most precise common base 
-class.  If an attribute is considered (i.e. read or written) through a 
-``SomeInstance`` of a parent class, then we assume that all subclasses 
-also have the same attribute, and that the same annotation applies to 
-them all (so code like ``return self.x`` in a method of a parent class 
-forces the parent class and all its subclasses to have an attribute 
-``x``, whose annotation is general enough to contain all the values that 
-all the subclasses might want to store in ``x``).  However, distinct 
-subclasses can have attributes of the same names with different, 
-unrelated annotations if they are not used in a general way through the 
-parent class.
-Prebuilt Constants
-(to be completed)
-Built-in functions and methods
-(to be completed)
-(to be completed)

Deleted: /pypy/dist/pypy/documentation/basicblock.asc
--- /pypy/dist/pypy/documentation/basicblock.asc	Sun Apr 17 23:41:34 2005
+++ (empty file)
@@ -1,36 +0,0 @@
-the CtlFlowObjSpace is supposed to produce the following object-model
-(currently just a linear list of objects referencing each other, sorry
-no UML diagream :-)
-the reason we want this objectmodel as the source for doing the
-translation is that otherwise we get very-hard-to-read low-level/pyrex
-code with lots of gotos etc.pp.
-With the CtlFlow-object model we can basically reconstruct some
-source-structure (without examining the bytecode).
-class BasicBlock:
- .inputargs = [list-of-input-locals]
- .locals = [list-of-all-locals-incluing-inputargs]
- .operations = [list-of-operations]
- .branch = <Branch or ConditionalBranch instance>
-class Variable:
-  pass
-class Constant:
-  .value = ...
-class SpaceOperation:
-  .opname = 'add'
-  .args = [list-of-variables]
-  .result = <Variable/Constant instance>
-class Branch:
-  .args = [list-of-variables]
-  .target = <BasicBlock instance>
-class ConditionalBranch:
-  .condition = <Variable instance>
-  .iftrue = <Branch instance>
-  .iffalse = <Branch instance>

Modified: pypy/dist/pypy/documentation/navlist
--- pypy/dist/pypy/documentation/navlist	(original)
+++ pypy/dist/pypy/documentation/navlist	Sun Apr 17 23:41:34 2005
@@ -2,7 +2,8 @@
-    'objspace.html', 
+    'objspace.html',
+    'translation.html',

Modified: pypy/dist/pypy/documentation/redirections
--- pypy/dist/pypy/documentation/redirections	(original)
+++ pypy/dist/pypy/documentation/redirections	Sun Apr 17 23:41:34 2005
@@ -19,6 +19,9 @@
     'goals.html'            : 'misc.html#goals', 
     'developers.html'       : 'misc.html#developers', 
-    'cmodules.html'         : 'misc.html#cmodules', 
+    'cmodules.html'         : 'misc.html#cmodules',
+    'annotation.html'       : 'translation.html#annotator',
+    'basicblock.asc'        : 'objspace.html#the-flow-model',

Modified: pypy/dist/pypy/documentation/test_redirections.py
--- pypy/dist/pypy/documentation/test_redirections.py	(original)
+++ pypy/dist/pypy/documentation/test_redirections.py	Sun Apr 17 23:41:34 2005
@@ -15,4 +15,6 @@
         yield checkexist, redir.dirpath(newname) 
 def test_navlist(): 
-    assert eval(redir.dirpath('navlist').read())
+    navlist = eval(redir.dirpath('navlist').read())
+    for entry in navlist:
+        yield checkexist, redir.dirpath(entry)

Modified: pypy/dist/pypy/documentation/theory.txt
--- pypy/dist/pypy/documentation/theory.txt	(original)
+++ pypy/dist/pypy/documentation/theory.txt	Sun Apr 17 23:41:34 2005
@@ -62,4 +62,4 @@
 .. _`quite general one`: http://codespeak.net/svn/pypy/dist/pypy/objspace/std/multimethod.py
 .. _StdObjSpace: objspace.html#the-standard-object-space
 .. _`short two-arguments-dispatching one`: http://codespeak.net/svn/pypy/dist/pypy/annotation/pairtype.py
-.. _annotator: annotation.html
+.. _annotator: translation.html#annotator

Copied: pypy/dist/pypy/documentation/translation.txt (from r10793, pypy/dist/pypy/documentation/annotation.txt)
--- pypy/dist/pypy/documentation/annotation.txt	(original)
+++ pypy/dist/pypy/documentation/translation.txt	Sun Apr 17 23:41:34 2005
@@ -1,3 +1,51 @@
+    Translation
+.. contents::
+.. sectnum::
+This document describes the tool chain that we developed to analyze and
+"compile" RPython_ programs (like PyPy itself) to various lower-level
+.. _RPython: coding-style.html#restricted-python
+XXX very preliminary documentation!
+The module `translator.py`_ is the common entry point to the various parts
+of the translation process.  It is available as an interactive utility to
+`play around`_.
+Here are the steps we follow to translate a given program:
+1. The complete program is imported.  If needed, extra initialization is performed.  Once this is done, the program must be present in memory is a form that is "static enough" in the sense of RPython_.
+2. The `Flow Object Space`_ processes the input program, turning each function independently into a `control flow graph`_ data structure recording sequences of basic operations in "single-style assignment".
+3. Optionally, the Annotator_ performs global type inference on the control flow graphs.  Each variable gets annotated with an inferred type.
+4. One of the Code Generators (XXX not documented yet) turns the optionally annotated flow graphs and produces a source file in a lower-level language: C_, LLVM_, `Common Lisp`_, Pyrex_, Java_, or `Python again`_ (this is used in PyPy to turn sufficiently RPythonic app-level code into interp-level code).
+.. _`translator.py`: http://codespeak.net/svn/pypy/dist/pypy/translator/translator.py
+.. _`play around`: getting_started.html#trying-out-the-translator
+.. _`Flow Object Space`: objspace.html#the-flow-object-space
+.. _`control flow graph`: objspace.html#the-flow-model
+.. _C: http://codespeak.net/svn/pypy/dist/pypy/translator/genc/
+.. _LLVM: http://codespeak.net/svn/pypy/dist/pypy/translator/llvm/
+.. _`Common Lisp`: http://codespeak.net/svn/pypy/dist/pypy/translator/gencl.py
+.. _Pyrex: http://codespeak.net/svn/pypy/dist/pypy/translator/genpyrex.py
+.. _Java: http://codespeak.net/svn/pypy/dist/pypy/translator/java/
+.. _`Python again`: http://codespeak.net/svn/pypy/dist/pypy/translator/geninterplevel.py
+.. _Annotator:
 The annotation pass
@@ -10,9 +58,8 @@
 (e.g. C/Lisp/Pyrex).
 The major goal of the annotator is to "annotate" each variable that 
 appears in a flow graph.  An "annotation" describes all the possible 
@@ -57,9 +104,8 @@
 the ``SomeXxx`` instance.
 The annotator itself (``pypy.translator.annrpython``) works by 
 propagating the annotations forward in the flow graphs, starting at some 
@@ -158,9 +204,8 @@
 by the annotation, and the annotator doesn't know it yet.
 Description of the available types
 The reference and the details for the annotation model is found in the 
 module ``pypy.annotation.model``.  We describe below the issues related 

More information about the Pypy-commit mailing list