[pypy-svn] r28896 - pypy/dist/pypy/doc
arigo at codespeak.net
arigo at codespeak.net
Fri Jun 16 18:56:47 CEST 2006
Date: Fri Jun 16 18:56:44 2006
New Revision: 28896
Finished (for now) the documentation about the ext compiler.
--- pypy/dist/pypy/doc/_ref.txt (original)
+++ pypy/dist/pypy/doc/_ref.txt Fri Jun 16 18:56:44 2006
@@ -32,7 +32,9 @@
.. _`pypy/module/`: ../../pypy/module
.. _`module/__builtin__/`: ../../pypy/module/__builtin__
.. _`pypy/module/__builtin__/__init__.py`: ../../pypy/module/__builtin__/__init__.py
-.. _`module/_sre/`: ../../pypy/module/_sre
+.. _`pypy/module/_demo`: ../../pypy/module/_demo
+.. _`pypy/module/_sre`: ../../pypy/module/_sre
.. _`pypy/module/readline`: ../../pypy/module/readline
.. _`module/recparser/`: ../../pypy/module/recparser
.. _`module/sys/`: ../../pypy/module/sys
--- pypy/dist/pypy/doc/coding-guide.txt (original)
+++ pypy/dist/pypy/doc/coding-guide.txt Fri Jun 16 18:56:44 2006
@@ -689,14 +689,14 @@
>>>> import sys
>>>> import operator
>>>> import types
- t>>>> types.__file__
+ >>>> types.__file__
>>>> import os
--- pypy/dist/pypy/doc/extcompiler.txt (original)
+++ pypy/dist/pypy/doc/extcompiler.txt Fri Jun 16 18:56:44 2006
@@ -2,7 +2,11 @@
PyPy Extension Compiler
-This document describes the PyPy extension compiler which is able to
+This document describes the PyPy extension compiler, which is able to
compile a set of source code to a PyPy or a CPython extension module.
**WARNING: this is beta software, APIs and details may change.**
@@ -12,14 +16,16 @@
edges are likely.
Understanding the ext compiler
PyPy provides a generic way to write modules for all Python
implementations: the so-called *mixed module* approach. A single mixed
module is implemented as a set of Python source files, making a
-subpackage of the ``pypy.module`` package. While running PyPy, each of
-these subpackages appears to be a single module, whose interface was
+subpackage of the `pypy/module/`_ package. While running PyPy, each of
+these subpackages appears to be a single module, whose interface is
specified in the ``__init__.py`` of the subpackage, and whose
implementation is lazily loaded from the various other ``.py`` files of
@@ -43,7 +49,7 @@
* compile it as a CPython extension module. Example::
- $ pypy/bin/compilemodule.py _demo
+ $ python pypy/bin/compilemodule.py _demo
[lots of output]
$ cd /tmp/usession-5/_demo
@@ -78,97 +84,133 @@
-- using rctypes for binding to external libraries
-- exchangeability of CPyObjspace and StdObjSpace
- * translation to CPython
- * translation to PyPy
Using the ext compiler
-XXXXXXXXXXXXXXXX THIS IS DOCUMENTATION WITHOUT CODE XXXXXXXXXXXXXXXXXXXXXX
-writing a module (provide example)
+Writing a module
-You have to put your module into its own directory in `pypy/module/`_
-of your local `pypy checkout`_.
+You have to put your module into its own directory in `pypy/module/`_ of
+your local `pypy checkout`_. See the following directories as guidelines:
-see `pypy/module/readline`_ for an example.
-XXX describe directory structure here
-.. _`pypy checkout`: getting-started.html#gettingpypy
-testing a module
+ A demo module showing a few of the more interesting features.
-testing only the (ctypes) bindings::
- python2.4 pypy/test_all.py pypy/module/readline/test/test_c_readline.py
-testing the (wrapped) package::
- python2.4 pypy/test_all.py pypy/module/readline/test/test_readline.py
- py.test pypy/module/readline/test/test_readline.py
+ A tiny, in-progress example giving bindings to the GNU ``readline``
+ An algorithmic example: the regular expression engine of CPython,
+ rewritten in RPython_.
+Modules can be based on ctypes. This is the case in the
+`pypy/module/readline`_ and `pypy/module/_demo`_ examples: they use
+ctypes to access functions in external C libraries. When translated to
+C, the calls in these examples become static, regular C function calls
+-- which means that most of the efficiency overhead of using ctypes
+disappears during the translation. However, some rules must be followed
+in order to make ctypes translatable; for more information, see the
+documentation of RCtypes_.
+All these modules are called "mixed" because they mix interpreter-level
+and application-level submodules to present a single, coherent module to
+the user. For a CPython extension module, you can think about
+`interpreter-level`_ as what will be compiled into C, and
+`application-level`_ as what will stay as it is, as Python code,
+included (as a frozen bytecode) within the C module. The capability to
+group several interpreter-level files into the final compiled module is
+similar to having a CPython extension module that was compiled from a
+set of C sources; but the capability to also include nicely integrated
+Python sources in the C extension module has no direct equivalent in
+hand-written C extension modules.
+The directory structure is described in the Coding Guide, section `mixed
+modules`_. The interpreter-level parts of a mixed module must follow
+the RPython_ programming style; this is necessary to allow them to be
+annotated and translated to C. Whenever they manipulate
+application-level (i.e. user-visible) objects, they must do so via
+object space operations (see `wrapping rules`_). The mixed modules can
+also export new user-visible types, as in the example of
+`pypy/module/_sre`_ (for reference, see the documentation about the
+`TypeDef`_ of the Module type itself to see how to expose types to
+.. _`pypy checkout`: getting-started.html#gettingpypy
+.. _`mixed modules`: coding-guide.html#mixed-modules
+.. _`interpreter-level`: coding-guide.html#interpreter-level
+.. _`application-level`: coding-guide.html#application-level
+.. _`wrapping rules`: coding-guide.html#wrapping-rules
+.. _`TypeDef`: interpreter.html#typedefs
+.. _RPython: coding-guide.html#rpython
+.. _RCtypes: rctypes.html
-translating a module to a CPython extension
- python2.4 pypy/bin/extcompiler.py readline #<needs to be in module dir
-The extension compiler imports the specified module/package
-and produces a shared library importable from your local
-python2.4 installation. The produced shared library is
-put into the current working directory by default.
+Testing a module
-XXX think about shared state of imported modules (e.g. linecache
- or other modules which might get imported from the ext-package)
+The readline mixed module has a minimal example for each of 4 different
+kind of tests:
-XXX what actually happens to applevel code in an extension module?
+* As this is a ctypes bindings module, we should test the ctypes bindings
+ directly to see if they work as expected::
+ python pypy/test_all.py pypy/module/readline/test/test_c_readline.py
+* We should then test that the mixed module wrapping is correct, using
+ the helper function ``pypy.interpreter.mixedmodule.testmodule()``.
+ Called from a normal CPython program, this function exposes a mixed
+ module as a plain CPython module for testing and inspection (it works
+ in the interactive interpreter too, of course: it is a good way to see
+ how mixed module wrappings look like in the end)::
-XXX integration as a PyPy module
+ python pypy/test_all.py pypy/module/readline/test/test_mixedmodule.py
+* We can also run the mixed module within PyPy, on top of CPython.
+ To do so from a test, we use a special ``AppTestXyz`` class::
+ python pypy/test_all.py pypy/module/readline/test/test_with_pypy.py
+* Finally, we can compile the mixed module to a CPython extension
+ module, re-import it into the running CPython interpreter, and test
+ it. Only this test will pick up the translation failures caused by
+ breaking the RPython rules. (To debug translation failures, though,
+ you should use ``compilemodule.py`` as described below: you will then
+ get a Pdb prompt and a flow graph viewer to look around.) ::
-Installation notes requirements
+ python pypy/test_all.py pypy/module/readline/test/test_compiler.py
-XXX more verbose, links
-you need to have a full PyPy checkout, refer to getting started
-required ctypes version
+Translating a module to a CPython extension
-As of this time, only `ctypes-0.9.9.6`_ is known to work.
+As seen in the introduction_, you translate a module into a CPython
+extension with the following command-line::
-.. _`ctypes-0.9.9.6`: http://sourceforge.net/project/showfiles.php?group_id=71702&package_id=71318&release_id=411554
+ python pypy/bin/compilemodule.py _demo
+The extension compiler imports the specified package from
+`pypy/module/`_ and produces a shared library importable from your local
+Python installation. The produced shared library is put into a
+temporary directory printed at the end (which on Linux is also
+accessible as `/tmp/usession-<username>/<modulename>/<modulename>.so`).
+Note that we recommend you to write and run tests_ for your module first.
+This is not only a matter of style: bogus modules are likely to make the
+translation tool-chain fail in mysterious ways.
+See the introduction_ for other things you can do with a mixed module.
-*ctypes 0.9.9.6 on OSX 10.3*: you need to change the ``RTLD_LOCAL`` default
-in ctypes/__init__.py line 293 to::
+Note that you obviously need to have a full `pypy checkout`_ first. If
+you have troubles compiling the demo modules, check out our
+ctypes-specific `installation notes`_.
- def __init__(self, name, mode=RTLD_GLOBAL, handle=None):
+.. _`installation notes`: rctypes.html#installation
-otherwise it will complain with "unable to open this file with RTLD_LOCAL"
-when trying to load the C library.
.. include:: _ref.txt
--- pypy/dist/pypy/doc/interpreter.txt (original)
+++ pypy/dist/pypy/doc/interpreter.txt Fri Jun 16 18:56:44 2006
@@ -286,6 +286,8 @@
annotator needs to keep track of the types of objects flowing
across those barriers.
Making interpreter-level functions available at application-level
--- pypy/dist/pypy/doc/rctypes.txt (original)
+++ pypy/dist/pypy/doc/rctypes.txt Fri Jun 16 18:56:44 2006
@@ -77,6 +77,24 @@
program, in addition to importing ctypes.
+ctypes version and platform notes
+As of this time, only `ctypes-0.9.9.6`_ is known to work correctly
+On Mac OSX 10.3 you need to change the ``RTLD_LOCAL`` default
+in ctypes/__init__.py line 293 to::
+ def __init__(self, name, mode=RTLD_GLOBAL, handle=None):
+otherwise it will complain with "unable to open this file with RTLD_LOCAL"
+when trying to load the C library.
@@ -279,7 +297,6 @@
CPython's Ctype's Behaviour
@@ -380,5 +397,6 @@
.. _`ctypes documentation`: http://docs.python.org/dev/lib/module-ctypes.html
.. _ctypes: http://starship.python.net/crew/theller/ctypes/
.. _RPython: coding-guide.html#rpython
+.. _`ctypes-0.9.9.6`: http://sourceforge.net/project/showfiles.php?group_id=71702&package_id=71318&release_id=411554
.. include:: _ref.txt
More information about the Pypy-commit