[Python-Dev] Parrot -- should life imitate satire?
Thomas Wouters
thomas@xs4all.net
Wed, 1 Aug 2001 15:36:07 +0200
On Tue, Jul 31, 2001 at 11:52:01PM -0400, Dan Sugalski wrote:
> [SRE] and the C extension interface are the two big Python things I've got
> on my "must peer deeply into" list.
I can't say much about SRE, but the C extension interface should be on
anyone's "must peer deeply into" list, except that there isn't a real need
to peer deeply into it :) Lets peer into it. I'm going to discuss the
pre-2.2 state of affairs, mostly because I know jack shit about the
2.2-type-dichotomy-healing :) In case you aren't aware, we are currently
into the 2.2 alphas, where this story is slightly different because you can
now subclass types. If Guido or anyone else involved in the descr-branch
wants to tune in and correct the explanation for Python 2.2, that's fine by
me.
As you might have heard, everything in Python is an object. This is true
even for the C side of things. A Python object, any Python object, is
essentially a struct like this:
stuct PyExample {
int ob_refcnt; /* reference count */
struct _typeobject *ob_type; /* type of this object */
/* type-specific data goes here */
}
The common header is made available as a #define, so that we can add extra
info in severe-debug mode (though this breaks binary compatibility, so it
should not be done in production builds.)
For instance, the struct to hold a (normal, unbounded) Python integer is
typedef struct {
PyObject_HEAD
long ob_ival;
} PyIntObject;
The struct _typeobject pointer is a pointer to a Type object, which also
acts as a vtable. It contains all the information about the type, including
how to operate on it. The typeobject is itself also an object! Its type is
the PyType type, which is its own type. The typeobject is actually what you
get back when you do 'type(object)' in Python code. All this is also
explained in Include/object.h, by the way.
The type object holds things like the type name, how to print it, howmuch
memory should be allocated, a couple of flags to implement binary
compatibility, and a whole boatload of function pointers. The function
pointers for operations are spread out over the type struct and several
separate structs, to which the type struct points: tp_as_number,
tp_as_sequence, and tp_as_mapping. Each of these can be NULL if the type
doesn't implement the operations.
Some of the operations that are defined on the type struct itself are
get-attribute, set-attribute, comparison, get-hashvalue, convert-to-string
and call-as-function.
The PyNumberMethods struct (tp_as_number) contains arithmatic operations
(such as adding, mulitplying, shifting, XORing, and in-place variations of
each for use with augmented assignment), truth-value-testing, and converting
to various other integer types.
The PySequenceMethods struct (tp_as_sequence) contains sequence operations
such as membership-test, item- and slice-retrieval and -assignment, and
sequence concatenation and repetition. The latter two are somewhat of an odd
duck, since they implement the same Python operation ("+" and "*") as the
'add' and 'multiply' methods from the PyNumberMethods struct. They're mainly
so the Python runtime can properly handle "number * list" as well as "list *
number" without attempting to convert list to an integer.
Lastly there's PyMappingMethods. I'm not at all sure why this is a separate
sturct, as the three operations it contains (length inquiry, item-retrieval
and item-assignment) are also covered by the PySequenceMethods, but I
suspect some kind of hysterical raisin crept in here :)
To see this construct in action, take a look at Objects/intobject.c, in the
Python source tree (doesn't really matter which version.) It defines a bunch
of functions to do the arithmatic, fills a PyNumberMethods struct with the
appropriate pointers (leaving unsupported ones NULL), and fills a new
PyTypeObject with the right info for the "int" type.
To contrast, the PyFunction type is nearly empty. It defines getattr and
setattr, and a 'repr' function (convert to string). You can't compare it (it
will always compare false to anything but itself), add it, index it, or get
its length, but you can get attributes from it. And the average function has
some interesting attributes: you can, from Python code and C code alike,
easily get at things like the bytecode it consists of, the number of
arguments it expects, the names of those arguments, etc. Each of those is,
of course, a Python object in its own right :) Function objects are defined
in Objects/funcobject.c, though the actual execution of function objects is
done elsewhere.
An 'instance' object is a slightly different case in this story (more so in
the post-2.2 story, but I'm not sure howmuch more.) An instance object is a
Python class, defined in Python code by the user/programmer. It can choose
to implement any of the above operations by defining magically named methods
on the class. For instance, to implement the 'getitem' protocol, it defines
a function __getitem__, and to implement addition, __add__. The Type object
for instance objects acts as a proxy for the Python class, though this does
represent some problems: instance objects need to be treated in some areas
of the interpreter (but this is almost certain to have been changed in 2.2.)
But back to generic Python objects. Obviously, when writing C code, you
don't want to manipulate Python objects by directly accessing all those
layers of C structs all the time. So each builtin type defines useful C
functions and macros to help. For instance, to create a new list, you call
PyList_New(size). To test whether something is a list, PyList_Check(object).
To index a list, PyList_GetItem(list, index), or a macro version of that,
PyList_GET_ITEM. The macro version doesn't do type or boundary checks, so it
should only be used when you already did those checks. Likewise, there is
PyInt_AsLong (convert a PyInt to a C 'long') and PyInt_AS_LONG.
But even that isn't that helpful in a dynamically typed language, since you
have to know an object's type in order to call the type-specific functions.
So we also have two more abstraction layers. The most abstract one is the
Object layer. Basically, regardless of the type of object, you can always
manipulate it using PyObject_*() functions. Some PyObject_ functions have a
direct relation to Python code, others are only useful to C code. There are
PyObject_ functions such as PyObject_IsTrue, PyObject_Compare,
PyObject_GetAttr, PyObject_CallObject, etcetera. All these work regardless
of the type. If the type (or instance) does not implement the operation, an
exception is raised (see below.)
The other layer is the slightly less abstract layer that differentiates in
the same way as the PyNumberMethods, PySequenceMethods, PyMappingMethods
structs. To add two objects, you call PyNumber_Add(o1, o2). To explicitly
concatenate two objects, you call PySequence_Concat(o1, o2). All of these
work for any type that defines the behaviour, including instance types. Note
that calling one of these functions can end up executing arbitrary Python
code, so they shouldn't be called in any time-critical situation :-)
As a last part of the API, there is the reference counting. As I showed a
couple of pages back, all objects contain a reference count. Operations that
copy a reference should increment the reference count, using Py_INCREF, and
deleting a reference should be done using Py_DECREF. The latter also does
the deallocation and cleanup if the DECREF causes the refcount to drop to 0.
All API functions that return a reference define whether they return a new
reference, which the caller has to DECREF when throwing it away, or a
borrowed reference, which shouldn't be DECREF'ed, or kept around without
INCREFing it.
Exceptions, by the way, are set by setting a global (or thread-local, in a
threaded interpreter) variable to contain the Python exception object, and
then returning an error flag (either NULL, -1 or 0, depending on the
function) to the caller. All code should check all return values and always
return an error value if a called function does so, unless it's prepared to
handle the exception itself. To compare errors with a particular error
'class', there is PyErr_ExceptionMatches (which also handles 'subclasses' of
exception types.) If the error value returned by a particular function is
also a real possible value (such as the value returned by PyNumber_AsLong),
there is PyErr_Occured() to see if it was a real exception, or a legitimate
-1 value.
So that's it for the API. All Python objects can be manipulated, created and
destroyed this way. Most extension types provide their functionality either
in builtin-operations (in the type object) or by providing methods that can
be called (using PyObject_GetAttr and PyObject_CallObject), and only rarely
need to export an explicit API of their own. (And as you probably know,
exporting an API from a shared library to another shared library is...
slightly tricky :)
Now, the Python bytecode-interpreter itself is very small. All it does is
interpret bytecode, and then call out to the API to make it do the actual
work. It has some special knowledge about some types, for optimzation, and
it does all the real exception handling and namespace handling (name lookup
and such), but the actual manipulation of objects is left to the API. Even
so, it has enough to do. In the case of exceptions, it has to search up the
function for a 'catching' block (a try/except or try/finally statement), or
break out of the function to let the error percolate to the caller.
Namespaces are a complete story in their own right. Python has three
distinct namespaces: the 'local' namespace inside a function (or 'code
block'), the 'global' or 'module-level' namespace at the toplevel of a file,
and the 'builtin' namespace which holds builtin functions and names. The
search order is always local->global->builtin, though for code that is
executed at the module level (not inside a function), local is the same as
global.
In Python, variables are defined by assignment. As a consequence, any
variable that is assigned to in a code block, is local to that block. Any
variable that isn't assigned to but is just referenced thus has to be
defined in the global namespace. The compiler keeps track of this, and
generates 'LOAD_FAST' (for loading from the local namespace, inside a
function), 'LOAD_GLOBAL' (for loading from the global or builtin namespace,
skipping the local namespace), and 'LOAD_NAME' (for occurances where it's
truly not known where a variable comes from, and both local and global
namespaces have to be searched.) The latter is necessary because Python has
a few operations that can modify a local namespace at runtime (such as 'from
module import *', and 'exec'.)
Since Python 2.1, Python has 'nested scopes', which makes code blocks
defined inside other code blocks slightly special: the 'parent' scope is
also searched for any variable that isn't local to the nested scope. Most of
this, but not all, is done at compiletime, IIRC. (Jeremy can correct me if
I'm wrong :)
I tried to make the distinction between a 'function' and a 'code block',
since the VM really deals with the second, not the first. I suspect that the
current Python VM's mechanics to deal with nested scopes could be
generalized so that Perl's lexical closures could fit as well (I assume that
won't change in perl6 :). Then again, Perl already knows where a variable
comes from, so the VM will probably want to put that to good use... The main
reason Python does the name-searching at runtime is that a global namespace
can be altered from outside the currently-executing codeblock (threads, or
other modules, can do 'module.range = my_fake_range', and suddenly the
range() function does something completely different throughout the entire
module.)
A good overview of how the VM uses the API can be obtained by looking at
Python/ceval.c, and looking for "switch (opcode)". You'll end up at the top
of the main bytecode switch. Most of the opcode names are pretty
descriptive, but if you want a better overview, take a look at the 'dis'
module's documentation. (www.python.org is still down, but you can use one
of the mirrors:
http://python.mirrors.netnumina.com/doc/current/lib/module-dis.html
http://python.mirrors.netnumina.com/doc/current/lib/bytecodes.html
And an easy way to see how a Python operation is actually implemented is
using the 'dis' module interactively:
>>> import dis
>>> def spam(x):
... print "Hello Perl World!"
>>> dis.dis(spam)
0 SET_LINENO 1
3 SET_LINENO 2
6 SETUP_LOOP 44 (to 53)
>> 9 SET_LINENO 2
12 LOAD_FAST 0 (x)
15 JUMP_IF_FALSE 33 (to 51)
18 POP_TOP
19 SET_LINENO 3
22 LOAD_CONST 1 ('Hello ')
25 LOAD_CONST 2 ('Perl ')
28 BINARY_ADD
29 LOAD_CONST 3 ('World!')
32 BINARY_ADD
33 PRINT_ITEM
34 PRINT_NEWLINE
35 SET_LINENO 4
38 LOAD_FAST 0 (x)
41 LOAD_CONST 4 (1)
44 INPLACE_SUBTRACT
45 STORE_FAST 0 (x)
48 JUMP_ABSOLUTE 9
>> 51 POP_TOP
52 POP_BLOCK
>> 53 LOAD_CONST 0 (None)
56 RETURN_VALUE
Keep in mind that this is stack-based, and all operations manipulate the
stack. (for instance, LOAD_FAST pushes the variable onto the stack, and
BINARY_ADD pops two values, adds them, and pushes the result.)
--
Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!