On Tue, Jul 31, 2001 at 11:52:01PM -0400, Dan Sugalski wrote:
[SRE] and the C extension interface are the two big Python things I've got on my "must peer deeply into" list.
I can't say much about SRE, but the C extension interface should be on anyone's "must peer deeply into" list, except that there isn't a real need to peer deeply into it :) Lets peer into it. I'm going to discuss the pre-2.2 state of affairs, mostly because I know jack shit about the 2.2-type-dichotomy-healing :) In case you aren't aware, we are currently into the 2.2 alphas, where this story is slightly different because you can now subclass types. If Guido or anyone else involved in the descr-branch wants to tune in and correct the explanation for Python 2.2, that's fine by me. As you might have heard, everything in Python is an object. This is true even for the C side of things. A Python object, any Python object, is essentially a struct like this: stuct PyExample { int ob_refcnt; /* reference count */ struct _typeobject *ob_type; /* type of this object */ /* type-specific data goes here */ } The common header is made available as a #define, so that we can add extra info in severe-debug mode (though this breaks binary compatibility, so it should not be done in production builds.) For instance, the struct to hold a (normal, unbounded) Python integer is typedef struct { PyObject_HEAD long ob_ival; } PyIntObject; The struct _typeobject pointer is a pointer to a Type object, which also acts as a vtable. It contains all the information about the type, including how to operate on it. The typeobject is itself also an object! Its type is the PyType type, which is its own type. The typeobject is actually what you get back when you do 'type(object)' in Python code. All this is also explained in Include/object.h, by the way. The type object holds things like the type name, how to print it, howmuch memory should be allocated, a couple of flags to implement binary compatibility, and a whole boatload of function pointers. The function pointers for operations are spread out over the type struct and several separate structs, to which the type struct points: tp_as_number, tp_as_sequence, and tp_as_mapping. Each of these can be NULL if the type doesn't implement the operations. Some of the operations that are defined on the type struct itself are get-attribute, set-attribute, comparison, get-hashvalue, convert-to-string and call-as-function. The PyNumberMethods struct (tp_as_number) contains arithmatic operations (such as adding, mulitplying, shifting, XORing, and in-place variations of each for use with augmented assignment), truth-value-testing, and converting to various other integer types. The PySequenceMethods struct (tp_as_sequence) contains sequence operations such as membership-test, item- and slice-retrieval and -assignment, and sequence concatenation and repetition. The latter two are somewhat of an odd duck, since they implement the same Python operation ("+" and "*") as the 'add' and 'multiply' methods from the PyNumberMethods struct. They're mainly so the Python runtime can properly handle "number * list" as well as "list * number" without attempting to convert list to an integer. Lastly there's PyMappingMethods. I'm not at all sure why this is a separate sturct, as the three operations it contains (length inquiry, item-retrieval and item-assignment) are also covered by the PySequenceMethods, but I suspect some kind of hysterical raisin crept in here :) To see this construct in action, take a look at Objects/intobject.c, in the Python source tree (doesn't really matter which version.) It defines a bunch of functions to do the arithmatic, fills a PyNumberMethods struct with the appropriate pointers (leaving unsupported ones NULL), and fills a new PyTypeObject with the right info for the "int" type. To contrast, the PyFunction type is nearly empty. It defines getattr and setattr, and a 'repr' function (convert to string). You can't compare it (it will always compare false to anything but itself), add it, index it, or get its length, but you can get attributes from it. And the average function has some interesting attributes: you can, from Python code and C code alike, easily get at things like the bytecode it consists of, the number of arguments it expects, the names of those arguments, etc. Each of those is, of course, a Python object in its own right :) Function objects are defined in Objects/funcobject.c, though the actual execution of function objects is done elsewhere. An 'instance' object is a slightly different case in this story (more so in the post-2.2 story, but I'm not sure howmuch more.) An instance object is a Python class, defined in Python code by the user/programmer. It can choose to implement any of the above operations by defining magically named methods on the class. For instance, to implement the 'getitem' protocol, it defines a function __getitem__, and to implement addition, __add__. The Type object for instance objects acts as a proxy for the Python class, though this does represent some problems: instance objects need to be treated in some areas of the interpreter (but this is almost certain to have been changed in 2.2.) But back to generic Python objects. Obviously, when writing C code, you don't want to manipulate Python objects by directly accessing all those layers of C structs all the time. So each builtin type defines useful C functions and macros to help. For instance, to create a new list, you call PyList_New(size). To test whether something is a list, PyList_Check(object). To index a list, PyList_GetItem(list, index), or a macro version of that, PyList_GET_ITEM. The macro version doesn't do type or boundary checks, so it should only be used when you already did those checks. Likewise, there is PyInt_AsLong (convert a PyInt to a C 'long') and PyInt_AS_LONG. But even that isn't that helpful in a dynamically typed language, since you have to know an object's type in order to call the type-specific functions. So we also have two more abstraction layers. The most abstract one is the Object layer. Basically, regardless of the type of object, you can always manipulate it using PyObject_*() functions. Some PyObject_ functions have a direct relation to Python code, others are only useful to C code. There are PyObject_ functions such as PyObject_IsTrue, PyObject_Compare, PyObject_GetAttr, PyObject_CallObject, etcetera. All these work regardless of the type. If the type (or instance) does not implement the operation, an exception is raised (see below.) The other layer is the slightly less abstract layer that differentiates in the same way as the PyNumberMethods, PySequenceMethods, PyMappingMethods structs. To add two objects, you call PyNumber_Add(o1, o2). To explicitly concatenate two objects, you call PySequence_Concat(o1, o2). All of these work for any type that defines the behaviour, including instance types. Note that calling one of these functions can end up executing arbitrary Python code, so they shouldn't be called in any time-critical situation :-) As a last part of the API, there is the reference counting. As I showed a couple of pages back, all objects contain a reference count. Operations that copy a reference should increment the reference count, using Py_INCREF, and deleting a reference should be done using Py_DECREF. The latter also does the deallocation and cleanup if the DECREF causes the refcount to drop to 0. All API functions that return a reference define whether they return a new reference, which the caller has to DECREF when throwing it away, or a borrowed reference, which shouldn't be DECREF'ed, or kept around without INCREFing it. Exceptions, by the way, are set by setting a global (or thread-local, in a threaded interpreter) variable to contain the Python exception object, and then returning an error flag (either NULL, -1 or 0, depending on the function) to the caller. All code should check all return values and always return an error value if a called function does so, unless it's prepared to handle the exception itself. To compare errors with a particular error 'class', there is PyErr_ExceptionMatches (which also handles 'subclasses' of exception types.) If the error value returned by a particular function is also a real possible value (such as the value returned by PyNumber_AsLong), there is PyErr_Occured() to see if it was a real exception, or a legitimate -1 value. So that's it for the API. All Python objects can be manipulated, created and destroyed this way. Most extension types provide their functionality either in builtin-operations (in the type object) or by providing methods that can be called (using PyObject_GetAttr and PyObject_CallObject), and only rarely need to export an explicit API of their own. (And as you probably know, exporting an API from a shared library to another shared library is... slightly tricky :) Now, the Python bytecode-interpreter itself is very small. All it does is interpret bytecode, and then call out to the API to make it do the actual work. It has some special knowledge about some types, for optimzation, and it does all the real exception handling and namespace handling (name lookup and such), but the actual manipulation of objects is left to the API. Even so, it has enough to do. In the case of exceptions, it has to search up the function for a 'catching' block (a try/except or try/finally statement), or break out of the function to let the error percolate to the caller. Namespaces are a complete story in their own right. Python has three distinct namespaces: the 'local' namespace inside a function (or 'code block'), the 'global' or 'module-level' namespace at the toplevel of a file, and the 'builtin' namespace which holds builtin functions and names. The search order is always local->global->builtin, though for code that is executed at the module level (not inside a function), local is the same as global. In Python, variables are defined by assignment. As a consequence, any variable that is assigned to in a code block, is local to that block. Any variable that isn't assigned to but is just referenced thus has to be defined in the global namespace. The compiler keeps track of this, and generates 'LOAD_FAST' (for loading from the local namespace, inside a function), 'LOAD_GLOBAL' (for loading from the global or builtin namespace, skipping the local namespace), and 'LOAD_NAME' (for occurances where it's truly not known where a variable comes from, and both local and global namespaces have to be searched.) The latter is necessary because Python has a few operations that can modify a local namespace at runtime (such as 'from module import *', and 'exec'.) Since Python 2.1, Python has 'nested scopes', which makes code blocks defined inside other code blocks slightly special: the 'parent' scope is also searched for any variable that isn't local to the nested scope. Most of this, but not all, is done at compiletime, IIRC. (Jeremy can correct me if I'm wrong :) I tried to make the distinction between a 'function' and a 'code block', since the VM really deals with the second, not the first. I suspect that the current Python VM's mechanics to deal with nested scopes could be generalized so that Perl's lexical closures could fit as well (I assume that won't change in perl6 :). Then again, Perl already knows where a variable comes from, so the VM will probably want to put that to good use... The main reason Python does the name-searching at runtime is that a global namespace can be altered from outside the currently-executing codeblock (threads, or other modules, can do 'module.range = my_fake_range', and suddenly the range() function does something completely different throughout the entire module.) A good overview of how the VM uses the API can be obtained by looking at Python/ceval.c, and looking for "switch (opcode)". You'll end up at the top of the main bytecode switch. Most of the opcode names are pretty descriptive, but if you want a better overview, take a look at the 'dis' module's documentation. (www.python.org is still down, but you can use one of the mirrors: http://python.mirrors.netnumina.com/doc/current/lib/module-dis.html http://python.mirrors.netnumina.com/doc/current/lib/bytecodes.html And an easy way to see how a Python operation is actually implemented is using the 'dis' module interactively:
import dis def spam(x): ... print "Hello Perl World!" dis.dis(spam) 0 SET_LINENO 1
3 SET_LINENO 2
6 SETUP_LOOP 44 (to 53)
>> 9 SET_LINENO 2
12 LOAD_FAST 0 (x)
15 JUMP_IF_FALSE 33 (to 51)
18 POP_TOP
19 SET_LINENO 3
22 LOAD_CONST 1 ('Hello ')
25 LOAD_CONST 2 ('Perl ')
28 BINARY_ADD
29 LOAD_CONST 3 ('World!')
32 BINARY_ADD
33 PRINT_ITEM
34 PRINT_NEWLINE
35 SET_LINENO 4
38 LOAD_FAST 0 (x)
41 LOAD_CONST 4 (1)
44 INPLACE_SUBTRACT
45 STORE_FAST 0 (x)
48 JUMP_ABSOLUTE 9
>> 51 POP_TOP
52 POP_BLOCK
>> 53 LOAD_CONST 0 (None)
56 RETURN_VALUE
Keep in mind that this is stack-based, and all operations manipulate the
stack. (for instance, LOAD_FAST pushes the variable onto the stack, and
BINARY_ADD pops two values, adds them, and pushes the result.)
--
Thomas Wouters