custom allocation and custom type
castironpi at gmail.com
Mon Aug 4 21:15:01 CEST 2008
I am trying to create a custom tuple type. I want it to live in a
custom memory region, which will be a memory-mapped file. Its
contents cannot be PyObject*. They have to be offsets into mapped
memory. GetItem( i ) would return:
(PyObject *)( t->ob_item[ i ]+ mmap_base_addr );
In the constructor, each item x would be stored as:
BYTE* x_offset= x- mmap_base_addr;
t->ob_item[ 0 ]= x_offset;
For starters, only ints, strings, and other tuples will be members.
I'll work out the list type and user-defined classes at a later time.
Each will be required to be already existing in the mmap; that is, be
of custom types too.
I had partial success with strings and ints just memcpy-ing the entire
PyObject structure. They are easier because they are immutable, and
don't contain references. That's why tuple is my next step: it's
immutable, but can contain references.
I want to back up a step and ask some questions about ints first.
What worries are there in memcpy-ing a PyIntObject? Would it be
better to compile my own MmapIntObject? I do Py_INCREF on the object
I return, to ensure it lives on, but I am getting an occasional crash
on exit. What did I miss? Do I need to readjust the ob_type pointer
every time I load the module (and lock for exclusive access)? Does
Python try to free the memory when it exits?
For the tuples, is there any way I can reuse the tuple code? The
constructor uses a custom pool for speed which I'd probably eliminate,
for the time. Slices would have to be MmapTupleObject instances too.
Should I also create MmapTupleIterObject?
PyObject_GC_New and PyObject_GC_NewVar are confusing. I can't follow
them to the actual call to 'malloc', since they themselves lead around
Python's custom memory pool. What steps do I need to be sure to take
in my own?
I'm having trouble building PerstTupleObject.c (persistent tuple
object) to try it out. I'm using MinGW on WinXP. I added #include
"persttupleobject.h" to the top of "persttupleobject.c" already, but
it's a duplicate otherwise, with the name replaced.
C>c:/programs/mingw/bin/g++ persttupleobject.c -c -Ic:/programs/
persttupleobject.c:650: warning: 'PyPerstTuple_Type' defined locally
referenced with dllimport linkage
C>c:/programs/mingw/bin/g++ -shared persttupleobject.o -o
persttupleobject.o(.text+0xce):persttupleobject.c: undefined reference
reference to `_PyG
I get some of the same errors when compiling tupleobject.c from the
C>c:/programs/mingw/bin/g++ tupleobject.c -c -Ic:/programs/python25/
tupleobject.c:649: warning: 'PyTuple_Type' defined locally after being
d with dllimport linkage
C>c:/programs/mingw/bin/g++ -shared tupleobject.o -o tupleobject.pyd -
ition of `PyTuple_Type'
tupleobject.o(.data+0x140):tupleobject.c: first defined here
tupleobject.o(.text+0x15c):tupleobject.c: undefined reference to
Thanks in advance.
More information about the Python-list