I wrote to this list a few weeks ago asking about Py_TPFLAGS_HEAPTYPE (http://thread.gmane.org/gmane.comp.python.devel/105648). It occurred to me today that I could probably make object instances INCREF and DECREF my type appropriately, without setting Py_TPFLAGS_HEAPTYPE, by writing my own tp_alloc and tp_dealloc functions. My tp_alloc function could be:
PyObject *my_tp_alloc(PyTypeObject *type, Py_ssize_t nitems) { PyObject *obj = PyType_GenericAlloc(type, nitems); if(obj) Py_INCREF(type); return obj; }
This seems right since it is PyType_GenericAlloc that contains this excerpt:
if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_INCREF(type);
I don't want to set Py_TPFLAGS_HEAPTYPE, but I want to get that Py_INCREF(), so far so good.
But I couldn't find the corresponding Py_DECREF() in typeobject.c to the above Py_INCREF(). Notably, object_dealloc() does not call Py_DECREF(self->ob_type) if self->ob_type has the Py_TPFLAGS_HEAPTYPE flag set.
So where does the Py_DECREF() for the above Py_INCREF() live? I expected to find this code snippet somewhere, but couldn't:
if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_DECREF(type);
Josh
So where does the Py_DECREF() for the above Py_INCREF() live? I expected to find this code snippet somewhere, but couldn't:
if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_DECREF(type);
For a regular heaptype, it's in subtype_dealloc:
/* Can't reference self beyond this point */ Py_DECREF(type);
HTH, Martin
On Sun, Aug 16, 2009 at 3:37 PM, "Martin v. Löwis"martin@v.loewis.de wrote:
So where does the Py_DECREF() for the above Py_INCREF() live? I expected to find this code snippet somewhere, but couldn't:
if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_DECREF(type);
For a regular heaptype, it's in subtype_dealloc:
/* Can't reference self beyond this point */ Py_DECREF(type);
Thanks for the pointer. I noticed that subtype_dealloc is only called for types that are allocated using type_new(). Does this mean that it is not safe to create types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on them? The documentation is not clear on this point.
Here is what I would like to do when I create my types dynamically:
- implement tp_alloc and tp_dealloc() to INCREF and DECREF the type. - not set Py_TPFLAGS_HEAPTYPE. - set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)
Does this seem safe? I notice that subtype_dealloc() does some funky GC/trashcan stuff. Is it safe for me not to call subtype_dealloc? Can I safely implement my tp_dealloc function like this?
void my_tp_dealloc(PyObject *obj) { obj->ob_type->tp_free(obj); Py_DECREF(obj->ob_type); }
Thanks, Josh
2009/8/16 Joshua Haberman joshua@reverberate.org:
On Sun, Aug 16, 2009 at 3:37 PM, "Martin v. Löwis"martin@v.loewis.de wrote:
So where does the Py_DECREF() for the above Py_INCREF() live? I expected to find this code snippet somewhere, but couldn't:
if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) Py_DECREF(type);
For a regular heaptype, it's in subtype_dealloc:
/* Can't reference self beyond this point */ Py_DECREF(type);
Thanks for the pointer. I noticed that subtype_dealloc is only called for types that are allocated using type_new(). Does this mean that it is not safe to create types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on them? The documentation is not clear on this point.
Here is what I would like to do when I create my types dynamically:
- implement tp_alloc and tp_dealloc() to INCREF and DECREF the type.
- not set Py_TPFLAGS_HEAPTYPE.
- set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)
[Note that this is really starting to get off topic for python-dev.]
Why do you need to set Py_TPFLAGS_HEAPTYPE on your C type? Is a normal static type not sufficient? The easiest way to create heaptypes is to simply call PyType_Type.
Benjamin Peterson wrote:
Why do you need to set Py_TPFLAGS_HEAPTYPE on your C type?
I think he *doesn't* want to set Py_TPFLAGS_HEAPTYPE, but does want to create the type dynamically.
But I suspect this is actually FUD, and that letting Py_TPFLAGS_HEAPTYPE be set wouldn't lead to anything disastrous happening.
Note that by not giving instances a __dict__, they will be prevented from having arbitrary attributes set on them, which is the most noticeable distinction between built-in and user-defined types.
Thanks for the pointer. I noticed that subtype_dealloc is only called for types that are allocated using type_new(). Does this mean that it is not safe to create types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on them? The documentation is not clear on this point.
As Benjamin says, this is getting off-topic - python-dev is not a place to ask for help in your project.
I believe setting flags on a type is inherently unsafe.
Here is what I would like to do when I create my types dynamically:
- implement tp_alloc and tp_dealloc() to INCREF and DECREF the type.
- not set Py_TPFLAGS_HEAPTYPE.
- set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)
Does this seem safe? I notice that subtype_dealloc() does some funky GC/trashcan stuff. Is it safe for me not to call subtype_dealloc? Can I safely implement my tp_dealloc function like this?
If you bypass documented API, you really need to study the code, understand its motivation, judge whether certain usage is "safe" wrt. to the current implementation, and judge the likelihood of this code not getting changed in future versions.
Regards, Martin
On Sun, Aug 16, 2009 at 11:53 PM, "Martin v. Löwis"martin@v.loewis.de wrote:
Thanks for the pointer. I noticed that subtype_dealloc is only called for types that are allocated using type_new(). Does this mean that it is not safe to create types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on them? The documentation is not clear on this point.
As Benjamin says, this is getting off-topic - python-dev is not a place to ask for help in your project.
Please let me know where is a more suitable place to discuss the implementation of the cPython as it pertains to C extensions. I wrote to python-dev only because the other lists appeared to be more focused on Python-the-language.
I believe setting flags on a type is inherently unsafe.
Clearly this is not true in general. Take Py_TPFLAGS_BASETYPE, which C types are expected to set if they can be subclassed. Or Py_TPFLAGS_HAVE_GC, which C types set if they participate in cyclic reference collection.
The docs do not distinguish (AFAICS) between flags that C types may set directly and those that they may not. My reading of the docs left me with the impression that a type could set Py_TPFLAGS_HEAPTYPE if it had allocated that type on the heap and wanted it INCREF'd and DECREF'd by instances. I now know that there is much more to this flag than I anticipated (see http://thread.gmane.org/gmane.comp.python.devel/105648), I am just giving you feedback about why the docs led me to this incorrect conclusion.
In any case, I think I will experiment with a different approach, where instead of creating types in C dynamically at runtime, I will create a type whose instances "pretend" to be types (they will create instances when called). Still, I would appreciate knowing where I should direct further questions of this type, which are not questions about how to use Python but rather questions about how to properly implement extensions.
Here is what I would like to do when I create my types dynamically:
- implement tp_alloc and tp_dealloc() to INCREF and DECREF the type.
- not set Py_TPFLAGS_HEAPTYPE.
- set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)
Does this seem safe? I notice that subtype_dealloc() does some funky GC/trashcan stuff. Is it safe for me not to call subtype_dealloc? Can I safely implement my tp_dealloc function like this?
If you bypass documented API, you really need to study the code, understand its motivation, judge whether certain usage is "safe" wrt. to the current implementation, and judge the likelihood of this code not getting changed in future versions.
It was not my intention to bypass the documented API. Py_TPFLAGS_HEAPTYPE is documented here, with no note that the flag should not be set explicitly by C types:
http://docs.python.org/c-api/typeobj.html#Py_TPFLAGS_HEAPTYPE
Also, INCREF'ing and DECREF'ing my type from the tp_new and tp_dealloc functions doesn't seem outside of the documented API.
Thanks, Josh
As Benjamin says, this is getting off-topic - python-dev is not a place to ask for help in your project.
Please let me know where is a more suitable place to discuss the implementation of the cPython as it pertains to C extensions. I wrote to python-dev only because the other lists appeared to be more focused on Python-the-language.
The general python-list, or, more specifically, capi-sig.
python-dev is exclusively reserved for current development of Python (including PEP discussions). It is out-of-scope to ask questions of the form "how do I do XYZ?".
Regards, Martin
On Mon, Aug 17, 2009 at 1:41 PM, "Martin v. Löwis"martin@v.loewis.de wrote:
As Benjamin says, this is getting off-topic - python-dev is not a place to ask for help in your project.
Please let me know where is a more suitable place to discuss the implementation of the cPython as it pertains to C extensions. I wrote to python-dev only because the other lists appeared to be more focused on Python-the-language.
The general python-list, or, more specifically, capi-sig.
python-dev is exclusively reserved for current development of Python (including PEP discussions). It is out-of-scope to ask questions of the form "how do I do XYZ?".
Ok, I will direct future questions of this sort to one of those two places -- thanks and sorry for posting off-topic.
Josh
participants (4)
-
"Martin v. Löwis" -
Benjamin Peterson -
Greg Ewing -
Joshua Haberman