On Sun, Aug 16, 2009 at 3:37 PM, "Martin v. Löwis"martin@v.loewis.de wrote:

...

So where does the Py_DECREF() for the above Py_INCREF() live?  I expected to find this code snippet somewhere, but couldn't:

if (type->tp_flags & Py_TPFLAGS_HEAPTYPE)     Py_DECREF(type);

For a regular heaptype, it's in subtype_dealloc:

/* Can't reference self beyond this point */                Py_DECREF(type);

Thanks for the pointer. I noticed that subtype_dealloc is only called for types that are allocated using type_new(). Does this mean that it is not safe to create types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on them? The documentation is not clear on this point.

Here is what I would like to do when I create my types dynamically:

- implement tp_alloc and tp_dealloc() to INCREF and DECREF the type. - not set Py_TPFLAGS_HEAPTYPE. - set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)

Does this seem safe? I notice that subtype_dealloc() does some funky GC/trashcan stuff. Is it safe for me not to call subtype_dealloc? Can I safely implement my tp_dealloc function like this?

void my_tp_dealloc(PyObject *obj) { obj->ob_type->tp_free(obj); Py_DECREF(obj->ob_type); }

Thanks, Josh

Benjamin Peterson wrote:

Why do you need to set Py_TPFLAGS_HEAPTYPE on your C type?

I think he *doesn't* want to set Py_TPFLAGS_HEAPTYPE, but does want to create the type dynamically.

But I suspect this is actually FUD, and that letting Py_TPFLAGS_HEAPTYPE be set wouldn't lead to anything disastrous happening.

Note that by not giving instances a __dict__, they will be prevented from having arbitrary attributes set on them, which is the most noticeable distinction between built-in and user-defined types.

On Sun, Aug 16, 2009 at 11:53 PM, "Martin v. Löwis"martin@v.loewis.de wrote:

...

Thanks for the pointer.  I noticed that subtype_dealloc is only called for types that are allocated using type_new().  Does this mean that it is not safe to create types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on them?  The documentation is not clear on this point.

As Benjamin says, this is getting off-topic - python-dev is not a place to ask for help in your project.

Please let me know where is a more suitable place to discuss the implementation of the cPython as it pertains to C extensions. I wrote to python-dev only because the other lists appeared to be more focused on Python-the-language.

I believe setting flags on a type is inherently unsafe.

Clearly this is not true in general. Take Py_TPFLAGS_BASETYPE, which C types are expected to set if they can be subclassed. Or Py_TPFLAGS_HAVE_GC, which C types set if they participate in cyclic reference collection.

The docs do not distinguish (AFAICS) between flags that C types may set directly and those that they may not. My reading of the docs left me with the impression that a type could set Py_TPFLAGS_HEAPTYPE if it had allocated that type on the heap and wanted it INCREF'd and DECREF'd by instances. I now know that there is much more to this flag than I anticipated (see http://thread.gmane.org/gmane.comp.python.devel/105648), I am just giving you feedback about why the docs led me to this incorrect conclusion.

In any case, I think I will experiment with a different approach, where instead of creating types in C dynamically at runtime, I will create a type whose instances "pretend" to be types (they will create instances when called). Still, I would appreciate knowing where I should direct further questions of this type, which are not questions about how to use Python but rather questions about how to properly implement extensions.

...

Here is what I would like to do when I create my types dynamically:

• implement tp_alloc and tp_dealloc() to INCREF and DECREF the type.
• not set Py_TPFLAGS_HEAPTYPE.
• set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)

Does this seem safe?  I notice that subtype_dealloc() does some funky GC/trashcan stuff.  Is it safe for me not to call subtype_dealloc?  Can I safely implement my tp_dealloc function like this?

If you bypass documented API, you really need to study the code, understand its motivation, judge whether certain usage is "safe" wrt. to the current implementation, and judge the likelihood of this code not getting changed in future versions.

It was not my intention to bypass the documented API. Py_TPFLAGS_HEAPTYPE is documented here, with no note that the flag should not be set explicitly by C types:

http://docs.python.org/c-api/typeobj.html#Py_TPFLAGS_HEAPTYPE

Also, INCREF'ing and DECREF'ing my type from the tp_new and tp_dealloc functions doesn't seem outside of the documented API.

Thanks, Josh

On Mon, Aug 17, 2009 at 1:41 PM, "Martin v. Löwis"martin@v.loewis.de wrote:

...

...

As Benjamin says, this is getting off-topic - python-dev is not a place to ask for help in your project.

Please let me know where is a more suitable place to discuss the implementation of the cPython as it pertains to C extensions. I wrote to python-dev only because the other lists appeared to be more focused on Python-the-language.

The general python-list, or, more specifically, capi-sig.

python-dev is exclusively reserved for current development of Python (including PEP discussions). It is out-of-scope to ask questions of the form "how do I do XYZ?".

Ok, I will direct future questions of this sort to one of those two places -- thanks and sorry for posting off-topic.

Josh