[Python-Dev] another Py_TPFLAGS_HEAPTYPE question

[Joshua Haberman]

On Sun, Aug 16, 2009 at 11:53 PM, "Martin v. Löwis"<martin at v.loewis.de> wrote:
>> Thanks for the pointer.  I noticed that subtype_dealloc is only called for types
>> that are allocated using type_new().  Does this mean that it is not
>> safe to create
>> types in C using just PyType_Ready() and set Py_TPFLAGS_HEAPTYPE on
>> them?  The documentation is not clear on this point.
>
> As Benjamin says, this is getting off-topic - python-dev is not a place
> to ask for help in your project.

Please let me know where is a more suitable place to discuss the
implementation of the cPython as it pertains to C extensions. I wrote
to python-dev only because the other lists appeared to be more focused
on Python-the-language.

> I believe setting flags on a type is inherently unsafe.

Clearly this is not true in general.  Take Py_TPFLAGS_BASETYPE, which
C types are expected to set if they can be subclassed.  Or
Py_TPFLAGS_HAVE_GC, which C types set if they participate in cyclic
reference collection.

The docs do not distinguish (AFAICS) between flags that C types may set
directly and those that they may not.  My reading of the docs left me with the
impression that a type could set Py_TPFLAGS_HEAPTYPE if it had allocated
that type on the heap and wanted it INCREF'd and DECREF'd by instances.
I now know that there is much more to this flag than I anticipated (see
http://thread.gmane.org/gmane.comp.python.devel/105648), I am just
giving you feedback about why the docs led me to this incorrect conclusion.

In any case, I think I will experiment with a different approach, where instead
of creating types in C dynamically at runtime, I will create a type
whose instances
"pretend" to be types (they will create instances when called).  Still, I would
appreciate knowing where I should direct further questions of this type, which
are not questions about how to use Python but rather questions about how to
properly implement extensions.

>> Here is what I would like to do when I create my types dynamically:
>>
>> - implement tp_alloc and tp_dealloc() to INCREF and DECREF the type.
>> - not set Py_TPFLAGS_HEAPTYPE.
>> - set Py_TPFLAGS_HAVE_GC (because instances of my obj can create cycles)
>>
>> Does this seem safe?  I notice that subtype_dealloc() does some funky
>> GC/trashcan stuff.  Is it safe for me not to call subtype_dealloc?  Can I
>> safely implement my tp_dealloc function like this?
>
> If you bypass documented API, you really need to study the code,
> understand its motivation, judge whether certain usage is "safe" wrt.
> to the current implementation, and judge the likelihood of this code
> not getting changed in future versions.

It was not my intention to bypass the documented API.
Py_TPFLAGS_HEAPTYPE is documented here, with no note that
the flag should not be set explicitly by C types:

http://docs.python.org/c-api/typeobj.html#Py_TPFLAGS_HEAPTYPE

Also, INCREF'ing and DECREF'ing my type from the tp_new and
tp_dealloc functions doesn't seem outside of the documented API.

Thanks,
Josh