[Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded

[Joshua Haberman]

I'm writing a C Python extension that needs to generate PyTypeObjects
dynamically.  Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded
in a way that makes it difficult to achieve this goal.

The documentation for Pt_TPFLAGS_HEAPTYPE says:

Py_TPFLAGS_HEAPTYPE

  This bit is set when the type object itself is allocated
  on the heap. In this case, the ob_type field of its
  instances is considered a reference to the type, and the
  type object is INCREF’ed when a new instance is created,
  and DECREF’ed when an instance is destroyed (this does not
  apply to instances of subtypes; only the type referenced
  by the instance’s ob_type gets INCREF’ed or DECREF’ed).

This sounds like exactly what I want.  I want my type object INCREF'd
and DECREF'd by its instances so it doesn't leak or get deleted
prematurely.  If this were all that Py_TPFLAGS_HEAPTYPE did, it would
work great for me.

Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean
"user-defined type" (as opposed to a built-in type).  It controls
numerous subtle behaviors such as:

- whether the type's name is module.type or just type.
- whether you're allowed to set __name__, __module__, or __bases__ on the type.
- whether you're allowed to set __class__ on instances of this type.
- whether the module name comes from the type name or the __module__ attribute.
- whether it will use type->tp_doc as the docstring
- whether its repr() calls it a "class" or a "type".
- whether you can set attributes of the type.
- whether someone is attempting the Carlo Verre hack.

So I'm stuck with an unenviable choice.  I think the lesser of two evils
is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will
happen is that my types will leak.  This is not as bad as having someone
set __class__ on one of my instances, or set attributes on my type, etc.

Ideally the interpreter would have a separate flag like
Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but
still make it possible to have dynamically generated built-in types get
garbage collected appropriately.

At the very least, the documentation I cited above should make it clear
that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets
INCREF'd and DECREF'd.  Based on the list of behaviors I discovered
above, it is almost certainly not correct for a C exension type to be
declared with Py_TPFLAGS_HEAPTYPE.

Josh