Hi All, The following extension module (AA) is a reduced example of what I'm doing to make extension classes in 2.2. I followed the examples given by typeobject.c. When I "import AA,pdb" I get a crash in GC. Investigating further, I see this makes sense: GC is enabled in class_metatype_object, yet class_type_object does not follow the first rule of objects whose type has GC enabled: "The memory for the object must be allocated using PyObject_GC_New() or PyObject_GC_VarNew()." So, I guess the question is, how does PyBaseObject_Type (also statically allocated) get away with it? TIA, Dave ---------------- // Copyright David Abrahams 2002. Permission to copy, use, // modify, sell and distribute this software is granted provided this // copyright notice appears in all copies. This software is provided // "as is" without express or implied warranty, and with no claim as // to its suitability for any purpose. #include <Python.h> PyTypeObject class_metatype_object = { PyObject_HEAD_INIT(0) 0, "Boost.Python.class", PyType_Type.tp_basicsize, 0, 0, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, // &PyType_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, // PyType_GenericNew /* tp_new */ }; // Get the metatype object for all extension classes. PyObject* class_metatype() { if (class_metatype_object.tp_dict == 0) { class_metatype_object.ob_type = &PyType_Type; class_metatype_object.tp_base = &PyType_Type; if (PyType_Ready(&class_metatype_object)) return 0; } Py_INCREF(&class_metatype_object); return (PyObject*)&class_metatype_object; } // Each extension instance will be one of these typedef struct instance { PyObject_HEAD void* objects; } instance; static void instance_dealloc(PyObject* inst) { instance* kill_me = (instance*)inst; inst->ob_type->tp_free(inst); } PyTypeObject class_type_object = { PyObject_HEAD_INIT(0) file://&class_metatype_object) 0, "Boost.Python.instance", sizeof(PyObject), 0, instance_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, file://&PyBaseObject_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ PyType_GenericAlloc, /* tp_alloc */ PyType_GenericNew }; PyObject* class_type() { if (class_type_object.tp_dict == 0) { class_type_object.ob_type = (PyTypeObject*)class_metatype(); class_type_object.tp_base = &PyBaseObject_Type; if (PyType_Ready(&class_type_object)) return 0; } Py_INCREF(&class_type_object); return (PyObject*)&class_type_object; } PyObject* make_class() { PyObject* bases, *args, *mt, *result; bases = PyTuple_New(1); PyTuple_SET_ITEM(bases, 0, class_type()); args = PyTuple_New(3); PyTuple_SET_ITEM(args, 0, PyString_FromString("AA")); PyTuple_SET_ITEM(args, 1, bases); PyTuple_SET_ITEM(args, 2, PyDict_New()); mt = class_metatype(); result = PyObject_CallObject(mt, args); Py_XDECREF(mt); Py_XDECREF(args); return result; } static PyMethodDef SpamMethods[] = { {NULL, NULL} /* Sentinel */ }; DL_EXPORT(void) initAA() { PyObject *m, *d; m = Py_InitModule("AA", SpamMethods); d = PyModule_GetDict(m); PyDict_SetItemString(d, "AA", make_class()); } +---------------------------------------------------------------+ David Abrahams C++ Booster (http://www.boost.org) O__ == Pythonista (http://www.python.org) c/ /'_ == resume: http://users.rcn.com/abrahams/resume.html (*) \(*) == email: david.abrahams@rcn.com +---------------------------------------------------------------+
On Thu, 21 Feb 2002, David Abrahams wrote:
The following extension module (AA) is a reduced example of what I'm doing to make extension classes in 2.2. I followed the examples given by typeobject.c. When I "import AA,pdb" I get a crash in GC. Investigating further, I see this makes sense: GC is enabled in class_metatype_object, yet class_type_object does not follow the first rule of objects whose type has GC enabled:
"The memory for the object must be allocated using PyObject_GC_New() or PyObject_GC_VarNew()."
So, I guess the question is, how does PyBaseObject_Type (also statically allocated) get away with it?
I doesn't have any time to really look at your code, but I thought I'd point out a trick that several extension modules use to protect statically allocated type objects. Here is the code from socketmodule.c: /* static PyTypeObject PySocketSock_Type = { . . . 0, /* set below */ /* tp_alloc */ PySocketSock_new, /* tp_new */ 0, /* set below */ /* tp_free */ }; /* buried in init_socket */ PySocketSock_Type.tp_alloc = PyType_GenericAlloc; PySocketSock_Type.tp_free = _PyObject_Del; This trick ensures that the static type object is never freed. Also, there is a funny-looking line in your PyTypeObject: 0, file://&PyBaseObject_Type, /* tp_base */ -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
Kevin Jacobs <jacobs@penguin.theopalgroup.com>:
I doesn't have any time to really look at your code, but I thought I'd point out a trick that several extension modules use to protect statically allocated type objects.
0, /* set below */ /* tp_alloc */ PySocketSock_new, /* tp_new */ 0, /* set below */ /* tp_free */
I don't think that has anything to do with protecting the type object. As I understand it, static type objects are protected by having their refcount statically initialised to 1, so that it will never drop to zero. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
From: "David Abrahams" <david.abrahams@rcn.com>
Hi All,
The following extension module (AA) is a reduced example of what I'm doing to make extension classes in 2.2. I followed the examples given by typeobject.c. When I "import AA,pdb" I get a crash in GC.
For me it crashes after import AA, gc gc.collect() (Win2k Prof, SP1, MSVC6.0) I'm not really sure, but it seems your code does not crash any longer if you remove the Py_TPFLAGS_HAVE_GC from your definition of class_metatype_object. This flag *will* be set by PyType_Ready(); I guess it is inherited from the base type (PyType_Type in your case). Thomas
PS: I have code very similar to yours, and a question: Why does your class_type_object have PyPyBase_ObjectType as tp_base? To implement a subtypable type this is not needed IMO, or do I miss something? Thomas
----- Original Message ----- From: "Thomas Heller" <thomas.heller@ion-tof.com>
From: "David Abrahams" <david.abrahams@rcn.com>
Hi All,
The following extension module (AA) is a reduced example of what I'm doing to make extension classes in 2.2. I followed the examples given by typeobject.c. When I "import AA,pdb" I get a crash in GC.
For me it crashes after import AA, gc gc.collect() (Win2k Prof, SP1, MSVC6.0)
I'm not really sure, but it seems your code does not crash any longer if you remove the Py_TPFLAGS_HAVE_GC from your definition of class_metatype_object.
Yes, I'm aware of that. What I don't understand is how the builtin metatype gets away with Py_TPFLAGS_HAVE_GC when some of its instance types are not even heap-allocated. -Dave
I'm not really sure, but it seems your code does not crash any longer if you remove the Py_TPFLAGS_HAVE_GC from your definition of class_metatype_object.
Yes, I'm aware of that. What I don't understand is how the builtin metatype gets away with Py_TPFLAGS_HAVE_GC when some of its instance types are not even heap-allocated.
Hm, I don't understand you, Are you talking about Py_TPFLAGS_HEAPTYPE? Thomas
I'm not really sure, but it seems your code does not crash any longer if you remove the Py_TPFLAGS_HAVE_GC from your definition of class_metatype_object.
Yes, I'm aware of that. What I don't understand is how the builtin
----- Original Message ----- From: "Thomas Heller" <thomas.heller@ion-tof.com> To: "David Abrahams" <david.abrahams@rcn.com>; <python-dev@python.org> Sent: Thursday, February 21, 2002 10:01 AM Subject: Re: [Python-Dev] A little GC confusion metatype
gets away with Py_TPFLAGS_HAVE_GC when some of its instance types are not even heap-allocated.
Hm, I don't understand you, Are you talking about Py_TPFLAGS_HEAPTYPE?
No, please re-read my initial posting. Py_TPFLAGS_HAVE_GC places requirements on the allocation method of instances, at least according to the docs.
Thomas Heller wrote:
David Abrahams wrote:
What I don't understand is how the builtin metatype gets away with Py_TPFLAGS_HAVE_GC when some of its instance types are not even heap-allocated.
Hm, I don't understand you, Are you talking about Py_TPFLAGS_HEAPTYPE?
I think David is asking about line 1404 of Objects/typeobject.c, where it says that PyType_Type is Py_TPFLAGS_HAVE_GC. How can it have GC when many instances are static objects, not allocated with PyObject_GC_VarNew()? I don't know the answer. ## Jason Orendorff http://www.jorendorff.com/
"Jason Orendorff" <jason@jorendorff.com> writes:
I think David is asking about line 1404 of Objects/typeobject.c, where it says that PyType_Type is Py_TPFLAGS_HAVE_GC. How can it have GC when many instances are static objects, not allocated with PyObject_GC_VarNew()?
Because the type type implements tp_is_gc (typeobject.c:1378), declaring static type objects as not being gc. In turn, garbage collection will not attempt to look at the GC header of these type objects. Regards, Martin
----- Original Message ----- From: "Martin v. Loewis" <martin@v.loewis.de>
"Jason Orendorff" <jason@jorendorff.com> writes:
I think David is asking about line 1404 of Objects/typeobject.c, where it says that PyType_Type is Py_TPFLAGS_HAVE_GC. How can it have GC when many instances are static objects, not allocated with PyObject_GC_VarNew()?
Because the type type implements tp_is_gc (typeobject.c:1378), declaring static type objects as not being gc. In turn, garbage collection will not attempt to look at the GC header of these type objects.
Aha! And the implementation is... static int type_is_gc(PyTypeObject *type) { return type->tp_flags & Py_TPFLAGS_HEAPTYPE; } so, wouldn't it make more sense that the Python source always checks Py_TPFLAGS_HEAPTYPE before tp_is_gc? Also, is there any guideline for which type slots get automatically copied from the base type? Since my slots are nearly all zero I expected to inherit most of the slots from type_type. -Dave
----- Original Message ----- From: "David Abrahams" <david.abrahams@rcn.com>
----- Original Message ----- From: "Martin v. Loewis" <martin@v.loewis.de>
Because the type type implements tp_is_gc (typeobject.c:1378), declaring static type objects as not being gc. In turn, garbage collection will not attempt to look at the GC header of these type objects.
Aha! And the implementation is...
static int type_is_gc(PyTypeObject *type) { return type->tp_flags & Py_TPFLAGS_HEAPTYPE; }
so, wouldn't it make more sense that the Python source always checks Py_TPFLAGS_HEAPTYPE before tp_is_gc?
Nice try, but no cigar I'm afraid: copying the tp_is_gc slot from PyType_Type into my metatype before PyType_Ready() doesn't prevent the crash. Does anyone really understand what happens here?
"David Abrahams" <david.abrahams@rcn.com> writes:
Nice try, but no cigar I'm afraid: copying the tp_is_gc slot from PyType_Type into my metatype before PyType_Ready() doesn't prevent the crash.
Does anyone really understand what happens here?
Understand why your code crashes? Because there is a bug in it... To understand what the bug is, one would have to study your code first. Regards, Martin
----- Original Message ----- From: "Martin v. Loewis" <martin@v.loewis.de>
"David Abrahams" <david.abrahams@rcn.com> writes:
Nice try, but no cigar I'm afraid: copying the tp_is_gc slot from PyType_Type into my metatype before PyType_Ready() doesn't prevent the crash.
Does anyone really understand what happens here?
Understand why your code crashes?
I'm not asking that. I'm asking if anyone really understands how the flags and tp_xxx slots are supposed to interact.
Because there is a bug in it...
I /guess/ there's a bug in my code if you measure it against the standard that says "if it doesn't work with the current Python source code, it's buggy". I'd consider that standard a bit more legitimate if I could find, for example, a mention of Py_TPFLAGS_HEAPTYPE *anywhere* in the Python docs. As it stands, your position seems a bit more unhelpful than neccessary. I can live with incomplete documentation if there's someone around who can explain how the software is supposed to be used; I just want to fill in the holes so that I know I'm not making important errors. I thought I was doing everything right until a few days ago when someone tried something new with my code and uncovered the GC crash. One can only cover so many cases with tests. Even if I repair this problem, how can I be sure I've got the rest of the formula right? Better docs would fix that problem, and give us an objective standard against which to judge which code has bugs. In lieu of that, I would hope that my questions would be answered in good faith. [In the meantime, GC remains turned off for my types and metatypes]
To understand what the bug is, one would have to study your code first.
I posted the code yesterday. Did you miss it? I'm sure you could figure out how to apply the simple modification described at the top of this message. -Dave
[David Abrahams]
I /guess/ there's a bug in my code if you measure it against the standard that says "if it doesn't work with the current Python source code, it's buggy". I'd consider that standard a bit more legitimate if I could find, for example, a mention of Py_TPFLAGS_HEAPTYPE *anywhere* in the Python docs.
I've been struggling with the meaning of the various TPFLAGS myself. I don't think it's documented anywhere, and I don't think anyone except Guido really understands what all the flags mean. One property of types that do not have define HEAPTYPE is that their __module__ attribute is always __builtin__. This makes them mighty hard to pickle. It further suggests that every type that isn't a builtin type should define HEAPTYPE. There are lots of other cases affected by HEAPTYPE. I imagine you've done the same grep that I did. Jeremy
[I wrote:]
One property of types that do not have define HEAPTYPE is that their __module__ attribute is always __builtin__. This makes them mighty hard to pickle. It further suggests that every type that isn't a builtin type should define HEAPTYPE.
I don't think I made much sense above. I meant to say: When my C types didn't define HEAPTYPE, it was impossible to pickle them. When I added the HEAPTYPE and defined __safe_for_unpickling__ as a data member, it became possible to pickle instances of those types. It was far from obvious, though, that I needed to do those two things to make pickling work. Jeremy
[Jeremy Hylton]
I've been struggling with the meaning of the various TPFLAGS myself. I don't think it's documented anywhere, and I don't think anyone except Guido really understands what all the flags mean.
I agree that at this point Guido is the only one who fully understands what they were all *intended* to mean, but I don't believe even Guido can tell you (without the same kinds of study and experimentation and hair-pulling you're doing) what the flags actually do today in all circumstances and combinations. A consequence is that neither can he (or anyone else) always predict what you need to do to get a desired result. What shipped in 2.2 was solid to the extent that it supported everything used by the Python core. You and David are pushing it in other directions, and while it was intended to support them, this stuff was never really *tried* at the C level beyond the demo xxsubtype.c module and some ExtensionClass fiddling. Most "weird experiments" were tried at the Python level instead, just because it's so much more time-efficient to try stuff in Python, and time was in short supply. So you're pioneers, and you've got to draw your own maps of the new territory. Luckily, God isn't resting yet, so He can still create new lifeforms if needed <wink>.
One property of types that do not have define HEAPTYPE is that their __module__ attribute is always __builtin__. This makes them mighty hard to pickle. It further suggests that every type that isn't a builtin type should define HEAPTYPE.
Yup, all kinds of questions get answered by "does it have HEAPTYPE?" that don't have any obvious connection to heaps. One of my favorites is this seemingly straightforward branch in type_repr(): if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) kind = "class"; else kind = "type"; The philosophical questions that raises could go on for pages <wink>.
"David Abrahams" <david.abrahams@rcn.com> writes:
Nice try, but no cigar I'm afraid: copying the tp_is_gc slot from PyType_Type into my metatype before PyType_Ready() doesn't prevent the crash.
Does anyone really understand what happens here?
After studying your code in a debugger, it turns out that the code now crashes for a different reason: The "AA" class created in make_class is traversed in subtract_refs. To do so, its type's traverse function is invoked, i.e. classtype_meta_object.tp_traverse. This is a null pointer, hence the crash. If you want objects (in your case, classes) to participate in GC, the of the objects (in your case, the metaclass) needs to implement the GC API. IOW, don't set Py_TPFLAGS_HAVE_GC in a type unless you also set tp_clear and tp_traverse in the same type, see http://www.python.org/doc/current/api/supporting-cycle-detection.html for details. This likely has been the problem all the time; if I remove tp_is_gc, but implement tp_traverse, your test case (import AA,pdb) does not crash anymore. BTW, gcc rejects the code you've posted, as you cannot use PyType_Type.tp_basicsize in an initializer of a global object (it's not a constant). HTH, Martin
"David Abrahams" <david.abrahams@rcn.com> writes:
static int type_is_gc(PyTypeObject *type) { return type->tp_flags & Py_TPFLAGS_HEAPTYPE; }
so, wouldn't it make more sense that the Python source always checks Py_TPFLAGS_HEAPTYPE before tp_is_gc?
No. Most GC objects do not have the HEAPTYPE flag (they actually aren't even type objects). Regards, Martin
"Jason Orendorff" <jason@jorendorff.com> writes:
How can it have GC when many instances are static objects, not allocated with PyObject_GC_VarNew()?
Martin v. Loewis writes:
Because the type type implements tp_is_gc (typeobject.c:1378), declaring static type objects as not being gc. In turn, garbage collection will not attempt to look at the GC header of these type objects.
I'm starting to really fear writing the documentation for all this! There are going to be a lot of mostly-inscrutible details to get right, and people are already asking the questions, so it really will need to be written down. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
participants (9)
-
David Abrahams -
Fred L. Drake, Jr. -
Greg Ewing -
Jason Orendorff -
Jeremy Hylton -
Kevin Jacobs -
martin＠v.loewis.de -
Thomas Heller -
Tim Peters