functions not showing up in pydoc
I'm running into a problem with functions at the module level not showing up in pydoc. The problem is that the type of the function object in the module is:
type(testfoo.foo) <type 'Boost.Python.function'>
Which isn't a sublcass of <type 'builtin_function_or_method'> When pydoc walks the module it only documents routines that are either builtin (according to inspect.isbuiltin) or are objects whose module can be deduced via inspect.getmodule. I modified function.cpp with: --- libs/python/src/object/function.cpp.orig 2005-02-07 17:11:52.000066304 -0800 +++ libs/python/src/object/function.cpp 2005-02-07 16:55:24.000120507 -0800 @@ -608,7 +608,7 @@ 0, /* tp_methods */ 0, // func_memberlist, /* tp_members */ function_getsetlist, /* tp_getset */ - 0, /* tp_base */ + &PyCFunction_Type, /* tp_base */ 0, /* tp_dict */ function_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */ and it seemed to fix it. Is this right, or is there a different way that functions should get exposed such that pydoc can understand them? Also, for functions with overloads, only one of the docstrings is printed out. It looks like void function::add_to_namespace( object const& name_space, char const* name_, object const& attribute, char const* doc) just replaces the old doc string instead of appending the new one to the old. -nick
--- Nick Rasmussen <nick@ilm.com> wrote:
I'm running into a problem with functions at the module level not showing up in pydoc. The problem is that the type of the function object in the module is:
type(testfoo.foo) <type 'Boost.Python.function'>
Which isn't a sublcass of <type 'builtin_function_or_method'>
When pydoc walks the module it only documents routines that are either builtin (according to inspect.isbuiltin) or are objects whose module can be deduced via inspect.getmodule.
I modified function.cpp with:
--- libs/python/src/object/function.cpp.orig 2005-02-07 17:11:52.000066304 -0800 +++ libs/python/src/object/function.cpp 2005-02-07 16:55:24.000120507 -0800 @@ -608,7 +608,7 @@ 0, /* tp_methods */ 0, // func_memberlist, /* tp_members */ function_getsetlist, /* tp_getset */ - 0, /* tp_base */ + &PyCFunction_Type, /* tp_base */ 0, /* tp_dict */ function_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */
and it seemed to fix it. Is this right, or is there a different way that functions should get exposed such that pydoc can understand them?
I applied this patch locally and tested it in 14 different configurations. It appears to work OK with Python 2.2, but leads to platform-specific fatal errors with Python 2.3 and 2.4 (Abort, Bus error, C assertions in Python, or plain wrong results). Here is the full damage: http://cci.lbl.gov/cctbx_build/show_error_summary.cgi?build_tag=2005_02_10_1... (This page will go away after a few days.) Using the exact same sources, but without the patch above, eliminates all errors. Before I knew about the Python 2.3 and 2.4 failures I asked David Abrahams regarding the patch. For the records:
With regards to this posting:
http://mail.python.org/pipermail/c++-sig/2005-February/008541.html
I've applied the patch locally. All the tests run fine. Can you think of any negative side-effects of applying the patch?
Well, I think it's probably benign.
It might mean that BPL function objects get bigger. Also, if something out there checks to see whether it's handling a PyCFunction and then manipulates the guts of the object on that basis, it might get unexpected results. I can't think of any other problems, but it's possible they're lurking. If I wanted to really know I'd have to go review what happens when you subclass a built-in type that way... and it's probably not documented, so that would mean reading the Python source.
We need a volunteer to do this research and to figure out what is wrong when using Python 2.3 and 2.4.
Also, for functions with overloads, only one of the docstrings is printed out. It looks like
void function::add_to_namespace( object const& name_space, char const* name_, object const& attribute, char const* doc)
just replaces the old doc string instead of appending the new one to the old.
Another case for a volunteer... Sorry that I cannot be more helpful. Cheers, Ralf __________________________________ Do you Yahoo!? Read only the mail you want - Yahoo! Mail SpamGuard. http://promotions.yahoo.com/new_mail
On Thu, 10 Feb 2005, Ralf W. Grosse-Kunstleve wrote:
--- Nick Rasmussen <nick@ilm.com> wrote:
I'm running into a problem with functions at the module level not showing up in pydoc. The problem is that the type of the function object in the module is:
type(testfoo.foo) <type 'Boost.Python.function'>
Which isn't a sublcass of <type 'builtin_function_or_method'>
When pydoc walks the module it only documents routines that are either builtin (according to inspect.isbuiltin) or are objects whose module can be deduced via inspect.getmodule.
I modified function.cpp with:
--- libs/python/src/object/function.cpp.orig 2005-02-07 17:11:52.000066304 -0800 +++ libs/python/src/object/function.cpp 2005-02-07 16:55:24.000120507 -0800 @@ -608,7 +608,7 @@ 0, /* tp_methods */ 0, // func_memberlist, /* tp_members */ function_getsetlist, /* tp_getset */ - 0, /* tp_base */ + &PyCFunction_Type, /* tp_base */ 0, /* tp_dict */ function_descr_get, /* tp_descr_get */ 0, /* tp_descr_set */
and it seemed to fix it. Is this right, or is there a different way that functions should get exposed such that pydoc can understand them?
I applied this patch locally and tested it in 14 different configurations. It appears to work OK with Python 2.2, but leads to platform-specific fatal errors with Python 2.3 and 2.4 (Abort, Bus error, C assertions in Python, or plain wrong results). Here is the full damage:
http://cci.lbl.gov/cctbx_build/show_error_summary.cgi?build_tag=2005_02_10_1...
(This page will go away after a few days.)
Using the exact same sources, but without the patch above, eliminates all errors.
Thanks for testing it out. It's been working fine for me here (suse9.1 amd64 with python2.3), but I'll see if I can figure out what the right fix is. I took a quick look at PyCFunction_Type, and it's tp_flags are set to: Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ Two things make me suspicious: absent is Py_TPFLAGS_BASETYPE, and I'm not sure if subclasses of gc-aware classes need to support garbage collection as well. I'll run it by our local python expert and see what he has to say. -nick
On Thu, 10 Feb 2005, Nick Rasmussen wrote:
Thanks for testing it out. It's been working fine for me here (suse9.1 amd64 with python2.3), but I'll see if I can figure out what the right fix is.
I took a quick look at PyCFunction_Type, and it's tp_flags are set to:
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
Two things make me suspicious: absent is Py_TPFLAGS_BASETYPE, and I'm not sure if subclasses of gc-aware classes need to support garbage collection as well.
I'll run it by our local python expert and see what he has to say.
I ran this question past python-dev, and they said that one way to do this is to have PyCFunction_Type as the __class__ attribute: http://thread.gmane.org/gmane.comp.python.devel/66557 I'm now running boost::python with the following patch applied (docstring concatenation and the __class__ attribute), and this seems to be working for me. -nick --- libs/python/src/object/function.cpp.orig Mon Feb 7 17:11:52 2005 +++ libs/python/src/object/function.cpp Wed Feb 16 12:13:38 2005 @@ -472,7 +472,12 @@ if (doc != 0) { object attr_copy(attribute); - attr_copy.attr("__doc__") = doc; + if (PyObject_HasAttrString(attr_copy.ptr(), "__doc__") && attr_copy.attr("__doc__")) { + attr_copy.attr("__doc__") += "\n\n"; + attr_copy.attr("__doc__") += doc; + } + else + attr_copy.attr("__doc__") = doc; } } @@ -566,11 +571,17 @@ else return python::incref(f->name().ptr()); } + + static PyObject* function_get_class(PyObject* op, void*) + { + return python::incref(reinterpret_cast<PyObject *>(&PyCFunction_Type)); + } } static PyGetSetDef function_getsetlist[] = { {"__name__", (getter)function_get_name, 0 }, {"func_name", (getter)function_get_name, 0 }, + {"__class__", (getter)function_get_class, 0 }, {"__doc__", (getter)function_get_doc, (setter)function_set_doc}, {"func_doc", (getter)function_get_doc, (setter)function_set_doc}, {NULL} /* Sentinel */
On Wed, 2005-02-16 at 15:26 -0800, Nick Rasmussen wrote:
On Thu, 10 Feb 2005, Nick Rasmussen wrote:
Thanks for testing it out. It's been working fine for me here (suse9.1 amd64 with python2.3), but I'll see if I can figure out what the right fix is.
I took a quick look at PyCFunction_Type, and it's tp_flags are set to:
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
Two things make me suspicious: absent is Py_TPFLAGS_BASETYPE, and I'm not sure if subclasses of gc-aware classes need to support garbage collection as well.
I'll run it by our local python expert and see what he has to say.
I ran this question past python-dev, and they said that one way to do this is to have PyCFunction_Type as the __class__ attribute:
http://thread.gmane.org/gmane.comp.python.devel/66557
I'm now running boost::python with the following patch applied (docstring concatenation and the __class__ attribute), and this seems to be working for me.
I'm more than a little leery of changing the __class__ attribute, since all instances of Boost.Python.function would be lying about what they are. If x and f.x are Boost.Python.function objects with docstrings, than inspect.getdoc(x) and inspect.getdoc(f.x) work. inspect.ismethoddescriptor(x) and inspect.ismethod(f.x) also return True. Where does pydoc test whether or not to document something as a function? -Jonathan
On Wed, 16 Feb 2005, Jonathan Brandmeyer wrote:
I'm more than a little leery of changing the __class__ attribute, since all instances of Boost.Python.function would be lying about what they are.
If x and f.x are Boost.Python.function objects with docstrings, than inspect.getdoc(x) and inspect.getdoc(f.x) work. inspect.ismethoddescriptor(x) and inspect.ismethod(f.x) also return True.
Where does pydoc test whether or not to document something as a function?
-Jonathan
for python2.3 it's at pydoc.py:559 for key, value in inspect.getmembers(object, inspect.isroutine): if inspect.isbuiltin(value) or inspect.getmodule(value) is object: if visiblename(key): funcs.append((key, value)) fdict[key] = '#-' + key if inspect.isfunction(value): fdict[value] = fdict[key] since inspect.isbuiltin(value) returns false and getmodule(value) returns None for non-python functions. My original solution was changing: if inspect.isbuiltin(value) or inspect.getmodule(value) is object: to: tmp = inspect.getmodule(value) if inspect.isbuiltin(value) or not tmp or tmp is object: in order to just document all entries in the module that have unknown provenance, but I still worry about what else may depend on isbuiltin. The other mechaism that I thought of was to do something like: import inspect inspect.isbuiltin = (some boost-aware version) When a boost module is imported. -nick
On Thu, 2005-02-17 at 10:06 -0800, Nick Rasmussen wrote:
On Wed, 16 Feb 2005, Jonathan Brandmeyer wrote:
Where does pydoc test whether or not to document something as a function?
-Jonathan
for python2.3 it's at pydoc.py:559
for key, value in inspect.getmembers(object, inspect.isroutine): if inspect.isbuiltin(value) or inspect.getmodule(value) is object: if visiblename(key): funcs.append((key, value)) fdict[key] = '#-' + key if inspect.isfunction(value): fdict[value] = fdict[key]
All of the routines have been filtered off already, so do you know why they are then filtering for only builtin types? If I comment out the "if inspect.isbuiltin(value)..." line, it appears to work just fine. -Jonathan
On Thu, 17 Feb 2005, Jonathan Brandmeyer wrote:
On Thu, 2005-02-17 at 10:06 -0800, Nick Rasmussen wrote:
On Wed, 16 Feb 2005, Jonathan Brandmeyer wrote:
Where does pydoc test whether or not to document something as a function?
-Jonathan
for python2.3 it's at pydoc.py:559
for key, value in inspect.getmembers(object, inspect.isroutine): if inspect.isbuiltin(value) or inspect.getmodule(value) is object: if visiblename(key): funcs.append((key, value)) fdict[key] = '#-' + key if inspect.isfunction(value): fdict[value] = fdict[key]
All of the routines have been filtered off already, so do you know why they are then filtering for only builtin types? If I comment out the "if inspect.isbuiltin(value)..." line, it appears to work just fine.
-Jonathan
They have it there so pydoc doesn't document functions pulled in from other modules, there's just special handling for builtins as there's no way to recover source modules from builtin functions, so it documents all builtin functions. -nick
On Thu, 2005-02-10 at 18:01 -0800, Ralf W. Grosse-Kunstleve wrote:
Also, for functions with overloads, only one of the docstrings is printed out. It looks like
void function::add_to_namespace( object const& name_space, char const* name_, object const& attribute, char const* doc)
just replaces the old doc string instead of appending the new one to the old.
Another case for a volunteer...
Sorry that I cannot be more helpful.
I have a patch that implements this feature, with the effect that sequential docstrings will have a single blank line separating them. Here is the documentation change that describes its behavior: (from doc/v2/class.html, Effects of the "docstring" argument to class::def) Value will be bound to the __doc__ attribute of the resulting method overload. If an earlier overload supplied a docstring, two newline characters and the new docstring are appended to it. I have attached the patch for comment. If there are no objections I can commit it tomorrow afternoon. -Jonathan
On Thu, 10 Feb 2005, Jonathan Brandmeyer wrote:
On Thu, 2005-02-10 at 18:01 -0800, Ralf W. Grosse-Kunstleve wrote:
Also, for functions with overloads, only one of the docstrings is printed out. It looks like
void function::add_to_namespace( object const& name_space, char const* name_, object const& attribute, char const* doc)
just replaces the old doc string instead of appending the new one to the old.
Another case for a volunteer...
Sorry that I cannot be more helpful.
I have a patch that implements this feature, with the effect that sequential docstrings will have a single blank line separating them. Here is the documentation change that describes its behavior:
(from doc/v2/class.html, Effects of the "docstring" argument to class::def) Value will be bound to the __doc__ attribute of the resulting method overload. If an earlier overload supplied a docstring, two newline characters and the new docstring are appended to it.
I have attached the patch for comment. If there are no objections I can commit it tomorrow afternoon.
Exactly what I was looking for. Thanks. -nick
participants (3)
-
Jonathan Brandmeyer -
Nick Rasmussen -
Ralf W. Grosse-Kunstleve