Extending documentation to c code
Hi All,
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters ---------- We don't need no stinkin' parameters.
Notes ----- The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
Chuck
On Mon, May 24, 2010 at 11:01 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hi All,
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters
We don't need no stinkin' parameters.
Notes
The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
Chuck
IMO it would be necessary to make such doc have the same status w.r.t. the Wiki as the Python source; how much tweaking of pydocweb would that require (Pauli is already over-committed in that regard; Joe, Perry, and I are taking steps to try to alleviate this, but nothing is close to materializing yet). I know that as far as Joe and I are concerned, getting pydocweb to support a dual review process is a much higher, longer-standing priority.
Also, quoting from the docstring standard: "An optional section for examples...while optional, this section is very strongly encouraged." (Personally, I think this section should be required, not optional, for functions, and methods which require their own docstrings.) But requiring docwriters to supply working (i.e., compilable, linkable, runable) c code examples (which would appear to be necessary because the coders appear to be loath to provide their docstrings with examples) might be asking too much (since we try to keep the doc writing effort open to persons at least comfortable w/ Python, though not necessarily w/ c).
Unless and until these concerns can be realistically and successfully addressed, I'm a strong "-1".
DG
On Mon, May 24, 2010 at 2:11 PM, David Goldsmith d.l.goldsmith@gmail.comwrote:
On Mon, May 24, 2010 at 11:01 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hi All,
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters
We don't need no stinkin' parameters.
Notes
The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
Chuck
IMO it would be necessary to make such doc have the same status w.r.t. the Wiki as the Python source; how much tweaking of pydocweb would that require (Pauli is already over-committed in that regard; Joe, Perry, and I are taking steps to try to alleviate this, but nothing is close to materializing yet). I know that as far as Joe and I are concerned, getting pydocweb to support a dual review process is a much higher, longer-standing priority.
Also, quoting from the docstring standard: "An optional section for examples...while optional, this section is very strongly encouraged." (Personally, I think this section should be required, not optional, for functions, and methods which require their own docstrings.) But requiring docwriters to supply working (i.e., compilable, linkable, runable) c code examples (which would appear to be necessary because the coders appear to be loath to provide their docstrings with examples) might be asking too much (since we try to keep the doc writing effort open to persons at least comfortable w/ Python, though not necessarily w/ c).
Unless and until these concerns can be realistically and successfully addressed, I'm a strong "-1".
I'm not interested in having this part of the standard user documentation since the c functions are mostly invisible to the user. What I want is documentation for maintainers/developers of the c code. The c code is essentially undocumented and that makes it difficult to work with, especially for new people. At one time in the past I suggested using doxygen but that didn't seem to arouse much interest. I've also tried generating a call graph but only managed to crash the system... Anyway, it needs to be done at some point and I'm looking for suggestions.
Chuck
On Mon, May 24, 2010 at 4:59 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Mon, May 24, 2010 at 2:11 PM, David Goldsmith d.l.goldsmith@gmail.comwrote:
On Mon, May 24, 2010 at 11:01 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hi All,
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters
We don't need no stinkin' parameters.
Notes
The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
Chuck
IMO it would be necessary to make such doc have the same status w.r.t. the Wiki as the Python source; how much tweaking of pydocweb would that require (Pauli is already over-committed in that regard; Joe, Perry, and I are taking steps to try to alleviate this, but nothing is close to materializing yet). I know that as far as Joe and I are concerned, getting pydocweb to support a dual review process is a much higher, longer-standing priority.
Also, quoting from the docstring standard: "An optional section for examples...while optional, this section is very strongly encouraged." (Personally, I think this section should be required, not optional, for functions, and methods which require their own docstrings.) But requiring docwriters to supply working (i.e., compilable, linkable, runable) c code examples (which would appear to be necessary because the coders appear to be loath to provide their docstrings with examples) might be asking too much (since we try to keep the doc writing effort open to persons at least comfortable w/ Python, though not necessarily w/ c).
Unless and until these concerns can be realistically and successfully addressed, I'm a strong "-1".
I'm not interested in having this part of the standard user documentation since the c functions are mostly invisible to the user. What I want is documentation for maintainers/developers of the c code. The c code is essentially undocumented and that makes it difficult to work with, especially for new people. At one time in the past I suggested using doxygen but that didn't seem to arouse much interest. I've also tried generating a call graph but only managed to crash the system... Anyway, it needs to be done at some point and I'm looking for suggestions.
Chuck
Not checking in un-/poorly documented new code would be a good start.
DG
On Mon, May 24, 2010 at 6:17 PM, David Goldsmith d.l.goldsmith@gmail.comwrote:
On Mon, May 24, 2010 at 4:59 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Mon, May 24, 2010 at 2:11 PM, David Goldsmith <d.l.goldsmith@gmail.com
wrote:
On Mon, May 24, 2010 at 11:01 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hi All,
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters
We don't need no stinkin' parameters.
Notes
The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
Chuck
IMO it would be necessary to make such doc have the same status w.r.t. the Wiki as the Python source; how much tweaking of pydocweb would that require (Pauli is already over-committed in that regard; Joe, Perry, and I are taking steps to try to alleviate this, but nothing is close to materializing yet). I know that as far as Joe and I are concerned, getting pydocweb to support a dual review process is a much higher, longer-standing priority.
Also, quoting from the docstring standard: "An optional section for examples...while optional, this section is very strongly encouraged." (Personally, I think this section should be required, not optional, for functions, and methods which require their own docstrings.) But requiring docwriters to supply working (i.e., compilable, linkable, runable) c code examples (which would appear to be necessary because the coders appear to be loath to provide their docstrings with examples) might be asking too much (since we try to keep the doc writing effort open to persons at least comfortable w/ Python, though not necessarily w/ c).
Unless and until these concerns can be realistically and successfully addressed, I'm a strong "-1".
I'm not interested in having this part of the standard user documentation since the c functions are mostly invisible to the user. What I want is documentation for maintainers/developers of the c code. The c code is essentially undocumented and that makes it difficult to work with, especially for new people. At one time in the past I suggested using doxygen but that didn't seem to arouse much interest. I've also tried generating a call graph but only managed to crash the system... Anyway, it needs to be done at some point and I'm looking for suggestions.
Chuck
Not checking in un-/poorly documented new code would be a good start.
So exactly how should the c code be documented? There is currently no documentation standard for c code.
Chuck
On Tue, May 25, 2010 at 3:01 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Hi All,
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters
We don't need no stinkin' parameters.
Notes
The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
I have thought about this for quite some time, but it is not easy. Docstrings are useful because of cross references, etc... and documentation for compiled code should contain signature extraction. For those reasons, I think a doc tool would need to parse C, which makes the problem that much harder.
Last time I looked, synopsis was interesting, but it does not seem to have caught up. Synopsis was interesting because it was modular, scriptable in python, and supported rest as a markup language within C code. OTOH, I hope that clang will change the game here - it gives a modular, robust C (and soon C++) parser, and having a documentation tool written from that is just a question of time I think.
Maybe as a first step, something that could extract function signature would be enough, and writing this should not take too much time (Sebastien B wrote something which could be a start, to autogenerate cython code from header:http://bitbucket.org/binet/cylon).
David
On Mon, May 24, 2010 at 8:06 PM, David Cournapeau cournape@gmail.comwrote:
On Tue, May 25, 2010 at 3:01 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Hi All,
I'm wondering if we could extend the current documentation format to the
c
source code. The string blocks would be implemented something like
/**NpyDoc """The Answer.
Answer the Ultimate Question of Life, the Universe, and Everything.
Parameters
We don't need no stinkin' parameters.
Notes
The run time of this routine may be excessive.
""" */ int answer_ultimate_question(void) { return 42; }
and the source scanned to generate the usual documentation. Thoughts?
I have thought about this for quite some time, but it is not easy. Docstrings are useful because of cross references, etc... and documentation for compiled code should contain signature extraction. For those reasons, I think a doc tool would need to parse C, which makes the problem that much harder.
Last time I looked, synopsis was interesting, but it does not seem to have caught up. Synopsis was interesting because it was modular, scriptable in python, and supported rest as a markup language within C code. OTOH, I hope that clang will change the game here - it gives a modular, robust C (and soon C++) parser, and having a documentation tool written from that is just a question of time I think.
Maybe as a first step, something that could extract function signature would be enough, and writing this should not take too much time (Sebastien B wrote something which could be a start, to autogenerate cython code from header:http://bitbucket.org/binet/cylon).
David
This does sound promising/a good first step. But it doesn't really answer Charles' question about a standard (which would be useful to have to help guide doc editor design). My proposal is that we start w/ what we have - the standard for our Python code - and figure out what makes sense to keep, add, change, and throw out. If we don't yet have an SEP process, perhaps this need could serve as a first test case; obviously, if we already do have an SEP, then we should follow that.
DG
On Tue, May 25, 2010 at 2:09 PM, David Goldsmith d.l.goldsmith@gmail.com wrote:
This does sound promising/a good first step. But it doesn't really answer Charles' question about a standard (which would be useful to have to help guide doc editor design).
it does - I looked into synopsis because we could use rest, and I don't think anyone wants to go the doxygen route. Just putting rest comments into sources is not useful (since just *extracting* them is non trivial for C/C++). I think the documentation project taught us that being able to build a decent looking document is required for people to actually document things.
Also, I may have not been clear, but when I said I thought about it, I meant I have tried it and it did not work after one hour of two of tinkering (then I realized that you need to parse C to do anything useful).
David
it does - I looked into synopsis because we could use rest, and I don't think anyone wants to go the doxygen route.
I am curious as to why doxygen isn't a viable option. While I don't have experience with the other suggestions, I have used doxygen in a few of my personall projects and have been quite happy with it serving as internal documentation. doxygen can still produce bare-bones documentation, call-graphs and cross-references from C/C++ code even if there are no special comments for it to parse.
While doxygen may not be perfect, I think it does well enough to produce good documentation for developers to use.
Just my 2 cents.
Ben Root
On Tue, May 25, 2010 at 11:18 PM, Benjamin Root ben.root@ou.edu wrote:
it does - I looked into synopsis because we could use rest, and I don't think anyone wants to go the doxygen route.
I am curious as to why doxygen isn't a viable option. While I don't have experience with the other suggestions, I have used doxygen in a few of my personall projects and have been quite happy with it serving as internal documentation.
It is yet another format to use inside C sources (I don't think doxygen supports rest), and I would rather have something that is similar, ideally integrated into sphinx. It also generates rather ugly doc by default,
David
Wed, 26 May 2010 06:57:27 +0900, David Cournapeau wrote: [clip: doxygen]
It is yet another format to use inside C sources (I don't think doxygen supports rest), and I would rather have something that is similar, ideally integrated into sphinx. It also generates rather ugly doc by default,
Anyway, we can probably nevertheless just agree on a readable plain-text/ rst format, and then just use doxygen to generate the docs, as a band-aid.
http://github.com/pv/numpycdoc
On Wed, May 26, 2010 at 2:59 AM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 06:57:27 +0900, David Cournapeau wrote: [clip: doxygen]
It is yet another format to use inside C sources (I don't think doxygen supports rest), and I would rather have something that is similar, ideally integrated into sphinx. It also generates rather ugly doc by default,
Anyway, we can probably nevertheless just agree on a readable plain-text/ rst format, and then just use doxygen to generate the docs, as a band-aid.
Neat. I didn't quite see the how how you connected the rst documentation and doxygen.
Chuck
Wed, 26 May 2010 07:15:08 -0600, Charles R Harris wrote:
On Wed, May 26, 2010 at 2:59 AM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 06:57:27 +0900, David Cournapeau wrote: [clip: doxygen]
It is yet another format to use inside C sources (I don't think doxygen supports rest), and I would rather have something that is similar, ideally integrated into sphinx. It also generates rather ugly doc by default,
Anyway, we can probably nevertheless just agree on a readable plain-text/ rst format, and then just use doxygen to generate the docs, as a band-aid.
Neat. I didn't quite see the how how you connected the rst documentation and doxygen.
I didn't :)
But I just did: doing this it was actually a 10 min job since Doxygen accepts HTML -- now it parses the comments as RST and renders it properly as HTML in the Doxygen output. Of course getting links etc. to work would require more effort, but that's left as an exercise for someone else to finish.
Pauli
On Wed, May 26, 2010 at 8:14 AM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 07:15:08 -0600, Charles R Harris wrote:
On Wed, May 26, 2010 at 2:59 AM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 06:57:27 +0900, David Cournapeau wrote: [clip: doxygen]
It is yet another format to use inside C sources (I don't think doxygen supports rest), and I would rather have something that is similar, ideally integrated into sphinx. It also generates rather ugly doc by default,
Anyway, we can probably nevertheless just agree on a readable plain-text/ rst format, and then just use doxygen to generate the docs, as a band-aid.
Neat. I didn't quite see the how how you connected the rst documentation and doxygen.
I didn't :)
But I just did: doing this it was actually a 10 min job since Doxygen accepts HTML -- now it parses the comments as RST and renders it properly as HTML in the Doxygen output. Of course getting links etc. to work would require more effort, but that's left as an exercise for someone else to finish.
Why don't you go ahead and merge this. If someone wants to substitute something else for doxygen at some point, then that is still open, meanwhile we can get started on writing some cdocs. In particular, it would be nice if the folks doing the code refactoring also documented any new functions. We can also put together a numpycdoc standard to go with it. I think your idea of combining the standard numpy doc format with the usual c code comment style is the way to go.
Chuck
On Thu, May 27, 2010 at 9:18 AM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Wed, May 26, 2010 at 8:14 AM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 07:15:08 -0600, Charles R Harris wrote:
On Wed, May 26, 2010 at 2:59 AM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 06:57:27 +0900, David Cournapeau wrote: [clip: doxygen]
It is yet another format to use inside C sources (I don't think doxygen supports rest), and I would rather have something that is similar, ideally integrated into sphinx. It also generates rather ugly doc by default,
Anyway, we can probably nevertheless just agree on a readable plain-text/ rst format, and then just use doxygen to generate the docs, as a band-aid.
Neat. I didn't quite see the how how you connected the rst documentation and doxygen.
I didn't :)
But I just did: doing this it was actually a 10 min job since Doxygen accepts HTML -- now it parses the comments as RST and renders it properly as HTML in the Doxygen output. Of course getting links etc. to work would require more effort, but that's left as an exercise for someone else to finish.
Why don't you go ahead and merge this. If someone wants to substitute something else for doxygen at some point, then that is still open, meanwhile we can get started on writing some cdocs. In particular, it would be nice if the folks doing the code refactoring also documented any new functions.
Thanks for being a voice for change! :-)
We can also put together a numpycdoc standard to go with it. I think your idea of combining the standard numpy doc format with the usual c code comment style is the way to go.
And certainly at this early stage something is better than nothing.
DG
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Excerpts from David Cournapeau's message of 2010-05-25 05:06:09 +0200:
On Tue, May 25, 2010 at 3:01 AM, Charles R Harris
[snip]
Maybe as a first step, something that could extract function signature would be enough, and writing this should not take too much time (Sebastien B wrote something which could be a start, to autogenerate cython code from header:http://bitbucket.org/binet/cylon).
note that llvm/clang is versatile enough to easily provide indices into the source code, which of course includes the comments... I am actually working on improving the python bindings to clang (which are already quite useful for this thread's topic as they are used for code completing C/C++ code - but are not yet complete enough for providing complete function signatures)
cheers, sebastien.
On Tue, May 25, 2010 at 4:04 PM, Sebastien Binet seb.binet@gmail.com wrote:
Excerpts from David Cournapeau's message of 2010-05-25 05:06:09 +0200:
On Tue, May 25, 2010 at 3:01 AM, Charles R Harris
[snip]
Maybe as a first step, something that could extract function signature would be enough, and writing this should not take too much time (Sebastien B wrote something which could be a start, to autogenerate cython code from header:http://bitbucket.org/binet/cylon).
note that llvm/clang is versatile enough to easily provide indices into the source code, which of course includes the comments... I am actually working on improving the python bindings to clang
Ah, nice - where do you put your work ? It seems that llvm-py does not have recent commits,
David
Excerpts from David Cournapeau's message of 2010-05-25 09:51:36 +0200:
On Tue, May 25, 2010 at 4:04 PM, Sebastien Binet seb.binet@gmail.com wrote:
Excerpts from David Cournapeau's message of 2010-05-25 05:06:09 +0200:
On Tue, May 25, 2010 at 3:01 AM, Charles R Harris
[snip]
Maybe as a first step, something that could extract function signature would be enough, and writing this should not take too much time (Sebastien B wrote something which could be a start, to autogenerate cython code from header:http://bitbucket.org/binet/cylon).
note that llvm/clang is versatile enough to easily provide indices into the source code, which of course includes the comments... I am actually working on improving the python bindings to clang
Ah, nice - where do you put your work ? It seems that llvm-py does not have recent commits,
I am talking about the clang python bindings, not the llvm ones :)
I am pushing stuff over there: http://llvm.org/viewvc/llvm-project/cfe/trunk/bindings/python/
FYI, my cylon project is waiting for this last patch to be applied: http://lists.cs.uiuc.edu/pipermail/cfe-dev/2010-May/009091.html
then world domination...
cheers, sebastien.
On 25/05/10 09:04, Sebastien Binet wrote:
note that llvm/clang is versatile enough to easily provide indices into the source code, which of course includes the comments... I am actually working on improving the python bindings to clang (which are already quite useful for this thread's topic as they are used for code completing C/C++ code - but are not yet complete enough for providing complete function signatures)
Have you seen http://code.google.com/p/pycparser/ ?
It is a pure python implementation with small or no external dependencies. I was thinking about using it for writing a simple cython interfaces generator. I do not know if it supports extracting comments, but I think it would be simple to extend it this way.
Cheers,
Excerpts from Daniele Nicolodi's message of 2010-05-25 11:37:50 +0200:
On 25/05/10 09:04, Sebastien Binet wrote:
note that llvm/clang is versatile enough to easily provide indices into the source code, which of course includes the comments... I am actually working on improving the python bindings to clang (which are already quite useful for this thread's topic as they are used for code completing C/C++ code - but are not yet complete enough for providing complete function signatures)
Have you seen http://code.google.com/p/pycparser/ ?
yes, and I have been using it to teach myself PyParsing. the problem is that I need to be able to parse C++, not just C.
Parsing (correctly) C++ is a full blown project and I prefer to rely on the work of others :)
cheers, sebastien.
Hi,
ma, 2010-05-24 kello 12:01 -0600, Charles R Harris kirjoitti:
I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like
[clip]
I'd perhaps stick closer to C conventions and use something like
/** * Spam foo out of the parrots. * * Parameters * ---------- * a * Amount of spam * b * Number of parrots */ int foo(int a, int b) { }
Using some JavaDoc-type syntax might also be possible, although I personally find it rather ugly. Also, parsing C might not be too difficult, as projects aimed doing that in Python exist, for instance http://code.google.com/p/pycparser/
participants (7)
-
Benjamin Root -
Charles R Harris -
Daniele Nicolodi -
David Cournapeau -
David Goldsmith -
Pauli Virtanen -
Sebastien Binet