Hi Guys, first post on the list, dont be shy ;)
In Blender3D were doing an API refactor and Im looking at better ways to write the Python C/API documentation.
At the moment we use epydoc, Its all Iv used so I don't have much to compare it to.
If you havnt used it, its basically a way to add formatting to your normal .py docstrings so they can be converted into HTML or PDF.
This works well with python only projects but is not somthing you can do in C without some manual docstring extraction from the C/API.
for blender3d we have C doctrings as well as python files that just contain functions with docstrings (no actual code) for epydoc to write them to html.
The C/API docstings often end up being brief and not that helpful, since the epydocs are what most people use so more effort is put there.
Here are the current epydocs. http://www.blender.org/documentation/244PythonDoc/index.html
Ideally Id like to only maintain 1 set of docs... so possibilities are them as HTML still)
- dont write epydocs, only use docstrings (we need a way we can browse
dont write C API/docstrings, just have empty strings where all the docstrings would go.
include epydoc formatting in C API/docstrings and pass these docstrings to epydoc for it to generate HTML.
Iv got 3) working as a testcase, the message attached details how this works with the pro's and con's of adding epydoc tags into C/API docstrings.
However Id be interested to know how other projects deal with the issue of maintaining docstrings and online docs.
Cheers
grrr my attachment was stripped
here it is inline..
Campbell Barton wrote:
Revision: 11145
http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=11145
Author: campbellbarton Date: 2007-07-02 14:22:30 +0200 (Mon, 02 Jul 2007)
Log Message:
Testing a way to have python docs in Blenders PyAPI docstrings. rather then maintaining 2 sets of docs which is what we have done for a long time.
Works for Group and Camera now, here are the results
http://members.optusnet.com.au/cjbarton/bpy_docs/camera.Camera-class.html
http://members.optusnet.com.au/cjbarton/bpy_docs/group.Group-class.html
This means the C docstrings need to contain epydoc formatting (humanly readable) but need to be changed in the C files. though editing be done in py files and the final version copied back into the .c file.
_epy_write_docstrings.py - extracts epydocs and examples into their own py files. _epy_docgen.sh - runs _epy_write_docstrings.py in background mode and then epydoc to make the html docs.
advantage is less work with duplicate docs, we can automate running all examples within the docs to make sure they work.
Hi, just committed an experimental C docstring/epydoc metrhod of maintaining our python API docs. its worth discussing the options for docs.
this is not ideal (epydocs in C are a bit of a kludge) but it as some good points and IMHO is better then what we have been doing.
pros:
- 1 set of docs, not 2
- built-in python methods are added, like getitem or __hash__
- we can do preprocessing since the epydocs are made from a script...
- better C docstrings (albeit with a bit of odd epydoc tags) - people do use the C docstrings sometimes and at the moment they are not well checked - some have mistakes or are quite uninformative.
- extracts examples to py files so we can automate running them all.
- its more flexible, we could for instance have all examples in py files and include them when the blender puts the docs together, this would also not bloat blender too much with really big docstrings,
- converting existing epydocs into C/epy/docstrings is not hard.
cons:
editing large docstrings with \n\ at the end of everyline is not nice.
nonstandard, uses a custom script (~200 lines) to extract and write docstrings
workflow to writing docs is longer since you need to edit the c file, compile blender and run the script. If there is a missing tab youd not want to do all that again. ... Realisticly its not so bad since you can edit the docs that blender spits out and then update the C file once your done, but the process is still longer for writing epydocs though arguably better then editing docs in 2 places.
one more hoop to jump through when writing docs - you have to learn our way of adding epydocs in C.
- Even though Id rather not make it harder for new people to come in and write docs, so far python scripters have not come up and offered to write docs, instead its been the C/API authors - who are alredy familiar with the C files, it also seems that very few people besides the API devs even use epydoc or have it installed so Im not too worried about
Campbell Barton wrote: this.
Have a look at Camera.c to see what the epy/docstrings look like.
If we deciede this is acceptable I can continue and move other finished areas of the new API to epy/docstrings.
Bf-python mailing list Bf-python@blender.org http://lists.blender.org/mailman/listinfo/bf-python
On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton wrote:
However Id be interested to know how other projects deal with the issue of maintaining docstrings and online docs.
I maintain separate docstrings and online docs for one simple reason: The online docs can be more extensive than docstrings. Docstrings are for brief memory-joggers. Online documentation can contain examples and generally more detail that's not appropriate for docstrings.
Just my two cents,
-- Carsten Haese http://informixdb.sourceforge.net
--- Carsten Haese carsten@uniqsys.com wrote:
However Id be interested to know how other
On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton wrote: projects deal with the issue
of maintaining docstrings and online docs.
I maintain separate docstrings and online docs for one simple reason: The online docs can be more extensive than docstrings. Docstrings are for brief memory-joggers. Online documentation can contain examples and generally more detail that's not appropriate for docstrings.
Can you elaborate on this a bit. What kind of tools do you use to generate/maintain your docs?
Martin
- Blender developer
Finding fabulous fares is fun.
Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains.
http://farechase.yahoo.com/promo-generic-14795097
On Mon, 2007-07-02 at 08:46 -0700, Martin Poirier wrote:
--- Carsten Haese carsten@uniqsys.com wrote:
However Id be interested to know how other
On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton wrote: projects deal with the issue
of maintaining docstrings and online docs.
I maintain separate docstrings and online docs for one simple reason: The online docs can be more extensive than docstrings. Docstrings are for brief memory-joggers. Online documentation can contain examples and generally more detail that's not appropriate for docstrings.
Can you elaborate on this a bit. What kind of tools do you use to generate/maintain your docs?
My needs are probably significantly different, i.e. smaller, than yours. My documentation is all in one moderately sized Restructured Text file. If you were expecting automatically generated hyperlinked API docs, you'll be disappointed.
I was merely making the point that in my opinion it's not a problem to have two sets of documentation because docstrings and online documentation serve different purposes. Your opinion may vary.
-- Carsten Haese http://informixdb.sourceforge.net
Martin Poirier wrote:
--- Carsten Haese carsten@uniqsys.com wrote:
However Id be interested to know how other
On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton wrote: projects deal with the issue
of maintaining docstrings and online docs. I maintain separate docstrings and online docs for one simple reason: The online docs can be more extensive than docstrings. Docstrings are for brief memory-joggers. Online documentation can contain examples and generally more detail that's not appropriate for docstrings.
Can you elaborate on this a bit. What kind of tools do you use to generate/maintain your docs?
Martin
- Blender developer
Its just one script that runs inside blender and writes its docstrings to py files epydoc can use.
At the moment it just takes a PyType, and writes a file with its methods and attributes. adding indenting and method names to the epydocs.
I wrote this last night and it only works with the Group and Camera types, so if we choose to keep using this method we can extend and modify as needed.
Barry pointed out that docs could be generated with epydoc inspecting the C api directly, Id be interested to know if you had much epydoc spesific stuff in these docstrings, though i was aware this was possible in some cases, you end up with nicer docs if they include formatting tags.
Carsten, agree Examples within the C docstrings is bloating them too much, Id like to add these in when generating the py files with some form of #include that inserts an existing Py file.
Thanks for your feedback - I may be over reacting about maintaining 2 sets of docstrings because when I first started with Blenders Python api there would be for instance.. mesh.getName() mesh.setName() mesh.name
And each needed a docstring and epydoc, so we had to have 6 different bits of documentation for each setting. We have decided not to use get/set style anymore so its only 2 - which may be a necessary annoyance and not that bad in practice.
This is the biggest PyType we have, to get an idea of what Im talking about...
file:///cvs_blender/blender/source/blender/python/api2_2x/doc/BPY_API/Object.Object-class.html
Campbell Barton wrote:
This is the biggest PyType we have, to get an idea of what Im talking about...
file:///cvs_blender/blender/source/blender/python/api2_2x/doc/BPY_API/Object.Object-class.html bugger... http://members.optusnet.com.au/cjbarton/BPY_API/Object.Object-class.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Jul 2, 2007, at 10:14 PM, Campbell Barton wrote:
Barry pointed out that docs could be generated with epydoc inspecting the C api directly, Id be interested to know if you had much epydoc spesific stuff in
these docstrings, though i was aware this was possible in some cases, you
end up with nicer docs if they include formatting tags.
I'm not sure if you were asking me this question, but I'll answer
anyway :). Yes! We put epydoc markup in our C API docstrings and
epydoc almost all the right things. As I mentioned there were a
couple of quirks due to import weirdness or inheritance but 98% of
the docstrings worked great.
I should note one other bit of funny business, Writing multiline
docstrings in C is a royal PITA of course, especially if you ever
have to re-fill a paragraph. Ain't no TQS's there! The one thing
that bit us fairly often was a limit on the number of characters
inside a string literal that Microsoft's compiler imposed on us. Our
code was portable between gcc and MSVC (Studio Enterprise .NET
Ultramarine Whizzy XP Light and Happiness Edition 2003 (tm)(r)(c)
(blah)). gcc never had a problem but with MSVC we couldn't use the \n
\ trick after 1024 characters. We'd have to close the literal and
start again. There was probably some magic config buried under
30,000 menu options, but we never found it.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin)
iQCVAwUBRorgpHEjvBPtnXfVAQLWnQQAhU+JslzLZYLeXEMl5/cSjugAgU1YJNdx 2xzp4qK1Oi/RYgRReoCBEvbNA/9+h3OARda+Uj9GiBhTEZziC+M3uqTqEFltFIt2 zVXKF9iHB4jNwiuu0L5e/hA3CxkFOKFCypt8JLzi9nJzOy3iTQv+KYqFfM8lMZvu hzheqIubQ6I= =WayZ -----END PGP SIGNATURE-----
Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Jul 2, 2007, at 10:14 PM, Campbell Barton wrote:
Barry pointed out that docs could be generated with epydoc inspecting the C api directly, Id be interested to know if you had much epydoc spesific stuff in these docstrings, though i was aware this was possible in some cases, you end up with nicer docs if they include formatting tags.
I'm not sure if you were asking me this question, but I'll answer anyway :). Yes! We put epydoc markup in our C API docstrings and epydoc almost all the right things. As I mentioned there were a couple of quirks due to import weirdness or inheritance but 98% of the docstrings worked great.
I should note one other bit of funny business, Writing multiline docstrings in C is a royal PITA of course, especially if you ever have to re-fill a paragraph. Ain't no TQS's there! The one thing that bit us fairly often was a limit on the number of characters inside a string literal that Microsoft's compiler imposed on us. Our code was portable between gcc and MSVC (Studio Enterprise .NET Ultramarine Whizzy XP Light and Happiness Edition 2003 (tm)(r)(c)(blah)). gcc never had a problem but with MSVC we couldn't use the \n\ trick after 1024 characters. We'd have to close the literal and start again. There was probably some magic config buried under 30,000 menu options, but we never found it.
- -Barry
For people who dont mind docstrings + online docs, how big are your projects?
I just did a quick check on Blender/Python epydocs and there are 1297 functions and 868 attributes and 73 classes == ~2238 doc strings.
(got my numbers from running this in the epydoc dir) cat *.py | grep "def " | grep ":" | wc -l cat *.py | grep "class " | grep ":" | wc -l cat *.py | grep "@ivar" | wc -l
Barry: Looks like this is a workable solution that Ill go ahead with.
I was hoping there was some kind of triple quoting in C :/ \n\ at the end of every line is a PITA, Though with search replace it can be added after writing..
- Thanks for the hint regarding 1024 chars.
EpyDocs as C strings aren't so bad, heres an example (for people other then Barry) of how it looks for getseters.
{"orthographic",
(getter)Camera_getOrtho, (setter)Camera_setOrtho, "\@ivar orthographic: When true this camera will render with no
perspective.\n
@type orthographic: bool",
NULL},
{"ipo",
(getter)Camera_getIpo, (setter)Camera_setIpo, "
@ivar ipo: The ipo linked to this camera data object.\n
Assign None to clear the ipo from this camera.\n
@type ipo: Blender Ipo",
NULL},
adding some special tag like ((reference some_example.py)) that can be inserted when generating the Py files is no big deal and would read well in the docstrings too.
Thanks again.
- Cam
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Jul 4, 2007, at 12:00 PM, Campbell Barton wrote:
For people who dont mind docstrings + online docs, how big are your projects?
Our project was pretty big. I don't have exact numbers any more, but
I'd say on the order of maybe two dozen C implemented classes with
several hundred methods and properties in total.
I just did a quick check on Blender/Python epydocs and there are 1297 functions and 868 attributes and 73 classes == ~2238 doc strings.
I'm not counting stuff implemented in Python, which was probably 2-5
times the size of our C stuff. OTOH, our C classes were the core of
our model so we took great pains to document them accurately.
Barry: Looks like this is a workable solution that Ill go ahead with.
I was hoping there was some kind of triple quoting in C :/ \n\ at the end of every line is a PITA, Though with search
replace it can be added after writing..
- Thanks for the hint regarding 1024 chars.
EpyDocs as C strings aren't so bad, heres an example (for people other then Barry) of how it looks for getseters.
{"orthographic", (getter)Camera_getOrtho, (setter)Camera_setOrtho, "
@ivar orthographic: When true this camera will render with no perspective.\n
@type orthographic: bool", NULL}, {"ipo", (getter)Camera_getIpo, (setter)Camera_setIpo, "
@ivar ipo: The ipo linked to this camera data object.\n
Assign None to clear the ipo from this camera.\n
@type ipo: Blender Ipo", NULL},
Just to relate my own experiences here. I would typically include a
PyDoc_STRVAR() call right before the function's static definition,
instead of including the docstring in the structure definition. I
found that was cleaner and made it easier to update the docstring
when the function was updated. YMMV of course.
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin)
iQCVAwUBRozo0XEjvBPtnXfVAQKkewQAinerU5Jc+jyMWD/6Drkx/3p7t9Z+MfFh 0t3NJt5fbY0VXJRh2R4ghApJpWXXNp1BQ3qy6YnRTb5zNhT+dz6V5iipenLdWyYg ShE5MoHdhEPrEZlQMxgDzJNHBiAYtDelGdyyJEwspgmE6expbt8z1s6QvML2dGgQ u147d9X2geo= =VYpg -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Jul 2, 2007, at 9:30 AM, Campbell Barton wrote:
At the moment we use epydoc, Its all Iv used so I don't have much
to compare it to.If you havnt used it, its basically a way to add formatting to your
normal .py docstrings so they can be converted into HTML or PDF.This works well with python only projects but is not somthing you
can do in C without some manual docstring extraction from the C/API.
This isn't necessarily true. In a previous life, I was doing tons of
C API stuff, essentially building and maintaining a big embedded/
extended Python application. We docstring'd all of our C API and
used epydoc to generate our API documentation. It worked very well,
with a few caveats.
First, epydoc had some problems in earlier versions, for example, the
fact that a class's docstring and its constructor's docstring both
live in the same slot in C. IIRC, epydoc also had some problems with
certain C inheritance style we were using, and a bunch of other minor
problems. AFAIK, these have mostly been addressed with newer epydoc
versions, though I honestly haven't looked at it for 6+ months.
Second, and more importantly, because of the nature of our app, we
had to run epydoc programmatically, inside the application's shell.
I think you're forced to use epydoc's import-based discovery
mechanisms when docstring'ing C code, so your modules have to be
importable. Because we were an embedded application, the package
namespace for our C code wasn't available except when the application
shell was run. Therefore we added a command line option that would
run the epydoc stuff, once Python was initialized and the extension
modules were available.
Other than that, we really didn't have to do much special to
docstring our C functions, provide them in normal Python interactive
interpreter help, and have nicely generated epydoc documentation.
All of our docstrings lived right in C and were attached to the C
functions using the normal Python C API structures.
I wish I could post some examples, but the code was closed-source and
the company no longer exists. ;)
- -Barry
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin)
iQCVAwUBRokyCXEjvBPtnXfVAQIYhwP/R/y9Jq18PaQzOkhDBdjHKsxo0bOx3FUN OFTYsDarPHpuxUVSRTqswjhi1Q/izP+NKcKeRRNA2KLqyrsMcVi0iWcMiqqZmosB ZZqSTmTgCXRqjoEZuCBHOb94YCYyn8J93ein95W0E2enUNVmYfNoL67/uqSVlUm1 wHS37QyBowQ= =UrSn -----END PGP SIGNATURE-----
In Blender3D were doing an API refactor and Im looking at better ways to write the Python C/API documentation.
At the moment we use epydoc, Its all Iv used so I don't have much to compare it to.
If you havnt used it, its basically a way to add formatting to your normal .py docstrings so they can be converted into HTML or PDF.
This works well with python only projects but is not somthing you can do in C without some manual docstring extraction from the C/API.
I use epydoc extensively, with a lot of C code. Actually, a lot of it is Pyrex, so it's much easier to write and is more natural, but the basics should be the same.
The latest versions of epydoc are much better. I would suggest getting the latest straight from subversion, it is pretty stable.
I think Barry may have mentioned this, but you can put __init__ signatures in the type docstring. I haven't experimented with this, much. This isn't completely documented, you can look in epydoc/docstringparser.py and look for SIGNATURE_RE. See this message from some detail:
http://sourceforge.net/mailarchive/message.php?msg_name=8D22D41A-730E-42B4-A...
I have one of my projects up where you can look at the docs generated by epydoc. The majority of this code is in Pyrex:
-Eric
Campbell Barton cbarton@metavr.com writes:
Ideally Id like to only maintain 1 set of docs... so possibilities are [...]
If you maintain only one set of docs, then your options are indeed limited. I tend to agree with Carsten that docstrings and full documentation are in fact different; docstrings tend to be briefer, don't provide examples, and in fact don't have real formatting. Attempts to add HTML-like formatting to docstrings lead to them not being readable from Python, which defeats the purpose. The Python source is a precedent for maintaining two sets of documentation and for the distinction between them.
We currently document the C-implemented classes with Doxygen by writing dummy classes with markup that Doxygen understands. The line numbers are off and Doxygen doesn't understand some Python features (and tries to shoehorn others into C++ terminology), but other than that, it serves us well. We haven't tried epydoc yet.
participants (6)
-
Barry Warsaw -
Campbell Barton -
Carsten Haese -
Eric Huss -
Hrvoje Niksic -
Martin Poirier