[Doc-SIG] Re: two subjects: docutils for python docs, and integrating documentation in source code

[Martin Blais]

On 4/22/05, Felix Wiemann <Felix.Wiemann at gmx.net> wrote:
> Martin Blais wrote:
> > 2. unifying source code and readable documentation
> 
> I'm not sure if we want that because source documentation is for
> developers and external documentation is for users, but it's certainly
> worth thinking about.

oh, yes, but i was implying "for the library modules documentation". 
in other words, there will always be a need to write documentation
outside of the code, but for the individual modules themselves, it
would make sense to be able to put the docs in the code itself; 
advantages are:

1. one source, easier for developers to keep it up to date
2. distribution of docs is easier
3. reuse of docstrings within the same file

we could conceptualize a template system like i mentioned before whose
template can live either inside the file, or separately (but still
able to fetch bits of docstrings).


> 
> > [...] it seems to me that auto-generated documentation tools should
> > really be part of python.
> 
> Docutils will be ready at the earliest in a few years.  Dunno about
> Epydoc though.

looks to me like docutils is very usable already... 


> > we might want to do something to couple the source code and the
> > written manual, e.g. where we could have a template that outlines the
> > "manual" versoin of the documentation, and our directive would go
> > fetch the docstring for the function or class and insert it inline in
> > our document automatically.  a class documentation could look like
> > this:
> >
> >  intro and example text
> >  .. classdescref:: module.classname
> >  ..methoddescref:: module.classname.foo1
> >  ..methoddescref:: module.classname.foo2
> >  ..methoddescref:: module.classname.foo3
> >  ..methoddescref:: module.classname.foo4
> >
> > this would allow arbitraty ordering of the methods (something which an
> > automated source to docs tool couldn't do)
> 
> No, let's pick up the methods automatically.  The methods *can* be
> reordered by just moving them around.

i thought about this, and by fiddling around with pydoc with some of
the stdlib module sources, came to a conclusion that this would not be
enough.

sometimes source code functions are grouped in a way that make sense
in a structured programming approach,  i.e. you can have a functoin
foohelper() directly after foo(), but foohelper() really should not be
in the documentation at all.  or perhaps it should be in a special
"Helper" section in the generated docs.  however in the code, some of
the data structures it uses were created in foo(), and as such it
really makes sense to put its implementation next to foohelper() 
(note that in C++ since you have separate declaration and definition
you don't have that problem and automated code generators use that,
they can use on the ordering in the header files).

also, and more obviously, many methods or functions simply don't need
to appear in the documentation.  try pydoc on some modules with
classes, you'll see what i mean.  you get some bits of content, and
lots of noise.  IMO this is very important if we're going to produce
documentation that is readable, like the one that currently exists, we
can't just pick up all the methods.

perhaps we could provide an easy way to pick up all the methods if
desired?  e.g.::

   ..classdescref:: modules.classname
   ..methods:: modules.classname


> > also, one thing that i can't see is how to generate index entries from
> > docutils.  also, for typed references e.g. \function{foo}, i don't
> > know how you would do that in docutils.
> 
> ::
> 
>     .. role:: function
> 
>     :function:`foo`
> 
> In the stylesheet for the new LaTeX writer which I'm currently
> developing::
> 
>     \providecommand{\DCfunction}[2]{\function{#2}}
> 
> I have a deadline next Wednesday, so I'll probably upload the new LaTeX
> writer then.

nice.

but note that these generate index entries for code declarations (i.e.
function, class, data).  these are "easy" since they have their own
syntactic units.  what i meant is that it is not currently possible to
generate generic index entries with docutils (not that i know of
anyway), e.g. in a paragraph of normal text, how do i indicate that a
word in the paragraph should appear in the index?

at some point we're entreing the zone where the question becomes: can
docutils be as complete a document preparation system as LaTeX?  and
the answer is obviously no and it never will be, but i suppose my
question is, how close can we get?  we don't need all the features
that latex offers for the technical documentation.  we should list all
the essential ones (not compromising on any the existing ones), and
see if they can all be implemented in docutils.

we should also see what can be improved.  in particular, the following
things should be improved about the current documentatoin input:

- the optional parameters markup should be grouped rather than nested with
  \optional.  it makes more sense to declare them this way in Python because
  of the keywords syntax for function calls (e.g. foot(bli=4));

- there is no way to declare the base classes.  The problem is that the syntax
  for the constructor (documentation) takes the space of inheritance (source
  code);

- there needs to be a way to write a documentation paragraph for many
  methods/functions at the same time.  i haven't found how to do this
nicely yet,
  apart from breaking the declarations of functions outside of the classdesc;

- there should be a formal syntax to describe the role of each of the arguments,
  as well as for the return value.  Parsing the code for doc
generation should validate
  that all the arguments are covered in the documentation string (this
could be an
  option, and a condition for merging code).  many people have suggested ways
  of doing this.

any ideas appreciated.
cheers,