[AstroPy] Documentation Guidelines

[Erik Tollerud]

I can definitely see the value of the numpy/scipy system, given the
reasons Joe and Michael have given here.  In particular, the actual
sectioning/organization of the docstring definitely makes sense for
our needs and consistency with numpy/scipy is an added bonus.

However, when I thought about it earlier, I saw a number of issues
with using the full numpy build doc system + pydocweb for the
particular needs of astropy.  I will say right now that any of these
may simply be due to my lack of knowledge of how the system works, so
if there are easily solutions I've missed, I'd love to hear them!

* First, and most importantly, it's not clear to me how to adapt the
numpy doc builder to our needs.  I've actually never gotten it to run
completely on numpy, despite trying on three different platforms.  I'm
willing to concede this could well be user error, but it implies that
it might be difficult to use on a bunch of new projects - there seem
to be 8 separate extensions in numpy's "sphinxet". Remember that the
plan we have set out here involves a large number of packages that
operate semi-independently until they are merged into the core, and
they must be able to build their documentation independently of the
core, at least until they are merged (and many will probably never be
merged).  Is there a separate project that is documented and can be
easily adapted/simplified to suit our needs?

* Does the pydocweb system still work since scipy moved to github (or
in general is it VCS-agnostic)? I have edited scipy docs using the
system before, but it was never clear to me how the web interface and
the VCS interplayed with each other.  I would say we want to keep the
docs themselves in sync with the VCS version of the code they apply to
- is that always the case with pydocweb?

*The pydocweb workflow diagram requires quite a few steps to
completion, and as Joe points out, it seems the review system ran out
of steam - and indeed looking at the numpy docs nearly everything is
still listed as "need review."  Given that our community is probably
smaller and has fewer resources to devote to reviewing docs, I'm
concerned this workflow will not work for astropy (particularly for
the affiliated packages).  My understanding is that the review system
was designed basically because there was a bunch of poorly-documented
functionality, so the idea was to bring other people in to do the
documentation.  With the vision in place for astropy, nothing even
makes it into the core in the first place if it doesn't have fairly
complete documentation.  That's one of the most important jobs of the
coordinating committee.  So with that in mind, might the full pydocweb
scheme actually be too *complex* for our needs? Or is it easy to
simplify it for our needs (perhaps just a single "review" stage that
comes immediately after the code is written?)

*How does pydocweb handle properties?  We definitely want to be able
to document properties directly, and have them still end up in the
sphinx docs alongside the docs for the class' attributes.

*pydocweb requires django to operate, according to it's docs - so that
implies that either we have to have a central host to hold all of the
affiliated projects' pydocweb projects while they are being written,
or we have to expect people to figure that out on their own... By
contrast, a simpler sphinx setup only requires "python setup.py
build_docs upload_docs" and then the latest docs are at
packages.python.org, and fully accessible via PyPI.  Or is there a way
to bring pydocweb onto packages.python.org ?

Given that sphinx now has tools to generate APIs entirely from
docstrings (the built-in autodoc and autosummary extensions) without
any extra scripts needed, might it make more sense to use the numpy
docstring format, but couple it instead to these sphinx tools? Sphinx
also has markup specific to all of the sections that the numpy docs
call for, also, so it's simple to use the ".. seealso::" and "..
notes" constructs to get the same effect as the numpy docstrings
without the additional numpy scripts.  This of would mean we'd either
have to change the syntax of the numpydoc scheme (although hopefully
not the sectioning), or write a sphinx extension to suit our needs.



> I think the alternate format discussed by Erik T. and Prasanth is OK,
> and I've encountered it before (e.g., while reading the AstroLibCoords
> documentation).  However, I think this format is more cumbersome to
> read because the use of the additional ":param" or ":returns" at the
> beginning of each variable definition, and the use of ":type".  For
> me, these phrases make the text more difficult to quickly search and
> find the variable name that I would be looking for when using the
> on-line help.  numpydoc's headings "Parameters", "Returns", etc. is
> much easier to read.

I agree this is a bit more cumbersome to read, but it produces (at
least, to my eye) much prettier sphinx APIs if you use it with
autodoc/autosummary.  So I guess the question is: do we want the
docstrings as they appear with ipython's "?" or help() to be
prioritized over the sphinx output, vice-versa, or what?

(Oh, and there's only need for one "returns" clause, and as I was
saying before, the "type" can be excluded if it's judged to make it
too hard to read.  The "param name" of course has to be there,
though).



On Mon, Jul 11, 2011 at 11:23 AM, Joe Harrington <jh at physics.ucf.edu> wrote:
> I apologize profusely for replying on the wrong thread!  I've
> re-posted here so discussion can follow in the right one.  Sorry for
> the extra traffic.
>
>> http://astropy.wikispaces.com/Astropy+Coding+Guidelines
>
> I would strongly encourage:
>
> 1. Use the numpy docstring format.  We spent a *lot* of time designing
>   this format and getting community feedback to ensure that projects
>   like this one can use it, in addition to the core numpy doc
>   project.  It's been vetted in the writing of over 1000 pages of
>   docs.  It can handle images, formulae, and cross references and
>   there are guidelines for how to do that and how to make sure that
>   the ASCII version is also sensible.  It's what numpy and scipy
>   users are now used to.  It's based on Sphinx, so it makes a PDF,
>   HTML, and of course ASCII, has doctests, etc.  There's a place for
>   references, and a way to separate the basic info everyone needs to
>   know from the details ("notes") only specialists care about.  There
>   is one mod that the Matplotlib folks have, for the special case of
>   families of routines that have dozens or hundreds of identical,
>   optional parameters (common for graphics routines).
>
>   Important: Keep the one-line summary truly to less than about 60
>   characters.  It can be used to make lists of routines, one line per
>   routine, including the name, for easy indexing.  This breaks if the
>   "one line" goes on and on.
>
>   Here is the standard:
>
>   https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
>
>   If you want to see what the results look like, just type
>   help(np.whatever), or look at the PDF or HTML pages at
>   docs.scipy.org.
>
> 2. Use the docs.scipy.org wiki system, pydocweb.  Make a project
>   either there or on the astropy system; the code is OSS.  This will
>   allow normal users who would not otherwise have access to the code
>   repo to submit doc fixes, examples, and tests; perform peer review;
>   and discuss specific pages.  This is extremely helpful if the
>   programmer is lazy (or even functionally illiterate ;-), or when
>   importing a package that's already written and has poor docs (e.g.,
>   numpy 3 years ago).  It can sync in both directions with the VCS,
>   gives metrics of who contributed what, what needs editing now, what
>   got worked on this week, etc.  You can see who wrote what, and
>   revert if there's a problem.  Check out these features at
>
>   http://docs.scipy.org/numpy/
>
>   A description of the workflow and links to the style guidelines,
>   templates, and examples are here:
>
>   http://docs.scipy.org/numpy/Front%20Page/
>
>   Source code to pydocweb is here:
>
>   http://code.google.com/p/pydocweb/
>
>   The project at numpy is fairly dormant now, the main docs having
>   been written (the community fell flat on its face and didn't step
>   up to review), but the folks are all available on the scipy-dev
>   mailing list if you want to ask questions about it.  The main
>   developer is Pauli Virtanen.  We wrote status articles in the 2008
>   and 2009 SciPy Conference proceedings:
>
>   http://conference.scipy.org/proceedings/SciPy2008/paper_7/
>   http://conference.scipy.org/proceedings/SciPy2009/paper_14/



-- 
Erik Tollerud