[Python-Dev] Re: PEP 287: reStructuredText Standard Docstring Format

[David Goodger]

Thanks for the many replies, Samuele.  I've put all my responses here.

[PEP 287]
>     1. To establish a standard docstring format by attaining
>        "accepted" status (Python community consensus; BDFL
>        pronouncement).  ...

[Samuele Pedroni]
> Really? establish in what sense?

PEP acceptance; community awareness; eventually culminating in tools
in the stdlib.  This is just one step.

[PEP 287]
>         Once a standard exists, people will start to use
>        it, and momentum will inevitably gather.

[Samuele Pedroni]
> This is rather naive.

Why do Perl programmers write inline docs in POD?  Because it's the
Perl standard, not because it's a great markup language.  Same for
Java and JavaDoc.  I'm not satisfied by those examples, barely
adequate IMHO.  Instead, I'm working on something *better* for Python.

Many people with experience in JavaDoc say "What's wrong with
JavaDoc?"  I tend to discount those opinions, for the same reason I
discount C programmers decrying Python's lack of variable declarations
or braces: they've been tainted by experience, and aren't entirely
objective.  (I don't claim to be, either, but I try to keep an open
mind.)

Not everybody agrees, true.  Not everbody agrees on *anything*, even
seemingly innocuous additions to Python like booleans.  PEP 287
doesn't introduce *any* changes to the language, just an optional
markup for docstrings.

People are already starting to use reStructuredText.  There are
already spin-off projects underway; PythonPoint integration is one.
I've received several unsolicited email testimonials in support.  I've
seen others on the web; do a Google search for "restructuredtext" and
you'll find some.

> it seems that the PEP is the result of a motivated self-selected
> group with difficulty to reach consensus and so neglecting
> minimality...

There is some truth to that.  There is a *lot* of relevant history on
the Doc-SIG.  ReStructuredText and PEP 287 are influenced by that
history.

Minimality is an oft-lauded goal, but minimal tools and markup
languages tend to be limited.

> From the PEP:
> 
>     [[Are these in fact the goals of the Doc-SIG members?  Anything
>     to add?]]
> 
>     Throughout the existence of the Doc-SIG, consensus on a single
>     standard docstring format has never been reached.
> 
> so is this just your proposal? how should we parse that?

The question [[in brackets]] was from a Doc-SIG post directed at
Doc-SIG members, before I posted to python-dev and comp.lang.python.
You must have read the PEP from there, from CVS, or from
python.org/peps/pep-0287.(txt|html).  That question was answered by
Doc-SIG members and was removed from the version I posted to
python-dev and comp.lang.python.  Sorry you had to see that.

> The PEP focus seems out of focus
...
> I don't see a clear picture of the input -> output relation in all
> the PEPs,

I see now that since it is part of a greater whole, the PEP needs to
locate itself with respect to the other parts; a "big picture"
overview, or a map with a big "You Are Here".  Then I think the focus
will become clear.  I will add such a map.

> it seems that the PEP goes well beyond the kind of structure that
> the average programmer need

Who is "the average programmer"?  I've learned from experience with
markup that an artificially limited feature set will limit adoption.
If that weren't the case, everyone would be using HappyDoc (the tool
is good, but the markup -- StructuredText -- is limited).  HappyDoc +
reStructuredText could be a killer combination.

Aim for mediocrity and we may get there, but we'll never reach
excellence.  I prefer to aim for excellence.

> what happens if I put 3 level of titles in the doc of a method?

You may get crappy docs; that's the author's problem.  But sometimes,
3 levels of titles may be necessary and appropriate!  If the markup
doesn't let you do something when you need it, you'll curse it and
stop using it.

> oops, ye the abstract says that the PEP is not concerned with this

True, it isn't.  That's intentional.  The Python docs don't tell you
what kind of programs you can or cannot write, either.

> the goal as stated are rich docstrings (for what?)

For inline documentation of Python modules and packages.  I'll
clarify.

> I'm puzzled, the PEP does not even refer or cite PEP 257, which OTOH
> seem also not to address completly the issue of doc-extraction tools
> interaction.
> 
> Honestly, PEP 256,257,258, PEP 287 where are we going?

As I've said, it's a big issue.  My approach has been to split it up
and deal with the pieces individually.  No other approach has worked
thus far...

Would it be better if I combined everything into one mega-PEP?
Roughly novel-length, I imagine!

> Maybe goal 1 should be honestly rephrased

(Not that it was ever dishonest, but) How's this?

    1. To establish reStructuredText as a standard structured
       plaintext format for docstrings (inline documentation of Python
       modules and packages), PEPs, README-type files and other
       standalone documents.  "Accepted" status will be sought through
       Python community consensus and eventual BDFL pronouncement.

       Please note that reStructuredText is being proposed as *a*
       standard, not *the only* standard.  Its use will be entirely
       optional.  Those who don't want to use it need not.

> you got at least the politics wrong IMHO

You got that right!  Nobody ever accused me of being a politician...

[PEP 287]
>       ... They may be used to implement docstring semantics, such as
>       identifying parameters, exceptions raised, etc.; such usage is
>       beyond the scope of this PEP.

[Samuele]
> how many eons would pass before we see a PEP about
> those *relevant* issues?

I don't know the answer to that one.  That's a policy issue, and I
won't be touching it until the technical issues (writing the code) is
much further along, if ever.  I think docstring semantics may be the
thorniest issue of all; people can accept a *syntax*, which is
low-level, but telling them how to write docs from a high level?

Frederic Giacometti's "iPhrase Python documentation conventions" is an
attachment to his Doc-SIG post of 2001-05-30
(http://mail.python.org/pipermail/doc-sig/2001-May/001840.html).
Interesting, but didn't go anywhere.

The approach I have taken is to break up the issues and try deal with
each of them in isolation, or as close as I can get.  IMO one cause of
the failures of other tools is that they imposed a fixed set of (input
syntax, semantics [docstring conventions and policies], processing
mechanisms, output formats, styles) on the users.  None of them have
gotten this set just right; I don't think it's possible to get it
right, at least not initially.  So I've broken them up, and the
Docutils is being designed modularly so that any of these components
can be swapped out easily.

> Presumably JavaDoc is a good start point for this <wink>.

Yes, it's a good start in some ways.

> More or less JavaDoc tags would map to fields

ReStructuredText field lists were partially modelled on JavaDoc tags.
The JavaDoc tag syntax was also considered but rejected (see the
alternatives doc for details).

[from PEP 287]
>     The lack of a standard syntax for docstrings has hampered the
>     development of standard tools ...

[Samuele Pedroni]
> Honestly, is that true?

PEP 256 lists several tools, of which only two are current: pydoc and
HappyDoc.  Pydoc is in the standard library, but it has two
deficiencies: it doesn't deal with markup *at all* (just treats
docstrings as literal blocks or <tt>); and it imports modules to
document, thereby losing information and potentially incurring a
security risk.  HappyDoc works well, but uses the older StructuredText
markup.  From HappyDoc's README:

    *How does an author write documentation so that it will be marked
    up and look fancy?* This is a perennial question for Python users,
    and seems to have introduced a roadblock into the development of
    more robust and useful documentation tools.  HappyDoc stands
    firmly on the fence and does not attempt to resolve the issue.

ReStructuredText is a more robust and useful markup, and Docutils
intends to be a more robust and useful toolset.  I hope that HappyDoc
(possibly with contributions from other tools) can inspire, influence,
subsume, or itself *become* Docutils.

> how does the PEP relate to producing from inline docstrings at least
> a first cut in a subset of the py-tex-markup format?

It doesn't.  It deals with input markup only.  At some point Docutils
should have a py-tex-markup writer component.  The Docutils PEP (258)
talks about output formats.

> I *honestly* wonder whether to the Python party really corresponds
> more-or-less just one documentation format party, a silent majority
> perhaps.

About as much as the Perl or Java parties, I expect.  The only
difference is they already have a doc format, and we don't.  When
there's one obvious choice, well, that's obviously what's chosen.

> And whether Docutils (formerly DPS) will be up to this ambitious
> design claim in PEP 258 (revision 1.2):
> 
>     Intermediate Data Structure
>     ===========================
>     ...
>     The DTD defines a rich set of elements, suitable for any input
>     syntax or output format

The DTD is quite generic, but it has been influenced by the one and
only markup available.  Given another markup, it would be even more
generic.  I try not to generalize prematurely.

> PS: the funny thing is that I do not dislike structured text formats
> in general, but auto-documenting is reducible to that ?

No, of course not.  It's just one piece of the puzzle.

-- 
David Goodger    goodger@users.sourceforge.net    Open-source projects:
 - Python Docstring Processing System: http://docstring.sourceforge.net
 - reStructuredText: http://structuredtext.sourceforge.net
 - The Go Tools Project: http://gotools.sourceforge.net