Thanks for the many replies, Samuele. I've put all my responses here. [PEP 287]

1. To establish a standard docstring format by attaining "accepted" status (Python community consensus; BDFL pronouncement). ...

[Samuele Pedroni]

Really? establish in what sense?

PEP acceptance; community awareness; eventually culminating in tools in the stdlib. This is just one step. [PEP 287]

Once a standard exists, people will start to use it, and momentum will inevitably gather.

[Samuele Pedroni]

This is rather naive.

Why do Perl programmers write inline docs in POD? Because it's the Perl standard, not because it's a great markup language. Same for Java and JavaDoc. I'm not satisfied by those examples, barely adequate IMHO. Instead, I'm working on something *better* for Python. Many people with experience in JavaDoc say "What's wrong with JavaDoc?" I tend to discount those opinions, for the same reason I discount C programmers decrying Python's lack of variable declarations or braces: they've been tainted by experience, and aren't entirely objective. (I don't claim to be, either, but I try to keep an open mind.) Not everybody agrees, true. Not everbody agrees on *anything*, even seemingly innocuous additions to Python like booleans. PEP 287 doesn't introduce *any* changes to the language, just an optional markup for docstrings. People are already starting to use reStructuredText. There are already spin-off projects underway; PythonPoint integration is one. I've received several unsolicited email testimonials in support. I've seen others on the web; do a Google search for "restructuredtext" and you'll find some.

it seems that the PEP is the result of a motivated self-selected group with difficulty to reach consensus and so neglecting minimality...

There is some truth to that. There is a *lot* of relevant history on the Doc-SIG. ReStructuredText and PEP 287 are influenced by that history. Minimality is an oft-lauded goal, but minimal tools and markup languages tend to be limited.

From the PEP:

[[Are these in fact the goals of the Doc-SIG members? Anything to add?]]

Throughout the existence of the Doc-SIG, consensus on a single standard docstring format has never been reached.

so is this just your proposal? how should we parse that?

The question [[in brackets]] was from a Doc-SIG post directed at Doc-SIG members, before I posted to python-dev and comp.lang.python. You must have read the PEP from there, from CVS, or from python.org/peps/pep-0287.(txt|html). That question was answered by Doc-SIG members and was removed from the version I posted to python-dev and comp.lang.python. Sorry you had to see that.

The PEP focus seems out of focus ... I don't see a clear picture of the input -> output relation in all the PEPs,

I see now that since it is part of a greater whole, the PEP needs to locate itself with respect to the other parts; a "big picture" overview, or a map with a big "You Are Here". Then I think the focus will become clear. I will add such a map.

it seems that the PEP goes well beyond the kind of structure that the average programmer need

Who is "the average programmer"? I've learned from experience with markup that an artificially limited feature set will limit adoption. If that weren't the case, everyone would be using HappyDoc (the tool is good, but the markup -- StructuredText -- is limited). HappyDoc + reStructuredText could be a killer combination. Aim for mediocrity and we may get there, but we'll never reach excellence. I prefer to aim for excellence.

what happens if I put 3 level of titles in the doc of a method?

You may get crappy docs; that's the author's problem. But sometimes, 3 levels of titles may be necessary and appropriate! If the markup doesn't let you do something when you need it, you'll curse it and stop using it.

oops, ye the abstract says that the PEP is not concerned with this

True, it isn't. That's intentional. The Python docs don't tell you what kind of programs you can or cannot write, either.

the goal as stated are rich docstrings (for what?)

For inline documentation of Python modules and packages. I'll clarify.

I'm puzzled, the PEP does not even refer or cite PEP 257, which OTOH seem also not to address completly the issue of doc-extraction tools interaction.

Honestly, PEP 256,257,258, PEP 287 where are we going?

As I've said, it's a big issue. My approach has been to split it up and deal with the pieces individually. No other approach has worked thus far... Would it be better if I combined everything into one mega-PEP? Roughly novel-length, I imagine!

Maybe goal 1 should be honestly rephrased

(Not that it was ever dishonest, but) How's this? 1. To establish reStructuredText as a standard structured plaintext format for docstrings (inline documentation of Python modules and packages), PEPs, README-type files and other standalone documents. "Accepted" status will be sought through Python community consensus and eventual BDFL pronouncement. Please note that reStructuredText is being proposed as *a* standard, not *the only* standard. Its use will be entirely optional. Those who don't want to use it need not.

you got at least the politics wrong IMHO

You got that right! Nobody ever accused me of being a politician... [PEP 287]

... They may be used to implement docstring semantics, such as identifying parameters, exceptions raised, etc.; such usage is beyond the scope of this PEP.

[Samuele]

how many eons would pass before we see a PEP about those *relevant* issues?

I don't know the answer to that one. That's a policy issue, and I won't be touching it until the technical issues (writing the code) is much further along, if ever. I think docstring semantics may be the thorniest issue of all; people can accept a *syntax*, which is low-level, but telling them how to write docs from a high level? Frederic Giacometti's "iPhrase Python documentation conventions" is an attachment to his Doc-SIG post of 2001-05-30 (http://mail.python.org/pipermail/doc-sig/2001-May/001840.html). Interesting, but didn't go anywhere. The approach I have taken is to break up the issues and try deal with each of them in isolation, or as close as I can get. IMO one cause of the failures of other tools is that they imposed a fixed set of (input syntax, semantics [docstring conventions and policies], processing mechanisms, output formats, styles) on the users. None of them have gotten this set just right; I don't think it's possible to get it right, at least not initially. So I've broken them up, and the Docutils is being designed modularly so that any of these components can be swapped out easily.

Presumably JavaDoc is a good start point for this <wink>.

Yes, it's a good start in some ways.

More or less JavaDoc tags would map to fields

ReStructuredText field lists were partially modelled on JavaDoc tags. The JavaDoc tag syntax was also considered but rejected (see the alternatives doc for details). [from PEP 287]

The lack of a standard syntax for docstrings has hampered the development of standard tools ...

[Samuele Pedroni]

Honestly, is that true?

PEP 256 lists several tools, of which only two are current: pydoc and HappyDoc. Pydoc is in the standard library, but it has two deficiencies: it doesn't deal with markup *at all* (just treats docstrings as literal blocks or <tt>); and it imports modules to document, thereby losing information and potentially incurring a security risk. HappyDoc works well, but uses the older StructuredText markup. From HappyDoc's README: *How does an author write documentation so that it will be marked up and look fancy?* This is a perennial question for Python users, and seems to have introduced a roadblock into the development of more robust and useful documentation tools. HappyDoc stands firmly on the fence and does not attempt to resolve the issue. ReStructuredText is a more robust and useful markup, and Docutils intends to be a more robust and useful toolset. I hope that HappyDoc (possibly with contributions from other tools) can inspire, influence, subsume, or itself *become* Docutils.

how does the PEP relate to producing from inline docstrings at least a first cut in a subset of the py-tex-markup format?

It doesn't. It deals with input markup only. At some point Docutils should have a py-tex-markup writer component. The Docutils PEP (258) talks about output formats.

I *honestly* wonder whether to the Python party really corresponds more-or-less just one documentation format party, a silent majority perhaps.

About as much as the Perl or Java parties, I expect. The only difference is they already have a doc format, and we don't. When there's one obvious choice, well, that's obviously what's chosen.

And whether Docutils (formerly DPS) will be up to this ambitious design claim in PEP 258 (revision 1.2):

Intermediate Data Structure =========================== ... The DTD defines a rich set of elements, suitable for any input syntax or output format

The DTD is quite generic, but it has been influenced by the one and only markup available. Given another markup, it would be even more generic. I try not to generalize prematurely.

PS: the funny thing is that I do not dislike structured text formats in general, but auto-documenting is reducible to that ?

No, of course not. It's just one piece of the puzzle. -- David Goodger goodger@users.sourceforge.net Open-source projects: - Python Docstring Processing System: http://docstring.sourceforge.net - reStructuredText: http://structuredtext.sourceforge.net - The Go Tools Project: http://gotools.sourceforge.net