[Doc-SIG] reStructuredText Field Lists: Deciding on Syntax

[David Goodger]

[Sending directly to Edward Loper as well. Comments, Edward?]

In "usage docstrings and bibliographic field lists" on Fri, 27 Jul 2001,
Garth Kidd <garth@netapp.com> wrote:
> I think we need to support field list option 3 (``reStructuredText.txt``
> revision 1.1.1.1 line 478) to support PEP-0257 style usage docstrings in
> Python scripts without preventing script authors from specifying bibliographic
> field lists.

He goes on to point out problems with field lists vis-a-vis the "special
context" of bibiographic fields (syntax alternative 2; the updated section
is appended to the bottom of this message), and how it conflicts with the
common practice of program "usage" documentation. For example, in a typical
script, the docstring will begin:

    """
    Usage: cmdname [options] arg1 arg2 ...
    ...
    """

I agree with Garth's analysis of the problem. The syntax for field lists has
been at the back of my mind ever since I included it in reStructuredText:
although obvious, it's ambiguous and potentially surprising. I've come to
the conclusion that the "special case" syntax was misguided, and only
unambiguous syntax of some kind (one of alternatives 3, 4, or 5 [new]) is
reasonable.

- alternative 3: use a directive::

      .. fields::

         Author: Me
         Version: 1

- alternative 4: use Javadoc-style::

      @Author: Me
      @Version: 1
      @param a: integer

- alternative 5: use leading colons::

      :Author: Me
      :Version: 1
      :param a: integer

  (I recently came up this syntax. It seems to have the best balance:
  unambiguous, obvious, and elegant. I like it best so far.)

Please comment; I'd really like to hear people's opinions about this. Any
other alternatives out there? Any answers to the questions asked in [1]_
below?

/David

P.S. Below, I've appended the current "Field Lists" section, including diff
marks (diff to release 0.3).

  Field Lists
  -----------
  
  DTD elements: field_list, field, field_name, field_argument,
  field_body.
  
!     Syntax under construction. Comments and suggestions welcome.
  
  Field lists are mappings from field names to field bodies, modeled on
  RFC822_ headers. A field name is made up of one or more letters,
  numbers, and punctuation, except colons (':') and whitespace. Field
  names are case-insensitive. A single colon and whitespace follows the
  field name, and this is followed by the field body. The field body may
  contain multiple body elements.
  
+     Why is RFC822 compliance important? It's a universal Internet
+     standard, and super obvious. Also, I'd like to support the PEP
+     format (ulterior motive: get PEPs to use reStructuredText as their
+     standard). But it *would* be easy to get used to an alternative
+     (easy even to convert PEPs; probably harder to convert
+     python-deviants ;-).
+ 
+     Unfortunately, without well-defined context (such as in email
+     headers: RFC822 applies before any blank lines only), the RFC822
+     format is very common in ordinary text. To implement field lists
+     unambiguously, we need explicit syntax.
+ 
  Applications of reStructuredText may recognize field names and
  transform fields or field bodies in certain contexts. Any
  untransformed fields remain in the field list as the document's first
  body element.
  
  The syntax for field lists has not been finalized. Syntax
  alternatives:
  
  1. Unadorned RFC822_ everywhere::
  
         Author: Me
         Version: 1
  
!    Advantages: clean, precedent (RFC822-compliant). Disadvantage:
!    ambiguous (these paragraphs are a prime example).
  
     Conclusion: rejected.
  
  2. Special case: use unadorned RFC822_ for the very first or very last
!    text block of a document::
  
         """
         Author: Me
         Version: 1
  
!        The rest of the document...
!        """
! 
!    Advantages: clean, precedent (RFC822-compliant). Disadvantages:
!    special case, flat (unnested) field lists only, still ambiguous::
! 
         """
+        Usage: cmdname [options] arg1 arg2 ...
  
!        We obviously *don't* want the like above to be interpreted as a
!        field list item. Or do we?
!        """
  
!    Conclusion: was accepted (although not yet implemented; see below);
!    currently reconsidering.
  
  3. Use a directive::
  
         .. fields::
  
            Author: Me
            Version: 1
  
!    Advantages: explicit and unambiguous, RFC822-compliant.
!    Disadvantage: cumbersome.
  
  4. Use Javadoc-style::
  
         @Author: Me
         @Version: 1
         @param a: integer
  
     Advantages: unambiguous, precedent, flexible. Disadvantages:
!    non-intuitive, ugly, not RFC822-compliant.
! 
! 5. Use leading colons::
  
!        :Author: Me
!        :Version: 1
! 
!    Advantages: unambiguous, obvious? (*almost* RFC822-compliant),
!    flexible, perhaps even elegant. Disadvantage: no precedent, not
!    quite RFC822-compliant.
! 
! One special context is defined for field lists [1]_. A field list as
! the very first non-comment block, or the second non-comment block
  immediately after a title, is interpreted as document bibliographic
! data. No special syntax is required, just unadorned RFC822_ [2]_. The
! first block ends with a blank line, therefore field bodies must be
! single paragraphs only and there may be no blank lines between fields.
! The following field names are recognized and transformed to the
  corresponding DTD elements listed, child elements of the 'document'
  element. No ordering is imposed on these fields:
  
  - Title: title
  - Subtitle: subtitle
  - Author: author
  - Authors: authors (a list of author elements)
  - Organization: organization
  - Contact: contact
  - Version: version
  - Status: status
  - Date: date
  - Copyright: copyright
  
  This field-name-to-element mapping can be extended, or replaced for
  other languages. See the implementation documentation for details.
  
+ .. _[1] Should "bibliographic field lists" be defined at the parser
+    level, or at the DPS transformation level? In other words, are they
+    reStructuredText-specific, or would they also be applicable to
+    another (many/every other?) syntax?
+ 
+ .. _[2] Note that this "no special syntax required" provision is being
+    reconsidered. An unambiguous syntax would also remove the
+    requirement for a bibliographic field list to end with a blank
+    line.