[Doc-SIG] looking for prior art
Doug Hellmann
doug@hellfly.net
Fri, 6 Dec 2002 08:27:49 -0500
On Thursday 05 December 2002 9:45 pm, David Goodger wrote:
> Doug Hellmann wrote:
> > I'm pretty sure HappyDoc was written before the compiler module was
> > generally available
>
> I suspected as much. Either that, or you're a glutton for punishment
> ;-)
Well, I didn't say that wasn't true. :-) I actually started with some
sample code included in the Python source distribution, so it wasn't too hard
to extend it and come up with a useful parser.
> > I've only had to make a few minor modifications to it in the past,
> > since the language syntax hasn't evolved that far.
>
> That's good to know. Still, the parser.suite() approach seems a lot
> harder.
If you're starting from scratch, I would definitely recommend trying the
compiler module first.
> > I'm working on a major overhaul of HappyDoc anyway, so now might be
> > the time to rewrite the parsing stuff to use the compiler module.
> > If you're interested in collaborating, let me know.
>
> I am, definitely. What I'd like to do is to take a module, read in
> the text, run it through the module parser (using compiler.py and
> tokenize.py) and produce a high-level AST full of nodes that are
> interesting from an auto-documentation standpoint. For example, given
> this module (x.py)::
[...]
> compiler.parse() provides most of what's needed for this AST. I think
> that "tokenize" can be used to get the rest, and all that's left is to
> hunker down and figure out how. We can determine the line number from
> the compiler.parse() AST, and a get_rhs(lineno) method would provide
> the rest.
Does compiler include comments? I had to write a separate parser to pull
comments out.
> The Docutils Python reader component will transform this AST into a
> Python-specific doctree, and then a `stylist transform`_ would further
> transform it into a generic doctree. Namespaces will have to be
> compiled for each of the scopes, but I'm not certain at what stage of
> processing.
Why perform all of those transformations? Why not go from the AST to a
generic doctree? Or, even from the AST to the final output?
> It's very important to keep all docstring processing out of this, so
> that it's a completely generic and not tool-specific.
Definitely.
Doug