On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen <screwtape@froup.com> wrote:
On Sat, Jul 10, 2010 at 02:58:40AM -0000, exarkun@twistedmatrix.com wrote:
> At last I've got a buildbot set up generating the sphinx docs.  The
> build results can be seen here:
>   http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/
> (or with a different revision number for different revisions; the
> containing directory is browseable).
> It would be great if everyone could go look around and report any
> problems you see.

   - In the Documentation Table Of Contents page, "Historical
     Documents" is listed between "Twisted Core" and "Twisted Lore"
     (presumably because of alphabetical order) but this seems
     a strange place for it. Could it be moved to the bottom, or an
     appendix, or something?

They're in alphabetical order.  The TOC page is generated dynamically by looking at the file structure of the docs and that's the order file-globbing lists them in.  We can fix this manually post-conversion.
   - The bottom of every page has a "Download in other formats: Plain
     Text" link that doesn't work; presumably scooped up from the Trac
     templates. It should be removed (the Sphinx "Show Source" link is
     in the right-hand sidebar anyway).

Yeah, this is a flaw in the trac theme.  Looks like I forgot to remove it.
   - When the breadcrumb navigation at the top of the page gets too
     long[1], it wraps to a second line, which looks kind of odd
     because the nice gradient background image loops. Changing the CSS
     background colour to be the same as the bottom of the gradient
     image would probably help.
   - Firefox's Error Console reports:
       - a bogus "%" on line 56 of twistedtrac.css,
       - bogus // comments on lines 559, 577, 588, 598 of trac.css (CSS
         only allows /* winged comments */)
   - Chromium's error console reports:
       - A 404 error for "/trac/chrome/common/js/jquery.js"
       - A 404 error for "/trac/chrome/common/js/trac.js"
       - A 404 error for "/trac/chrome/common/js/search.js"
       - A 404 error for "/builds/sphinx-html-15615/_static/dots.gif"

Ah! This explains why the search quit working.  Not sure why these files are missing,  Probably something to do with the custom theme.
   - The Twisted.Conch tutorial[1] has a number of things surrounded
     with double-backticks like "``Deferred``" or
     "``ClientTransport``". I'm not sure if that's deliberate or if
     it's markup gone horribly wrong.

This is probably due to missing spaces before/after the markup in the rst source.  One of those things that will need to be fixed manually.  I spent a lot of time trying to fix this in lore2sphinx and when you fix it in one place, it breaks somewhere else.  Right now the automated conversion is about as good as it is likely to get with a sane amount of effort.
   - On the same page, the text ":api:`
     < twisted.internet.interface.Transport>`" appears, which also
     looks wrong.

   - The Conch code examples (as linked from the code examples page[2])
     are sent with a Python mime-type (which Firefox tries to download)
     while the .tac files are sent as text/html; this may be an
     artifact of the docs being hosted on the buildbot machine rather
     than Sphinx, but I think it would be nice if by default they were
     displayed in the browser, syntax-highlighted, with an option to
     download the original files (much like Trac does).

This is a web server configuration thing.  The files are actually .py and .tac files (and probably need to remain so, if we ever want to get automated example code tests going).

Maybe we can do some web server magic to get them nicely displayed in the browser, but I see that as a secondary issue for the moment.  Anyone should feel free to give a shout if they disagree, though.
   - The Twisted Split FAQ[3] has more visible markup:
     <core-upgrades-2.0-split-protocols> `". The footnote target is
     kind of messed up, too.

Another spacing issue.
   - The Twisted Split FAQ[3] has a heading "Why are arr the packages
     still named twisted.*subproject*?", but it's not obvious whether
     the author was trying to use italics (in which case it's
     a lore2sphinx bug) or globbing (in which case it's not).

Pretty sure it was supposed to be italics.  Spacing again.  There are ways to make this display properly, but it'll need to be done manually.

Also, this is pretty outdated stuff...we might consider just removing some of this.  Or perhaps moving some of it into a more obvious place.
   - The Twisted Zope Interfaces FAQ[4] has inline Python source
     examples that don't appear to be marked up properly (although this
     is possibly a problem with the original Lore source). The answer
     to question "How can I update my own code?" is one example.

The code bits are marked up this way because of the attributes used on the elements in the Lore source.  We can change it easily
   - The Banana Protocol Specifications[5], under the heading
     "Examples" mentions "the type bytes are marked in bold" but
     actually they're surrounded by double-asterisks.

Double asterisks are the markup for bold in rst.  Another spacing and/or nested inline markup issue.
   - The Twisted Coding Standard[6], mentions "the complete test
     suite in trunk@HEAD" - and "trunk@HEAD" is converted into
     a mailto: link.

Huh.  I guess that's Sphinx trying to be helpful.  Good catch.  That'll definitely need fixing.
   - Links followed by punctuation seem to leave a space between the
     link text and the punctuation. See "test-driven ." and "Test
     Standard ." in the Twisted Coding Standard[6], but I've observed
     the same problem on multiple pages.

This is the spacing issue again, but in reverse.  It's relatively easy for lore2sphinx to detect links and add a space at the end, so that's what it does.  There's ways to get around this and get rid of the space by escaping it, but it rapidly get mind-boggling, so this is another thing that will need manual fixing.
   - The Twisted Coding Standard[6] mentions "If you modify, or write
     a new, HOWTO, please read the Lore documentation to learn how to
     format the docs"; if there's not already a follow-up ticket for
     the lore-sphinx conversion titled "Remove references to Lore from
     Twisted's non-Lore-related documentation", there should be, and
     this should be in it.

There isn't, but there will be.  See the transition plan here: http://twistedsphinx.funsize.net/transition_plan.html
   - The Twisted Coding Standard[6], under the heading "Modules" says
     "Use this template:" and then
     "../listings/new_module_template.py".. and *then* includes the
     content of said template inline. Presumably it should either link
     to it or include it, but not both.

It actually is including an external file there (i.e. the contents are not in the rst source, but in the
"../listings/new_module_template.py" file.  What looks like a link there is actually a sort of heading to
let the reader see where it's coming from.  I think this can probably be made clearer with some theme/css changes.
   - The HTML Documentation Standard for Twisted[7] has more visible
     markup: in the list of allowable markup, most tags are rendered
     properly except for "``<tr>``".

Missing space after preceding comma.  Another manual fix.
   - It seems that the HTML Documentation Standard for Twisted[7] is
     actually "how to write Lore documentation", despite the name.
     Maybe this file should be scheduled for the post-transition purge,
     or at least moved to the Lore documentation.

It pretty much is "how to write Lore documentation".  I think it should probably be moved into the Lore docs, and replaced with the
(yet to be written, but again see the transition plan) planned Twisted Documentation Guide.
   - Working from Twisted's Subversion Repository[8] has visible markup
     that looks like an attempt to link "the Subversion homepage" under
     the heading "Checkout".

Missing preceding space again.
   - Under the heading "Alternate tree names", the document uses
     in 'directory other than”Twisted” .' Looks like a problem in the
     original markup. Also note the space between the LEFT DOUBLE
     QUOTATION MARK and the punctuation.

The missing preceding space actually is what's causing the quotes to be wrong.
   - The same document has more broken links under the "Combinator"

Missing preceding spaces again.
   - The same document has broken markup under "Running tests":
     "``twisted/protocols/imap4.py``" and later
   - The same document has broken markup under "Building docs":
     "``doc/development/policy/svn-dev.xhtml`` ," (also trailing space
     before punctuation) and
     "``doc/development/policy/svn-dev.html`` :" (again).
   - The same document has a broken link under "Committing and
     Post-commit Hooks", attempting to link the text
   - The same document has broken markup under "Emacs":
   - The same document has a broken link under "Building Debian
     packages", attempting to link th text "stdeb".

All bad spacing issues.

That's probably enough feedback to be getting on with; most of the
problems appear to be from normalising "\n" in Lore docs to "" instead
of " ", and also from adding whitespace after things. It is generally
looking pretty great, though!

Yeah, that's pretty much it.  As I said above though, if you "fix" it one place, it breaks in another, so I tried to balance things in such a way that the least broken markup appears in the output.

Almost all of the remaining problems will need to be fixed manually.

Some extra thoughts:
   - The ReviewingDocumentation wiki page has a section called "Editing
     man pages" that describes how to turn the nicely-formatted
     manpages into Lore input files. Would it be possible to do that as
     part of the lore2sphinx run, have the manpages included in the
     Sphinx documentation, and from then on generate the manpages from
     the .rst files instead of the other way around?

Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this.

So you're suggesting convert the man pages to Lore format -> use lore2sphinx to convert the resulting Lore docs to rst -> build as part of the Sphinx process, yes?

I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket).  lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around.

[1]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[2]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[3]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[4]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[5]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[6]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[7]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
[8]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/

Thanks for the fantastic (and nicely detailed) feedback, Tim!

Please take a look at the transition plan.  In a few days (maybe sooner), I should have the base docs in a branch, and the "chunk tickets" referenced in the transition plan created.  This is pretty much _exactly_ what I'd like to see in those "chunk tickets".  Hopefully you haven't already burned up your brain staring at markup issues. :) We could really use this kind of help throughout the process.

Kevin Horn