Mailman 3 [Twisted-Python] Lore to Sphinx Conversion Progress Report 6 - Twisted

newer
[Twisted-Python] Weekly Bug Summary

[Twisted-Python] Lore to Sphinx Conversion Progress Report 6

Kevin Horn

4 Mar 2010 4 Mar '10

10:49 p.m.

It's been a while, I know you were all waiting with baited breath... First a few fixes: - Thanks to Tim Allen and Steve Steiner, several theme issues were fixed. This fixes a few minor display bugs and should make pages validate properly. - inline markup will now include child contents. This causes rst not to display properly in cases of "nested inline markup", but should make it easier to manually fix these places later - :download: links are now generated for non-rst files ending in .py (most of the examples), as welll as .tac and .sql files - toc entries which end in a "/" character now have "index appended to them in the ReST output, which fixes a few issues where the built-Sphinx docs would end up with broken navigation links. A couple of issues were also fixed at the PyCon sprints: - <cite> tags are now handled...sort of. According to Glyph, these aren't really very important, so we just return the contents of the tag without any special markup. If we feel the need to put citation markup back into the docs we can do that post-transition. - Glyph also found an easy way to improve the CSS of the theme somewhat. The issue where the sidebar drops below the main content should now be less of a problem, though it's not gone entirely. - Glyph and I also discussed a few different ways to handle links to the API docs and agreed on a way forward. Basically, well change the <code class='API'> tags into something like: :api:`path.to.documented object <label>`, and then create a docutils extension to convert that into a link. The beginnings of this are in the hg repos, but it looks like it won't be quite as simple as we thought. First, the way pydoctor generates links is deceptively simple. If you look up a module, like "twisted.internet.defer", the link looks like this: http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.html If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this: http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr... Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so: http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred... So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet. Second, I haven't yet figured out how to access the Sphinx config object from a docutils role function. I'm sure this is just a matter of finding an example or getting an answer from the mailing list, but I just haven't gotten to it yet. I think the API links are the only major issue still remaining until we start doing the conversion, so once this issue is handled, I'll probably file a ticket and create a branch in SVN for the conversion. If anyone has any docs tickets open, now is the time to get them fixed! And if anyone has some great ideas about how to organize the new docs in the repository, I'd love to hear them. As always, converted docs are here: http://twistedsphinx.funsize.net/ The lore2Sphinx Hg repos is here: http://bitbucket.org/khorn/lore2sphinx/ Comments, ideas, patches and pull requests gratefully accepted (or at least discussed). Kevin Horn

Attachments:

attachment.htm (text/html — 4.2 KB)

Show replies by thread

exarkun＠twistedmatrix.com

4 Mar 4 Mar

11:05 p.m.

On 10:49 pm, kevin.horn@gmail.com wrote:

...

If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this:

http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr...

Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so:

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet.

Did you notice that http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred... also exists? Jean-Paul

Kevin Horn

11:31 p.m.

On Thu, Mar 4, 2010 at 5:05 PM, wrote:

...

On 10:49 pm, kevin.horn@gmail.com wrote:

...
If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this:

http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr...

...
Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so:

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

...
So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet.

Did you notice that

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred... also exists?

Jean-Paul

I hadn't actually. If all functions and methods have a similar page in the PyDoctor output, then I think what I have now will pretty much work as-is (almost). Do we think that just generating links to the individual function/method pages is acceptable? Kevin Horn

Michael Hudson-Doyle

5 Mar 5 Mar

12:10 a.m.

On 5 March 2010 12:31, Kevin Horn wrote:

...

On Thu, Mar 4, 2010 at 5:05 PM, wrote:

...
On 10:49 pm, kevin.horn@gmail.com wrote:

...
If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this:

...
...
http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr...

Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so:

...
...
http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet.

Did you notice that

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred... also exists?

Jean-Paul

I hadn't actually. If all functions and methods have a similar page in the PyDoctor output, then I think what I have now will pretty much work as-is (almost). Do we think that just generating links to the individual function/method pages is acceptable?

It's what the lore-processed documentation currently links to. It would be nice to fix this (and I sort of vaguely know how), but that should in no way hold up the move away from lore! Cheers, mwh

Kevin Horn

12:16 a.m.

On Thu, Mar 4, 2010 at 6:10 PM, Michael Hudson-Doyle wrote:

...

...
On Thu, Mar 4, 2010 at 5:05 PM, wrote:

...
On 10:49 pm, kevin.horn@gmail.com wrote:

...
If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this:

...
...
http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr...

...
...
Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so:

...
...
http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet.

Did you notice that

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

...
also exists?

Jean-Paul

I hadn't actually. If all functions and methods have a similar page in

On 5 March 2010 12:31, Kevin Horn wrote: the

...
PyDoctor output, then I think what I have now will pretty much work as-is (almost). Do we think that just generating links to the individual function/method pages is acceptable?

It's what the lore-processed documentation currently links to. It would be nice to fix this (and I sort of vaguely know how), but that should in no way hold up the move away from lore!

Cheers, mwh

OK, if that's the way Lore does it, then I'm happy to call that "good enough for now". It seems to work here in my local version of the docs, I'll update the online stuff at http://twistedsphinx.funsize.net soonish. Kevin Horn

Drew Smathers

3:30 p.m.

On Thu, Mar 4, 2010 at 5:49 PM, Kevin Horn wrote:

...

It's been a while, I know you were all waiting with baited breath...

First a few fixes:

- Thanks to Tim Allen and Steve Steiner, several theme issues were fixed. This fixes a few minor display bugs and should make pages validate properly.

- inline markup will now include child contents. This causes rst not to display properly in cases of "nested inline markup", but should make it easier to

manually fix these places later

- :download: links are now generated for non-rst files ending in .py (most of the examples), as welll as .tac and .sql files

- toc entries which end in a "/" character now have "index appended to them in the ReST output, which fixes a few issues where the built-Sphinx docs would end up with broken navigation links.

A couple of issues were also fixed at the PyCon sprints:

- <cite> tags are now handled...sort of. According to Glyph, these aren't really very important, so we just return the contents of the tag without any special markup. If we feel the need to put citation markup back into the docs we can do that post-transition.

- Glyph also found an easy way to improve the CSS of the theme somewhat. The issue where the sidebar drops below the main content should now be less of a problem, though it's not gone entirely.

- Glyph and I also discussed a few different ways to handle links to the API docs and agreed on a way forward. Basically, well change the <code class='API'> tags into something like: :api:`path.to.documented object <label>`, and then create a docutils extension to convert that into a link. The beginnings of this are in the hg repos, but it looks like it won't be quite as simple as we thought.

First, the way pydoctor generates links is deceptively simple.

If you look up a module, like "twisted.internet.defer", the link looks like this:

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.html

If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this:

http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr...

Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so:

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet.

Second, I haven't yet figured out how to access the Sphinx config object from a docutils role function. I'm sure this is just a matter of finding an example or getting an answer from the mailing list, but I just haven't gotten to it yet.

I think the API links are the only major issue still remaining until we start doing the conversion, so once this issue is handled, I'll probably file a ticket and create a branch in SVN for the conversion.

If anyone has any docs tickets open, now is the time to get them fixed!

How can I filter for tickets related to lore2sphinx transition in trac? I've noticed a few documents that don't seem properly converted. Here's an example: http://twistedsphinx.funsize.net/projects/web/howto/web-in-60/asynchronous.h... ... which is missing syntax highlighting on code examples and the code examples are also truncated. -Drew

Kevin Horn

5:33 p.m.

On Fri, Mar 5, 2010 at 9:30 AM, Drew Smathers wrote:

...

On Thu, Mar 4, 2010 at 5:49 PM, Kevin Horn wrote:

...
It's been a while, I know you were all waiting with baited breath...

First a few fixes:

- Thanks to Tim Allen and Steve Steiner, several theme issues were fixed. This fixes a few minor display bugs and should make pages validate properly.

- inline markup will now include child contents. This causes rst not to display properly in cases of "nested inline markup", but should make it easier to manually fix these places later

- :download: links are now generated for non-rst files ending in .py (most of the examples), as welll as .tac and .sql files

- toc entries which end in a "/" character now have "index appended to them in the ReST output, which fixes a few issues where the built-Sphinx docs would end up with broken navigation links.

A couple of issues were also fixed at the PyCon sprints:

- <cite> tags are now handled...sort of. According to Glyph, these aren't really very important, so we just return the contents of the tag without any special markup. If we feel the need to put citation markup back into the docs we can do that post-transition.

- Glyph also found an easy way to improve the CSS of the theme somewhat. The issue where the sidebar drops below the main content should now be less of a problem, though it's not gone entirely.

- Glyph and I also discussed a few different ways to handle links to the API docs and agreed on a way forward. Basically, well change the <code class='API'> tags into something like: :api:`path.to.documented object <label>`, and then create a docutils extension to convert that into a link. The beginnings of this are in the hg repos, but it looks like it won't be quite as simple as we thought.

First, the way pydoctor generates links is deceptively simple.

If you look up a module, like "twisted.internet.defer", the link looks like this:

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.html

If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this:

http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferr...

Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so:

http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferred...

So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet.

Second, I haven't yet figured out how to access the Sphinx config object from a docutils role function. I'm sure this is just a matter of finding an example or getting an answer from the mailing list, but I just haven't gotten to it yet.

I think the API links are the only major issue still remaining until we start doing the conversion, so once this issue is handled, I'll probably file a ticket and create a branch in SVN for the conversion.

If anyone has any docs tickets open, now is the time to get them fixed!

How can I filter for tickets related to lore2sphinx transition in trac? I've noticed a few documents that don't seem properly converted. Here's an example:

http://twistedsphinx.funsize.net/projects/web/howto/web-in-60/asynchronous.h...

... which is missing syntax highlighting on code examples and the code examples are also truncated.

-Drew

As far as I know, there aren't any tickets in Trac yet. I was planning on making some (or asking others to do so) once a "master ticket" and a branch are created. As for the specific issue you linked to, the ReST markup is similar to that on other pages, but it looks like Sphinx isn't recognizing the code samples as Python for some reason. I'll look into this later on. Kevin Horn

Jonathan Lange

6 Jul 6 Jul

6:16 p.m.

On Fri, Mar 5, 2010 at 7:33 PM, Kevin Horn wrote: ...

...

As far as I know, there aren't any tickets in Trac yet. I was planning on making some (or asking others to do so) once a "master ticket" and a branch are created.

I have filed such a master ticket here: http://twistedmatrix.com/trac/ticket/4553 And created a milestone here: http://twistedmatrix.com/trac/milestone/Lore%20to%20Sphinx You should probably add any new tickets you file to the milestone. Personally, I'd love to see the transition happen. jml

Jonathan Lange

6:27 p.m.

On Tue, Jul 6, 2010 at 8:16 PM, Jonathan Lange wrote:

...

On Fri, Mar 5, 2010 at 7:33 PM, Kevin Horn wrote: ...

...
As far as I know, there aren't any tickets in Trac yet. I was planning on making some (or asking others to do so) once a "master ticket" and a branch are created.

I have filed such a master ticket here: http://twistedmatrix.com/trac/ticket/4553

And created a milestone here: http://twistedmatrix.com/trac/milestone/Lore%20to%20Sphinx

You should probably add any new tickets you file to the milestone.

Hey and look a ticket was already filed (#4500) and http://twistedsphinx.funsize.net/transition_plan.html is out of date :) jml

Kevin Horn

8:09 p.m.

On Tue, Jul 6, 2010 at 1:27 PM, Jonathan Lange wrote:

...

On Tue, Jul 6, 2010 at 8:16 PM, Jonathan Lange wrote:

...
On Fri, Mar 5, 2010 at 7:33 PM, Kevin Horn wrote: ...

...
As far as I know, there aren't any tickets in Trac yet. I was planning on making some (or asking others to do so) once a "master ticket" and a branch are created.

I have filed such a master ticket here: http://twistedmatrix.com/trac/ticket/4553

And created a milestone here: http://twistedmatrix.com/trac/milestone/Lore%20to%20Sphinx

You should probably add any new tickets you file to the milestone.

Hey and look a ticket was already filed (#4500) and http://twistedsphinx.funsize.net/transition_plan.html is out of date :)

jml

Doh! Sorry about that... I didn't realize anyone was actually looking at it. :) So now that Jonathan has cut the 10.1 release, my plan is to start the conversion process pretty soon (meaning sometime between tonight and Wednesday). Once I have the branch created and the basic conversion completed it would be really great to have some help with the (yet to be created) "chunk tickets". We're getting really close... Kevin Horn

exarkun＠twistedmatrix.com

10 Jul 10 Jul

2:58 a.m.

On 6 Jul, 08:09 pm, kevin.horn@gmail.com wrote:

...

So now that Jonathan has cut the 10.1 release, my plan is to start the conversion process pretty soon (meaning sometime between tonight and Wednesday).

Once I have the branch created and the basic conversion completed it would be really great to have some help with the (yet to be created) "chunk tickets".

We're getting really close...

At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here: http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/ (or with a different revision number for different revisions; the containing directory is browseable). It would be great if everyone could go look around and report any problems you see. Here's one from me. On <http://buildbot.twistedmatrix.com/builds /sphinx-html-15615/projects/core/howto/pb-intro.html>, there are a bunch of API links which don't appear to have been rendered properly. Jean-Paul

Glyph Lefkowitz

3:52 a.m.

On Jul 9, 2010, at 10:58 PM, exarkun@twistedmatrix.com wrote:

...

At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

Thanks for setting this up. In the table of contents, both the "Core" and "Conch" outlines have a section called "Twisted Documentation". I assume this is a placeholder for a directory that Lore didn't have a title for? It strikes me as somewhat ironic that the table on http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/lore/how... came out rather nicely, though.

Tim Allen

7:40 a.m.

On Sat, Jul 10, 2010 at 02:58:40AM -0000, exarkun@twistedmatrix.com wrote:

...

At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the containing directory is browseable).

It would be great if everyone could go look around and report any problems you see.

- In the Documentation Table Of Contents page, "Historical Documents" is listed between "Twisted Core" and "Twisted Lore" (presumably because of alphabetical order) but this seems a strange place for it. Could it be moved to the bottom, or an appendix, or something? - The bottom of every page has a "Download in other formats: Plain Text" link that doesn't work; presumably scooped up from the Trac templates. It should be removed (the Sphinx "Show Source" link is in the right-hand sidebar anyway). - When the breadcrumb navigation at the top of the page gets too long[1], it wraps to a second line, which looks kind of odd because the nice gradient background image loops. Changing the CSS background colour to be the same as the bottom of the gradient image would probably help. - Firefox's Error Console reports: - a bogus "%" on line 56 of twistedtrac.css, - bogus // comments on lines 559, 577, 588, 598 of trac.css (CSS only allows /* winged comments */) - Chromium's error console reports: - A 404 error for "/trac/chrome/common/js/jquery.js" - A 404 error for "/trac/chrome/common/js/trac.js" - A 404 error for "/trac/chrome/common/js/search.js" - A 404 error for "/builds/sphinx-html-15615/_static/dots.gif" - The Twisted.Conch tutorial[1] has a number of things surrounded with double-backticks like "``Deferred``" or "``ClientTransport``". I'm not sure if that's deliberate or if it's markup gone horribly wrong. - On the same page, the text ":api:` twisted.internet.interface.Transport < twisted.internet.interface.Transport>`" appears, which also looks wrong. - The Conch code examples (as linked from the code examples page[2]) are sent with a Python mime-type (which Firefox tries to download) while the .tac files are sent as text/html; this may be an artifact of the docs being hosted on the buildbot machine rather than Sphinx, but I think it would be nice if by default they were displayed in the browser, syntax-highlighted, with an option to download the original files (much like Trac does). - The Twisted Split FAQ[3] has more visible markup: "twisted.protocols:superscript::ref:`[1] `". The footnote target is kind of messed up, too. - The Twisted Split FAQ[3] has a heading "Why are arr the packages still named twisted.*subproject*?", but it's not obvious whether the author was trying to use italics (in which case it's a lore2sphinx bug) or globbing (in which case it's not). - The Twisted Zope Interfaces FAQ[4] has inline Python source examples that don't appear to be marked up properly (although this is possibly a problem with the original Lore source). The answer to question "How can I update my own code?" is one example. - The Banana Protocol Specifications[5], under the heading "Examples" mentions "the type bytes are marked in bold" but actually they're surrounded by double-asterisks. - The Twisted Coding Standard[6], mentions "the complete test suite in trunk@HEAD" - and "trunk@HEAD" is converted into a mailto: link. - Links followed by punctuation seem to leave a space between the link text and the punctuation. See "test-driven ." and "Test Standard ." in the Twisted Coding Standard[6], but I've observed the same problem on multiple pages. - The Twisted Coding Standard[6] mentions "If you modify, or write a new, HOWTO, please read the Lore documentation to learn how to format the docs"; if there's not already a follow-up ticket for the lore-sphinx conversion titled "Remove references to Lore from Twisted's non-Lore-related documentation", there should be, and this should be in it. - The Twisted Coding Standard[6], under the heading "Modules" says "Use this template:" and then "../listings/new_module_template.py".. and *then* includes the content of said template inline. Presumably it should either link to it or include it, but not both. - The HTML Documentation Standard for Twisted[7] has more visible markup: in the list of allowable markup, most tags are rendered properly except for "``<tr>``". - It seems that the HTML Documentation Standard for Twisted[7] is actually "how to write Lore documentation", despite the name. Maybe this file should be scheduled for the post-transition purge, or at least moved to the Lore documentation. - Working from Twisted's Subversion Repository[8] has visible markup that looks like an attempt to link "the Subversion homepage" under the heading "Checkout". - Under the heading "Alternate tree names", the document uses RIGHT DOUBLE QUOTATION MARK instead of LEFT DOUBLE QUOTATION MARK in 'directory other than”Twisted” .' Looks like a problem in the original markup. Also note the space between the LEFT DOUBLE QUOTATION MARK and the punctuation. - The same document has more broken links under the "Combinator" heading. - The same document has broken markup under "Running tests": "``twisted/protocols/imap4.py``" and later "``twisted.mail.test.test_imap``". - The same document has broken markup under "Building docs": "``doc/development/policy/svn-dev.xhtml`` ," (also trailing space before punctuation) and "``doc/development/policy/svn-dev.html`` :" (again). - The same document has a broken link under "Committing and Post-commit Hooks", attempting to link the text "trac-post-commit-hook". - The same document has broken markup under "Emacs": "``emacs/twisted-dev.el``". - The same document has a broken link under "Building Debian packages", attempting to link th text "stdeb". That's probably enough feedback to be getting on with; most of the problems appear to be from normalising "\n" in Lore docs to "" instead of " ", and also from adding whitespace after things. It is generally looking pretty great, though! Some extra thoughts: - The ReviewingDocumentation wiki page has a section called "Editing man pages" that describes how to turn the nicely-formatted manpages into Lore input files. Would it be possible to do that as part of the lore2sphinx run, have the manpages included in the Sphinx documentation, and from then on generate the manpages from the .rst files instead of the other way around? [1]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ conch/howto/conch_client.html> [2]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ conch/examples/index.html> [3]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/upgrades/2.0/split.html> [4]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/upgrades/2.0/components.html> [5]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/specifications/banana.html> [6]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/development/policy/coding-standard.html> [7]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/development/policy/doc-standard.html> [8]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/development/policy/svn-dev.html>

Kevin Horn

7:34 p.m.

On Fri, Jul 9, 2010 at 9:58 PM, wrote:

...

On 6 Jul, 08:09 pm, kevin.horn@gmail.com wrote:

...
So now that Jonathan has cut the 10.1 release, my plan is to start the conversion process pretty soon (meaning sometime between tonight and Wednesday).

Once I have the branch created and the basic conversion completed it would be really great to have some help with the (yet to be created) "chunk tickets".

We're getting really close...

At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the containing directory is browseable).

cool, good work.

...

It would be great if everyone could go look around and report any problems you see.

Here's one from me. On <http://buildbot.twistedmatrix.com/builds /sphinx-html-15615/projects/core/howto/pb-intro.html>, there are a bunch of API links which don't appear to have been rendered properly.

Ah, welcome to the joys of "nested inline markup", which xml/xhtml does fine and RestructuredText doesn't. I expect this issue and extra or missing space characters are the two main things which will have to be fixed manually. Kevin Horn

Kevin Horn

7:38 p.m.

On Fri, Jul 9, 2010 at 10:52 PM, Glyph Lefkowitz wrote:

...

On Jul 9, 2010, at 10:58 PM, exarkun@twistedmatrix.com wrote:

At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

Thanks for setting this up.

In the table of contents, both the "Core" and "Conch" outlines have a section called "Twisted Documentation". I assume this is a placeholder for a directory that Lore didn't have a title for?

Nope, that's what the titles of those documents are in the source documents. I presume the Conch one is supposed to say something else.

...

It strikes me as somewhat ironic that the table on < http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/lore/howto/lore.html> came out rather nicely, though.

Yes, I spent probably too much time on that table, but it sure was handy. I spent a lot of time during this process referring to it, and it was much nicer on the eyes that the old one :). I had originally intended to just convert the tables by hand (I think there's only two), but I found some code on the net that did almost what I wanted and adapted it, so it actually wasn't too onerous. Kevin Horn

Kevin Horn

8:17 p.m.

On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen wrote:

...

On Sat, Jul 10, 2010 at 02:58:40AM -0000, exarkun@twistedmatrix.com wrote:

...
At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the containing directory is browseable).

It would be great if everyone could go look around and report any problems you see.

- In the Documentation Table Of Contents page, "Historical Documents" is listed between "Twisted Core" and "Twisted Lore" (presumably because of alphabetical order) but this seems a strange place for it. Could it be moved to the bottom, or an appendix, or something?

They're in alphabetical order. The TOC page is generated dynamically by looking at the file structure of the docs and that's the order file-globbing lists them in. We can fix this manually post-conversion.

...

- The bottom of every page has a "Download in other formats: Plain Text" link that doesn't work; presumably scooped up from the Trac templates. It should be removed (the Sphinx "Show Source" link is in the right-hand sidebar anyway).

Yeah, this is a flaw in the trac theme. Looks like I forgot to remove it.

...

- When the breadcrumb navigation at the top of the page gets too long[1], it wraps to a second line, which looks kind of odd because the nice gradient background image loops. Changing the CSS background colour to be the same as the bottom of the gradient image would probably help. - Firefox's Error Console reports: - a bogus "%" on line 56 of twistedtrac.css, - bogus // comments on lines 559, 577, 588, 598 of trac.css (CSS only allows /* winged comments */) - Chromium's error console reports: - A 404 error for "/trac/chrome/common/js/jquery.js" - A 404 error for "/trac/chrome/common/js/trac.js" - A 404 error for "/trac/chrome/common/js/search.js" - A 404 error for "/builds/sphinx-html-15615/_static/dots.gif"

Ah! This explains why the search quit working. Not sure why these files are missing, Probably something to do with the custom theme.

...

- The Twisted.Conch tutorial[1] has a number of things surrounded with double-backticks like "``Deferred``" or "``ClientTransport``". I'm not sure if that's deliberate or if it's markup gone horribly wrong.

This is probably due to missing spaces before/after the markup in the rst source. One of those things that will need to be fixed manually. I spent a lot of time trying to fix this in lore2sphinx and when you fix it in one place, it breaks somewhere else. Right now the automated conversion is about as good as it is likely to get with a sane amount of effort.

...

- On the same page, the text ":api:` twisted.internet.interface.Transport < twisted.internet.interface.Transport>`" appears, which also looks wrong.

DItto.

...

- The Conch code examples (as linked from the code examples page[2]) are sent with a Python mime-type (which Firefox tries to download) while the .tac files are sent as text/html; this may be an artifact of the docs being hosted on the buildbot machine rather than Sphinx, but I think it would be nice if by default they were displayed in the browser, syntax-highlighted, with an option to download the original files (much like Trac does).

This is a web server configuration thing. The files are actually .py and .tac files (and probably need to remain so, if we ever want to get automated example code tests going). Maybe we can do some web server magic to get them nicely displayed in the browser, but I see that as a secondary issue for the moment. Anyone should feel free to give a shout if they disagree, though.

...

- The Twisted Split FAQ[3] has more visible markup: "twisted.protocols:superscript::ref:`[1] `". The footnote target is kind of messed up, too.

Another spacing issue.

...

- The Twisted Split FAQ[3] has a heading "Why are arr the packages still named twisted.*subproject*?", but it's not obvious whether the author was trying to use italics (in which case it's a lore2sphinx bug) or globbing (in which case it's not).

Pretty sure it was supposed to be italics. Spacing again. There are ways to make this display properly, but it'll need to be done manually. Also, this is pretty outdated stuff...we might consider just removing some of this. Or perhaps moving some of it into a more obvious place.

...

- The Twisted Zope Interfaces FAQ[4] has inline Python source examples that don't appear to be marked up properly (although this is possibly a problem with the original Lore source). The answer to question "How can I update my own code?" is one example.

The code bits are marked up this way because of the attributes used on the elements in the Lore source. We can change it easily

...

- The Banana Protocol Specifications[5], under the heading "Examples" mentions "the type bytes are marked in bold" but actually they're surrounded by double-asterisks.

Double asterisks are the markup for bold in rst. Another spacing and/or nested inline markup issue.

...

- The Twisted Coding Standard[6], mentions "the complete test suite in trunk@HEAD" - and "trunk@HEAD" is converted into a mailto: link.

Huh. I guess that's Sphinx trying to be helpful. Good catch. That'll definitely need fixing.

...

- Links followed by punctuation seem to leave a space between the link text and the punctuation. See "test-driven ." and "Test Standard ." in the Twisted Coding Standard[6], but I've observed the same problem on multiple pages.

This is the spacing issue again, but in reverse. It's relatively easy for lore2sphinx to detect links and add a space at the end, so that's what it does. There's ways to get around this and get rid of the space by escaping it, but it rapidly get mind-boggling, so this is another thing that will need manual fixing.

...

- The Twisted Coding Standard[6] mentions "If you modify, or write a new, HOWTO, please read the Lore documentation to learn how to format the docs"; if there's not already a follow-up ticket for the lore-sphinx conversion titled "Remove references to Lore from Twisted's non-Lore-related documentation", there should be, and this should be in it.

There isn't, but there will be. See the transition plan here: http://twistedsphinx.funsize.net/transition_plan.html

...

- The Twisted Coding Standard[6], under the heading "Modules" says "Use this template:" and then "../listings/new_module_template.py".. and *then* includes the content of said template inline. Presumably it should either link to it or include it, but not both.

It actually is including an external file there (i.e. the contents are not in the rst source, but in the "../listings/new_module_template.py" file. What looks like a link there is actually a sort of heading to let the reader see where it's coming from. I think this can probably be made clearer with some theme/css changes.

...

- The HTML Documentation Standard for Twisted[7] has more visible markup: in the list of allowable markup, most tags are rendered properly except for "``<tr>``".

Missing space after preceding comma. Another manual fix.

...

- It seems that the HTML Documentation Standard for Twisted[7] is actually "how to write Lore documentation", despite the name. Maybe this file should be scheduled for the post-transition purge, or at least moved to the Lore documentation.

It pretty much is "how to write Lore documentation". I think it should probably be moved into the Lore docs, and replaced with the (yet to be written, but again see the transition plan) planned Twisted Documentation Guide.

...

- Working from Twisted's Subversion Repository[8] has visible markup that looks like an attempt to link "the Subversion homepage" under the heading "Checkout".

Missing preceding space again.

...

- Under the heading "Alternate tree names", the document uses RIGHT DOUBLE QUOTATION MARK instead of LEFT DOUBLE QUOTATION MARK in 'directory other than”Twisted” .' Looks like a problem in the original markup. Also note the space between the LEFT DOUBLE QUOTATION MARK and the punctuation.

The missing preceding space actually is what's causing the quotes to be wrong.

...

- The same document has more broken links under the "Combinator" heading.

Missing preceding spaces again.

...

- The same document has broken markup under "Running tests": "``twisted/protocols/imap4.py``" and later "``twisted.mail.test.test_imap``". - The same document has broken markup under "Building docs": "``doc/development/policy/svn-dev.xhtml`` ," (also trailing space before punctuation) and "``doc/development/policy/svn-dev.html`` :" (again). - The same document has a broken link under "Committing and Post-commit Hooks", attempting to link the text "trac-post-commit-hook". - The same document has broken markup under "Emacs": "``emacs/twisted-dev.el``". - The same document has a broken link under "Building Debian packages", attempting to link th text "stdeb".

All bad spacing issues.

...

That's probably enough feedback to be getting on with; most of the problems appear to be from normalising "\n" in Lore docs to "" instead of " ", and also from adding whitespace after things. It is generally looking pretty great, though!

Yeah, that's pretty much it. As I said above though, if you "fix" it one place, it breaks in another, so I tried to balance things in such a way that the least broken markup appears in the output. Almost all of the remaining problems will need to be fixed manually.

...

Some extra thoughts: - The ReviewingDocumentation wiki page has a section called "Editing man pages" that describes how to turn the nicely-formatted manpages into Lore input files. Would it be possible to do that as part of the lore2sphinx run, have the manpages included in the Sphinx documentation, and from then on generate the manpages from the .rst files instead of the other way around?

Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this. So you're suggesting convert the man pages to Lore format -> use lore2sphinx to convert the resulting Lore docs to rst -> build as part of the Sphinx process, yes? I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket). lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around.

...

[1]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ conch/howto/conch_client.html> [2]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ conch/examples/index.html> [3]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/upgrades/2.0/split.html> [4]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/upgrades/2.0/components.html> [5]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/specifications/banana.html> [6]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/development/policy/coding-standard.html> [7]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/development/policy/doc-standard.html> [8]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/ core/development/policy/svn-dev.html>

Thanks for the fantastic (and nicely detailed) feedback, Tim! Please take a look at the transition plan. In a few days (maybe sooner), I should have the base docs in a branch, and the "chunk tickets" referenced in the transition plan created. This is pretty much _exactly_ what I'd like to see in those "chunk tickets". Hopefully you haven't already burned up your brain staring at markup issues. :) We could really use this kind of help throughout the process. Kevin Horn

Tim Allen

11 Jul 11 Jul

1:19 a.m.

On Sat, Jul 10, 2010 at 03:17:03PM -0500, Kevin Horn wrote:

...

On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen wrote:

...
That's probably enough feedback to be getting on with; most of the problems appear to be from normalising "\n" in Lore docs to "" instead of " ", and also from adding whitespace after things. It is generally looking pretty great, though!

Yeah, that's pretty much it. As I said above though, if you "fix" it one place, it breaks in another, so I tried to balance things in such a way that the least broken markup appears in the output.

Almost all of the remaining problems will need to be fixed manually.

It sounds like a lot of the manual fixes will involve rewriting things so that inline markup does not appear at the end of a sentence next to the punctuation. That seems like a recipe for awkward prose. :/

...

...
Some extra thoughts: - The ReviewingDocumentation wiki page has a section called "Editing man pages" that describes how to turn the nicely-formatted manpages into Lore input files. Would it be possible to do that as part of the lore2sphinx run, have the manpages included in the Sphinx documentation, and from then on generate the manpages from the .rst files instead of the other way around?

Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this.

So you're suggesting convert the man pages to Lore format -> use lore2sphinx to convert the resulting Lore docs to rst -> build as part of the Sphinx process, yes?

Yes. At least for trial, there's a bunch of stuff that's *only* in the man page and not the online docs, and a bunch of stuff that's *only* in the online docs and not the man page (and stuff that's *only* in the core/development/policy section of the docs, and not in the Trial section...). Hopefully if they're all part of the same doc system, it'll be easier to have everything in the one place and easy to find.

...

I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket). lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around.

Something to add to the "open ticket for: anything else???" section of your transition plan, then? :)

...

Thanks for the fantastic (and nicely detailed) feedback, Tim!

Please take a look at the transition plan. In a few days (maybe sooner), I should have the base docs in a branch, and the "chunk tickets" referenced in the transition plan created. This is pretty much _exactly_ what I'd like to see in those "chunk tickets".

Having done pretty much the first two of your suggested "chunks" in my previous mail, they look to be about the right size.

...

Hopefully you haven't already burned up your brain staring at markup issues. :) We could really use this kind of help throughout the process.

I'm looking forward to learning a little more about Sphinx and ReST. :)

Glyph Lefkowitz

2:21 a.m.

On Jul 10, 2010, at 4:17 PM, Kevin Horn wrote:

...

On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen wrote: On Sat, Jul 10, 2010 at 02:58:40AM -0000, exarkun@twistedmatrix.com wrote:

...
At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the containing directory is browseable).

It would be great if everyone could go look around and report any problems you see.

- In the Documentation Table Of Contents page, "Historical Documents" is listed between "Twisted Core" and "Twisted Lore" (presumably because of alphabetical order) but this seems a strange place for it. Could it be moved to the bottom, or an appendix, or something?

They're in alphabetical order. The TOC page is generated dynamically by looking at the file structure of the docs and that's the order file-globbing lists them in. We can fix this manually post-conversion.

It's worth noting that even in this peculiar order, the existence of this document is an upgrade. It did not previously exist. Or, rather, it did, but I think the analogous page in the current system is this: http://twistedmatrix.com/documents/10.1.0/, which is certainly not as nice. (If you were expecting to see something like http://twistedmatrix.com/documents/10.1.0/core/howto/index.html, that's here: http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/core/how....)

...

- The bottom of every page has a "Download in other formats: Plain Text" link that doesn't work; presumably scooped up from the Trac templates. It should be removed (the Sphinx "Show Source" link is in the right-hand sidebar anyway).

Yeah, this is a flaw in the trac theme. Looks like I forgot to remove it.

This seems like a simple fix - if you could do it quickly so we can see if fixes get propagated to the buildbot, that would be great (so we can get the build logistics out of the way for future, more complex issues).

...

- When the breadcrumb navigation at the top of the page gets too long[1], it wraps to a second line, which looks kind of odd because the nice gradient background image loops. Changing the CSS background colour to be the same as the bottom of the gradient image would probably help.

The breadcrumb names are also kinda weird, and very wordy. http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/conch/ho... says "Twisted >> Twisted Documentation Table of Contents >> Twisted Conch Documentation >> Twisted Documentation". This is altogether too many "Twisted"s ;-). It should really be more like 'Twisted >> Documentation >> Conch >> "Writing a client with Twisted.Conch"' Again: this is a new feature, so I'm not too concerned if we defer fixing this; go ahead if it's easy, but if it's hard, it's still better than the status quo, which is to say, not having any navigation at all.

...

- Chromium's error console reports: - A 404 error for "/trac/chrome/common/js/jquery.js" - A 404 error for "/trac/chrome/common/js/trac.js" - A 404 error for "/trac/chrome/common/js/search.js" - A 404 error for "/builds/sphinx-html-15615/_static/dots.gif"

Ah! This explains why the search quit working. Not sure why these files are missing, Probably something to do with the custom theme.

They're missing because this code is hosted on buildbot.twistedmatrix.com, not twistedmatrix.com. It's probably most expedient to just make those links include the netloc for now.

...

- The Twisted.Conch tutorial[1] has a number of things surrounded with double-backticks like "``Deferred``" or "``ClientTransport``". I'm not sure if that's deliberate or if it's markup gone horribly wrong.

This is probably due to missing spaces before/after the markup in the rst source. One of those things that will need to be fixed manually. I spent a lot of time trying to fix this in lore2sphinx and when you fix it in one place, it breaks somewhere else. Right now the automated conversion is about as good as it is likely to get with a sane amount of effort.

Pardon my ignorance, but why not just always add the spaces when generating this type of markup? It doesn't seem like it should be that hard; docutils claims to be able to treat ReST as structured, right? Isn't there an API to construct some objects that will do the quoting for you?

...

- The Conch code examples (as linked from the code examples page[2]) are sent with a Python mime-type (which Firefox tries to download) while the .tac files are sent as text/html; this may be an artifact of the docs being hosted on the buildbot machine rather than Sphinx, but I think it would be nice if by default they were displayed in the browser, syntax-highlighted, with an option to download the original files (much like Trac does).

This is a web server configuration thing. The files are actually .py and .tac files (and probably need to remain so, if we ever want to get automated example code tests going).

...

Maybe we can do some web server magic to get them nicely displayed in the browser, but I see that as a secondary issue for the moment. Anyone should feel free to give a shout if they disagree, though.

I agree completely. We should fix it so the .tac files are text/x-python as well, so your browser can DTRT with them; when I click on an example, I often want to download it. Perhaps the lore markup should be adjusted to format them inline as well, though?

...

- The Twisted Split FAQ[3] has a heading "Why are arr the packages still named twisted.*subproject*?", but it's not obvious whether the author was trying to use italics (in which case it's a lore2sphinx bug) or globbing (in which case it's not).

Pretty sure it was supposed to be italics. Spacing again. There are ways to make this display properly, but it'll need to be done manually.

Also, this is pretty outdated stuff...we might consider just removing some of this. Or perhaps moving some of it into a more obvious place.

Let's try to have those discussions in separate threads so they can be resolved individually. I don't have an opinion at the moment and would not like to develop one in the middle of a monster thread about the whole conversion :).

...

- The Twisted Coding Standard[6] mentions "If you modify, or write a new, HOWTO, please read the Lore documentation to learn how to format the docs"; if there's not already a follow-up ticket for the lore-sphinx conversion titled "Remove references to Lore from Twisted's non-Lore-related documentation", there should be, and this should be in it.

There isn't, but there will be. See the transition plan here: http://twistedsphinx.funsize.net/transition_plan.html

FWIW, you should really start linking to the ticket http://twistedmatrix.com/trac/ticket/4500, rather than the "transition plan" page, so that folks can comment, link to other tickets, bookmark it for review when it comes up, etc.

...

[...] All bad spacing issues. [...] Almost all of the remaining problems will need to be fixed manually.

It really seems like there's a lot of these. I'm grateful to Tim for such a thorough job finding them, but I'm sure he missed some - maybe it would be worthwhile to get someone else to help make the tool be correct more often?

...

Thanks for the fantastic (and nicely detailed) feedback, Tim!

+1. (I had some thoughts about manual pages but those deserved their own message, so I'll send them separately.)

Glyph Lefkowitz

2:42 a.m.

On Jul 10, 2010, at 4:17 PM, Kevin Horn wrote:

...

Some extra thoughts: - The ReviewingDocumentation wiki page has a section called "Editing man pages" that describes how to turn the nicely-formatted manpages into Lore input files. Would it be possible to do that as part of the lore2sphinx run, have the manpages included in the Sphinx documentation, and from then on generate the manpages from the .rst files instead of the other way around?

Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this.

So you're suggesting convert the man pages to Lore format -> use lore2sphinx to convert the resulting Lore docs to rst -> build as part of the Sphinx process, yes?

I think there may be some confusion here. However, the current situation is extremely confused, so I will try to shed some light... First, the goal: we should include the content of the man pages that you get from a UNIX command prompt, but formatted in a manner consistent with the rest of our documentation. As Tim noted in another message:

...

At least for trial, there's a bunch of stuff that's *only* in the man page and not the online docs, and a bunch of stuff that's *only* in the online docs and not the man page (and stuff that's *only* in the core/development/policy section of the docs, and not in the Trial section...). Hopefully if they're all part of the same doc system, it'll be easier to have everything in the one place and easy to find.

It would definitely be better to have this information more centrally located and maintained. This is a good goal, and something we are nominally already supposed to be doing. (And in the process of writing this up, I discovered, oops, we're not, but more on that later.) Lore already has a plugin which attempts to convert man pages (groff input) into HTML. To be clear: it creates lore *input* documents, which are intended to be an intermediate state; they're supposed to be re-processed in the same way as the rest of the lore input, to produce lore output. Lest it seem like editing groff directly is a thing only an insane person would want to do (and arguably, Moshe Zadka, the author of this feature, is in fact an insane person (we love you Moshe)), he designed it this way for a good reason. The thinking at the time among several Twisted developers was that it would be more useful to go from groff (which has a fairly limited vocabulary) to HTML (which is pretty expressive) than the other way around, since you'd have to give up if you ran into some HTML nuance that couldn't be expressed in groff. This is slightly good news for including the manpages in the current process being tested on the buildbot, because it just means you have to add one additional command-line to generate the input, and the .rst output can be generated along the same lines as what is already being done. At least, this is all you need to do to get HTML that probably looks reasonable. Getting manpages out is another matter. Unfortunately there's no 'man2rst', so we can't keep man pages around as input for sphinx. So, in order to continue with the same workflow we've had - with groff as source - we'd have to do several undesirable things: Produce a convoluted document conversion process where we start with groff input, go through HTML, then end up with rst, then some more, different HTML generated from that. One goal of this process would be for the document generation pipeline to be a simple, single command (I am hoping something like 'sphinx doc' and that's it...). Keep maintaining lore2sphinx. This was an explicit non-goal of the migration process, given that there are going to be some manual fixes required. (Despite my suggestion that I think we should have fewer of these, I don't think that maintaining it long-term is a good idea.) Keep maintaining lore, since that's where the manpage conversion code lives. Since the whole point of this process is so that we can dump lore out of the nearest airlock, this strikes me as a dealbreaker by itself. There is, however, an 'rst2man'. http://docutils.sourceforge.net/sandbox/manpage-writer/rst2man.txt, which appears to be the basis for sphinx's manpage generator. And this is where we encounter the problem where it's somewhat difficult to get manpages as output: it seems to have exactly the issues that the lore groff-input thingy was designed to prevent. It just chokes and dies on groff-unfriendly markup, and it requires several annotations which can't be provided in the HTML input. That means the output generated from the groff->lore->rst pipeline will be missing a bunch of stuff that rst2man wants, such as ":Manual section:" and ":Manual group:" fields. Which, in turn, means that this task will require a distinct set of manual (no pun intended) post-processing that the rest of the documents won't. Of course, these limitations and the extra work might be worth living with for the reasons listed at the beginning of this message. But there's good(-ish) news. We don't currently generate readable HTML from manpages anywhere. So if we don't have it in the next release, it won't be a regression. The code used by the current release process, here - http://twistedmatrix.com/trac/wiki/ReleaseProcess#Buildhowtodocumentsforwebs... - has a bug in it, where it generates the lore input, but then uses a dictionary to avoid processing those generated input documents. Nobody ever sees this half-generated input, because it's only available here http://twistedmatrix.com/documents/10.1.0/core/man/, a secret index to which I can't find a link anywhere in the browsable bits of the lore documentation! Anyway, I apologize for the extremely long-winded enumeration, because ultimately it's just to say I agree with this:

...

I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket). lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around.

I shouldn't let all the effort of composing this message go to waste though, so I'll be more specific. We should use lore and then lore2sphinx to convert these documents to rst, in a one-shot process similar to the one we're using to convert the rest of the documentation, manually add in the information required by 'rst2man', and add 'rst2man' as part of our build process. Of course, part of this task will be making sure that the documentation buildbot turns red if we have any rst in the desginated 'man' documents that doesn't get turned into valid groff. And we'll have to keep lore in the tree until this is done, because otherwise it'll be a pain to get the code installed for the man-page-conversion pipeline. Again, since we're lacking it now, I don't think this is a high-priority task. I would be happy to have even a couple of releases with no pretense at HTML manpages, until we can do this separate conversion. Until then, I encourage our users to please enjoy these fine links from die.net: http://linux.die.net/man/1/twistd, http://linux.die.net/man/1/trial, http://linux.die.net/man/1/conch, http://linux.die.net/man/1/cftp, http://linux.die.net/man/1/lore. (Since this is so low-priority, It's too bad I just spent over an hour writing an email about it. Oh well.)

Kevin Horn

2:48 a.m.

On Sat, Jul 10, 2010 at 8:19 PM, Tim Allen wrote:

...

...
On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen wrote:

...
That's probably enough feedback to be getting on with; most of the problems appear to be from normalising "\n" in Lore docs to "" instead of " ", and also from adding whitespace after things. It is generally looking pretty great, though!

Yeah, that's pretty much it. As I said above though, if you "fix" it one place, it breaks in another, so I tried to balance things in such a way

On Sat, Jul 10, 2010 at 03:17:03PM -0500, Kevin Horn wrote: that

...
the least broken markup appears in the output.

Almost all of the remaining problems will need to be fixed manually.

It sounds like a lot of the manual fixes will involve rewriting things so that inline markup does not appear at the end of a sentence next to the punctuation. That seems like a recipe for awkward prose. :/

Not necessarily. There are ways around a lot of these things, but they make the markup a bit more complex and thus more complicated to generate automatically. For example, if the markup requires that there be a space, but you don't actually want a space there, you insert an 'escaped space' ("\ ", that's a backslash-space). But the exact rules for what can abut the beginning and end of inline markup are more complicated than that, and accounting for every special case would be....er... Let's just say there's a reason that there's not an official docutils tool to _create_ restructuredText.

...

...
...
Some extra thoughts: - The ReviewingDocumentation wiki page has a section called "Editing man pages" that describes how to turn the nicely-formatted manpages into Lore input files. Would it be possible to do that as part of the lore2sphinx run, have the manpages included in the Sphinx documentation, and from then on generate the manpages from the .rst files instead of the other way around?

Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this.

So you're suggesting convert the man pages to Lore format -> use lore2sphinx to convert the resulting Lore docs to rst -> build as part of the Sphinx process, yes?

Yes. At least for trial, there's a bunch of stuff that's *only* in the man page and not the online docs, and a bunch of stuff that's *only* in the online docs and not the man page (and stuff that's *only* in the core/development/policy section of the docs, and not in the Trial section...). Hopefully if they're all part of the same doc system, it'll be easier to have everything in the one place and easy to find.

...
I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket). lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around.

Something to add to the "open ticket for: anything else???" section of your transition plan, then? :)

Sounds like it.

...

...
Thanks for the fantastic (and nicely detailed) feedback, Tim!

Please take a look at the transition plan. In a few days (maybe sooner), I should have the base docs in a branch, and the "chunk tickets" referenced in the transition plan created. This is pretty much _exactly_ what I'd like to see in those "chunk tickets".

Having done pretty much the first two of your suggested "chunks" in my previous mail, they look to be about the right size.

...
Hopefully you haven't already burned up your brain staring at markup issues. :) We could really use this kind of help throughout the process.

I'm looking forward to learning a little more about Sphinx and ReST. :)

Kevin Horn

Glyph Lefkowitz

3:39 a.m.

On Jul 10, 2010, at 10:48 PM, Kevin Horn wrote:

...

Let's just say there's a reason that there's not an official docutils tool to _create_ restructuredText.

Is that really true? If so, it concerns me. Isn't that what docutils.writer.UnfilteredWriter does? Even if not, it should be possible - reasonable, even - to write a docutils.writer.Writer which just emits reST. Sphinx is a nice tool, but reST's grossness bothers me more and more. If there are really no tools which can emit reStructuredText from a data structure, that means we have no hope of ever processing our source documentation in deterministic, correct ways (as a trivial example, let's say we wanted to change the name of a package). It's not a lot of fun to process XML documents, but there are plenty of APIs to do it and to be sure that the output is still XML when you're done. This is really a theoretical concern (in the years we've been using lore, we never really managed to use any XML tools with it), and on balance all the features that sphinx brings to the table are almost certainly worth living with it, but it still bugs me. Where should I file and/or vote for a bug report or feature request? Is this sphinx's problem, docutils, or both?

Kevin Horn

3:49 a.m.

On Sat, Jul 10, 2010 at 9:21 PM, Glyph Lefkowitz wrote:

...

On Jul 10, 2010, at 4:17 PM, Kevin Horn wrote:

On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen wrote:

...
On Sat, Jul 10, 2010 at 02:58:40AM -0000, exarkun@twistedmatrix.comwrote:

...
At last I've got a buildbot set up generating the sphinx docs. The build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the containing directory is browseable).

It would be great if everyone could go look around and report any problems you see.

- In the Documentation Table Of Contents page, "Historical Documents" is listed between "Twisted Core" and "Twisted Lore" (presumably because of alphabetical order) but this seems a strange place for it. Could it be moved to the bottom, or an appendix, or something?

They're in alphabetical order. The TOC page is generated dynamically by looking at the file structure of the docs and that's the order file-globbing lists them in. We can fix this manually post-conversion.

It's worth noting that even in this peculiar order, the existence of this document is an upgrade. It did not previously exist. Or, rather, it did, but I think the analogous page in the current system is this: < http://twistedmatrix.com/documents/10.1.0/>, which is certainly not as nice.

(If you were expecting to see something like < http://twistedmatrix.com/documents/10.1.0/core/howto/index.html>, that's here: < http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/core/how...

...
.)

As I said, the order can be changed manually.

...

...
- The bottom of every page has a "Download in other formats: Plain Text" link that doesn't work; presumably scooped up from the Trac templates. It should be removed (the Sphinx "Show Source" link is in the right-hand sidebar anyway).

Yeah, this is a flaw in the trac theme. Looks like I forgot to remove it.

This seems like a simple fix - if you could do it quickly so we can see if fixes get propagated to the buildbot, that would be great (so we can get the build logistics out of the way for future, more complex issues).

Hmmm...it won't be propagated to the buildbot automatically, since it's in the theme files in the lore2sphinx repo. From what exarkun said it sounds like it would require another manual checkout of that repo. I'll change it pretty soon, though.

...

- When the breadcrumb navigation at the top of the page gets too

...
long[1], it wraps to a second line, which looks kind of odd because the nice gradient background image loops. Changing the CSS background colour to be the same as the bottom of the gradient image would probably help.

The breadcrumb names are also kinda weird, and very wordy.

< http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/conch/howto/conch_client.html> says "Twisted >> Twisted Documentation Table of Contents >> Twisted Conch Documentation >> Twisted Documentation". This is altogether too many "Twisted"s ;-).

It should really be more like

'Twisted >> Documentation >> Conch >> "Writing a client with Twisted.Conch"'

Again: this is a new feature, so I'm not too concerned if we defer fixing this; go ahead if it's easy, but if it's hard, it's still better than the status quo, which is to say, not having any navigation at all.

The breadcrumbs use document titles. Of course, in my opinion the word Twisted is used far too often in the document titles, which I think will be less necessary as we move the docs toward a cohesive whole, rather than a bunch of separate articles (which I see as a long-term, out-of-scope-for-this-discussion goal).

...

- Chromium's error console reports:

...
- A 404 error for "/trac/chrome/common/js/jquery.js" - A 404 error for "/trac/chrome/common/js/trac.js" - A 404 error for "/trac/chrome/common/js/search.js" - A 404 error for "/builds/sphinx-html-15615/_static/dots.gif"

Ah! This explains why the search quit working. Not sure why these files are missing, Probably something to do with the custom theme.

They're missing because this code is hosted on *buildbot*. twistedmatrix.com, not twistedmatrix.com. It's probably most expedient to just make those links include the netloc for now.

<facepalm> of course.

...

- The Twisted.Conch tutorial[1] has a number of things surrounded

...
with double-backticks like "``Deferred``" or "``ClientTransport``". I'm not sure if that's deliberate or if it's markup gone horribly wrong.

This is probably due to missing spaces before/after the markup in the rst source. One of those things that will need to be fixed manually. I spent a lot of time trying to fix this in lore2sphinx and when you fix it in one place, it breaks somewhere else. Right now the automated conversion is about as good as it is likely to get with a sane amount of effort.

Pardon my ignorance, but why not just *always* add the spaces when generating this type of markup? It doesn't seem like it should be that hard; docutils claims to be able to treat ReST as structured, right? Isn't there an API to construct some objects that will do the quoting for you?

It sounds reasonable doesn't it? But it's harder than that, because extra spaces in the wrong place can throw off indentation, which is a whole other mess. I would much rather go back through and add/remove a bunch of isolated spaces than try to reconstruct indentation in some of these complex document structures.

...

- The Conch code examples (as linked from the code examples page[2])

...
are sent with a Python mime-type (which Firefox tries to download) while the .tac files are sent as text/html; this may be an artifact of the docs being hosted on the buildbot machine rather than Sphinx, but I think it would be nice if by default they were displayed in the browser, syntax-highlighted, with an option to download the original files (much like Trac does).

This is a web server configuration thing. The files are actually .py and .tac files (and probably need to remain so, if we ever want to get automated example code tests going).

Maybe we can do some web server magic to get them nicely displayed in the browser, but I see that as a secondary issue for the moment. Anyone should feel free to give a shout if they disagree, though.

I agree completely.

We should fix it so the .tac files are text/x-python as well, so your browser can DTRT with them; when I click on an example, I often want to download it.

Perhaps the lore markup should be adjusted to format them inline as well, though?

- The Twisted Split FAQ[3] has a heading "Why are arr the packages

...
still named twisted.*subproject*?", but it's not obvious whether the author was trying to use italics (in which case it's a lore2sphinx bug) or globbing (in which case it's not).

Pretty sure it was supposed to be italics. Spacing again. There are ways to make this display properly, but it'll need to be done manually.

Also, this is pretty outdated stuff...we might consider just removing some of this. Or perhaps moving some of it into a more obvious place.

Let's try to have those discussions in separate threads so they can be resolved individually. I don't have an opinion at the moment and would not like to develop one in the middle of a monster thread about the whole conversion :).

Good idea.

...

- The Twisted Coding Standard[6] mentions "If you modify, or write

...
a new, HOWTO, please read the Lore documentation to learn how to format the docs"; if there's not already a follow-up ticket for the lore-sphinx conversion titled "Remove references to Lore from Twisted's non-Lore-related documentation", there should be, and this should be in it.

There isn't, but there will be. See the transition plan here: http://twistedsphinx.funsize.net/transition_plan.html

FWIW, you should really start linking to the ticket < http://twistedmatrix.com/trac/ticket/4500>, rather than the "transition plan" page, so that folks can comment, link to other tickets, bookmark it for review when it comes up, etc.

Hmmm...maybe. Though the transition plan (which I guess I should update some) covers more than the ticket, IMO. I _did_ ask for feedback on it back when, but nobody spoke up...and silence implies consent :) At any rate, if there need to be changes made to my sketch of a transition plan, let's get that figured out ASAP.

...

[...] All bad spacing issues. [...] Almost all of the remaining problems will need to be fixed manually.

It really seems like there's a lot of these. I'm grateful to Tim for such a thorough job finding them, but I'm sure he missed some - maybe it would be worthwhile to get someone else to help make the tool be correct more often?

If someone else can do it great. What's there at the moment is about as good as I can make it (unless someone can pay me truly ridiculous amounts of money...I always have that proviso :) ).

...

Thanks for the fantastic (and nicely detailed) feedback, Tim!

+1.

(I had some thoughts about manual pages but those deserved their own message, so I'll send them separately.)

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

5031

Age (days ago)

5160

Last active (days ago)

List overview

Download

21 comments

7 participants

participants (7)

Drew Smathers
exarkun＠twistedmatrix.com
Glyph Lefkowitz
Jonathan Lange
Kevin Horn
Michael Hudson-Doyle
Tim Allen