[Twisted-Python] Proposal (almost) to switch from Lore to Sphinx
Back in July, there was a long discussion thread regarding Twisted Documentation, and the idea of switching to Sphinx came up. Here's probably the culmination of that thread: http://twistedmatrix.com/pipermail/twisted-python/2009-July/020114.html To summarize, Glyph laid out the following items that would need to happen in order for Twisted to move from Lore to another documentation system: 1. Someone would have to volunteer to maintain the documentation infrastructure and teach others to use it. 2. Submit a complete proposal outlining benefits for the move so that it could be discussed rationally without having to chase down lots of little bits in email, irc, etc. 3. Address the transition process. So I went away and did some thinking, and some research and some experimenting, and I decided that I really want this to happen. So I am hereby volunteering to manage a Sphinx-based documentation infrastructure (should there eventually be one) for the Twisted project as well as the transition to said system. I have written up a proposal here: http://twistedsphinx.funsize.net/proposal.html that I think addresses at least the main points of what should need to happen in order to move from Lore to Sphinx. It's not quite finished, but I think it's close enough for people to start discussing it. In particular, there are a few claims I've made about Lore and it's capabilities that I'm not quite sure of (just search for the question marks). I would very much like to get some feedback from the community (especially the core developers) on this proposal. I'd especially like to hear about: - things which are not addressed and should be. - anything that is addressed in the proposal, but you think should be done in a different way. - Lore's capabilities. Please let me know what you think. Thanks, Kevin Horn
On Nov 11, 2009, at 5:52 PM, Kevin Horn wrote:
Please let me know what you think.
Initial reaction: awesome! I don't really care about the documentation format or tools, I'm just thrilled that someone is willing to step forward and maintain them in _any_ form. Secondary reaction: automated testing of code and examples is great. Necessary, even, especially for a new tool. But doctests are not so much a testing plan as a disease that comes to infest your code. (You are free to disagree with me about the utility or quality of doctest as a tool, but doing so on this mailing list is counterproductive, as my mind is pretty well made up. Despite the fact that I have fond feelings towards lore's design, this is the *only* point I'd veto a migration over.) Automated tests for documentation should look roughly like this: http://divmod.org/trac/browser/trunk/Nevow/nevow/test/test_howtolistings.py?... but the stuff that does "sys.modules.clear()" and so on should be a lot further away from the actual test cases. The only thing that is necessary for testing like that to work is the ability to reference an external .py file as a code listing. This is also a good thing because then users can just run the Python files straight in the examples directory, rather than copying some stuff from a web browser into a text editor, mucking around with path variables for a while, trying to make sure they got the indentation right etc. Can sphinx do that out of the box? Tertiary reaction, since I had to go to divmod.org to dig up that example: a migration plan should include the Divmod projects, especially Nevow, since a lot of people use that. But, if there's an automated tool that can handle the Twisted conversion, there really shouldn't be any additional effort to just run it over the Divmod codebase and just check in the result. The only real work will be in adjusting the buildbot. This is really just an argument for an automated, repeatable conversion process rather than manual editing.
On Wed, Nov 11, 2009 at 5:56 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Well, that's encouraging. :)
I never use doctests in my own projects...but the functionality comes with Sphinx (as a "built-in extension"...which such a fantastic oxymoron it makes me smile). There's lot's of other work being done in the Sphinx testing arena, though, notably: Sphinxtest: http://groups.google.com/group/sphinx-dev/browse_thread/thread/fd9fc42648fc3... Manuel: http://packages.python.org/manuel/ and something-I-read-about-the-other-day-but-can't-find-now.
Yes. The "literalinclude" directive pulls in an external file and sticks it in a literal block (the ReST equivalent of a <pre> tag). So you could use whatever testing framework you wanted and just reference those files.
Hmmm...I hadn't considered the Divmod stuff, but you're right. I think the automated conversion plan is the way to go...up to a point. As I laid out in the proposal, I think at some point, the Sphinx docs (and the automated conversion process) will become "good enough", and then edits should be done manually to the ReST/Sphinx source files, rather than the Lore files. Otherwise you just end up with another tool to maintain forever. And in my opinion, Twisted needs to get away from having it's own collection of self-maintained tools as much as possible. :) I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work. There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some. I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible. That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds). If there's interest, I can put up what I can generate to this point, though it's awfully early days yet. Also: Were my claims/assumption about Lore pretty much correct? Did I miss anything? Kevin
On Nov 11, 2009, at 7:37 PM, Kevin Horn wrote:
Please! Be encouraged! And therefore maintain our documentation and documentation toolchain! :)
There's lot's of other work being done in the Sphinx testing arena, though, notably: ...
It's great to see that the testing vibe is out there!
Can sphinx do that out of the box?
Yes. The "literalinclude" directive pulls in an external file and sticks it in a literal block (the ReST equivalent of a <pre> tag). So you could use whatever testing framework you wanted and just reference those files.
Excellent. In that case (modulo several weeks of work on your part), I'm sold.
Hmmm...I hadn't considered the Divmod stuff, but you're right.
I'm glad you think so. I do tend to be, especially about Twisted ;-).
I think the automated conversion plan is the way to go...up to a point. As I laid out in the proposal, I think at some point, the Sphinx docs (and the automated conversion process) will become "good enough", and then edits should be done manually to the ReST/Sphinx source files, rather than the Lore files. Otherwise you just end up with another tool to maintain forever.
Well, sure. The goal of this process is to delete all the lore files and check in a bunch of sphinx files which replace them. This should be done in a branch, and reviewed like usual; the automated tool is just to make the process easier and more reliable. You have two options regarding the Divmod projects; either you can make the tool good enough that you just run it, or you can commit to putting the work into the minor edits to correct the tool's deficiencies on the Divmod lore docs _before_ you can remove / desupport twisted.lore. Another thing to note in the migration plan: despite <http://twistedmatrix.com/trac/wiki/CompatibilityPolicy>, no deprecation period should be necessary for lore, it can just be removed. It has no users that I'm aware of, and if anyone does still use it they can manually copy their 'twisted/lore' directory and 'lore' script into new installations until they have a chance to run your script. It's not designed to be used as a library anyway; it's a bit of an accident that it's even present in the twisted/* tree. One thing to note in the mig ration plan is the closing of outstanding branches with lore changes in them. This would be a good topic for a sprint.
And in my opinion, Twisted needs to get away from having it's own collection of self-maintained tools as much as possible. :)
Not just your opinion :).
I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work. There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some. I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible. That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds).
If there's interest, I can put up what I can generate to this point, though it's awfully early days yet.
Release early, release often. Most efforts like this end up moribund: you sound serious right now, but never underestimate the extent to which life will get in the way. If you put up what you've got somewhere public ASAP, (A) some people will probably show up to egg you on, (B) it's possible for them to contribute (and those contributions will be a motivation on days when this seems like a ridiculous amount of effort just to make Twisted's docs look nicer), and (C) they can pick up the slack if you need to disappear, either temporarily or permanently. *don't* try to get your migration tools to the state where they can get through Twisted code review, etc. Hopefully we can throw these away quickly.
Also: Were my claims/assumption about Lore pretty much correct? Did I miss anything?
Your claim about highlighting only python is correct. See twisted.lore.tree.munge (is it any wonder that we need to replace this thing), specifically the bit where it calls fontifyPython and there is no definition of fontifyAnythingElse. The "docs errors throw build warnings" thing is an extremely confusing header, although I understand what you're saying. I'd re-word the whole thing in terms of test integration. But the important point has nothing to do with doctest, it's that sphinx has various bits of testing support, including "leave the examples outside the documentation so they're just regular python and don't need to be parsed". There's a "*" for "requires plugin" - it's worth mentioning that Sphinx has a usable, and more importantly *actually used by some people*, extension API, whereas lore's extension mechanisms are... ahem. Let's not talk about lore's extension mechanism. You do need to address linking *from* sphinx documentation *to* API documentation, and ideally, vice versa. Lore can do the former, via 'class="API"', although pydoctor still has no way to do the latter (see <http://twistedmatrix.com/trac/ticket/2801>). I don't know of any other projects (besides the ones on divmod.org) which currently use lore. The only other one I'm aware of that ever did is <https://launchpad.net/thinkcspy>, and they switched to using sphinx input a while ago. Maybe you could bug Jeff Elkner and see if he did any automation which might help? You're correct that lore cannot include partial external files. In your description of 'phase 0', you say "once it has been verified": be specific here. Nobody is going to want to be the one who actually pulls a trigger on a change this big, but it needs to get done. Propose a specific mechanism for the verification. Lay out a timeline which culminates in the ReST files being committed to trunk. It's too early for specific dates, but you can say "first we will do X, then we will do Y". In general though, your representation of Lore seems accurate. Thanks again for taking this on. Good luck!
On 05:22 am, glyph@twistedmatrix.com wrote:
On Nov 11, 2009, at 7:37 PM, Kevin Horn wrote:
You're correct that lore cannot include partial external files.
Actually, it has limited support for this. The PB documentation relies on it (search for "skipLines" in doc/core/howto). And if it were implemented better, I think I would find it a useful feature. Jean-Paul
On Wed, Nov 11, 2009 at 11:22 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote: Excellent. In that case (modulo several weeks of work on your part), I'm
sold.
Hooray!
I think the second option is probably what will end up happening, though it may not be necessary to do too much of this. So far, I've only got 3 files in the Twisted Lore docs that wil definitely need reworking before they can be processed. And they have some really strange markup. So maybe there won't be too much of this in the Divmod stuff. (crossing fingers!)
Yikes. Another thing I hadn't considered. Depending on how many branches there, this could get nasty. I'll have to do some thinking on this.
Glad to hear it :). I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work. There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some. I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible. That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds). If there's interest, I can put up what I can generate to this point, though it's awfully early days yet. Release early, release often. Most efforts like this end up moribund: you
I'll try to put up what I've got later today.
Roger that. Also:
Noted.
I've just removed this whole section, as it _was_ confusing, and also mostly redundant.
I've added something about this.
This should be doable. Right now I'm thinking of doing something similar to an existing extension, which creates links to generated Epydoc API docs. Shouldn't be too hard to do something similar for pyDoctor. I'll get something in the proposal.
I'll try to get this updated a bit later.
In general though, your representation of Lore seems accurate.
Thanks again for taking this on. Good luck!
Thanks. I have a feeling I'll need it :) Kevin Horn
On Thu, Nov 12, 2009 at 11:36 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
OK, finally got the current results of the conversion process up. Sorry for the delay, my weekend got hijacked by the Swine Flu. Some of that "life getting in the way", I guess. I want to reiterate that this is _not_ even _close_ to finished. But it's what I've got so far. Kevin Horn
On Mon, Nov 16, 2009 at 11:57 AM, ssteinerX@gmail.com <ssteinerx@gmail.com>wrote:
er, sorry, forgot to post the link... here it is: http://twistedsphinx.funsize.net/ Kevin Horn
On Nov 16, 2009, at 1:09 PM, Kevin Horn wrote:
That gets an "awesome" first pass grade. It sure does look different presented that way, now, doesn't it? I know what a huge bunch'o'stuff the docs are as I looked into just collecting what there was in the docs, on the Trac, and in the wild a while ago. Great start! S
On 06:09 pm, kevin.horn@gmail.com wrote:
Thanks. It's good to be able to see (at least an early draft of) what the outcome of this proposal will be. I'm glad the reactions have been good so far, but I want to be a wet blanket for a moment and suggest that we try to avoid being influenced excessively by the "new look" effect. It would be easy to give the existing documentation a face lift without switching systems entirely - it's just a matter of editing the template and/or the css. What are the other benefits of the Sphinx output, in particular, the ones which it would be difficult to achieve with Lore? I'll start with one possibility - the index. Am I right in thinking this is automatically generated? This is something Lore isn't currently capable of (there's a branch which addresses this, but I suspect it is far too much of a mess to ever merge, so we can ignore it). Going along with this, how much control do we have over how the index is generated? Some of what shows up here probably doesn't merit being displayed as prominently as this - for example, the historical documents are mostly of, well, historical interesting. What options do we have for changing how these are presented? Jean-Paul
On Mon, Nov 16, 2009 at 3:11 PM, <exarkun@twistedmatrix.com> wrote:
Wet blanket away! Every project needs a good "devil's advocate".
Well, IMO, the absolute #1 benefit of using Sphinx over Lore is simply not having to maintain Lore, which has nothing at all to do with the actual output. The next "tier" of advantages (again IMO) also have nothing to do with the output, those advantages being: - a more approachable document format (in the sense that I think there are probably more people familiar with the Sphinx dialect of RestructuredText than are familiar with Lore's subset of XHTML...and the number of Sphinx users is growing) - a much larger user base, and clear extension API, which allows us to take advantage of existing extensions, etc. Now having said that, switching to Sphinx *output* does have some advantages (though by themselves they might not be compelling enough to switch). - better navigation (prev & next links, table of contents) - JS-based full text search - CHTML output (very nice for those of us who use Win32...though I realize we're a minority here in the Twisted world) - Direct PDF output (somewhat immature and provided by a 3rd party extension, but mature enough that someone has actually "typeset" a printed version of the Python Tutorial with it) You're absolutely right that just giving the docs a "facelift" wouldn't require nearly so much effort, but I don't think it would buy us as much with regard to the ultimate maintainability of the documentation. I'll start with one possibility - the index. Am I right in thinking
There are (at least) 3 meanings of index as it relates to Sphinx. - general index - currently empty - module index - currently empty and, in fact, missing - search index - auto-generated and used for the JS-based full text search feature However from you comments above I think you probably meant what Sphinx calls the "table of contents" (which shows up in the index.html, contents.html, etc.). Not trying to nitpick here, just trying to be clear. If I'm incorrect in assuming that's what you meant, then you can ignore the following and correct me, and we'll go from there. In the meantime, I'll assume we're talking about the table of contents (TOC). RestructuredText (in a plain vanilla docutils install) does not have any way of connecting documents together other than hyperlinks. Sphinx defines a "toctree" directive to get around this limitation. This is a directive that goes into your Sphinx source documents that looks something like this: | ..toctree: | | projects/conch/index | projects/core/index | ..etc Typically these would be created manually in your Sphinx source, but in this instance I am creating them using my conversion script which generates ReST source from Lore source. At Sphinx build-time, Sphinx expects every ReST source file to show up someplace in one of these toctree directives, and will generate warnings if a file doesn't. Sphinx gathers up the connections defined by these toctree directives throughout the project into a tree-shaped data structure and uses this to generate the handy navigation links (prev & next), and also to display all or part of the tree as a series of hyperlinks in a table of contents format. The depth of the tree displayed (or even displaying it at all) is easily configurable. Right now I'm showing quite a lot, because it makes it easier to test my toctree-generating code, and it makes browsing my output a bit simpler. There's no reason we'd need to do this in the final product. Of course, if you mean the "index" index, then the answer is entirely different :) Kevin Horn
On Nov 16, 2009, at 5:37 PM, Kevin Horn wrote:
Well! If it's time for criticism I've got some ;-). I'd really like to see the stylesheets on this site adjusted to look as much as possible like the current Twisted style. Use the verbatim stylesheets from twistedmatrix.com as much as possible. I realize you'll have to have some custom elements, but it should be possible to at least avoid creating any new images and to stitch together the existing styles rather than make new ones wholesale. If we ever have the resources to do a site redesign again (and I hope we do, it would be nice to get a fresh look every few years) I want to be able to re-style the _entire_ site. We could do this with the Lore documentation too, but if we're going to make the investment I'd rather it be on something easier to maintain, as your conversion project promises to be. I also suspect that it would be a lot easier to style the output from sphinx, as it is explicitly designed to be themable, and has actually been modified to have multiple, radically different stylesheets already. Actually integrating this with the L&F of the rest of the Twisted site would go a long way towards demonstrating Sphinx's superiority in this area :). On the "things to do" side, while the mere presence of this page is itself unintentionally hilarious - <http://twistedsphinx.funsize.net/projects/lore/howto/lore.html> - and it will likely be removed from the final output, it does contain a good enumeration of all the things that need to be converted in one place. I suspect that if you made all those UNHANDLED_TAG output warts go away, we'd be most of the way towards converting all of the documentation.
I agree.
The next "tier" of advantages (again IMO) also have nothing to do with the output, those advantages being:
- a more approachable document format (in the sense that I think there are probably more people familiar with the Sphinx dialect of RestructuredText than are familiar with Lore's subset of XHTML...and the number of Sphinx users is growing)
Actually, I think that this particular often-touted advantage is a wash. The "approachableness" of ReST is questionable, especially once you get into the weird corner cases where the syntax just completely falls apart and turns into an incomprehensible line-noise jumble. I mean, the syntax for tables reads like a joke about how trying to make the plain-text input look like the output is a bad idea: <http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#tables>. Consider trying to re-flow the contents of a table cell after adding text to it, for example. Plus, a _lot_ of people know HTML -- still many more than know ReST, I would argue -- and lore adds relatively little to HTML. However, the motivation for choosing HTML was that "any day now" there would be a good, commonly used, GUI editor for HTML documents and we could easily annotate the output of one of those with the extra metadata that lore wanted. That hasn't happened. What *has* happened is that despite the difficulty involved in parsing and emitting ReST as compared to something fairly regular like HTML or XML, tools like these have been emerging: http://kib2.free.fr/reSTinPeace/ http://blog.enthought.com/?p=127 which let users edit ReST documents in a sort-of-wsyiwyg way, so you get immediate feedback when you put a "|" or a "`" in some obscure spot that makes your entire document disappear, rather than trying to hunt it down after an hour of writing. Plus, docutils provides a document model for ReST so you can transform it programmatically and ignore its syntactic peculiarities. Which all comes down to the fact that I'm mainly just restating this point as the primary advantage:
- a much larger user base, and clear extension API, which allows us to take advantage of existing extensions, etc.
The active user community means that not only is somebody else mantaining the basics, lots of other people are working on *tools* that we can use for more intense usage. I am personally looking forward to using some of those GUI tools for actually editing the documentation; it's a much more streamlined workflow than running the 'lore' commandline and reloading in a browser.
On Tue, Nov 17, 2009 at 2:13 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
The current look is the "sphinxdoc" theme, that comes with Sphinx. My intention is to create a new one for the Twisted docs that will be very similar to the Twisted site. I haven't really even started on this yet, though.
Heh, that is pretty funny. If you'll notice though, almost all the UNHANDLED_TAG stuff (on that page anyway) is related to the table on that page. I haven't even touched tables yet. I don't think they'll be too difficult, as long as there aren't any nested tables (there aren't in the Twisted docs...I haven't looked at the Divmod stuff yet). Since I think there's only 2 tables in the Twisted docs, there kind of low on the priority list. FYI, those UNHANDLED_TAG messages are intentionally ugly, to make it easier to find things that haven't been handled yet. As far as what still needs to be done on conversion, this falls into a couple of main areas: - handling unhandled tags - fixing some whitespace handling issues (notice that nested lists are totally borked at the moment) - going back through the "Using Lore" document, making sure all the various tags are being handled correctly, and fixing any problems - building a "twisteddocs" theme After this stuff is done, then the real grunt-work begins...fixing the bits manually that can't be easily converted. So far, I think this only really involves 3 pages of Lore XHTML, but I'm sure I'll find other problems. :)
I'm not sure I agree here. While it's true that lots of people know HTML, I'm not sure that that helps more than it hurts. First of all, XHTML is subtly different from HTML, but more importantly, everyone who "knows" HTML doesn't know it to the same level, and has different opinions about what different elements should be used for. The "Using Lore" page spells things out pretty clearly, but even some of those clear rules aren't followed very well in the actual Twisted docs. I think because people "know" HTML, they just use what they know instead of paying close attention to the actual Lore rules. (going way out on the opinion limb here, though) I guess my point is that I think it takes about the same amount of effort to really learn the Lore subset of XHTML as it does to learn Sphinx's dialect of RestructuredText. And more people know Sphinx-RestructuredText to the level practically usable for documentation (from using it on other projects). There's arguments either way I guess. On a personal level, having written docs in other XML/XHTML dialects and in Sphinx/ReST, I know I can write them way faster and with much less thought to format in ReST. And, IMO, those corner cases are very much in the corner. They don't come up very often, at least for me. There's no defending ReST tables though. You're totally right there. Yuck.
I think the underlying reason this has happened is that ReST is less flexible than HTML, and therefore less ambiguous. I couldn't tell you why I think that, though. Just a feeling.
Kevin Horn
Not to detract from any of the points you've made regarding reST's table syntax, but I've found it to be quite livable-with using table.el. Emacs users might be interested: http://table.sourceforge.net/ Laurens
On Nov 17, 2009, at 3:58 PM, Laurens Van Houtven wrote:
Wow. Uh, erm. This does detract significantly. Docutils' API for parsing it mitigates my parsing concerns, and this now mitigates my UI concerns. Although... I've been using it and editing text inside of tables, resize and reformat them for the last 20 minutes or so and it is both mesmerising and disturbing. I thought artist-mode would be the weirdest thing I'd ever see in Emacs, but this tops it easily.
2009/11/11 Kevin Horn <kevin.horn@gmail.com>:
I have written up a proposal here:
In this proposal you say: Development Work * fabric fabfile for automation of docs workflow Future Work * This page has some stuff on using cog in RestructuredText comments to generate the output of sample scripts. The second point (in Future Work) uses cog that is integrated in paver: * Paver allow automation of docs workflow (fabric is not needed). * With paver you can create an bootstraped instalation: http://www.blueskyonmars.com/projects/paver/getting_started.html#but-people-... * Paver is compatible with setuptools and distutils. I think that fabric is great for cluster distributions, but paver is better for manage docs. Regards, Javi
On Tue, Nov 17, 2009 at 4:00 AM, lasizoillo <lasizoillo@gmail.com> wrote:
For the moment, I plan to continue using fabric, for the following reasons: - I'm already familiar with it - it does what I need (run local commands, and send stuff over SSH) I think Paver might be very useful down the road, though. I'll definitely check into it more as this project progresses. Kevin Horn
On Tue, Nov 17, 2009 at 1:16 PM, <exarkun@twistedmatrix.com> wrote:
Yes. For right now at least, this is the case. I think one or the other should probably be used in the future to automate/streamline Twisted releases, since this has been a long-standing desire of the development team (at least that's my perception...and there are a couple of tickets in Trac about it). This would prevent maintenance of yet another homegrown Twisted tool to perform said automation. But that's a discussion for another day. Or at least another thread. Right now I'm using fabric strictly as a (mostly) platform-neutral alternative to shell scripts (or some other solution) to be able to convert docs to ReST, build the Sphinx project, and deploy them to the server (sort of...still working out some kinks) with a single command. Once the conversion takes place (assuming this project is successful) much of my current need for it will go away, though it still might be useful for other things (as I mentioned above). Kevin Horn
On Nov 11, 2009, at 5:52 PM, Kevin Horn wrote:
Please let me know what you think.
Initial reaction: awesome! I don't really care about the documentation format or tools, I'm just thrilled that someone is willing to step forward and maintain them in _any_ form. Secondary reaction: automated testing of code and examples is great. Necessary, even, especially for a new tool. But doctests are not so much a testing plan as a disease that comes to infest your code. (You are free to disagree with me about the utility or quality of doctest as a tool, but doing so on this mailing list is counterproductive, as my mind is pretty well made up. Despite the fact that I have fond feelings towards lore's design, this is the *only* point I'd veto a migration over.) Automated tests for documentation should look roughly like this: http://divmod.org/trac/browser/trunk/Nevow/nevow/test/test_howtolistings.py?... but the stuff that does "sys.modules.clear()" and so on should be a lot further away from the actual test cases. The only thing that is necessary for testing like that to work is the ability to reference an external .py file as a code listing. This is also a good thing because then users can just run the Python files straight in the examples directory, rather than copying some stuff from a web browser into a text editor, mucking around with path variables for a while, trying to make sure they got the indentation right etc. Can sphinx do that out of the box? Tertiary reaction, since I had to go to divmod.org to dig up that example: a migration plan should include the Divmod projects, especially Nevow, since a lot of people use that. But, if there's an automated tool that can handle the Twisted conversion, there really shouldn't be any additional effort to just run it over the Divmod codebase and just check in the result. The only real work will be in adjusting the buildbot. This is really just an argument for an automated, repeatable conversion process rather than manual editing.
On Wed, Nov 11, 2009 at 5:56 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Well, that's encouraging. :)
I never use doctests in my own projects...but the functionality comes with Sphinx (as a "built-in extension"...which such a fantastic oxymoron it makes me smile). There's lot's of other work being done in the Sphinx testing arena, though, notably: Sphinxtest: http://groups.google.com/group/sphinx-dev/browse_thread/thread/fd9fc42648fc3... Manuel: http://packages.python.org/manuel/ and something-I-read-about-the-other-day-but-can't-find-now.
Yes. The "literalinclude" directive pulls in an external file and sticks it in a literal block (the ReST equivalent of a <pre> tag). So you could use whatever testing framework you wanted and just reference those files.
Hmmm...I hadn't considered the Divmod stuff, but you're right. I think the automated conversion plan is the way to go...up to a point. As I laid out in the proposal, I think at some point, the Sphinx docs (and the automated conversion process) will become "good enough", and then edits should be done manually to the ReST/Sphinx source files, rather than the Lore files. Otherwise you just end up with another tool to maintain forever. And in my opinion, Twisted needs to get away from having it's own collection of self-maintained tools as much as possible. :) I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work. There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some. I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible. That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds). If there's interest, I can put up what I can generate to this point, though it's awfully early days yet. Also: Were my claims/assumption about Lore pretty much correct? Did I miss anything? Kevin
On Nov 11, 2009, at 7:37 PM, Kevin Horn wrote:
Please! Be encouraged! And therefore maintain our documentation and documentation toolchain! :)
There's lot's of other work being done in the Sphinx testing arena, though, notably: ...
It's great to see that the testing vibe is out there!
Can sphinx do that out of the box?
Yes. The "literalinclude" directive pulls in an external file and sticks it in a literal block (the ReST equivalent of a <pre> tag). So you could use whatever testing framework you wanted and just reference those files.
Excellent. In that case (modulo several weeks of work on your part), I'm sold.
Hmmm...I hadn't considered the Divmod stuff, but you're right.
I'm glad you think so. I do tend to be, especially about Twisted ;-).
I think the automated conversion plan is the way to go...up to a point. As I laid out in the proposal, I think at some point, the Sphinx docs (and the automated conversion process) will become "good enough", and then edits should be done manually to the ReST/Sphinx source files, rather than the Lore files. Otherwise you just end up with another tool to maintain forever.
Well, sure. The goal of this process is to delete all the lore files and check in a bunch of sphinx files which replace them. This should be done in a branch, and reviewed like usual; the automated tool is just to make the process easier and more reliable. You have two options regarding the Divmod projects; either you can make the tool good enough that you just run it, or you can commit to putting the work into the minor edits to correct the tool's deficiencies on the Divmod lore docs _before_ you can remove / desupport twisted.lore. Another thing to note in the migration plan: despite <http://twistedmatrix.com/trac/wiki/CompatibilityPolicy>, no deprecation period should be necessary for lore, it can just be removed. It has no users that I'm aware of, and if anyone does still use it they can manually copy their 'twisted/lore' directory and 'lore' script into new installations until they have a chance to run your script. It's not designed to be used as a library anyway; it's a bit of an accident that it's even present in the twisted/* tree. One thing to note in the mig ration plan is the closing of outstanding branches with lore changes in them. This would be a good topic for a sprint.
And in my opinion, Twisted needs to get away from having it's own collection of self-maintained tools as much as possible. :)
Not just your opinion :).
I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work. There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some. I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible. That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds).
If there's interest, I can put up what I can generate to this point, though it's awfully early days yet.
Release early, release often. Most efforts like this end up moribund: you sound serious right now, but never underestimate the extent to which life will get in the way. If you put up what you've got somewhere public ASAP, (A) some people will probably show up to egg you on, (B) it's possible for them to contribute (and those contributions will be a motivation on days when this seems like a ridiculous amount of effort just to make Twisted's docs look nicer), and (C) they can pick up the slack if you need to disappear, either temporarily or permanently. *don't* try to get your migration tools to the state where they can get through Twisted code review, etc. Hopefully we can throw these away quickly.
Also: Were my claims/assumption about Lore pretty much correct? Did I miss anything?
Your claim about highlighting only python is correct. See twisted.lore.tree.munge (is it any wonder that we need to replace this thing), specifically the bit where it calls fontifyPython and there is no definition of fontifyAnythingElse. The "docs errors throw build warnings" thing is an extremely confusing header, although I understand what you're saying. I'd re-word the whole thing in terms of test integration. But the important point has nothing to do with doctest, it's that sphinx has various bits of testing support, including "leave the examples outside the documentation so they're just regular python and don't need to be parsed". There's a "*" for "requires plugin" - it's worth mentioning that Sphinx has a usable, and more importantly *actually used by some people*, extension API, whereas lore's extension mechanisms are... ahem. Let's not talk about lore's extension mechanism. You do need to address linking *from* sphinx documentation *to* API documentation, and ideally, vice versa. Lore can do the former, via 'class="API"', although pydoctor still has no way to do the latter (see <http://twistedmatrix.com/trac/ticket/2801>). I don't know of any other projects (besides the ones on divmod.org) which currently use lore. The only other one I'm aware of that ever did is <https://launchpad.net/thinkcspy>, and they switched to using sphinx input a while ago. Maybe you could bug Jeff Elkner and see if he did any automation which might help? You're correct that lore cannot include partial external files. In your description of 'phase 0', you say "once it has been verified": be specific here. Nobody is going to want to be the one who actually pulls a trigger on a change this big, but it needs to get done. Propose a specific mechanism for the verification. Lay out a timeline which culminates in the ReST files being committed to trunk. It's too early for specific dates, but you can say "first we will do X, then we will do Y". In general though, your representation of Lore seems accurate. Thanks again for taking this on. Good luck!
On 05:22 am, glyph@twistedmatrix.com wrote:
On Nov 11, 2009, at 7:37 PM, Kevin Horn wrote:
You're correct that lore cannot include partial external files.
Actually, it has limited support for this. The PB documentation relies on it (search for "skipLines" in doc/core/howto). And if it were implemented better, I think I would find it a useful feature. Jean-Paul
On Wed, Nov 11, 2009 at 11:22 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote: Excellent. In that case (modulo several weeks of work on your part), I'm
sold.
Hooray!
I think the second option is probably what will end up happening, though it may not be necessary to do too much of this. So far, I've only got 3 files in the Twisted Lore docs that wil definitely need reworking before they can be processed. And they have some really strange markup. So maybe there won't be too much of this in the Divmod stuff. (crossing fingers!)
Yikes. Another thing I hadn't considered. Depending on how many branches there, this could get nasty. I'll have to do some thinking on this.
Glad to hear it :). I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work. There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some. I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible. That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds). If there's interest, I can put up what I can generate to this point, though it's awfully early days yet. Release early, release often. Most efforts like this end up moribund: you
I'll try to put up what I've got later today.
Roger that. Also:
Noted.
I've just removed this whole section, as it _was_ confusing, and also mostly redundant.
I've added something about this.
This should be doable. Right now I'm thinking of doing something similar to an existing extension, which creates links to generated Epydoc API docs. Shouldn't be too hard to do something similar for pyDoctor. I'll get something in the proposal.
I'll try to get this updated a bit later.
In general though, your representation of Lore seems accurate.
Thanks again for taking this on. Good luck!
Thanks. I have a feeling I'll need it :) Kevin Horn
On Thu, Nov 12, 2009 at 11:36 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
OK, finally got the current results of the conversion process up. Sorry for the delay, my weekend got hijacked by the Swine Flu. Some of that "life getting in the way", I guess. I want to reiterate that this is _not_ even _close_ to finished. But it's what I've got so far. Kevin Horn
On Mon, Nov 16, 2009 at 11:57 AM, ssteinerX@gmail.com <ssteinerx@gmail.com>wrote:
er, sorry, forgot to post the link... here it is: http://twistedsphinx.funsize.net/ Kevin Horn
On Nov 16, 2009, at 1:09 PM, Kevin Horn wrote:
That gets an "awesome" first pass grade. It sure does look different presented that way, now, doesn't it? I know what a huge bunch'o'stuff the docs are as I looked into just collecting what there was in the docs, on the Trac, and in the wild a while ago. Great start! S
On 06:09 pm, kevin.horn@gmail.com wrote:
Thanks. It's good to be able to see (at least an early draft of) what the outcome of this proposal will be. I'm glad the reactions have been good so far, but I want to be a wet blanket for a moment and suggest that we try to avoid being influenced excessively by the "new look" effect. It would be easy to give the existing documentation a face lift without switching systems entirely - it's just a matter of editing the template and/or the css. What are the other benefits of the Sphinx output, in particular, the ones which it would be difficult to achieve with Lore? I'll start with one possibility - the index. Am I right in thinking this is automatically generated? This is something Lore isn't currently capable of (there's a branch which addresses this, but I suspect it is far too much of a mess to ever merge, so we can ignore it). Going along with this, how much control do we have over how the index is generated? Some of what shows up here probably doesn't merit being displayed as prominently as this - for example, the historical documents are mostly of, well, historical interesting. What options do we have for changing how these are presented? Jean-Paul
On Mon, Nov 16, 2009 at 3:11 PM, <exarkun@twistedmatrix.com> wrote:
Wet blanket away! Every project needs a good "devil's advocate".
Well, IMO, the absolute #1 benefit of using Sphinx over Lore is simply not having to maintain Lore, which has nothing at all to do with the actual output. The next "tier" of advantages (again IMO) also have nothing to do with the output, those advantages being: - a more approachable document format (in the sense that I think there are probably more people familiar with the Sphinx dialect of RestructuredText than are familiar with Lore's subset of XHTML...and the number of Sphinx users is growing) - a much larger user base, and clear extension API, which allows us to take advantage of existing extensions, etc. Now having said that, switching to Sphinx *output* does have some advantages (though by themselves they might not be compelling enough to switch). - better navigation (prev & next links, table of contents) - JS-based full text search - CHTML output (very nice for those of us who use Win32...though I realize we're a minority here in the Twisted world) - Direct PDF output (somewhat immature and provided by a 3rd party extension, but mature enough that someone has actually "typeset" a printed version of the Python Tutorial with it) You're absolutely right that just giving the docs a "facelift" wouldn't require nearly so much effort, but I don't think it would buy us as much with regard to the ultimate maintainability of the documentation. I'll start with one possibility - the index. Am I right in thinking
There are (at least) 3 meanings of index as it relates to Sphinx. - general index - currently empty - module index - currently empty and, in fact, missing - search index - auto-generated and used for the JS-based full text search feature However from you comments above I think you probably meant what Sphinx calls the "table of contents" (which shows up in the index.html, contents.html, etc.). Not trying to nitpick here, just trying to be clear. If I'm incorrect in assuming that's what you meant, then you can ignore the following and correct me, and we'll go from there. In the meantime, I'll assume we're talking about the table of contents (TOC). RestructuredText (in a plain vanilla docutils install) does not have any way of connecting documents together other than hyperlinks. Sphinx defines a "toctree" directive to get around this limitation. This is a directive that goes into your Sphinx source documents that looks something like this: | ..toctree: | | projects/conch/index | projects/core/index | ..etc Typically these would be created manually in your Sphinx source, but in this instance I am creating them using my conversion script which generates ReST source from Lore source. At Sphinx build-time, Sphinx expects every ReST source file to show up someplace in one of these toctree directives, and will generate warnings if a file doesn't. Sphinx gathers up the connections defined by these toctree directives throughout the project into a tree-shaped data structure and uses this to generate the handy navigation links (prev & next), and also to display all or part of the tree as a series of hyperlinks in a table of contents format. The depth of the tree displayed (or even displaying it at all) is easily configurable. Right now I'm showing quite a lot, because it makes it easier to test my toctree-generating code, and it makes browsing my output a bit simpler. There's no reason we'd need to do this in the final product. Of course, if you mean the "index" index, then the answer is entirely different :) Kevin Horn
On Nov 16, 2009, at 5:37 PM, Kevin Horn wrote:
Well! If it's time for criticism I've got some ;-). I'd really like to see the stylesheets on this site adjusted to look as much as possible like the current Twisted style. Use the verbatim stylesheets from twistedmatrix.com as much as possible. I realize you'll have to have some custom elements, but it should be possible to at least avoid creating any new images and to stitch together the existing styles rather than make new ones wholesale. If we ever have the resources to do a site redesign again (and I hope we do, it would be nice to get a fresh look every few years) I want to be able to re-style the _entire_ site. We could do this with the Lore documentation too, but if we're going to make the investment I'd rather it be on something easier to maintain, as your conversion project promises to be. I also suspect that it would be a lot easier to style the output from sphinx, as it is explicitly designed to be themable, and has actually been modified to have multiple, radically different stylesheets already. Actually integrating this with the L&F of the rest of the Twisted site would go a long way towards demonstrating Sphinx's superiority in this area :). On the "things to do" side, while the mere presence of this page is itself unintentionally hilarious - <http://twistedsphinx.funsize.net/projects/lore/howto/lore.html> - and it will likely be removed from the final output, it does contain a good enumeration of all the things that need to be converted in one place. I suspect that if you made all those UNHANDLED_TAG output warts go away, we'd be most of the way towards converting all of the documentation.
I agree.
The next "tier" of advantages (again IMO) also have nothing to do with the output, those advantages being:
- a more approachable document format (in the sense that I think there are probably more people familiar with the Sphinx dialect of RestructuredText than are familiar with Lore's subset of XHTML...and the number of Sphinx users is growing)
Actually, I think that this particular often-touted advantage is a wash. The "approachableness" of ReST is questionable, especially once you get into the weird corner cases where the syntax just completely falls apart and turns into an incomprehensible line-noise jumble. I mean, the syntax for tables reads like a joke about how trying to make the plain-text input look like the output is a bad idea: <http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#tables>. Consider trying to re-flow the contents of a table cell after adding text to it, for example. Plus, a _lot_ of people know HTML -- still many more than know ReST, I would argue -- and lore adds relatively little to HTML. However, the motivation for choosing HTML was that "any day now" there would be a good, commonly used, GUI editor for HTML documents and we could easily annotate the output of one of those with the extra metadata that lore wanted. That hasn't happened. What *has* happened is that despite the difficulty involved in parsing and emitting ReST as compared to something fairly regular like HTML or XML, tools like these have been emerging: http://kib2.free.fr/reSTinPeace/ http://blog.enthought.com/?p=127 which let users edit ReST documents in a sort-of-wsyiwyg way, so you get immediate feedback when you put a "|" or a "`" in some obscure spot that makes your entire document disappear, rather than trying to hunt it down after an hour of writing. Plus, docutils provides a document model for ReST so you can transform it programmatically and ignore its syntactic peculiarities. Which all comes down to the fact that I'm mainly just restating this point as the primary advantage:
- a much larger user base, and clear extension API, which allows us to take advantage of existing extensions, etc.
The active user community means that not only is somebody else mantaining the basics, lots of other people are working on *tools* that we can use for more intense usage. I am personally looking forward to using some of those GUI tools for actually editing the documentation; it's a much more streamlined workflow than running the 'lore' commandline and reloading in a browser.
On Tue, Nov 17, 2009 at 2:13 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
The current look is the "sphinxdoc" theme, that comes with Sphinx. My intention is to create a new one for the Twisted docs that will be very similar to the Twisted site. I haven't really even started on this yet, though.
Heh, that is pretty funny. If you'll notice though, almost all the UNHANDLED_TAG stuff (on that page anyway) is related to the table on that page. I haven't even touched tables yet. I don't think they'll be too difficult, as long as there aren't any nested tables (there aren't in the Twisted docs...I haven't looked at the Divmod stuff yet). Since I think there's only 2 tables in the Twisted docs, there kind of low on the priority list. FYI, those UNHANDLED_TAG messages are intentionally ugly, to make it easier to find things that haven't been handled yet. As far as what still needs to be done on conversion, this falls into a couple of main areas: - handling unhandled tags - fixing some whitespace handling issues (notice that nested lists are totally borked at the moment) - going back through the "Using Lore" document, making sure all the various tags are being handled correctly, and fixing any problems - building a "twisteddocs" theme After this stuff is done, then the real grunt-work begins...fixing the bits manually that can't be easily converted. So far, I think this only really involves 3 pages of Lore XHTML, but I'm sure I'll find other problems. :)
I'm not sure I agree here. While it's true that lots of people know HTML, I'm not sure that that helps more than it hurts. First of all, XHTML is subtly different from HTML, but more importantly, everyone who "knows" HTML doesn't know it to the same level, and has different opinions about what different elements should be used for. The "Using Lore" page spells things out pretty clearly, but even some of those clear rules aren't followed very well in the actual Twisted docs. I think because people "know" HTML, they just use what they know instead of paying close attention to the actual Lore rules. (going way out on the opinion limb here, though) I guess my point is that I think it takes about the same amount of effort to really learn the Lore subset of XHTML as it does to learn Sphinx's dialect of RestructuredText. And more people know Sphinx-RestructuredText to the level practically usable for documentation (from using it on other projects). There's arguments either way I guess. On a personal level, having written docs in other XML/XHTML dialects and in Sphinx/ReST, I know I can write them way faster and with much less thought to format in ReST. And, IMO, those corner cases are very much in the corner. They don't come up very often, at least for me. There's no defending ReST tables though. You're totally right there. Yuck.
I think the underlying reason this has happened is that ReST is less flexible than HTML, and therefore less ambiguous. I couldn't tell you why I think that, though. Just a feeling.
Kevin Horn
Not to detract from any of the points you've made regarding reST's table syntax, but I've found it to be quite livable-with using table.el. Emacs users might be interested: http://table.sourceforge.net/ Laurens
On Nov 17, 2009, at 3:58 PM, Laurens Van Houtven wrote:
Wow. Uh, erm. This does detract significantly. Docutils' API for parsing it mitigates my parsing concerns, and this now mitigates my UI concerns. Although... I've been using it and editing text inside of tables, resize and reformat them for the last 20 minutes or so and it is both mesmerising and disturbing. I thought artist-mode would be the weirdest thing I'd ever see in Emacs, but this tops it easily.
2009/11/11 Kevin Horn <kevin.horn@gmail.com>:
I have written up a proposal here:
In this proposal you say: Development Work * fabric fabfile for automation of docs workflow Future Work * This page has some stuff on using cog in RestructuredText comments to generate the output of sample scripts. The second point (in Future Work) uses cog that is integrated in paver: * Paver allow automation of docs workflow (fabric is not needed). * With paver you can create an bootstraped instalation: http://www.blueskyonmars.com/projects/paver/getting_started.html#but-people-... * Paver is compatible with setuptools and distutils. I think that fabric is great for cluster distributions, but paver is better for manage docs. Regards, Javi
On Tue, Nov 17, 2009 at 4:00 AM, lasizoillo <lasizoillo@gmail.com> wrote:
For the moment, I plan to continue using fabric, for the following reasons: - I'm already familiar with it - it does what I need (run local commands, and send stuff over SSH) I think Paver might be very useful down the road, though. I'll definitely check into it more as this project progresses. Kevin Horn
On Tue, Nov 17, 2009 at 1:16 PM, <exarkun@twistedmatrix.com> wrote:
Yes. For right now at least, this is the case. I think one or the other should probably be used in the future to automate/streamline Twisted releases, since this has been a long-standing desire of the development team (at least that's my perception...and there are a couple of tickets in Trac about it). This would prevent maintenance of yet another homegrown Twisted tool to perform said automation. But that's a discussion for another day. Or at least another thread. Right now I'm using fabric strictly as a (mostly) platform-neutral alternative to shell scripts (or some other solution) to be able to convert docs to ReST, build the Sphinx project, and deploy them to the server (sort of...still working out some kinks) with a single command. Once the conversion takes place (assuming this project is successful) much of my current need for it will go away, though it still might be useful for other things (as I mentioned above). Kevin Horn
participants (7)
-
Alex Clemesha -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Kevin Horn -
lasizoillo -
Laurens Van Houtven -
ssteinerX＠gmail.com