On Mon, Nov 30, 2009 at 4:39 PM, Marco Giusti <marco.giusti@gmail.com>wrote:

On Mon, Nov 30, 2009 at 03:05:30PM -0600, Kevin Horn wrote:

...

[...] Thanks for "listening"!

some notices about what i get[1]. mainly i'd like if you could make the documentation more readable when the browser's window is not really big. usually i do not use the browser with the window maximized so will be really helpful if you can check the following points:

* in the navigation bar "twisted web documentation" is repeated twice and is breaked in two lines.

* the table of contents steals a lot of vertical space while it is visible only at the top of the page. in this circumstances i prefer "the lore way", ie. showing the table of contents in the normal content flow at the top and not in a side column.

* again the left blue column steal vertical space an maybe can be reduced.

* would be great if the layout will fit in the page width: as you can see the index link is half hidden.

obiviusly with a bigger window all the aboves are not more valid and all looks pretty.

ciao m.

[1] http://img44.imageshack.us/img44/1953/sphinx.png

Marco, Thanks for the feedback. I really haven't worked at all on formatting/theming the docs as of yet (what you see is just a theme included with Sphinx), but I will certainly keep your comments in mind when I get to that point. Kevin Horn

On 2 Dec, 11:04 pm, kevin.horn@gmail.com wrote:

...

[1] http://twistedsphinx.funsize.net/proposal.html [2] http://twistedsphinx.funsize.net/index.html

Any of the core Twisted devs care to sound of regarding the proposed timelines?

Silence implies consent. Or in this case approval... :)

Timeline might not be the right word, since as far as I can tell there's little that's actually about time there. :) Let me know if I overlooked something. As far as the (I'm going to call it) roadmap goes, the thought that's pushed its way to the fore for me is that I'd like to try this with something smaller and simpler than Twisted first. It would be nice if the Divmod projects would qualify here, but they may not be active enough for any real experience to accumulate. pyOpenSSL might work. It uses the original CPython tex documentation system, so the conversion would (presumably) be done with whatever tools were used for CPython's own documentation, not the new Lore->Sphinx converter. It would still be a useful way to get more familiar with Sphinx, though. The downside is that I'm pretty much the only Twisted developer who works on pyOpenSSL, so it wouldn't help anyone else get any experience. Jean-Paul

On Dec 3, 2009, at 1:29 AM, Kevin Horn wrote:

On Wed, Dec 2, 2009 at 8:38 PM, <exarkun@twistedmatrix.com> wrote:

As far as the (I'm going to call it) roadmap goes, the thought that's pushed its way to the fore for me is that I'd like to try this with something smaller and simpler than Twisted first. It would be nice if the Divmod projects would qualify here, but they may not be active enough for any real experience to accumulate.

I haven't run my lore2sphinx script against the Divmod stuff yet, but I could certainly try it. The intention is to convert the Divmod stuff as well as the Twisted stuff eventually though.

I've been concentrating on the Twisted docs, since they seemed the highest priority, but I could detour a bit if you like. Obviously that would slow progress on the Twisted stuff a little.

While converting pyOpenSSL's documentation would be a worthwhile goal in itself, I don't actually like this idea. Mainly I disagree with the premise that accumulation of experience is necessary for the conversion. But this is not a strong disagreement, as I don't understand the motivation for saying so in the first place :). The beauty of this plan, as far as I see it, is that the only person who has to get any significant experience with Sphinx in order for the conversion to happen is Kevin. Mostly what the rest of us will do is read the documentation and make sure it looks OK. Obviously we all need to learn ReST *after* that conversion to write documentation, but there is so much documentation of ReST and sphinx available that I'm not really worried about that part. More importantly, as JP already noted, the pyOpenSSL documentation is in a different format and the conversion would use a different toolchain, so even if we do have to get some practice, it's not a particularly helpful place to start. If we *do* need practice for some reason, I think Nevow would be the best place to start, but then, I don't see why the activity level of the project makes any difference. I have a few issues with the roadmap too, though: "blah, blah": I'm not sure what that's supposed to mean. "branches containing lore docs changes should be separated into two groups..." - no criteria are specified for deciding which go into which group. Ideally we could just get all of those changes merged; if you are making progress on the conversion I'm sure we could organize a sprint to evaluate those changes and either abandon them or get them into review. Identifying tickets which propose lore functionality is pretty easy; just look at <http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&component=lore&order=priority>. Any tickets which have not been classified properly and therefore don't show up in that list can be closed later, as we discover them. I strongly feel that we do *not* need to conduct an exhaustive review of the entire ticket tracker and get everything perfectly in order in order to do this conversion, as long as it's clear to everyone what is supposed to happen to lore-related tickets in the future. I see the biggest risk at the "docs freeze" step, that the doc conversion guy (or team, as the case may be by that point) will start work, then get distracted and walk away for 6 months, leaving a long period of time where nobody is supposed to write or edit documentation. The whole point of our branch-based review process is to avoid this sort of situation. We can't always avoid it (for example, the immense outstanding Conch branches that made everyone afraid to edit those warning-filled tests for years) but I think it's best to follow the same plan as for any branch, and have no formal "freeze" duration, just a point where the conversion branch gets merged to trunk. It's OK if a few stale doc branches get left out in the cold during the conversion; if they're still stale once the conversion is ready, they must not be terribly actively worked on anyway. Even in the worst case, where a branch is left stale well after the docs have been deleted from trunk, presumably the author of the change could run lore2sphinx against the conversion, copy the sphinxified doc to their trunk working copy, and make a new branch. Assuming that the output of the conversion tool is deterministic, the diff should be small and readable. I'm not saying that we shouldn't identify those changes... It would be good to identify the branches with outstanding doc changes so that we would at least *know* how many changes will be broken, and perhaps motivate their authors to fix them ahead of time. I'm mainly interested in the "phase 0" outlined in the roadmap; I think that the stuff for phase 1 and 2 sounds good, but I don't think we need any special planning for it, since it fits into our normal development workflow pretty neatly. People will be filing bugs for documentation typos pretty much forever ;-). I'd like it if you could break down the "phase 0" a bit more clearly with regard to what happens when, since the review *before* the Big Switch gets thrown to put these changes in trunk is the most important part to get timely feedback from the community.

On Dec 3, 2009, at 10:06 AM, exarkun@twistedmatrix.com wrote:

I want some experience with Sphinx so that can verify the premise that Sphinx is better than Lore. So, the experience isn't necessary for the conversion, it's necessary for me to be comfortable that the conversion is a good idea.

Oh. I am definitely taking that as a given. Or rather: having frequently interacted with the output of both sphinx and lore, I was already confident that it was better than Lore in most ways before we began this conversation. I had a few specific misgivings, which Kevin has since addressed. Sphinx clearly has the *features* that we need; Kevin's analysis looks pretty thorough, and even the current error-laden converted documentation serves as a reasonable proof of concept. Plus, the plan specifically includes resolving the remaining issues with that documentation as a prerequisite to getting it merged. So while I could believe that there will be bad things about Sphinx which may surprise me, I find it hard to believe that I could be surprised by any bug in Sphinx so awful that it would be *enough worse* than Lore to make continuing to maintain Lore a preferable option, given the presence of a documentation maintainer willing to do the drudgework of conversion. That said, it might be a good idea to do the Divmod project conversions first for other reasons. They have less documentation, so there's less to review, and the process could be completed more quickly, letting us get some benefit out of the process earlier on. Since these projects are the only known remaining users of lore, we could get rid of lore immediately when twisted itself is converted, rather than letting it linger until its users have converted.

On Thu, Dec 3, 2009 at 3:00 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:

On Dec 3, 2009, at 1:29 AM, Kevin Horn wrote:

On Wed, Dec 2, 2009 at 8:38 PM, <exarkun@twistedmatrix.com> wrote:

...

As far as the (I'm going to call it) roadmap goes, the thought that's pushed its way to the fore for me is that I'd like to try this with something smaller and simpler than Twisted first. It would be nice if the Divmod projects would qualify here, but they may not be active enough for any real experience to accumulate.

I haven't run my lore2sphinx script against the Divmod stuff yet, but I could certainly try it. The intention is to convert the Divmod stuff as well as the Twisted stuff eventually though.

I've been concentrating on the Twisted docs, since they seemed the highest priority, but I could detour a bit if you like. Obviously that would slow progress on the Twisted stuff a little.

While converting pyOpenSSL's documentation would be a worthwhile goal in itself, I don't actually like this idea.

Mainly I disagree with the premise that accumulation of experience is necessary for the conversion. But this is not a strong disagreement, as I don't understand the motivation for saying so in the first place :).

The beauty of this plan, as far as I see it, is that the only person who has to get any significant experience with Sphinx in order for the conversion to happen is Kevin. Mostly what the rest of us will do is read the documentation and make sure it looks OK. Obviously we all need to learn ReST *after* that conversion to write documentation, but there is so much documentation of ReST and sphinx available that I'm not really worried about that part.

More importantly, as JP already noted, the pyOpenSSL documentation is in a different format and the conversion would use a different toolchain, so even if we do have to get some practice, it's not a particularly helpful place to start. If we *do* need practice for some reason, I think Nevow would be the best place to start, but then, I don't see why the activity level of the project makes any difference.

I have a few issues with the roadmap too, though:

"blah, blah": I'm not sure what that's supposed to mean.

Doh! Forgot to go back and change that...pretend it sayssomething like: "(list of branches here)" I'll fix it shortly. "branches containing lore docs changes should be separated into two

groups..." - no criteria are specified for deciding which go into which group. Ideally we could just get all of those changes merged; if you are making progress on the conversion I'm sure we could organize a sprint to evaluate those changes and either abandon them or get them into review.

Well, I'm not entirely certain what criteria should be used until we at least know what branches exist with lore docs in them. If it's only say 2 or 3 branches that are relatively easy to do, we just do all of them. If it's a whole bunch, and some of them are those "everlasting branches" which have been around for years with no sign of being merged any time soon, well...that's group B. :) Identifying tickets which propose lore functionality is pretty easy; just

look at < http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&component=lore&order=priority>. Any tickets which have not been classified properly and therefore don't show up in that list can be closed later, as we discover them. I strongly feel that we do *not* need to conduct an exhaustive review of the entire ticket tracker and get everything perfectly in order in order to do this conversion, as long as it's clear to everyone what is supposed to happen to lore-related tickets in the future.

Yeah, I don't expect this to be too difficult. Basically thought I'd run a couple of searches (including the very one you mentioned above) and go from there. I'd like to get most of them though. One of the problems I think Twisted has is that there are just too dogone many open tickets. It gets tricky to figure out what really needs to be done unless you've been doing it for years (as the core devs have) because of all the stuff getting in the way. Which is why I've been trying to find outstanding tickets that are almost finished and "shepherd" them to completion. But that's a discussion for another day/thread.

I see the biggest risk at the "docs freeze" step, that the doc conversion guy (or team, as the case may be by that point) will start work, then get distracted and walk away for 6 months, leaving a long period of time where nobody is supposed to write or edit documentation.

The whole point of our branch-based review process is to avoid this sort of situation. We can't always avoid it (for example, the immense outstanding Conch branches that made everyone afraid to edit those warning-filled tests for years) but I think it's best to follow the same plan as for any branch, and have no formal "freeze" duration, just a point where the conversion branch gets merged to trunk. It's OK if a few stale doc branches get left out in the cold during the conversion; if they're still stale once the conversion is ready, they must not be terribly actively worked on anyway.

To be honest, I'm not even sure such a freeze would be necessary. I just thought I would throw it out there since the actual changeover would involve a rather large "structural" change to that part of the Twisted source tree. And I didn't expect it to last more than a few days. Maybe even hours. Perhaps I'm just used to having this kind of step. If the consensus is that it doesn't need to happen, I'm fine with that. (FYI, I'm currently working in an environment where version control is ACTIVELY DISCOURAGED, so it makes me a little paranoid that some kind of disaster could happen.)

Even in the worst case, where a branch is left stale well after the docs have been deleted from trunk, presumably the author of the change could run lore2sphinx against the conversion, copy the sphinxified doc to their trunk working copy, and make a new branch. Assuming that the output of the conversion tool is deterministic, the diff should be small and readable.

Hmmm. This kind of procedure could sort of work. Pondering.... The only potential problem I see with this idea is that I think we'll need to do some manual cleanups on the converted docs. But I guess the author could do as you suggested, then do their own (presumably small) manual cleanup of their working copy after the conversion of their modified branch. Should be workable, if not _quite_ as simple as you've laid it out (though almost as simple).

I'm not saying that we shouldn't identify those changes... It would be good to identify the branches with outstanding doc changes so that we would at least *know* how many changes will be broken, and perhaps motivate their authors to fix them ahead of time.

I think we're on the same page here.

I'm mainly interested in the "phase 0" outlined in the roadmap; I think that the stuff for phase 1 and 2 sounds good, but I don't think we need any special planning for it, since it fits into our normal development workflow pretty neatly. People will be filing bugs for documentation typos pretty much forever ;-). I'd like it if you could break down the "phase 0" a bit more clearly with regard to what happens when, since the review *before* the Big Switch gets thrown to put these changes in trunk is the most important part to get timely feedback from the community.

Looking at this after having been away from it for a day or so, I can see that it's a bit chaotic :) I guess I had been looking at it too long. I'll try to update this soonish. Anything specific that is unclear? Kevin Horn