On Mar 1, 2013, at 9:44 AM, Kevin Horn <kevin.horn@gmail.com> wrote:

On Fri, Mar 1, 2013 at 2:29 AM, Glyph <glyph@twistedmatrix.com> wrote:
Jean-Paul recently closed a Lore ticket as invalid, and suggested we have a discussion about Lore's future direction.  This strikes me as a very good idea, and so I wrote a message which is a bit too long (for which I apologize) to kick that off.

I don't think these two paths (lore2sphinx and continuing to maintain lore) are necessarily mutually exclusive.  Also I think it implies something about the current state of affairs that isn't accurate - e.g. that the Twisted team has agreed that Sphinx will surely replace Lore and that we are making progress on that process of placement more than we are maintaining Lore itself.

Unfortunately, I think it will be clear to anyone following its progress that lore2sphinx is unmaintained and the sphinx migration effort is stalled.  Nobody has committed to <https://bitbucket.org/khorn/lore2sphinx> in a year and a half, about the same amount of time that <http://twistedmatrix.com/trac/browser/branches/sphinx-conversion-4500> has been idle as well.  By contrast, <http://twistedmatrix.com/trac/browser/trunk/twisted/lore> has seen commits - albeit not many - within only a couple of weeks.  So, empirically, we're already maintaining lore and lore2sphinx is currently "obsolete"; really the question should be if we want to reverse that path.


Some what orthagonal to your point, but this is incorrect.  lore2sphinx was split some time ago into "lore2sphinx-ng" and "rstgen".

Hi Kevin!  Long time no see!  (Too long, obviously!)


This was initially done as an experiment in using a more explicit "formatting model" for the generation for the Sphinx docs (and somewhat due to _your_ prodding, Glyph), and so I didn't initially make a big announcement or anything.

I do remember this.  The previous output of lore2sphinx really was unreliable enough that it was creating a never-ending treadmill of irrelevant / unpredictable Lore source fixes that were really dragging the whole process out.  Thanks for working on improving it.

Once it became apparent that it was actually going to work out better, I sent out some emails to those who had expressed interest in helping with the whole lore2sphinx project, though I don't believe I sent out anything to the twisted list in general, as I probably should have.  I'll point out that I can count people who have shown interest in moving this forward on one hand, though.

More discussion on this list would be almost always be better.  We are a *long* way from too much traffic here.  (And, this update is honestly a surprise to me.)

And I've specifically mentioned that I had done said forking to you, Glyph, in IRC  ;)
(though it's IRC after all...who remembers what happens in IRC?)

Based on this exchange, my understanding was simply that you had started to try to improve lore2sphinx, but then wandered off again.

I thought I had put a notice up in the readme file in the lore2sphinx repo, but as it isn't there, I presume I either forgot, or never got it merged, or something.

So, totally my bad for not communicating better, but I have NOT given up on converting things from Lore into Sphinx.
(Nor do I intend to.)

OK.  Let's move things along then.  Several people showed up on IRC yesterday and voiced an interest in helping out, although what to do next - especially what to do next for a new contributor who does *not* want to try to reverse-engineer the conversion itself - needs to be made much, much clearer.

Thinking about it, I suppose I've been somewhat reticent to do much communicating about any work I do on this, as what seems to happen is that it just gives everyone an excuse bring up some new objection to actually getting the conversion done.  I hadn't really realized this consciously until just now, though.

Communicate constantly.  The biggest objection that _I_ have to getting the conversion done at this point is that the people working on it (well, okay: you) are uncommunicative, unreliable and frequently unavailable. ;-)  If you were just keeping us all up to date - even just to complain! - I'd be much more sanguine about the whole thing.  And apparently some of your misconceptions would have been corrected a lot earlier.

I also have no objection if someone wants to complete the lore2sphinx work, but if the lore2sphinx buildbot were to die tomorrow and go offline, I wouldn't be particularly anxious to spend a lot of resources to fix it.

My position on this was always that if someone wanted to improve the documentation, they were welcome to do so, and if they wanted to use Sphinx to do it, that's great too.  I just wasn't willing to tolerate any period where our toolchain was broken and we couldn't generate documentation for a release.  And a good thing we didn't, by the way!  If we had said "go ahead, pull the trigger, whatever, it's OK to break trunk for a little while!" we wouldn't have had any documentation toolchain for the last 2 years.


And since we didn't break the toolchain, I've been in no particular hurry.  I've accepted that this will take approximately a billion years.  So no rush.

It does not have to take a billion years.  The criteria ought to be clear - and if they aren't, you should have asked for clarification :).

On the other hand, I have at several points been willing to make the "cutover", and for various different reasons, been told it wasn't happening until things were closer to "perfect" (for some value of "perfect") than they were at the time.

Let's be specific: <http://twistedmatrix.com/trac/ticket/5312> is in need of some final code-review.  Despite several reviews and an apparently extensive final response pass, it's not currently in review, which means it's still in your court for some reason.  There is no reason to hold back on this and try to do *everything* in one big bang: this code just needs to be production-quality and land on trunk _before_ the ReST sources themselves are ready to go.

Probably something needs to happen to the buildbot build steps, too, since there's this nastiness that did an end-run around our development process to get checked in to the buildbot config without tests instead of into twisted with tests, <http://buildbot.twistedmatrix.com/builders/documentation/builds/2994/steps/process-docs/logs/stdio>, and that needs to be replaced with a command that's just like "build the docs, whether they be lore or sphinx or docbook or whatever".  But, Tom's got your back here; if you can get this done during his fellowship (see today's post, <http://labs.twistedmatrix.com/2013/03/welcome-our-new-twisted-fellow-tom.html>) I estimate you will see a completed reconfiguration within hours.

Once that's done, then it's a matter of putting <http://tm.tl/4500> into code-review with the output of the lore2sphinx builder.  That review can be somewhat expedited, and can be done in parallel by lots of people since there are no unit tests to be worried about, and formatting fixes can be done quickly by multiple people, we don't need a big formal code review.

The current output of the old lore2sphinx branch is functional, though has a few warts (mostly extraneous spaces in the output).  These warts were apparently enough to block adoption.

Let's not under-state the problem: thanks to the jaw-droppingly weird arbitrariness of the ReST format, "extraneous spaces" can mean "arbitrarily mangled output".  But no, even these "warts" were not enough to block adoption.  What blocked adoption is that the painstakingly hand-tweaked lore sources that did not have any more "warts" were left to languish (and bit-rot, and now probably require more manual fixing) while we waited for 2 years for someone to actually finish the sphinx development and release management tools and get them finalized.  As I recall we basically finished fixing them all up, at the time.

There were three reasons that I personally kept pressing for a more thorough lore -> sphinx converter.  One is not necessarily necessary.

First, and most importantly, is the bit-rot problem: people are working on lore docs in parallel with this effort.  And, despite this exchange, I want to be clear that they should keep doing so: nobody should stop working on docs in the meanwhile, since we have no way to tell how much longer this will take.  Looking at the modified docs on the sphinx buildbot is challenging, and keeping track of random whitespace jiggling is not documented on <http://twistedmatrix.com/trac/wiki/ReviewProcess#Reviewers:Howtoreviewachange>.  *I* can't even remember how to do the math to associate one of the results in <http://buildbot.twistedmatrix.com/builds/sphinx-html/>.  And now that there have been so many changes (as I predicted there might be) we have to figure out what's changed, and re-review to make sure that everything (or at least a big enough majority of everything) is OK to go to trunk.  If the tool itself could be verified to produce correct output for all the cases we've encountered where it falls over, we wouldn't have to do this manual verification step; we could just trust that it was right, because it has tests that indicate it's correct.  Of course it's possible there might be *some* corner-case it still doesn't handle and that we didn't find, but if the tool is known to be broken in a large number of cases that we just have to magically know to avoid, then it's likely people will keep unknowingly re-introducing those problems.

Second, there are going to be some doc patches in-progress whenever the cutover happens.  Now, this is a bit less of a concern, because we can just manually translate one or two paragraphs to the new markup if necessary.  But it would still be nice to have a tool that does the job well enough that someone could grab the buildbot output for an in-progress doc fix and keep working on it without having to learn how to re-express everything in Sphinx first.

Third, the output is just hella grody right now.  Have a look here, for example: <http://buildbot.twistedmatrix.com/builds/sphinx-html/989-37334/_sources/projects/web/howto/twisted-templates.txt>.  *Tons* of peculiarly and unnecessary vertical whitespace, and very ragged right edges where the word wrap doesn't seem to respect line lengths.  This means that every change that hits these documents is going to produce a lot of unnecessary delta when authors try to clean up some of this mess to make it nicer to edit.

Spot-checking some of the output now, it seems like the tool must have been upgraded, or we've been lucky, since I can't spot any obvious bit-rot (and I could swear the docs look a lot less grody; the problems I mentioned there).  So maybe you've already addressed these problems, or they're not actually that serious any more.  But, as I said in the first point, spot-checking isn't enough.

It has been a pretty discouraging effort at times, I have to say, as I seem to garner agreement/support/buy-in/whatever for a particular course of action (e.g. getting 99% of the way there, and then fixing Sphinx markup manually, which was the original plan, way back when), and focusing my efforts in that direction.  Then when we're ready to proceed on that basis, had another task/challenge/set of requirements/whatever added to the work that needs to be done.  In fact I still think that if the Twisted community had actually wanted to, we could have switched over to Sphinx at the first PyCon Atlanta (2010?).

By 'actually wanted to' you mean 'be willing to abandon the development process for this one thing'.

We do not abandon the development process.  Every past attempt at doing so to facilitate some feature has been a road to ruin.  Although this process has been frustrating for you, I am still happier with the current outcome (Twisted has perfectly functional documentation in our downloads and on our website) than with the alternative (create a situation where we could not produce a release for two years because the tools were languishing unfinished while we waited for you to say something about it).

I'm sorry that this has been a frustrating process for you.  And I'm not just saying that to be polite: I genuinely *am* sorry that our communication has not been clear, and that we have had wasted effort all around because of that.  But I am fairly sure that we have had basically the same requirements for this process from day one.  Let me state them here:

  1. We need to have release-automation tools that allow developers to produce a release, including documentation.  These tools need to be subjected to the same development process as the rest of those tools, which is to say the same process as for the rest of Twisted.
  2. The documentation itself needs to be able to be generated from any version of trunk.  While one or two formatting snafus are acceptable to be fixed after the fact, the documentation needs to be in a comprehensible state in every revision of trunk, which means that in order to land on trunk, the ReST output.

Really, most of the work has been done here already.  The docs appear to be in a mostly-workable state.  lore2sphinx looks like maybe it's doing a good enough job, maybe better than the last time I looked at it.  The _major_ hang-up is getting the release management tools over their final hump and just driving the trac tickets to completion.  With Tom keeping the review queue basically empty right now, this is an excellent opportunity to get that done.

It may make sense to schedule an event where we all show up on IRC, everyone claims a documentation component, and we all do a final review pass to make sure that the formatting problems aren't too bad before going to trunk with the cut-over.  This pre-supposes that the release/building tools are done and on trunk though.

Anyway, I'm not giving up.  If nothing else, I'll end up with a nice restructuredText-generating library.  And if Twisted never ends up adopting Sphinx as a doc tool, eventually I'll still be able to read the Twisted docs in a format that I can navigate and doesn't hurt my eyes to look at. :)

But I'd really rather see Twisted adopt Sphinx, and get rid of Lore.

Help accepted.

All right!  I hope this exchange has gotten some people fired up to cross the finish line.  It's surprisingly close!  Thanks for updating us, Kevin - better late than never :).

-glyph

P.S.: apologies for any errors.  I didn't even really have the time to write this email, let alone copy-edit it.