Hey Kevin! Glyph and Jean-Paul got me up to speed (somewhat) on the whole lore2sphinx thing last night—though they were rather unclear as to how close it is to completion. (which you addressed later in this thread)
I certainly want to combine our efforts as much as possible and I'm not too worried about working at cross purposes; your main task seems to be converting the existing documentation while mine is reorganizing, editing, and standardizing it. I'm sure our paths will start to cross when the lore conversion is complete and you have a huge Sphinx project ready to be properly organized and catalogued. My hope is we can then begin the process of merging the two sides of this coin.
My current plan (which should become a bit more clear throughout this massive reply) is to start from scratch but modify existing documentation wherever possible. If a piece of documentation fits in a "hole" in the structure and I can get by with a mv and some editing, that's what I'd like to do—it seems the most pragmatic approach.
This. Point of fact, (3) is already implemented; that example task simply links to projects/web/tasks/serve. This strikes me as the most logical layout as the tasks are all specific to a particular project (the only iffy one being Core which is sort of a few projects plus the foundation)—we just want to highlight some of them.
On to Glyph's reply...
There are many outstanding documentation branches which are substantial improvements, which need to be edited and merged - the trial tutorial among them. It would be great if your efforts could start with getting those landed, turning the crank on the process to get our users better documentation in the interim, as you survey the existing documentation.
I definitely agree that resolving the low-hanging fruit first is a good idea. For finishing "docs branch X" to make sense, my personal belief is that X should:
- Still be relevant in terms of best practices and simply what's available
- If project documentation, have outstanding issues that a passing familiarity with the project [right now] will be sufficient to close them (I could spend time learning one project, or the same time improving all the documentation)
- Adhere to whatever documentation standards we agree upon, if much is to left to do.
I guess my overall opinion here is that, yes, if relatively minor edits can bring a branch close enough to completion that we can get it out there to help newcomers now, let's do that. If a branch is more of a draft and requires a good deal of fleshing out (or is simply stale), it's probably worth nailing down the structure and previously mentioned docs standards before I just create more work for myself (or others) down the road.
Finally, my biggest hurdle right now is not knowing how to find said branches. I don't see "documentation" as a category in Trac and common keyword searches didn't show up much for me. I'm sure this is an easy question to answer, though.
What should a newcomer who reads this document know by the end of it?
I'm not sure because I can't see how a practical guide to creating something so generic really fits in the grand scheme of things. I think if you want to create a TCP server from scratch you must first create the Universe! In this case, that means learning how Twisted addresses the concept of a server before ever bothering to write one so generic. My general beef is that many documents seem to make an attempt to appeal to everybody and in doing so don't sufficiently help anybody. Maybe I can justify that claim better with examples of "better" (at least more targeted) documents.
A massive pile of improved documentation would of course be useful, but a good start would be a clear statement of requirements and audience. (As well as an enumeration ofdifferent audiences that different documents might serve.)
I completely agree. The updates I made last night (hopefully) address the issue of different audiences and needs, at least as a beginning stage. What Jason said about the Python docs is a great parallel: Tutorials, [Project] Reference, [Core Concepts] Reference. All of these are important and cater to a different need. I think we're quite on the same page here, though.
(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"?
It's inconsistent and I completely agree (and the consensus seems to be) that event-* is superior.
You will probably have to press us core developers on this one, and you may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
Debates are great! And I fully intend to bitch about the lack of "best ways" to do certain things—having one clear way to accomplish something is, after all, a core tenant of Python ;) -- more than that, it's really annoying to everyone involved (readers, writers, and developers) when there are many different ways to do something and none of them have very clear use cases or strengths / weaknesses. Definitely a TODO.
The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
Kevin touched on this already, but I really think if we make maintaining and growing a Project something that is both honorable and accessible, more people will want to do it. If you like Python and DNS or SMTP or whatever the hell else, what has the potential to be a more awesome implementation than its Twisted one? Let's help people find out for themselves that the answer is nothing. Then they'll want it to be their project.
As problematic as the current situation is, there is a definite potential for some baby vs. bathwater confusion in improving it... Also, while you're energized now, please consider what happens to our documentation efforts if you run out of time or out of motivation halfway through this process. Incremental improvements may not necessarily provide the best outcome, but they _do_ mean that users get to see some results from these volunteer efforts even if the original volunteer gets busy with other aspects of life and can't complete their project.
I have no intention of rewriting everything, even if it sounds absurdly entertaining to make myself learn and write practical examples of every Twisted project. I am (just) slightly more realistic than that. Having the Sphinx conversion complete will be a huge help, if for no other reason than the documentation will be easy to port, edit, and find in the first place.
On the subject of loss of motivation or time, sure. It happens to all of us. And it's not my intention to keep the repository on GitHub completely separate and wait until it's perfect to get it to users. Once all the documentation is under Sphinx, there will be no reason for that. The existing documentation can be sliced and diced into the newly-minted structure and then combed over piece by piece. Some of it will likely need to be thrown out wholesale, but I imagine much of it will live on with minor edits (and probably new code) at new URLs.
As for Launchpad, etc.—I want to do whatever makes you guys (a) most comfortable and (b) most likely to contribute. I chose to start the GitHub project because I know Git well and it allowed me to spend all my time writing and none of it figuring out something foreign. I don't understand the link between Twisted Trac and Twisted on Launchpad, so an explanation of that would be awesome. Provided that, though, how complicated could Bazaar possibly be? We can leave Twisted's migration to GitHub as task #2 on my list ;)
I'd really like any effort that undertakes to substantially change the structure of the documentation to make a concerted effort to hit the ground running with test-driven examples which can be automatically verified, so that we have some idea of the impact of code changes on the docs.
Most definitely. Documentation requires the same maintenance that code does and can benefit from the same automated verification; and it's trivial to write entire programs and show code samples using literalinclude and the :lines: option. Failing code can be fixed and grep'd for realignment in the documentation.
Indeed, as the author of many of these original documents, I did not feel insulted in either my person or my work :). Hopefully you didn't feel that way either after reading this reply!
Not at all!
* * *
I believe I covered the main thoughts / concerns / queries related to my original post. My apologies if I missed anybody! And it's great to finally be working with you :)
Cheers,
Tom
On Thu, Jan 20, 2011 at 8:20 PM, Tim Allen
<screwtape@froup.com> wrote:
On Thu, Jan 20, 2011 at 06:34:17PM -0600, Kevin Horn wrote:
> On Thu, Jan 20, 2011 at 5:57 PM, Tim Allen <
screwtape@froup.com> wrote:
> > You mean these DTDs?
> >
> > twisted/lore/xhtml1-strict.dtd
> > twisted/lore/xhtml1-transitional.dtd
> >
> > They reference the xhtml-*.ent entity definitions which are also in the
> > same directory. It would be neat if lore2sphinx could be taught to use
> > the DTDs packaged with lore instead of having to download them from the
> > Internet every time.
>
> Huh. Never even knew that was there. It probably could, and the reason it
> downloads from the internet was because that's the default way of doing it
> in lxml. I've since figured out how to override that behavior (which is how
> the caching works) so maybe that wouldn't even be hard. The easiest/fastest
> fix for the moment though would probably be to pre-populate the cache as I
> mentioned before, since IIRC, this would just involve adding the file to my
> hg repo. I'll have to look into it though, it may be just as easy to do it
> the other way, though I don't want to depend necessarily on having Twisted's
> code available (remember, this is supposed to work on the various divmod
> projects, and anything else that uses Lore, too).
Well, you wouldn't necessarily be depending on Twisted, just depending
on Lore - and I imagine any not-Twisted project whose documentation
depends on Lore has already made peace with the idea of depending on
Lore. :)
If it's easier to just copy these well-known DTDs into the lore2sphinx
repository, I guess that's a good plan too - it's not like the W3C is
going to suddenly issue updated XHTML DTDs.