I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the
trial-tutorial branch which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly,
twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge.
Reading code to find answers isn't rocket science; I've been programming long enough to be comfortable doing it. But I probably have to resort to reading Twisted's code about 8 billion percent more often than any other codebase. And reading code is a hurdle. Reading through Twisted's semi-random, 45-point FAQ is a hurdle—and recommending it as a starting point is unhelpful at best. The core documentation isn't awesome either, given that it has a tendency to be overly cryptic and link to API documentation that is often incomplete or generally unhelpful.
- It's tutorial page #1, but basically tells me I need to read howto/plugins.html first if I am writing an "application" (whatever that is), as opposed to a "TCP, SSL, and Unix socket server". And it's the wrong place for UDP.
- It attempts to introduce Protocols and Factories—two of Twisted's most important concepts—and does neither particularly well. I know that Protocols (usually) inherit from t.i.p.Protocol and may be instantiated for a variety of reasons and aren't (usually) persistent. I also know that Factories instantiate Protocols and give a reference to themselves so protocols can "access and modify the persistent configuration". I am told I need to implement some interface (or something) to actually listen on a host/port. I think.
- At one point "TCP4ServerEndpoint" is instantiated (but never imported); its explanation is left to a digression into the endpoints API, which has its own issues. Suffice it to say the document doesn't give me sufficient reason to actually use the endpoints API.
- Later on, we just use reactor.listenTCP()—which our previous digression (if bothered to click through and read) claims is not preferable.
By the end of the servers tutorial (and after reading some linked documentation), here's all I really know:
- Factories create protocols somehow
- Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events
- There are other protocols with other methods. One that I know of, anyway.
- I may need to write a state machine (???)
- I should use an Endpoint or maybe a Service or reactor (but probably not!)
- I should also use Application for serious business
Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server.
I think the point has been made. My real point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine and despite its decade of existence it remains largely confusing and obscure to the majority of Python programmers who come upon it. It contains concepts and standards that are alien to the average Python programmer, but they make a lot of sense and have a lot more consistency and predictability than the documentation conveys.
I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later.
There are some things I do want to accomplish early on:
- Make the docs accessible (a lot is hidden and hard to find)
- Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use)
- Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications.
- Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too.
There are also some things I don't want to accomplish, at least initially:
- Explain asynchronous programming in depth
- Get into the level of detail that the Krondo series does (but I plan to borrow from it!)
There are things I'll need help with:
- The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible").
- Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge.
Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:
I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them.
Cheers,
Tom