[Twisted-Python] Refactoring Documentation
I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the trial-tutorial branch<http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2> which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge. Reading code to find answers isn't rocket science; I've been programming long enough to be comfortable doing it. But I probably have to resort to reading Twisted's code about 8 billion percent more often than any other codebase. And reading code is a hurdle. Reading through Twisted's semi-random, 45-point FAQ is a hurdle—and recommending it as a starting point is unhelpful at best. The core documentation isn't awesome either, given that it has a tendency to be overly cryptic and link to API documentation that is often incomplete or generally unhelpful. As one very basic example, see: http://twistedmatrix.com/documents/current/core/howto/servers.html. Let's just review a few things wrong with this page: - It's tutorial page #1, but basically tells me I need to read howto/plugins.html first if I am writing an "application" (whatever that is), as opposed to a "TCP, SSL, and Unix socket server". And it's the wrong place for UDP. - It attempts to introduce Protocols and Factories—two of Twisted's most important concepts—and does neither particularly well. I know that Protocols (usually) inherit from t.i.p.Protocol and may be instantiated for a variety of reasons and aren't (usually) persistent. I also know that Factories instantiate Protocols and give a reference to themselves so protocols can "access and modify the persistent configuration". I am told I need to implement some interface (or something) to actually listen on a host/port. I think. - At one point "TCP4ServerEndpoint" is instantiated (but never imported); its explanation is left to a digression into the endpoints API, which has its own issues. Suffice it to say the document doesn't give me sufficient reason to actually use the endpoints API. - Later on, we just use reactor.listenTCP()—which our previous digression (if bothered to click through and read) claims is not preferable. By the end of the *servers* tutorial (and after reading some linked documentation), here's all I *really* know: 1. Factories create protocols somehow 2. Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events 3. There are other protocols with other methods. One that I know of, anyway. 4. I may need to write a state machine (???) 5. I should use an Endpoint or maybe a Service or reactor (but probably not!) 6. I should also use Application for serious business Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server. I think the point has been made. My *real* point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine and despite its decade of existence it remains largely confusing and obscure to the majority of Python programmers who come upon it. It contains concepts and standards that are alien to the average Python programmer, but they make a lot of sense and have a lot more consistency and predictability than the documentation conveys. I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later. There are some things I *do* want to accomplish early on: - Make the docs accessible (a lot is hidden and hard to find) - Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use) - Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications. - Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too. There are also some things I *don't* want to accomplish, at least initially: - Explain asynchronous programming in depth - Get into the level of detail that the Krondo series does (but I plan to borrow from it!) There are things I'll *need help with*: - The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible"). - Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge. Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here: https://github.com/tdavis/twisted-docs I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them. Cheers, Tom
On Thu, Jan 20, 2011 at 12:10 AM, Tom Davis <tom@recursivedream.com> wrote:
I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the trial-tutorial branch<http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2> which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge.
Reading code to find answers isn't rocket science; I've been programming long enough to be comfortable doing it. But I probably have to resort to reading Twisted's code about 8 billion percent more often than any other codebase. And reading code is a hurdle. Reading through Twisted's semi-random, 45-point FAQ is a hurdle—and recommending it as a starting point is unhelpful at best. The core documentation isn't awesome either, given that it has a tendency to be overly cryptic and link to API documentation that is often incomplete or generally unhelpful.
As one very basic example, see: http://twistedmatrix.com/documents/current/core/howto/servers.html. Let's just review a few things wrong with this page:
- It's tutorial page #1, but basically tells me I need to read howto/plugins.html first if I am writing an "application" (whatever that is), as opposed to a "TCP, SSL, and Unix socket server". And it's the wrong place for UDP. - It attempts to introduce Protocols and Factories—two of Twisted's most important concepts—and does neither particularly well. I know that Protocols (usually) inherit from t.i.p.Protocol and may be instantiated for a variety of reasons and aren't (usually) persistent. I also know that Factories instantiate Protocols and give a reference to themselves so protocols can "access and modify the persistent configuration". I am told I need to implement some interface (or something) to actually listen on a host/port. I think. - At one point "TCP4ServerEndpoint" is instantiated (but never imported); its explanation is left to a digression into the endpoints API, which has its own issues. Suffice it to say the document doesn't give me sufficient reason to actually use the endpoints API. - Later on, we just use reactor.listenTCP()—which our previous digression (if bothered to click through and read) claims is not preferable.
By the end of the *servers* tutorial (and after reading some linked documentation), here's all I *really* know:
1. Factories create protocols somehow 2. Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events 3. There are other protocols with other methods. One that I know of, anyway. 4. I may need to write a state machine (???) 5. I should use an Endpoint or maybe a Service or reactor (but probably not!) 6. I should also use Application for serious business
Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server.
I think the point has been made. My *real* point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine and despite its decade of existence it remains largely confusing and obscure to the majority of Python programmers who come upon it. It contains concepts and standards that are alien to the average Python programmer, but they make a lot of sense and have a lot more consistency and predictability than the documentation conveys.
I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later.
There are some things I *do* want to accomplish early on:
- Make the docs accessible (a lot is hidden and hard to find) - Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use) - Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications. - Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too.
There are also some things I *don't* want to accomplish, at least initially:
- Explain asynchronous programming in depth - Get into the level of detail that the Krondo series does (but I plan to borrow from it!)
There are things I'll *need help with*:
- The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible"). - Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge.
Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:
https://github.com/tdavis/twisted-docs
I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them.
Cheers,
Tom
Tom, I very much like this idea, and in fact I have some ideas for this, but I've been putting off doing anything about it until the (neraly done! really!) conversion of the current documentation to Sphinx is complete (see ticket #4500). I got off track with that the last part of last year, but one of my New Year's Resolutions is to complete that project. How can we combine these efforts, or at least keep from working at cross purposes? I see from your link above that you are building your own Sphinx project. Perhaps you would be better off working from the results of the Lore2sphinx conversion? Are you modifying existing docs or working from scratch? Let's get together on this! Kevin Horn
Let me begin by saying that I'm _very_ excited about this prospect and I'm looking forward to the output of this project. So, some of what I'm about to say may sound like harsh criticism but please do not let it discourage you. I agree with pretty much everything you're saying about problems with the current documentation. They need to be fixed. I'm offering criticism at this early stage in the hopes that it will be useful and feed into your project, not in the hopes that it will stop you. On Jan 20, 2011, at 1:10 AM, Tom Davis wrote:
I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the trial-tutorial branch which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge.
Yet, the problem _starts_ with a branch merge. There are many outstanding documentation branches which are substantial improvements, which need to be edited and merged - the trial tutorial among them. It would be great if your efforts could start with getting those landed, turning the crank on the process to get our users better documentation in the interim, as you survey the existing documentation.
Factories create protocols somehow Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events There are other protocols with other methods. One that I know of, anyway. I may need to write a state machine (???) I should use an Endpoint or maybe a Service or reactor (but probably not!) I should also use Application for serious business Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server.
I think the point has been made.
This kind of tear-down is pretty easy: I'd really like to hear your idea of its inverse, though. What should a newcomer who reads this document know by the end of it? A massive pile of improved documentation would of course be useful, but a good start would be a clear statement of requirements and audience. (As well as an enumeration of different audiences that different documents might serve.)
My real point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine
(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"? I find that especially in documentation it's a lot easier to explain "event-driven", because you can enumerate what the events are, instead of explaining the etymology of "synchronicity"...)
I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later.
There are some things I do want to accomplish early on: Make the docs accessible (a lot is hidden and hard to find) Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use) Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications. Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too. These all sound like excellent goals.
There are also some things I don't want to accomplish, at least initially: Explain asynchronous programming in depth Get into the level of detail that the Krondo series does (but I plan to borrow from it!) There are things I'll need help with: The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible"). You will probably have to press us core developers on this one, and you may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things. Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge. The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:
There have been several abortive efforts to do something grand to re-invent all of the Twisted documentation in the universe, or a complete overhaul of the website, including several false starts that I've made, and most recently the (somewhat arduous, arguably "mostly complete" (fingers crossed on that one)) attempt to do a sphinx migration. By the way, it seemed tonight that you weren't that familiar with the existing Sphinx efforts, so you may want to have a look at this and see if it is an improvement on our existing documentation index: http://twistedsphinx.funsize.net/contents.html (I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process. Anyone know what this is about? <http://buildbot.twistedmatrix.com/builds/sphinx-html-15724/contents.html>) I'm really glad to see that Kevin beat me to the punch on posting his reply though, hopefully we'll see this moving along soon :). Before you continue too much further in that github repository though, please, consider this: ~/Projects/Twisted/trunk/doc$ find conch core mail names pair web words -name '*.xhtml' -or -name '*.py' | xargs wc -l . . . 28836 total That's 28 thousand lines of documentation and sample code. It might not all be the best documentation, but some of it is quite good. Jean-Paul's "twisted web in 60 seconds" tutorial is in there, and that's definitely worth reading. Would you volunteer to rewrite 28 thousand lines of code all at once? Or submit a 28 thousand line patch for review? As problematic as the current situation is, there is a definite potential for some baby vs. bathwater confusion in improving it. I am open to the idea that documentation needs different review practices from code, and there are things about documentation (like narrative flow) which are impossible to automatically verify and may require significant changes at multiple layers simultaneously. These features may not have an obvious parallel in code - which can typically be evolved incrementally, despite the occasional frustrated lament that everything needs to be rewritten. But, parts of the documentation are code. Another major problem with the current documentation is that it is out of date. We know when code goes out of date because it starts raising exceptions and emitting warnings from our test suite, but we only find out about documentation examples are going out of date when we do a release and some poor soul encounters an error when trying to follow along. I'd really like any effort that undertakes to substantially change the structure of the documentation to make a concerted effort to hit the ground running with test-driven examples which can be automatically verified, so that we have some idea of the impact of code changes on the docs. Also, while you're energized now, please consider what happens to our documentation efforts if you run out of time or out of motivation halfway through this process. Incremental improvements may not necessarily provide the best outcome, but they _do_ mean that users get to see some results from these volunteer efforts even if the original volunteer gets busy with other aspects of life and can't complete their project. Many of your bullet-points above are well suited to individual efforts (some of which are actually issues with the web site, rather than the documentation per se). I'd like to suggest that, at the very least, you create a branch of Twisted (perhaps on Launchpad?) and integrate your work into the doc/ hierarchy of the current tree. That way, at least, it would be easier for someone else to pick up where you've left off and fold things into the right place. This would also provide a way to more easily integrate the doc test suite into the main test suite on our existing build and release infrastructure.
I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them.
Indeed, as the author of many of these original documents, I did not feel insulted in either my person or my work :). Hopefully you didn't feel that way either after reading this reply!
On Thu, Jan 20, 2011 at 2:55 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"? I find that especially in documentation it's a lot easier to explain "event-driven", because you can enumerate what the events are, instead of explaining the etymology of "synchronicity"...)
I was also surprised by the choice of "asynchronous". From wikipedia: In programming, asynchronous events are those occurring independently of the
main program flow. Asynchronous actions are actions executed in a non-blocking scheme, allowing the main program flow to continue processing.
My understanding is that this is the opposite of what twisted is. The reactor hands-off control and must wait until control is relinquished. It handles events when it checks for them, not necessarily when they happen. The reactor and application code blocks, possibly halting the entire application if it is not written cooperatively. This was a major point of confusion for me when I started to use twisted. I see that if I had carefully read the first few sentences under "Reactor Basics" in http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html, I might not have been so confused. But, this is one of about 50 links on the "core" documentation page, which is one of about 20 links on the main documentation page. I'm guessing that simply editing and reorganizing existing documentation would go a long way toward fixing the documentation problem. Personally, I'd love to see documentation organization that mimics Python's, especially the Tutorial/Library Reference/Language Reference breakdown. Users tend to know the level of understanding they are looking for and Python's documentation reflects that. Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Hear! Hear! My major disappointment with Twisted is its documentation. I've used many many packages over the years, some with books and books of documentation (HP OpenView, e.g.), but I've never encountered a package with poorer help for a newbie. I finally started to get it when I stumbled upon http://krondo.com/blog/?page_id=1327. This has to be one of the best tutorials I've seen on any topic. It should be the *first* link in any Twisted tutorial, IMO. Vic On Thu, Jan 20, 2011 at 8:57 AM, Jason Rennie <jrennie@gmail.com> wrote:
On Thu, Jan 20, 2011 at 2:55 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"? I find that especially in documentation it's a lot easier to explain "event-driven", because you can enumerate what the events are, instead of explaining the etymology of "synchronicity"...)
I was also surprised by the choice of "asynchronous". From wikipedia:
In programming, asynchronous events are those occurring independently of
the main program flow. Asynchronous actions are actions executed in a non-blocking scheme, allowing the main program flow to continue processing.
My understanding is that this is the opposite of what twisted is. The reactor hands-off control and must wait until control is relinquished. It handles events when it checks for them, not necessarily when they happen. The reactor and application code blocks, possibly halting the entire application if it is not written cooperatively.
This was a major point of confusion for me when I started to use twisted. I see that if I had carefully read the first few sentences under "Reactor Basics" in http://twistedmatrix.com/documents/current/core/howto/reactor-basics.html, I might not have been so confused. But, this is one of about 50 links on the "core" documentation page, which is one of about 20 links on the main documentation page. I'm guessing that simply editing and reorganizing existing documentation would go a long way toward fixing the documentation problem.
Personally, I'd love to see documentation organization that mimics Python's, especially the Tutorial/Library Reference/Language Reference breakdown. Users tend to know the level of understanding they are looking for and Python's documentation reflects that.
Cheers,
Jason
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.” -- Antoine de Saint Exupéry
On Thu, Jan 20, 2011 at 1:55 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Let me begin by saying that I'm _very_ excited about this prospect and I'm looking forward to the output of this project. So, some of what I'm about to say may sound like harsh criticism but *please do not let it discourage you*. I agree with pretty much everything you're saying about problems with the current documentation. They need to be fixed. I'm offering criticism at this early stage in the hopes that it will be useful and feed into your project, not in the hopes that it will stop you.
On Jan 20, 2011, at 1:10 AM, Tom Davis wrote:
I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the trial-tutorial branch<http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2> which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge.
Yet, the problem _starts_ with a branch merge.
There are many outstanding documentation branches which are substantial improvements, which need to be edited and merged - the trial tutorial among them. It would be great if your efforts could start with getting those landed, turning the crank on the process to get our users better documentation in the interim, as you survey the existing documentation.
1. Factories create protocols somehow 2. Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events 3. There are other protocols with other methods. One that I know of, anyway. 4. I may need to write a state machine (???) 5. I should use an Endpoint or maybe a Service or reactor (but probably not!) 6. I should also use Application for serious business
Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server.
I think the point has been made.
This kind of tear-down is pretty easy: I'd really like to hear your idea of its inverse, though. What *should* a newcomer who reads this document know by the end of it? A massive pile of improved documentation would of course be useful, but a good start would be a clear statement of requirements and audience. (As well as an enumeration of *different* audiences that different documents might serve.)
My *real* point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine
(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"? I find that especially in documentation it's a lot easier to explain "event-driven", because you can enumerate what the events are, instead of explaining the etymology of "synchronicity"...)
I also find that if you use "event-driven" you can compare to event-driven GUI apps, when giving an explanation. Many, many people who have no idea about what "asynchronous networking" is will get the idea pretty easily when you make this comparison. Not everybody, certainly, but a lot.
I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later.
There are some things I *do* want to accomplish early on:
- Make the docs accessible (a lot is hidden and hard to find) - Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use) - Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications. - Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too.
These all sound like excellent goals.
There are also some things I *don't* want to accomplish, at least initially:
- Explain asynchronous programming in depth - Get into the level of detail that the Krondo series does (but I plan to borrow from it!)
There are things I'll *need help with*:
- The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible").
You will probably have to press us core developers on this one, and you may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
- Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge.
The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
I think this right here is the main reason that the docs need to be improved. Easier newbie experience translates to more users translates to more developers translates to more maintainers. Especially if the perception of Twisted as a "languishing" or even "mostly dead" project can be undone. (Yes, I've heard this a number of times. No, I don't know where they get it. I correct it whenever I can. It's something that needs to be addressed, but that's a different conversation.). Fixing up the docs will at least help with this.
Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:
https://github.com/tdavis/twisted-docs
There have been several abortive efforts to do something grand to re-invent all of the Twisted documentation in the universe, or a complete overhaul of the website, including several false starts that I've made, and most recently the (somewhat arduous, arguably "mostly complete" (fingers crossed on that one)) attempt to do a sphinx migration.
Seriously man, we're close. 3 more "chunk tickets" in the "edit the lore source" phase (and one of those is finished I think and just needs to be merged). Then another round of "chunk tickets" to manually fix any other little typographical issues in the Sphinx source, and done. I suppose there will also need to be a website deployment process. And probably lots of other minor things that we'll discover as we go. I've had some major speedbumps, had to find a new job, kids were sick several times, etc. You know...life. I'm getting fired up again though, and thijs seems to be as well, and this coming up now just pumps the bellows. By the way, it seemed tonight that you weren't that familiar with the
existing Sphinx efforts, so you may want to have a look at this and see if it is an improvement on our existing documentation index:
http://twistedsphinx.funsize.net/contents.html
(I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process. Anyone know what this is about? < http://buildbot.twistedmatrix.com/builds/sphinx-html-15724/contents.html>)
I'm really glad to see that Kevin beat me to the punch on posting his reply though, hopefully we'll see this moving along soon :).
I was glad to see just about anyone else interested! :) It looks like Tom and I have some similar ideas on where we should be going, and I think his Sphinx skeleton is a great example of what things should eventually look like. As I said earlier, I've had a number of similar ideas, though it looks like maybe Tom's are a little more concrete and/or fleshed-out. I just really wanted to get the Sphinx convo "out the door", before I took on another huge project. I'll elaborate in a separate email.
Before you continue too much further in that github repository though, please, consider this:
~/Projects/Twisted/trunk/doc$ find conch core mail names pair web words -name '*.xhtml' -or -name '*.py' | xargs wc -l . . . 28836 total
That's 28 thousand lines of documentation and sample code. It might not all be the best documentation, but some of it is quite good. Jean-Paul's "twisted web in 60 seconds" tutorial is in there, and that's definitely worth reading. Would you volunteer to rewrite 28 thousand lines of code all at once? Or submit a 28 thousand line patch for review? As problematic as the current situation is, there is a definite potential for some baby vs. bathwater confusion in improving it.
I am open to the idea that documentation needs different review practices from code, and there are things about documentation (like narrative flow) which are impossible to automatically verify and may require significant changes at multiple layers simultaneously. These features may not have an obvious parallel in code - which can typically be evolved incrementally, despite the occasional frustrated lament that everything needs to be rewritten.
But, parts of the documentation are code. Another major problem with the current documentation is that it is out of date. We know when code goes out of date because it starts raising exceptions and emitting warnings from our test suite, but we only find out about documentation examples are going out of date when we do a release and some poor soul encounters an error when trying to follow along. I'd really like any effort that undertakes to substantially change the structure of the documentation to make a concerted effort to hit the ground running with test-driven examples which can be automatically verified, so that we have some idea of the impact of code changes on the docs.
There's a ticket for writing tests for the code samples in Trac.
Also, while you're energized now, please consider what happens to our documentation efforts if you run out of time or out of motivation halfway through this process. Incremental improvements may not necessarily provide the best outcome, but they _do_ mean that users get to see some results from these volunteer efforts even if the original volunteer gets busy with other aspects of life and can't complete their project. Many of your bullet-points above are well suited to individual efforts (some of which are actually issues with the web site, rather than the documentation per se).
@Tom: agree here. Let's get _some_ benefit out of your energy, rather than you getting discouraged and having half a great set of docs floating out there someplace confusing users. I've been through this myself a couple of times (I've been pretty much ready to "pull the trigger" at least three times!) and have gotten discouraged and/or off track. After a while, my "voom" comes back, but in the meantime, the trunk has progressed, and there's that much more work to do. My advice is to try and get many small changes made, and get them _done_ rather than a few huge changes.
I'd like to suggest that, at the very least, you create a branch of Twisted (perhaps on Launchpad?) and integrate your work into the doc/ hierarchy of the current tree. That way, at least, it would be easier for someone else to pick up where you've left off and fold things into the right place. This would also provide a way to more easily integrate the doc test suite into the main test suite on our existing build and release infrastructure.
I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them.
Indeed, as the author of many of these original documents, I did not feel insulted in either my person or my work :). Hopefully you didn't feel that way either after reading this reply!
I think this has been pretty constructive on both sides. I look forward to more. Kevin Horn
I alluded in my earlier mail to some of the improvement ideas I have had regarding Twisted's docs (other than the conversion to Sphinx), and I though I'd elaborate Here's some things I had "planned" (yes, I'm using the term loosely) to try and add/improve/fix in the Twisted docs after the Sphinx conversion was finished: (urg, I know I've been keeping a list of these someplace, and some are in trac tickets, etc. but I can't find it right now, so I may ramble a bit) - Add a basic intro to Twisted. This would introduce some basic ideas similar to the krondoblog tutorial, but in less detail. - "What is Twisted good for" - Explain the most important parts of Twisted for people to get started with. IMO, this is a) the reactor, b) deferreds, c) some of the basic interfaces/classes, esp. Factories/Protocols (I'd like to see some docs on Transports as well) - more/better UDP stuff - "How to test your Twisted apps" (e.g., the idea of faking up a transport never occurred to me until I read a test that did it, and it's been one of the most useful techniques I've found) - Task-based docs - how to serve a web page, how to send a mail, how to write an IRC client (just to cut down on the questions!) etc. The basic stuff at first (what newbie's will be looking for), maybe eventually turning into a "cookbook" of sorts. - A revamped section on "How to write twisted documentation", since that would likely be very different after the Sphinx conversion ( What Sphinx markup to use, and I also have some custom extensions, etc. which need to be documented). - "How to _build_ the documentation" - Beginner's tutorial - better glossary - move a bunch of stuff out of the Trac wiki, and into the Sphinx project. There's stuff there that has become de-facto documentation, which really needs to be version-controlled, etc. - install docs So after looking over Tom's "skeleton" Sphinx project and a night's sleep, I think we're pretty close to on the same page (or at least pages in the same book). It looks like a lot of this would be covered in Tom's Task-based docs and the Base Documentation section ("Suiting Up", "Diving in"). So here's what I'm kind of thinking as far as combining efforts: 1) I'll continue with the "Project Documentation" conversion, while Tom works on the other bits. Should be fairly easy to combine them, I'm thinking. 2) Let's leave the "Project Documentation" pretty much as-is for the moment, until the Sphinx convo is done. 3) I wonder if at least some of the "task-based docs" shouldn't be put into the project sections, and then just linked to from the task-based section? Thoughts, Tom? [As an aside, is there any way to get the Combinator stuff from the old Trac wiki back online, or at least readable someplace? (Hrmmm, it looks like Google's cache has it for the moment). Also, it would be much easier to get new contributors, especially for Windows, if Combinator worked out of the box on Windows. There's (or were) 3 or 4 patches in the Divmod SVN repo that needed to be applied in order to get it to work, and now that the Divmod site is offline it's really hard to set it up on a new machine, even for me, and I have a working example to go from. Yeah it's off topic, sue me.]
I have a lot of input here, but as the guys actually doing the work, the final say is mostly with you and Tom. So, please feel free to take everything here with a grain of salt. On Jan 20, 2011, at 2:58 PM, Kevin Horn wrote:
Here's some things I had "planned" (yes, I'm using the term loosely) to try and add/improve/fix in the Twisted docs after the Sphinx conversion was finished: (urg, I know I've been keeping a list of these someplace, and some are in trac tickets, etc. but I can't find it right now, so I may ramble a bit)
Thanks for writing up this list. I'm re-ordering just a little bit, to start with what I think are the good bits...
- Task-based docs - how to serve a web page, how to send a mail, how to write an IRC client (just to cut down on the questions!) etc. The basic stuff at first (what newbie's will be looking for), maybe eventually turning into a "cookbook" of sorts.
Yes, yes, yes. I think everything should be driven from this, as I will repeat many times through this email :).
- "How to test your Twisted apps" (e.g., the idea of faking up a transport never occurred to me until I read a test that did it, and it's been one of the most useful techniques I've found)
And this is clearly my second priority. Personally, I really wish there were a tutorial to Twisted itself which were test-driven, rather than separating out "here's how you get started" and "here's how you write your tests". It may be too much material to present at once, but on the other hand, the trial command-line is very friendly, so that
- Add a basic intro to Twisted. This would introduce some basic ideas similar to the krondoblog tutorial, but in less detail.
This is where everybody wants to start, but I'd like to play devil's advocate here. Before anybody tries another grand restart on tutorial / introductory docs (the last one was the now frequently vilified Finger Tutorial), we should ask these questions: Who wants to read this introduction? What do they want to learn? What problem are they actually trying to solve, and how is that going to be furthered by their reading this document? I think the place to start is really to focus on particular tasks that people want to accomplish, with a pointer to more abstract documentation once they want to learn more about how it's all put together. Anyway, I'm not going to _completely_ repeat everything I said to Tom at the meetup, but we should take a look at the individual project pages and make them things people want to read, and have tutorials that are focused on each project, since almost everyone who is approaching Twisted as a newbie really wants to get some specific task done, not learn generally about event-driven networking.
- "What is Twisted good for"
Again: it depends what you want to use it for. One sprawling page that explains every benefit that Twisted has is really hard to articulate.
- Explain the most important parts of Twisted for people to get started with. IMO, this is a) the reactor, b) deferreds, c) some of the basic interfaces/classes, esp. Factories/Protocols (I'd like to see some docs on Transports as well)
... okay I'm totally harping on this now, but: I think that focusing on the higher-level stuff and getting people running with actual applications should be more of a focus. There's already a lot of conceptual / introductory material, but it lacks a coherent narrative or clearly-defined audience, and I think that's why it's bad. In some places this may require some actual code changes. For example, twisted.names doesn't really have an API to speak of, and while writing a tutorial on how to use it, you're going to bump into that. That might motivate somebody to go and write the 20 lines of glue which would actually be required to write a dynamic DNS server without subclassing some internal stuff and twiddling undocumented attributes, and that would be great. (But, lest I get ahead of myself: let's get started on the documentation first; starting off with a bunch of features because "maybe they'll be useful for docs" is an even worse trap to fall into :)) But, I don't think a lot of people are actually developing custom protocols from scratch; and those who are, should probably be using a construction kit like AMP, or PB, or Foolscap, so that they can skip over the tedious and easy-to-get-wrong bits about framing and figuring out a language of how to document messages.
- more/better UDP stuff
Let's please make this the lowest priority and the last thing that gets done. I don't think I've ever heard anyone who _actually_ needed to use UDP asking questions about it on IRC or on the mailing list. If you're using Twisted for DNS (arguably the most common usage of UDP), it's already done for you. Almost universally, questions about UDP come from people who don't really understand that they should be using TCP because they read on some MMORPG-design forum somewhere that UDP is "faster". People who already understand the nuances of UDP and know why it's reasonable for movement packets but not for in-game micropayments, for example, might hit a speedbump or two, but they'll get through it pretty easily. Let's address the audience that really needs the documentation before we start adding it for people whose lives will only get very slightly easier.
- A revamped section on "How to write twisted documentation", since that would likely be very different after the Sphinx conversion ( What Sphinx markup to use, and I also have some custom extensions, etc. which need to be documented). - "How to _build_ the documentation"
Writing this alongside designing the new documentation build process would be a big help :). (Although really, it should be at most three steps: "install sphinx", "install epydoc", and then "run ./admin/build-twisted-documentation". If it's more than that then something is probably wrong...)
- Beginner's tutorial
... again, tutorial for what?
- better glossary
Wow, do we even have a glossary? It seems to me this task might be better served by links to the API docs.
- move a bunch of stuff out of the Trac wiki, and into the Sphinx project. There's stuff there that has become de-facto documentation, which really needs to be version-controlled, etc.
+1
- install docs
So after looking over Tom's "skeleton" Sphinx project and a night's sleep, I think we're pretty close to on the same page (or at least pages in the same book). It looks like a lot of this would be covered in Tom's Task-based docs and the Base Documentation section ("Suiting Up", "Diving in").
Great.
So here's what I'm kind of thinking as far as combining efforts: 1) I'll continue with the "Project Documentation" conversion, while Tom works on the other bits. Should be fairly easy to combine them, I'm thinking.
Getting Tim to help out with some conversion stuff (and you helping with some of his stuff, as well) might accelerate the review process, which has been part of the bottleneck.
2) Let's leave the "Project Documentation" pretty much as-is for the moment, until the Sphinx convo is done. 3) I wonder if at least some of the "task-based docs" shouldn't be put into the project sections, and then just linked to from the task-based section?
Thoughts, Tom?
[As an aside, is there any way to get the Combinator stuff from the old Trac wiki back online, or at least readable someplace? (Hrmmm, it looks like Google's cache has it for the moment). Also, it would be much easier to get new contributors, especially for Windows, if Combinator worked out of the box on Windows. There's (or were) 3 or 4 patches in the Divmod SVN repo that needed to be applied in order to get it to work, and now that the Divmod site is offline it's really hard to set it up on a new machine, even for me, and I have a working example to go from. Yeah it's off topic, sue me.]
The divmod migration is underway, but we're all busy (life, etc) so it's going slowly. I have a bunch of tasks on my plate that are part of that which I haven't gotten to yet. Kevin & Tom, you guys should probably stay away from that time sink and keep forging ahead with this great documentation effort, but if anyone else would like to help out we're on #divmod; drop by, say hi, and maybe we can come up with something for you to do...
On Jan 20, 2011, at 1:54 PM, Kevin Horn wrote:
The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
I think this right here is the main reason that the docs need to be improved. Easier newbie experience translates to more users translates to more developers translates to more maintainers. Especially if the perception of Twisted as a "languishing" or even "mostly dead" project can be undone. (Yes, I've heard this a number of times. No, I don't know where they get it. I correct it whenever I can. It's something that needs to be addressed, but that's a different conversation.). Fixing up the docs will at least help with this.
Wow, that is weird. Maybe in the meanwhile, refer people with this peculiar misconception to <https://www.ohloh.net/p/twisted> - "Large, active development team", "Mature, well-established codebase". (Also tell them to click "I use this".)
Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:
There have been several abortive efforts to do something grand to re-invent all of the Twisted documentation in the universe, or a complete overhaul of the website, including several false starts that I've made, and most recently the (somewhat arduous, arguably "mostly complete" (fingers crossed on that one)) attempt to do a sphinx migration.
Seriously man, we're close.
So prove me wrong, and get it done! ;-)
3 more "chunk tickets" in the "edit the lore source" phase (and one of those is finished I think and just needs to be merged). Then another round of "chunk tickets" to manually fix any other little typographical issues in the Sphinx source, and done. I suppose there will also need to be a website deployment process. And probably lots of other minor things that we'll discover as we go.
I think that the main problem right now is that these "chunk tickets" are too big, and especially with the sphinx builder in this half-working state, nearly impossible to review. As Jean-Paul were recently discussing, he bit the bullet and plowed through one of these (overlarge) ticket reviews, assuming "how much of a problem could it be, it's just whitespace", and ultimately (after trying his best to examine it closely) gave it a passing review. And yet, there were still a couple of bugs filed that were introduced by that branch, including things like word being accidentally deleted. Breaking these up even more into smaller, easier-to-digest fixes, and then having a docs review sprint, should be able to get us over that hump.
I've had some major speedbumps, had to find a new job, kids were sick several times, etc. You know...life.
No worries. That's why we've been taking this conservative approach :).
I'm getting fired up again though, and thijs seems to be as well, and this coming up now just pumps the bellows.
Yes! Woo!
It looks like Tom and I have some similar ideas on where we should be going, and I think his Sphinx skeleton is a great example of what things should eventually look like. As I said earlier, I've had a number of similar ideas, though it looks like maybe Tom's are a little more concrete and/or fleshed-out. I just really wanted to get the Sphinx convo "out the door", before I took on another huge project. I'll elaborate in a separate email.
+1
There's a ticket for writing tests for the code samples in Trac.
Always good to have a link: http://twistedmatrix.com/trac/ticket/2205
My advice is to try and get many small changes made, and get them _done_ rather than a few huge changes.
I think I want this to be on my tombstone :).
I think this has been pretty constructive on both sides. I look forward to more.
Same here.
On Thu, Jan 20, 2011 at 1:55 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
(I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process. Anyone know what this is about? < http://buildbot.twistedmatrix.com/builds/sphinx-html-15724/contents.html>)
I was going to address this as well. I've been seeing what are apparently intermittent problems with the buildbot, though I haven't seen that one before, since the beginning of the year when I started seriously looking at this stuff again. I was able to build the docs properly on the buildbot yesterday, and in fact couldn't replicate any of the issues I had previously seen (which, in my case, seem to mostly have to do with missing CSS files). when I was talking to exarkun about it on IRC yesterday. I'm not sure whether it has to do with the buildbot configuration, maybe stuff timing out, network issues, problems in the actual conversion process, or what. Ideas welcome. Kevin Horn
On 10:05 pm, kevin.horn@gmail.com wrote:
On Thu, Jan 20, 2011 at 1:55 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
(I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process. Anyone know what this is about? < http://buildbot.twistedmatrix.com/builds/sphinx- html-15724/contents.html>)
I was going to address this as well. I've been seeing what are apparently intermittent problems with the buildbot, though I haven't seen that one before, since the beginning of the year when I started seriously looking at this stuff again.
I was able to build the docs properly on the buildbot yesterday, and in fact couldn't replicate any of the issues I had previously seen (which, in my case, seem to mostly have to do with missing CSS files). when I was talking to exarkun about it on IRC yesterday.
I'm not sure whether it has to do with the buildbot configuration, maybe stuff timing out, network issues, problems in the actual conversion process, or what.
Does this help? http://buildbot.twistedmatrix.com/builders/sphinx- documentation/builds/123/steps/shell_4/logs/stdio The salient point perhaps being: File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 347, in open_http 'got a bad status line', None) IOError: ('http protocol error', 0, 'got a bad status line', None) Jean-Paul
On Thu, Jan 20, 2011 at 4:45 PM, <exarkun@twistedmatrix.com> wrote:
On 10:05 pm, kevin.horn@gmail.com wrote:
On Thu, Jan 20, 2011 at 1:55 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
(I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process. Anyone know what this is about? < http://buildbot.twistedmatrix.com/builds/sphinx- html-15724/contents.html>)
I was going to address this as well. I've been seeing what are apparently intermittent problems with the buildbot, though I haven't seen that one before, since the beginning of the year when I started seriously looking at this stuff again.
I was able to build the docs properly on the buildbot yesterday, and in fact couldn't replicate any of the issues I had previously seen (which, in my case, seem to mostly have to do with missing CSS files). when I was talking to exarkun about it on IRC yesterday.
I'm not sure whether it has to do with the buildbot configuration, maybe stuff timing out, network issues, problems in the actual conversion process, or what.
Does this help?
http://buildbot.twistedmatrix.com/builders/sphinx- documentation/builds/123/steps/shell_4/logs/stdio
The salient point perhaps being:
File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 347, in open_http 'got a bad status line', None) IOError: ('http protocol error', 0, 'got a bad status line', None)
Jean-Paul
Hmmm...yes, I think it does. It looks like the l2s_builder script is choking on fetching the DTD for some reason. lore2sphinx itself is supposed to cache the DTD between runs, but I'm betting it doesn't work in the buildbot due to the (hopefully) clean environment it has every time it runs. I can change it to not care about that, but if I do, then it won't be able to resolve HTML entities (of which there are a few in the Lore sources, mostly em-dashes IIRC). So the XML parsing will choke on those, unless I tell it to ignore them, in which case it might ignore other errors...etc. I'll see what I can do to fix it, I haven't worked on that bit for a long time. I'll see if possible to add a "pre-cached" DTD file to teh lore2sphinx repo, and that should fix that problem at least. Thanks for the research! (I sometimes forget that buildbot keeps a log of the stdio. You wouldn't think that I would, since I use the analogous logs in hudson fairly frequently, but I do. Sigh.) Kevin Horn
On Thu, Jan 20, 2011 at 05:22:55PM -0600, Kevin Horn wrote:
Hmmm...yes, I think it does. It looks like the l2s_builder script is choking on fetching the DTD for some reason. lore2sphinx itself is supposed to cache the DTD between runs, but I'm betting it doesn't work in the buildbot due to the (hopefully) clean environment it has every time it runs.
I can change it to not care about that, but if I do, then it won't be able to resolve HTML entities (of which there are a few in the Lore sources, mostly em-dashes IIRC).
You mean these DTDs? twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time.
On 20 Jan, 11:57 pm, screwtape@froup.com wrote:
On Thu, Jan 20, 2011 at 05:22:55PM -0600, Kevin Horn wrote:
Hmmm...yes, I think it does. It looks like the l2s_builder script is choking on fetching the DTD for some reason. lore2sphinx itself is supposed to cache the DTD between runs, but I'm betting it doesn't work in the buildbot due to the (hopefully) clean environment it has every time it runs.
I can change it to not care about that, but if I do, then it won't be able to resolve HTML entities (of which there are a few in the Lore sources, mostly em-dashes IIRC).
You mean these DTDs?
twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd
They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time.
It could even use Lore's parser, twisted.lore.tree.parseFileAndReport ;) Jean-Paul
On Thu, Jan 20, 2011 at 6:06 PM, <exarkun@twistedmatrix.com> wrote:
On 20 Jan, 11:57 pm, screwtape@froup.com wrote:
On Thu, Jan 20, 2011 at 05:22:55PM -0600, Kevin Horn wrote:
Hmmm...yes, I think it does. It looks like the l2s_builder script is choking on fetching the DTD for some reason. lore2sphinx itself is supposed to cache the DTD between runs, but I'm betting it doesn't work in the buildbot due to the (hopefully) clean environment it has every time it runs.
I can change it to not care about that, but if I do, then it won't be able to resolve HTML entities (of which there are a few in the Lore sources, mostly em-dashes IIRC).
You mean these DTDs?
twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd
They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time.
It could even use Lore's parser, twisted.lore.tree.parseFileAndReport ;)
Jean-Paul
Yes, but: - The docs (well, docstrings) aren't very clear about exactly what that does, or what quirks it might or might not have. - Is the DOM document it returns a typical XHTML DOM document? Is there anything special/different about it since it's a Lore XHTML doc? - Consider the number of tickets open involving Lore's parser (or at least there were several when I started this whole business). - The whole point of this project is to stop using (and by extension supporting/maintaining) Lore, so it seems kind of backwards to me to depend on it. - I'd rather deal with an etree-a-like, than a straight DOM document (admittedly, personal preference) - I certainly don't want to rewrite everything at this point ;) Kevin Horn
On Jan 20, 2011, at 7:43 PM, Kevin Horn wrote:
It could even use Lore's parser, twisted.lore.tree.parseFileAndReport ;)
Jean-Paul
Yes, but:
- The docs (well, docstrings) aren't very clear about exactly what that does, or what quirks it might or might not have.
The source is pretty short, and has some very useful comments: <http://twistedmatrix.com/trac/browser/trunk/twisted/lore/tree.py#L1021> It is a thin wrapper around xml.sax.make_parser().parse() which uses a fixed set of DTDs, never downloads them, and remembers the file name and line number of errors so that the exceptions are actually useful to someone editing the XML.
- Is the DOM document it returns a typical XHTML DOM document? Is there anything special/different about it since it's a Lore XHTML doc?
Yep, it's plain vanilla XML objects.
- Consider the number of tickets open involving Lore's parser (or at least there were several when I started this whole business).
This isn't the soupy mess of Lore's tag munging. It's just the thing it uses to get the DOM loaded in the first place.
- The whole point of this project is to stop using (and by extension supporting/maintaining) Lore, so it seems kind of backwards to me to depend on it.
parseFileAndReport is actually useful enough that I'd like to see it move somewhere else - maybe somewhere in twisted.python, because this is useful knowledge.
- I'd rather deal with an etree-a-like, than a straight DOM document (admittedly, personal preference)
Isn't there a function to do the conversion somewhere?
- I certainly don't want to rewrite everything at this point ;)
If you can't use this function, the methods that it calls should be simple enough that you can just crib them :).
On Thu, Jan 20, 2011 at 5:57 PM, Tim Allen <screwtape@froup.com> wrote:
On Thu, Jan 20, 2011 at 05:22:55PM -0600, Kevin Horn wrote:
Hmmm...yes, I think it does. It looks like the l2s_builder script is choking on fetching the DTD for some reason. lore2sphinx itself is supposed to cache the DTD between runs, but I'm betting it doesn't work in the buildbot due to the (hopefully) clean environment it has every time it runs.
I can change it to not care about that, but if I do, then it won't be able to resolve HTML entities (of which there are a few in the Lore sources, mostly em-dashes IIRC).
You mean these DTDs?
twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd
They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time.
Huh. Never even knew that was there. It probably could, and the reason it downloads from the internet was because that's the default way of doing it in lxml. I've since figured out how to override that behavior (which is how the caching works) so maybe that wouldn't even be hard. The easiest/fastest fix for the moment though would probably be to pre-populate the cache as I mentioned before, since IIRC, this would just involve adding the file to my hg repo. I'll have to look into it though, it may be just as easy to do it the other way, though I don't want to depend necessarily on having Twisted's code available (remember, this is supposed to work on the various divmod projects, and anything else that uses Lore, too). Kevin Horn
On Thu, Jan 20, 2011 at 06:34:17PM -0600, Kevin Horn wrote:
On Thu, Jan 20, 2011 at 5:57 PM, Tim Allen <screwtape@froup.com> wrote:
You mean these DTDs?
twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd
They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time.
Huh. Never even knew that was there. It probably could, and the reason it downloads from the internet was because that's the default way of doing it in lxml. I've since figured out how to override that behavior (which is how the caching works) so maybe that wouldn't even be hard. The easiest/fastest fix for the moment though would probably be to pre-populate the cache as I mentioned before, since IIRC, this would just involve adding the file to my hg repo. I'll have to look into it though, it may be just as easy to do it the other way, though I don't want to depend necessarily on having Twisted's code available (remember, this is supposed to work on the various divmod projects, and anything else that uses Lore, too).
Well, you wouldn't necessarily be depending on Twisted, just depending on Lore - and I imagine any not-Twisted project whose documentation depends on Lore has already made peace with the idea of depending on Lore. :) If it's easier to just copy these well-known DTDs into the lore2sphinx repository, I guess that's a good plan too - it's not like the W3C is going to suddenly issue updated XHTML DTDs.
Wow! Okay, so, not being really familiar with the list etiquette combined with the fact that the topic has digressed a bit, I hope you'll forgive the non-inline reply here. I want to hit the main points of what everybody said / asked, but let it be known that I read all of it! How can we combine these efforts, or at least keep from working at cross
purposes?
I see from your link above that you are building your own Sphinx project. Perhaps you would be better off working from the results of the Lore2sphinx conversion? Are you modifying existing docs or working from scratch?
Let's get together on this!
Hey Kevin! Glyph and Jean-Paul got me up to speed (somewhat) on the whole lore2sphinx thing last night—though they were rather unclear as to how close it is to completion. (which you addressed later in this thread) I certainly want to combine our efforts as much as possible and I'm not too worried about working at cross purposes; your main task seems to be converting the existing documentation while mine is reorganizing, editing, and standardizing it. I'm sure our paths will start to cross when the lore conversion is complete and you have a huge Sphinx project ready to be properly organized and catalogued. My hope is we can then begin the process of merging the two sides of this coin. My current plan (which should become a bit more clear throughout this massive reply) is to start from scratch *but* modify existing documentation wherever possible. If a piece of documentation fits in a "hole" in the structure and I can get by with a *mv* and some editing, that's what I'd like to do—it seems the most pragmatic approach. So here's what I'm kind of thinking as far as combining efforts:
1) I'll continue with the "Project Documentation" conversion, while Tom works on the other bits. Should be fairly easy to combine them, I'm thinking. 2) Let's leave the "Project Documentation" pretty much as-is for the moment, until the Sphinx convo is done. 3) I wonder if at least some of the "task-based docs" shouldn't be put into the project sections, and then just linked to from the task-based section?
This. Point of fact, (3) is already implemented; that example task simply links to *projects/web/tasks/serve*. This strikes me as the most logical layout as the tasks are all specific to a particular project (the only iffy one being Core which is sort of a few projects plus the foundation)—we just want to highlight some of them. On to Glyph's reply... There are many outstanding documentation branches which are substantial
improvements, which need to be edited and merged - the trial tutorial among them. It would be great if your efforts could start with getting those landed, turning the crank on the process to get our users better documentation in the interim, as you survey the existing documentation.
I definitely agree that resolving the low-hanging fruit first is a good idea. For finishing "docs branch X" to make sense, my personal belief is that X should: - Still be relevant in terms of best practices and simply what's available - If project documentation, have outstanding issues that a passing familiarity with the project [right now] will be sufficient to close them (I could spend time learning one project, or the same time improving *all* the documentation) - Adhere to whatever documentation standards we agree upon, if much is to left to do. I guess my overall opinion here is that, yes, if relatively minor edits can bring a branch close enough to completion that we can get it out there to help newcomers *now*, let's do that. If a branch is more of a draft and requires a good deal of fleshing out (or is simply stale), it's probably worth nailing down the structure and previously mentioned docs standards before I just create more work for myself (or others) down the road. Finally, my biggest hurdle right now is not knowing how to *find* said branches. I don't see "documentation" as a category in Trac and common keyword searches didn't show up much for me. I'm sure this is an easy question to answer, though. What *should* a newcomer who reads this document know by the end of it? I'm not sure because I can't see how a practical guide to creating something so generic really fits in the grand scheme of things. I think if you want to create a TCP server from scratch you must first create the Universe! In this case, that means learning how Twisted addresses the *concept* of a server before ever bothering to write one so generic. My general beef is that many documents seem to make an attempt to appeal to everybody and in doing so don't sufficiently help anybody. Maybe I can justify that claim better with examples of "better" (at least more targeted) documents. A massive pile of improved documentation would of course be useful, but a
good start would be a clear statement of requirements and audience. (As well as an enumeration of*different* audiences that different documents might serve.)
I completely agree. The updates I made last night (hopefully) address the issue of different audiences and needs, at least as a beginning stage. What Jason said about the Python docs is a great parallel: Tutorials, [Project] Reference, [Core Concepts] Reference. All of these are important and cater to a different need. I think we're quite on the same page here, though. (minor nitpick: I really like "event-based" or "event-driven", as you've
said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"?
It's inconsistent and I completely agree (and the consensus seems to be) that event-* is superior. You will probably have to press us core developers on this one, and you may
spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
Debates are great! And I fully intend to bitch about the lack of "best ways" to do certain things—having one clear way to accomplish something is, after all, a core tenant of Python ;) -- more than that, it's really annoying to everyone involved (readers, writers, and developers) when there are many different ways to do something and none of them have very clear use cases or strengths / weaknesses. Definitely a TODO. The biggest problem with this is that you will find that a very small group
of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
Kevin touched on this already, but I really think if we make maintaining and growing a Project something that is both honorable *and* accessible, more people will want to do it. If you like Python and DNS or SMTP or whatever the hell else, what has the potential to be a more awesome implementation than its Twisted one? Let's help people find out for themselves that the answer is *nothing*. Then they'll want it to be *their* project. As problematic as the current situation is, there is a definite potential
for some baby vs. bathwater confusion in improving it... Also, while you're energized now, please consider what happens to our documentation efforts if you run out of time or out of motivation halfway through this process. Incremental improvements may not necessarily provide the best outcome, but they _do_ mean that users get to see some results from these volunteer efforts even if the original volunteer gets busy with other aspects of life and can't complete their project.
I have no intention of rewriting everything, even if it sounds absurdly entertaining to make myself learn and write practical examples of every Twisted project. I am (just) slightly more realistic than that. Having the Sphinx conversion complete will be a huge help, if for no other reason than the documentation will be easy to port, edit, and find in the first place. On the subject of loss of motivation or time, sure. It happens to all of us. And it's not my intention to keep the repository on GitHub completely separate and wait until it's perfect to get it to users. Once all the documentation is under Sphinx, there will be no reason for that. The existing documentation can be sliced and diced into the newly-minted structure and then combed over piece by piece. Some of it will likely need to be thrown out wholesale, but I imagine much of it will live on with minor edits (and probably new code) at new URLs. As for Launchpad, etc.—I want to do whatever makes you guys (a) most comfortable and (b) most likely to contribute. I chose to start the GitHub project because I know Git well and it allowed me to spend all my time writing and none of it figuring out something foreign. I don't understand the link between Twisted Trac and Twisted on Launchpad, so an explanation of that would be awesome. Provided that, though, how complicated could Bazaar possibly be? We can leave Twisted's migration to GitHub as task #2 on my list ;) I'd really like any effort that undertakes to substantially change the
structure of the documentation to make a concerted effort to hit the ground running with test-driven examples which can be automatically verified, so that we have some idea of the impact of code changes on the docs.
Most definitely. Documentation requires the same maintenance that code does and can benefit from the same automated verification; and it's trivial to write entire programs and show code samples using *literalinclude* and the * :lines:* option. Failing code can be fixed and grep'd for realignment in the documentation. Indeed, as the author of many of these original documents, I did not feel
insulted in either my person or my work :). Hopefully you didn't feel that way either after reading this reply!
Not at all! * * * I believe I covered the main thoughts / concerns / queries related to my original post. My apologies if I missed anybody! And it's great to finally be working with you :) Cheers, Tom On Thu, Jan 20, 2011 at 8:20 PM, Tim Allen <screwtape@froup.com> wrote:
On Thu, Jan 20, 2011 at 06:34:17PM -0600, Kevin Horn wrote:
On Thu, Jan 20, 2011 at 5:57 PM, Tim Allen <screwtape@froup.com> wrote:
You mean these DTDs?
twisted/lore/xhtml1-strict.dtd twisted/lore/xhtml1-transitional.dtd
They reference the xhtml-*.ent entity definitions which are also in the same directory. It would be neat if lore2sphinx could be taught to use the DTDs packaged with lore instead of having to download them from the Internet every time.
Huh. Never even knew that was there. It probably could, and the reason it downloads from the internet was because that's the default way of doing it in lxml. I've since figured out how to override that behavior (which is how the caching works) so maybe that wouldn't even be hard. The easiest/fastest fix for the moment though would probably be to pre-populate the cache as I mentioned before, since IIRC, this would just involve adding the file to my hg repo. I'll have to look into it though, it may be just as easy to do it the other way, though I don't want to depend necessarily on having Twisted's code available (remember, this is supposed to work on the various divmod projects, and anything else that uses Lore, too).
Well, you wouldn't necessarily be depending on Twisted, just depending on Lore - and I imagine any not-Twisted project whose documentation depends on Lore has already made peace with the idea of depending on Lore. :)
If it's easier to just copy these well-known DTDs into the lore2sphinx repository, I guess that's a good plan too - it's not like the W3C is going to suddenly issue updated XHTML DTDs.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 04:20 am, tom@recursivedream.com wrote:
Finally, my biggest hurdle right now is not knowing how to *find* said branches. I don't see "documentation" as a category in Trac and common keyword searches didn't show up much for me. I'm sure this is an easy question to answer, though.
As for Launchpad, etc. 14I want to do whatever makes you guys (a) most comfortable and (b) most likely to contribute. I chose to start the GitHub project because I know Git well and it allowed me to spend all my time writing and none of it figuring out something foreign. I don't understand the link between Twisted Trac and Twisted on Launchpad, so an explanation of that would be awesome. Provided that, though, how complicated could Bazaar possibly be? We can leave Twisted's migration to GitHub as task #2 on my list ;)
I don't think it makes sense to put anything on Launchpad for now. Eventually it needs to go into the canonical Twisted repository, but if it's easier to leave it in GitHub for now, that's fine. Jean-Paul
On Jan 21, 2011, at 8:29 AM, exarkun@twistedmatrix.com wrote:
I don't think it makes sense to put anything on Launchpad for now. Eventually it needs to go into the canonical Twisted repository, but if it's easier to leave it in GitHub for now, that's fine.
The only reason I even suggested Launchpad in the first place was to suggest a place where you could immediately stick a branch of the Twisted tree proper without having commit access, so that you could experiment within that tree. ('bzr get lp:twisted'). I don't believe we have a mirror on github, but maybe somebody could correct me. As long as we're all on the same page as far as the eventual target of these docs (i.e. in the Twisted tree, part of some official structure) then please feel free to use whatever tools make that easier for you, your existing github repo included.
On Fri, Jan 21, 2011 at 5:35 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Jan 21, 2011, at 8:29 AM, exarkun@twistedmatrix.com wrote:
I don't think it makes sense to put anything on Launchpad for now. Eventually it needs to go into the canonical Twisted repository, but if it's easier to leave it in GitHub for now, that's fine.
The only reason I even suggested Launchpad in the first place was to suggest a place where you could immediately stick a branch of the Twisted tree proper without having commit access, so that you could experiment within that tree. ('bzr get lp:twisted'). I don't believe we have a mirror on github, but maybe somebody could correct me.
As long as we're all on the same page as far as the eventual target of these docs (i.e. in the Twisted tree, part of some official structure) then please feel free to use whatever tools make that easier for you, your existing github repo included.
We are definitely on the same page. I never wanted the new documentation to remain separate from Twisted proper. It should be part of the main repository, build process, etc.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 21 January 2011 22:35, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
I don't believe we have a mirror on github, but maybe somebody could correct me.
There are 79 repos on github that match the word twisted. This one claims to be updated every 15 minutes and seems to be pretty up to date (has a commit from 13 hours ago) https://github.com/powdahound/twisted PS Having moved from subversion to Git there is no going back, especially once you get using rebase to avoid merge commits.
On 21 January 2011 22:35, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
�I don't believe we have a mirror on github, but maybe somebody could correct me.
There are 79 repos on github that match the word twisted.
This one claims to be updated every 15 minutes and seems to be pretty up to date (has a commit from 13 hours ago)
Here's what might pass for an "official" git mirror of Twisted svn: http://repo.or.cz/w/twisted.git It seems to have diverged a while back from powdahound's github one, due to what looks like a bogus commit in powdahound's. (I do need to go back and redo the "releases" branches/tags at some point though; they're not being copied at the right level.) James
On Jan 20, 2011, at 11:20 PM, Tom Davis wrote:
I definitely agree that resolving the low-hanging fruit first is a good idea. For finishing "docs branch X" to make sense, my personal belief is that X should: Still be relevant in terms of best practices and simply what's available If project documentation, have outstanding issues that a passing familiarity with the project [right now] will be sufficient to close them (I could spend time learning one project, or the same time improving all the documentation) Adhere to whatever documentation standards we agree upon, if much is to left to do. I guess my overall opinion here is that, yes, if relatively minor edits can bring a branch close enough to completion that we can get it out there to help newcomers now, let's do that. If a branch is more of a draft and requires a good deal of fleshing out (or is simply stale), it's probably worth nailing down the structure and previously mentioned docs standards before I just create more work for myself (or others) down the road.
If branches that are out there don't meet these standards, commenting on their tickets and getting them deleted or closed as invalid (as appropriate) would be a big help too. Lots of languishing tickets that nobody knows what to do with is not a good thing, and there's plenty of opportunities for interested parties to reopen tickets, attach new patches, and object in various ways, so you shouldn't be too concerned about stepping on toes. Focus is a valuable commodity. Part of my comment about low-hanging fruit was to help you get familiar with and integrated into the development process. Going through the process of getting patches reviewed and accepted will be _much_ easier if you go through the motions of doing a few trivial things first. In fact you may want to just pick up a couple of trivial non-docs patches as well, which might help you on documenting the development process :). <bit.ly/easy-twisted-tickets> might help you there. Mostly, I really don't want you to write a gigantic pile of new documentation and then find, when you're "done", that you missed some nuance of the coding standard, or the patch is too big to be reasonably reviewed, and that now you have three months of additional work to do before it's all really done. Experience with the process will mitigate that problem significantly. (And in fact I hope that you don't actually have a gigantic pile of stuff to commit all at once at any point, and can continue this work incrementally as a series of small tickets, but I realize that later on some of the index reorganization stuff may need to be big. This is mostly just restating what Kevin already said in his message, but it bears repeating.)
Finally, my biggest hurdle right now is not knowing how to find said branches. I don't see "documentation" as a category in Trac and common keyword searches didn't show up much for me. I'm sure this is an easy question to answer, though.
<http://twistedmatrix.com/trac/query?status=assigned&status=new&status=reopened&order=priority&col=id&col=summary&col=status&col=owner&col=type&col=priority&col=milestone&keywords=%7Edocumentation> There is, unfortunately, no "has branch" column in that report, but it will at least give you some data to work with.
What should a newcomer who reads this document know by the end of it?
I'm not sure because I can't see how a practical guide to creating something so generic really fits in the grand scheme of things. I think if you want to create a TCP server from scratch you must first create the Universe! In this case, that means learning how Twisted addresses the concept of a server before ever bothering to write one so generic. My general beef is that many documents seem to make an attempt to appeal to everybody and in doing so don't sufficiently help anybody. Maybe I can justify that claim better with examples of "better" (at least more targeted) documents.
I don't think you need a clear definition of "better" for this particular document, if it doesn't really fit into your scheme; I don't mind if it eventually becomes irrelevant. I just want a clear statement of goals for the documentation _in advance_ of writing that new documentation, so that we can discuss whether it's actually better for the intended audience or whether it's just more suited to a new author's tastes.
You will probably have to press us core developers on this one, and you may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
Debates are great!
Debates that reach some kind of conclusion are great :). Debates that just go in circles until everybody feels crappy about the topic aren't. So I'm really just asking you to help us make these debates into the great kind. (The rest of your reply seems to agree with that, I just wanted to be clear.)
The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
Kevin touched on this already, but I really think if we make maintaining and growing a Project something that is both honorable and accessible, more people will want to do it.
Okay, so, this is really a tangent, but maybe you could let me know what you think of <http://tm.tl/2372>.
If you like Python and DNS or SMTP or whatever the hell else, what has the potential to be a more awesome implementation than its Twisted one? Let's help people find out for themselves that the answer is nothing. Then they'll want it to be their project.
If you do this, you will be my hero forever. OK, I'm going to skip the rest of the reply so that I don't write another book-length thing here, and I agree with most of the rest of what you've said. So get to it!
On 03:00 am, glyph@twistedmatrix.com wrote:
Finally, my biggest hurdle right now is not knowing how to find said branches. I don't see "documentation" as a category in Trac and common keyword searches didn't show up much for me. I'm sure this is an easy question to answer, though.
There is, unfortunately, no "has branch" column in that report, but it will at least give you some data to work with.
I gave this better URL a few days ago. Here's an even better one, that does have a branch column (and some other arbitrary things I added): <http://bit.ly/i4Zj8U> Jean-Paul
On Fri, Jan 21, 2011 at 10:00 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Jan 20, 2011, at 11:20 PM, Tom Davis wrote:
I definitely agree that resolving the low-hanging fruit first is a good idea. For finishing "docs branch X" to make sense, my personal belief is that X should:
- Still be relevant in terms of best practices and simply what's available - If project documentation, have outstanding issues that a passing familiarity with the project [right now] will be sufficient to close them (I could spend time learning one project, or the same time improving *all* the documentation) - Adhere to whatever documentation standards we agree upon, if much is to left to do.
I guess my overall opinion here is that, yes, if relatively minor edits can bring a branch close enough to completion that we can get it out there to help newcomers *now*, let's do that. If a branch is more of a draft and requires a good deal of fleshing out (or is simply stale), it's probably worth nailing down the structure and previously mentioned docs standards before I just create more work for myself (or others) down the road.
If branches that are out there *don't* meet these standards, commenting on their tickets and getting them deleted or closed as invalid (as appropriate) would be a big help too. Lots of languishing tickets that nobody knows what to do with is not a good thing, and there's plenty of opportunities for interested parties to reopen tickets, attach new patches, and object in various ways, so you shouldn't be too concerned about stepping on toes. Focus is a valuable commodity.
I was wondering to what extent it would be helpful to actually reply to all the tickets, or just the ones that seem to have actionable next steps. I will try to find something to ask or opine on in each of the documentation tickets so we can get them moving along or removed. Part of my comment about low-hanging fruit was to help you get familiar with
and integrated into the development process. Going through the process of getting patches reviewed and accepted will be _much_ easier if you go through the motions of doing a few trivial things first. In fact you may want to just pick up a couple of trivial non-docs patches as well, which might help you on documenting the development process :). < bit.ly/easy-twisted-tickets> might help you there.
I will find an easy ticket or few to bang out. I just replied here ( http://twistedmatrix.com/trac/ticket/2491) in an attempt to get started on one that doesn't already have a long history. After spending about an hour going over the "easy" tickets, it seems many of them are in odd states. Either they're done and waiting on something undefined or there is an incomplete debate in the comments or the owner disappeared or... well, there are lots of examples. Maybe this is just me being dense or whatever, but I think (at least as a newcomer) I could mass-update all these tickets with "Guidance!" and it would more often than not be a relevant comment given the state of the ticket. Here's a great example of what I'm talking about (and I apologize for the mid-message digression, but I think it's relevant...): http://twistedmatrix.com/trac/ticket/4636. This seems totally trivial, but five months later __all__ was never changed in t.i.main and JP's buildbot link is broken. Whether the offending class should be removed from __all__ or imported instead was never even mentioned. Anyway, I took a stab at it and attached a trivial patch so this isn't just another complaint (and it looks like JP closed it while I was drafting this email, so that's great!). But I do agree that working tickets would really help document the process!
Mostly, I really don't want you to write a gigantic pile of new documentation and then find, when you're "done", that you missed some nuance of the coding standard, or the patch is too big to be reasonably reviewed, and that now you have three months of additional work to do before it's all *really* done. Experience with the process will mitigate that problem significantly. (And in fact I hope that you don't actually have a gigantic pile of stuff to commit all at once at any point, and can continue this work incrementally as a series of small tickets, but I realize that later on some of the index reorganization stuff may need to be big. This is mostly just restating what Kevin already said in his message, but it bears repeating.)
The more I think about this, the more I agree with you. My initial inclination was just to start from scratch and move over existing docs that I found I could use. This doesn't conflict with the reuse I've been supportive of (and for good reason: there's a lot of decent documentation already there) but it does conflict with the Twisted development policies. At the end of the day, I can't really submit patches to existing documentation until said documentation is in Sphinx form. I guess one way this could work to the happiness of all involved is: 1. Get re-org nailed down; basic samples for whatever pages I think should have standardized format / elements. 2. Mixed in here, find existing docs and non-docs tickets to work on trunk. 3. When the re-org structure is ready, create a new branch from whatever the current Sphinx conversion branch is (hopefully it'll be "done" at this point) 4. Migrate new structure to branch, moving the existing docs to semi-logical-but-possibly-temporary places in the new structure. 5. Submit tickets / patches against existing documentation in said branch to move docs to new(er) locations, submit new TDD-style code samples, general edits, new docs, etc. 6. When everything is complete, submit a final ticket to merge to trunk Does that make sense or am I still cognitively off the mark in terms of existing development practices?
Finally, my biggest hurdle right now is not knowing how to *find* said branches. I don't see "documentation" as a category in Trac and common keyword searches didn't show up much for me. I'm sure this is an easy question to answer, though.
There is, unfortunately, no "has branch" column in that report, but it will at least give you some data to work with.
What *should* a newcomer who reads this document know by the end of it?
I'm not sure because I can't see how a practical guide to creating something so generic really fits in the grand scheme of things. I think if you want to create a TCP server from scratch you must first create the Universe! In this case, that means learning how Twisted addresses the * concept* of a server before ever bothering to write one so generic. My general beef is that many documents seem to make an attempt to appeal to everybody and in doing so don't sufficiently help anybody. Maybe I can justify that claim better with examples of "better" (at least more targeted) documents.
I don't think you need a clear definition of "better" for this particular document, if it doesn't really fit into your scheme; I don't mind if it eventually becomes irrelevant. I just want a clear statement of goals for the documentation _in advance_ of writing that new documentation, so that we can discuss whether it's actually better for the intended audience or whether it's just more suited to a new author's tastes.
I will be fleshing out those goals more in the coming days, but some of it is implicit within the new documentation structure that's already in my repo. As for a statement of goals when replacing/removing specific documentation, sure, I can do that as necessary. It's impossible to completely remove the element of taste but I will certainly avoid the "because I like this better" argument for making changes.
You will probably have to press us core developers on this one, and you may
spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
Debates are great!
Debates *that reach some kind of conclusion* are great :). Debates that just go in circles until everybody feels crappy about the topic aren't. So I'm really just asking you to help us make these debates into the great kind. (The rest of your reply seems to agree with that, I just wanted to be clear.)
Yeah, as I've already mentioned in this reply, the "limbo syndrome" of many tickets drives me nuts. I'm sure you guys hate it too, but having somebody new around who isn't afraid to ruffle some feathers for the greater good and isn't yet at some "acceptance" level with it should be helpful! That being said, I will do my best not to be a jerk about it; you guys have been doing this for much, much longer and I respect that.
The biggest problem with this is that you will find that a very small group
of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).
Kevin touched on this already, but I really think if we make maintaining and growing a Project something that is both honorable *and* accessible, more people will want to do it.
Okay, so, this is really a tangent, but maybe you could let me know what you think of <http://tm.tl/2372>.
Commented in ticket.
If you like Python and DNS or SMTP or whatever the hell else, what has the potential to be a more awesome implementation than its Twisted one? Let's help people find out for themselves that the answer is *nothing*. Then they'll want it to be *their* project.
If you do this, you will be my hero forever.
Twisted is a naturally-superior Python choice for most supported protocols by virtue of the fact that most network "stuff" benefits from event-based solutions. Like, find me a Python DNS server that *isn't* implemented in Twisted. People have these weird misconceptions about Twisted; it's really hard to grasp, ugly, unmaintained, etc. Ultimately, I want to correct these misconceptions, have *twisted.names* show up first on Google for "Python DNS Server", and all the other shit that really should be the case but isn't. Resolving these issues should ultimately resolve the problem of maintainers, community involvement, etc. I can't promise that in the course of this crusade I won't cause grief by trying to change things, but I will always strive to have a rational reason for wanting to do so!
OK, I'm going to skip the rest of the reply so that I don't write another book-length thing here, and I agree with most of the rest of what you've said. So get to it!
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jan 22, 2011, at 3:18 PM, Tom Davis wrote:
On Fri, Jan 21, 2011 at 10:00 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: On Jan 20, 2011, at 11:20 PM, Tom Davis wrote:
If branches that are out there don't meet these standards, commenting on their tickets and getting them deleted or closed as invalid (as appropriate) would be a big help too. Lots of languishing tickets that nobody knows what to do with is not a good thing, and there's plenty of opportunities for interested parties to reopen tickets, attach new patches, and object in various ways, so you shouldn't be too concerned about stepping on toes. Focus is a valuable commodity.
I was wondering to what extent it would be helpful to actually reply to all the tickets, or just the ones that seem to have actionable next steps. I will try to find something to ask or opine on in each of the documentation tickets so we can get them moving along or removed.
Getting rid of dead tickets is almost as important as actually getting valid tickets moved along. I think this effort will be hugely valuable.
Part of my comment about low-hanging fruit was to help you get familiar with and integrated into the development process. Going through the process of getting patches reviewed and accepted will be _much_ easier if you go through the motions of doing a few trivial things first. In fact you may want to just pick up a couple of trivial non-docs patches as well, which might help you on documenting the development process :). <bit.ly/easy-twisted-tickets> might help you there.
I will find an easy ticket or few to bang out. I just replied here (http://twistedmatrix.com/trac/ticket/2491) in an attempt to get started on one that doesn't already have a long history.
After spending about an hour going over the "easy" tickets, it seems many of them are in odd states. Either they're done and waiting on something undefined or there is an incomplete debate in the comments or the owner disappeared or... well, there are lots of examples. Maybe this is just me being dense or whatever, but I think (at least as a newcomer) I could mass-update all these tickets with "Guidance!" and it would more often than not be a relevant comment given the state of the ticket.
No, this is not you being dense. At least among the core developers in Boston, this is a widely-recognized and frequently-complained-about problem, and it's something I'd like everyone doing ticket reviews to please think about. Reviewers: if you make a comment on a ticket, but you don't say what you want to happen next, then you have effectively killed progress on that ticket until some other reviewer comes along and contradicts you to get things moving again. This is especially true if you make one trivial comment on the ticket and remove the review keyword, but don't say "please address these issues and then merge" or "please address these issues and then resubmit for review". If you've done a partial review, and made a comment like "I don't like this aspect of the design" or "please update copyright dates" or "your docstring formatting is wrong", please note in your comment that this is not a complete review, and don't remove the review keyword. Removing the keyword will introduce additional latency for the contributor, when other reviewers might still come along and attach more comprehensive feedback. There are few things more discouraging than having one free weekend every six months to work on a ticket, and to come back every time to "oops, you forgot to update the copyright date and insert a blank line in one file of your 300-line patch". So, Tom: mass updating those tickets wouldn't be helpful, but an update every couple of days with a specific question on one of these I-don't-know-what-to-do tickets would be great. Your question on tm.tl/2491 was a definite step forward.
Here's a great example of what I'm talking about (and I apologize for the mid-message digression, but I think it's relevant...): http://twistedmatrix.com/trac/ticket/4636. This seems totally trivial, but five months later __all__ was never changed in t.i.main and JP's buildbot link is broken. Whether the offending class should be removed from __all__ or imported instead was never even mentioned. Anyway, I took a stab at it and attached a trivial patch so this isn't just another complaint (and it looks like JP closed it while I was drafting this email, so that's great!). But I do agree that working tickets would really help document the process!
As you have discovered here, drawing attention to a ticket in this "stuck" state will often cause it to get un-stuck. So please keep doing that.
Mostly, I really don't want you to write a gigantic pile of new documentation and then find, when you're "done", that you missed some nuance of the coding standard, or the patch is too big to be reasonably reviewed, and that now you have three months of additional work to do before it's all really done. Experience with the process will mitigate that problem significantly. (And in fact I hope that you don't actually have a gigantic pile of stuff to commit all at once at any point, and can continue this work incrementally as a series of small tickets, but I realize that later on some of the index reorganization stuff may need to be big. This is mostly just restating what Kevin already said in his message, but it bears repeating.)
The more I think about this, the more I agree with you. My initial inclination was just to start from scratch and move over existing docs that I found I could use. This doesn't conflict with the reuse I've been supportive of (and for good reason: there's a lot of decent documentation already there) but it does conflict with the Twisted development policies.
At the end of the day, I can't really submit patches to existing documentation until said documentation is in Sphinx form.
Aaaaaaaauuuuugh. When you say this, the first thing I think is "okay then, the sphinx migration is now introducing roadblocks into the documentation process and preventing people from contributing documentation, so let's just cancel it". So please don't say things like this :). The whole reason that we insisted on the current process for the sphinx migration was to avoid this kind of blockage. You can submit documentation patches right now, in Lore format - which, I would like to remind you, is 99% plain-vanilla HTML and very easy to edit - and get them accepted. You can see on the sphinx buildbot (once a few minor issues are fixed, as discussed earlier in this thread) what those changes will look like once converted to ReST->Sphinx. Or you can simply run the conversion yourself locally - hopefully Kevin will chime in on how to set that up so I don't need to look it up :). As long as you aren't trying to do anything fancy with diagrams or tables (and most of the documentation really should not need elaborate diagrams or tables), you shouldn't run into any issues. Heck, if you want to write your documentation patches as ReST snippets and attach them to tickets, I'm sure you will find many willing contributors (myself included) who will jump in and do the format-munging manually to get them integrated into exiting lore documents so that they can make it to trunk immediately. Or, you can just jump in with ticket reviews, or breaking up chunk tickets, to move along the Sphinx migration quicker. But "I will work over here in a corner until somehow this gigantic pile of work gets finished by somebody else" is not a good strategy. (I'm not saying that that's exactly what you're proposing to do, but the aspects of what your proposing which align with that may not be the best way to proceed.)
I guess one way this could work to the happiness of all involved is: Get re-org nailed down; basic samples for whatever pages I think should have standardized format / elements. Mixed in here, find existing docs and non-docs tickets to work on trunk. When the re-org structure is ready, create a new branch from whatever the current Sphinx conversion branch is (hopefully it'll be "done" at this point) Migrate new structure to branch, moving the existing docs to semi-logical-but-possibly-temporary places in the new structure. Submit tickets / patches against existing documentation in said branch to move docs to new(er) locations, submit new TDD-style code samples, general edits, new docs, etc. When everything is complete, submit a final ticket to merge to trunk Does that make sense or am I still cognitively off the mark in terms of existing development practices?
With the caveat of what I said above, most of these things sound like generally good things to do. Except, first, I'd still like to see a ticket for what "the re-org" is actually going to be and what the point of it is. If I've only learned one thing in maintaining Twisted for 10 years, it's that a description of what we are trying to do separate from how we are going to do it early on is essential to make sure that others can give you useful feedback - and, more importantly, provide you with resources. In this case, those resources would be pointers to other areas in the documentation that you may not have noticed which already do some of what you are trying to do, but may be poorly integrated.
I will be fleshing out those goals more in the coming days, but some of it is implicit within the new documentation structure that's already in my repo.
$ python -c 'import this' | grep Explicit # :-)
As for a statement of goals when replacing/removing specific documentation, sure, I can do that as necessary. It's impossible to completely remove the element of taste but I will certainly avoid the "because I like this better" argument for making changes.
Again, this is more about having a good statement of purpose up front, so that everyone is focused on the same thing, than having an argument that you can trot out later once everyone starts nitpicking from a hundred different perspectives :). Plus, such a statement of purpose can serve as a focus for you as well.
You will probably have to press us core developers on this one, and you may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
Debates are great!
Debates that reach some kind of conclusion are great :). Debates that just go in circles until everybody feels crappy about the topic aren't. So I'm really just asking you to help us make these debates into the great kind. (The rest of your reply seems to agree with that, I just wanted to be clear.)
Yeah, as I've already mentioned in this reply, the "limbo syndrome" of many tickets drives me nuts. I'm sure you guys hate it too, but having somebody new around who isn't afraid to ruffle some feathers for the greater good and isn't yet at some "acceptance" level with it should be helpful! That being said, I will do my best not to be a jerk about it; you guys have been doing this for much, much longer and I respect that.
Tickets in limbo need to be eliminated. I think you will find that just about everyone is happy to pick up these discussions with an injection of a little bit of attention and energy from a contributor. We all know that these sap the will of new contributors and thereby decrease the overall pool of available time, so I think you may have to try really, really hard to actually ruffle any feathers :).
If you like Python and DNS or SMTP or whatever the hell else, what has the potential to be a more awesome implementation than its Twisted one? Let's help people find out for themselves that the answer is nothing. Then they'll want it to be their project.
If you do this, you will be my hero forever.
Twisted is a naturally-superior Python choice for most supported protocols by virtue of the fact that most network "stuff" benefits from event-based solutions. Like, find me a Python DNS server that isn't implemented in Twisted. People have these weird misconceptions about Twisted; it's really hard to grasp, ugly, unmaintained, etc. Ultimately, I want to correct these misconceptions, have twisted.names show up first on Google for "Python DNS Server", and all the other shit that really should be the case but isn't. Resolving these issues should ultimately resolve the problem of maintainers, community involvement, etc.
+10000.
I can't promise that in the course of this crusade I won't cause grief by trying to change things, but I will always strive to have a rational reason for wanting to do so!
Please keep the grief coming! It'll all be worth it, I'm sure :). Everything that you've said so far that I disagree with has just been an opportunity to communicate some of the Twisted development philosophy which may not yet be written down - and should probably be part of the documentation eventually. So you have yet to do anything which has put me out even in the slightest. We need more new contributors feeling comfortable being bold and trying to Just Do It.
On Sat, Jan 22, 2011 at 10:02 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Jan 22, 2011, at 3:18 PM, Tom Davis wrote:
On Fri, Jan 21, 2011 at 10:00 PM, Glyph Lefkowitz <glyph@twistedmatrix.com
wrote:
On Jan 20, 2011, at 11:20 PM, Tom Davis wrote:
If branches that are out there *don't* meet these standards, commenting on their tickets and getting them deleted or closed as invalid (as appropriate) would be a big help too. Lots of languishing tickets that nobody knows what to do with is not a good thing, and there's plenty of opportunities for interested parties to reopen tickets, attach new patches, and object in various ways, so you shouldn't be too concerned about stepping on toes. Focus is a valuable commodity.
I was wondering to what extent it would be helpful to actually reply to all the tickets, or just the ones that seem to have actionable next steps. I will try to find something to ask or opine on in each of the documentation tickets so we can get them moving along or removed.
Getting rid of dead tickets is almost as important as actually getting valid tickets moved along. I think this effort will be hugely valuable.
Will continue commenting on tickets and asserting non-existent power to get them resolved! ;)
Part of my comment about low-hanging fruit was to help you get familiar
with and integrated into the development process. Going through the process of getting patches reviewed and accepted will be _much_ easier if you go through the motions of doing a few trivial things first. In fact you may want to just pick up a couple of trivial non-docs patches as well, which might help you on documenting the development process :). < bit.ly/easy-twisted-tickets> might help you there.
I will find an easy ticket or few to bang out. I just replied here ( http://twistedmatrix.com/trac/ticket/2491) in an attempt to get started on one that doesn't already have a long history.
After spending about an hour going over the "easy" tickets, it seems many of them are in odd states. Either they're done and waiting on something undefined or there is an incomplete debate in the comments or the owner disappeared or... well, there are lots of examples. Maybe this is just me being dense or whatever, but I think (at least as a newcomer) I could mass-update all these tickets with "Guidance!" and it would more often than not be a relevant comment given the state of the ticket.
No, this is not you being dense. At least among the core developers in Boston, this is a widely-recognized and frequently-complained-about problem, and it's something I'd like everyone doing ticket reviews to please think about.
Reviewers: if you make a comment on a ticket, but you don't say what you want to happen next, then you have effectively killed progress on that ticket until some other reviewer comes along and contradicts you to get things moving again. This is especially true if you make one trivial comment on the ticket and remove the review keyword, but don't say "please address these issues and then merge" or "please address these issues and then resubmit for review". If you've done a partial review, and made a comment like "I don't like this aspect of the design" or "please update copyright dates" or "your docstring formatting is wrong", please note in your comment that this is not a complete review, and *don't* remove the review keyword. Removing the keyword will introduce additional latency for the contributor, when other reviewers might still come along and attach more comprehensive feedback. There are few things more discouraging than having one free weekend every six months to work on a ticket, and to come back every time to "oops, you forgot to update the copyright date and insert a blank line in one file of your 300-line patch".
So, Tom: mass updating those tickets wouldn't be helpful, but an update every couple of days with a specific question on one of these I-don't-know-what-to-do tickets would be great. Your question on tm.tl/2491 was a definite step forward.
Awesome! I'm sure a lot of this has to do with the fact that once you maintain and work on something like this for a decade you take a lot for granted, especially with regard to your own policies and procedures. It becomes tedious to provide (or just difficult to enumerate) all the details necessary to new people. Hopefully as I become more familiar with them I can ensure some of the tedious/forgotten corners become documented in a way that makes it easier. And just point out when the current status of something is way too ambiguous to be actionable for more than a couple folks who are too busy to deal with it.
Here's a great example of what I'm talking about (and I apologize for the mid-message digression, but I think it's relevant...): http://twistedmatrix.com/trac/ticket/4636. This seems totally trivial, but five months later __all__ was never changed in t.i.main and JP's buildbot link is broken. Whether the offending class should be removed from __all__ or imported instead was never even mentioned. Anyway, I took a stab at it and attached a trivial patch so this isn't just another complaint (and it looks like JP closed it while I was drafting this email, so that's great!). But I do agree that working tickets would really help document the process!
As you have discovered here, drawing attention to a ticket in this "stuck" state will often cause it to get un-stuck. So please keep doing that.
Mostly, I really don't want you to write a gigantic pile of new
documentation and then find, when you're "done", that you missed some nuance of the coding standard, or the patch is too big to be reasonably reviewed, and that now you have three months of additional work to do before it's all *really* done. Experience with the process will mitigate that problem significantly. (And in fact I hope that you don't actually have a gigantic pile of stuff to commit all at once at any point, and can continue this work incrementally as a series of small tickets, but I realize that later on some of the index reorganization stuff may need to be big. This is mostly just restating what Kevin already said in his message, but it bears repeating.)
The more I think about this, the more I agree with you. My initial inclination was just to start from scratch and move over existing docs that I found I could use. This doesn't conflict with the reuse I've been supportive of (and for good reason: there's a lot of decent documentation already there) but it does conflict with the Twisted development policies.
At the end of the day, I can't really submit patches to existing documentation until said documentation is in Sphinx form.
Aaaaaaaauuuuugh.
When you say this, the first thing I think is "okay then, the sphinx migration is now introducing roadblocks into the documentation process and preventing people from contributing documentation, so let's just cancel it". So please don't say things like this :). The whole reason that we insisted on the current process for the sphinx migration was to avoid this kind of blockage.
Sorry! I didn't mean to suggest the migration is a roadblock. It's *inconvenient* that it isn't already done, but it's certainly not stopping anything to the point of cancelation (and doing so would probably do more harm than good since every new document we *can* convert is one that doesn't have to be converted manually)
You can submit documentation patches right now, in Lore format - which, I would like to remind you, is 99% plain-vanilla HTML and very easy to edit - and get them accepted. You can see on the sphinx buildbot (once a few minor issues are fixed, as discussed earlier in this thread) what those changes will look like once converted to ReST->Sphinx. Or you can simply run the conversion yourself locally - hopefully Kevin will chime in on how to set that up so I don't need to look it up :). As long as you aren't trying to do anything fancy with diagrams or tables (and *most* of the documentation really should not need elaborate diagrams or tables), you shouldn't run into any issues.
Yes, aside from things like toctrees and glossary references, it should be largely similar (although I tend to get pretty verbose with Sphinx/reST declarations because they're so handy). However, there will likely be times where it is either wildly inefficient or flat out impossible to port something to/from Sphinx. Specifically, any new/modified documentation that involves messing with code listings is going to be problematic. The current system calls for multiple code listing modules with copious code duplication and a general lack of testability; the new system calls for tested, combined modules who's display can be managed by Sphinx (see: http://sphinx.pocoo.org/markup/code.html#includes). I'm sort of at a loss for what to do in these situations and no simple solution jumps out at me. Having to make the cognitive switch between "I will write a dozen incremental, untestable listings" and "I will write compact, TDD listings" seems like it could cause a lot of latency (and extra work) regardless of which side of the fence you start on. Maybe we'll just have to take these sorts of conflicts one at a time and try to make decisions as quickly as possible as to which require backwards compatibility and which don't? *shrug*
Heck, if you want to write your documentation patches as ReST snippets and attach them to tickets, I'm sure you will find *many* willing contributors (myself included) who will jump in and do the format-munging manually to get them integrated into exiting lore documents so that they can make it to trunk immediately.
That may make sense for some issues. However, I am definitely up for modifying Lore docs when the changes wouldn't create a significant discrepancy between formats that would need to be manually resolved later (see: http://twistedmatrix.com/trac/ticket/2018).
Or, you can just jump in with ticket reviews, or breaking up chunk tickets, to move along the Sphinx migration quicker. But "I will work over here in a corner until somehow this gigantic pile of work gets finished by somebody else" is not a good strategy. (I'm not saying that that's exactly what you're proposing to do, but the aspects of what your proposing which align with that may not be the best way to proceed.)
Fair enough. I am trying to split my time equally between current, future, and new docs issues; I spent all day managing/working tickets and emailing, for instance. I definitely don't want to create a separatist thing here. :)
I guess one way this could work to the happiness of all involved is:
1. Get re-org nailed down; basic samples for whatever pages I think should have standardized format / elements. 2. Mixed in here, find existing docs and non-docs tickets to work on trunk. 3. When the re-org structure is ready, create a new branch from whatever the current Sphinx conversion branch is (hopefully it'll be "done" at this point) 4. Migrate new structure to branch, moving the existing docs to semi-logical-but-possibly-temporary places in the new structure. 5. Submit tickets / patches against existing documentation in said branch to move docs to new(er) locations, submit new TDD-style code samples, general edits, new docs, etc. 6. When everything is complete, submit a final ticket to merge to trunk
Does that make sense or am I still cognitively off the mark in terms of existing development practices?
With the caveat of what I said above, most of these things sound like generally good things to do. Except, first, I'd still like to see a ticket for what "the re-org" is actually going to be and what the point of it is. If I've only learned one thing in maintaining Twisted for 10 years, it's that a description of *what we are trying to do* separate from *how we are going to do it* early on is essential to make sure that others can give you useful feedback - and, more importantly, provide you with resources. In this case, those resources would be pointers to other areas in the documentation that you may not have noticed which already do some of what you are trying to do, but may be poorly integrated.
Okay. I will formalize the reorganization based on what we've already discussed on here and at the meet-up and hopefully that will make things at least as clear as mud! I have an official one for the new howto/Task standardization already (http://twistedmatrix.com/trac/ticket/4818), let me know if it's at all useful and/or what you mean here.
I will be fleshing out those goals more in the coming days, but some of it is implicit within the new documentation structure that's already in my repo.
$ python -c 'import this' | grep Explicit # :-)
As for a statement of goals when replacing/removing specific documentation, sure, I can do that as necessary. It's impossible to completely remove the element of taste but I will certainly avoid the "because I like this better" argument for making changes.
Again, this is more about having a good statement of purpose up front, so that everyone is focused on the same thing, than having an argument that you can trot out later once everyone starts nitpicking from a hundred different perspectives :). Plus, such a statement of purpose can serve as a focus for you as well.
You will probably have to press us core developers on this one, and you
may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
Debates are great!
Debates *that reach some kind of conclusion* are great :). Debates that just go in circles until everybody feels crappy about the topic aren't. So I'm really just asking you to help us make these debates into the great kind. (The rest of your reply seems to agree with that, I just wanted to be clear.)
Yeah, as I've already mentioned in this reply, the "limbo syndrome" of many tickets drives me nuts. I'm sure you guys hate it too, but having somebody new around who isn't afraid to ruffle some feathers for the greater good and isn't yet at some "acceptance" level with it should be helpful! That being said, I will do my best not to be a jerk about it; you guys have been doing this for much, much longer and I respect that.
Tickets in limbo need to be eliminated. I think you will find that just about everyone is happy to pick up these discussions with an injection of a little bit of attention and energy from a contributor. We all *know* that these sap the will of new contributors and thereby decrease the overall pool of available time, so I think you may have to try really, really hard to actually ruffle any feathers :).
Great, I was hoping I wasn't the only person who saw them this way!
If you like Python and DNS or SMTP or whatever the hell else, what has the
potential to be a more awesome implementation than its Twisted one? Let's help people find out for themselves that the answer is *nothing*. Then they'll want it to be *their* project.
If you do this, you will be my hero forever.
Twisted is a naturally-superior Python choice for most supported protocols by virtue of the fact that most network "stuff" benefits from event-based solutions. Like, find me a Python DNS server that *isn't* implemented in Twisted. People have these weird misconceptions about Twisted; it's really hard to grasp, ugly, unmaintained, etc. Ultimately, I want to correct these misconceptions, have *twisted.names* show up first on Google for "Python DNS Server", and all the other shit that really should be the case but isn't. Resolving these issues should ultimately resolve the problem of maintainers, community involvement, etc.
+10000.
I can't promise that in the course of this crusade I won't cause grief by trying to change things, but I will always strive to have a rational reason for wanting to do so!
Please keep the grief coming! It'll all be worth it, I'm sure :). Everything that you've said so far that I disagree with has just been an opportunity to communicate some of the Twisted development philosophy which may not yet be written down - and should probably be part of the documentation eventually. So you have yet to do anything which has put me out even in the slightest. We need more new contributors feeling comfortable being bold and trying to Just Do It.
That's good to hear! This is probably a discussion for another thread, but Twisted has always seemed a bit imposing in terms of ease of contribution. That may be a public relations issue as much as a procedural one, but making contribution a relatively frictionless, step-by-step process will go a looooong way to getting more people to do it. I also think that reviewer conduct is as important as the tone of the documentation (which should probably start with something like "These are Twisted's contributor rules and processes. We urge new contributors to start by simply writing a patch, though."); JP helped me out earlier today ( http://twistedmatrix.com/trac/ticket/4636) by accepting my patch despite a couple missing details. That gave me a nice sense of accomplishment without feeling like it was diluted by a few trivial details. I think there is *very* little cost to these sorts of things because either you just got (a) a one-time contribution you wouldn't have with a dozen steps, or (b) a new long-term contributor thanks to quick feedback, appreciation, and flexibility on the part of the team. But I'm drifting now, so shall drift in the direction of bed and pick things up tomorrow...
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jan 23, 2011, at 12:55 AM, Tom Davis wrote:
But I'm drifting now, so shall drift in the direction of bed and pick things up tomorrow...
I think this thread has reached a nice point of consensus, so rather than keep flooding everybody's inboxes with 'me too', I'm going to withdraw until we have some more specific things to discuss :). Still excited to see the documentation getting fixed though!
On Sun, Jan 23, 2011 at 2:08 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Jan 23, 2011, at 12:55 AM, Tom Davis wrote:
But I'm drifting now, so shall drift in the direction of bed and pick things up tomorrow...
I think this thread has reached a nice point of consensus, so rather than keep flooding everybody's inboxes with 'me too', I'm going to withdraw until we have some more specific things to discuss :).
Agreed, so I won't reply point by point, but I will just mention this" As of about 5 mins ago, with exarkun's kind assistance, the sphinx builder appears to be fixed: http://buildbot.twistedmatrix.com/builds/sphinx-html/sphinx-html-15731/
Still excited to see the documentation getting fixed though!
Yesyesyes! Kevin Horn
participants (9)
-
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
James Y Knight -
Jason Rennie -
Kevin Horn -
Michael Thompson -
Tim Allen -
Tom Davis -
Victor Norman