[Twisted-web] Thoughts on documentation, wading into Twisted, Nevow, etc.
Here's a summary of thoughts I've had over the last month or so. Please try to take the following as positive / constructive. I'm happy to be involved in trying to alleviate what some people may agree are problems. Or people may not agree that they're problems, which is fine too - more on this below. A few months ago I started looking at various (mainly Python) frameworks with an eye to using one for a project I'm working on. When I found Twisted (and friends) I was really happy. I've been writing code for what seems like a long time (30 years) and I think I'm pretty critical and demanding when it comes to good and clean design. I certainly had not expected to find something like Twisted and its various additional components. Without having written a line of Twisted code, I made the decision to throw away a full year of my own low-level, efficient, debugged, C code (networking, custom RPC, argument marshaling, tons of sweat). That was a pretty serious step, not taken lightly, and, I still think, a good one. That's why I'm still here, spending hours writing this email, wanting to press ahead, and willing to put in time and effort to help. In one corner then, we have a great foundation - elegant, dynamic, well designed, full of potential, nice programming language, written by talented and responsive developers, etc. But... in the other corner, documentation that very often seems to be a complete shambles. There, I've said it. I don't claim that this is actually a problem. That depends. It may well be that I've happened along at a moment which is not optimal for my needs and abilities; that there is a well-organized design, development, and documentation trajectory; and that I just happened to arrive at a bad point on that curve. I fully sympathize with things being less well documented than they could otherwise be. I love writing code. Following the rise of highlighting editors I've even become quite fond of comments. But I'm not even slightly fond of writing external docs. I think an important part of getting things right in design is to implement something that feels right and then rework it until it really feels right and clean, and is also useful. There's little point in documenting things before they have settled. Writing code, debugging, re-designing, building apps on top of libraries - all those things, to me, should take precedence over documentation. I don't believe in writing fully fleshed-out specs, docs, and tests ahead of implementation. I even have little time or sympathy for people (like myself in this case) who don't have what it takes to use emerging tools before they're fully documented for use. I've spent plenty of my own free time in the last 20 years writing, giving away, and supporting, code I felt like producing. So I think I understand the equation pretty well. But, Twisted and its related components are released. Valuable (to the project) people are trying to use them. This can be frustrating. I think I have a reasonably high frustration threshold (but maybe I'm wrong - maybe I'm just old & cranky). It's quite general to talk about "Twisted and its related components". There are many moving parts (some conveniently accompanied by moving names). So let me just pick on Nevow - it's perhaps in the middle in terms of maturity, documentation, frustration level, etc. I actually don't mean to single out Nevow in particular, but it has qualities that make it a good example target for discussion. Apologies in advance to anyone this seems to pick on - that's not my intention, I have no idea who wrote (or didn't write) some of this stuff. I know Matt is involved as he's been kind enough to several times send me long explanations and code in email. Again, I'm not claiming there's a problem here. I find Nevow to be extremely promising and yet quite cryptic. Sadly, I've built dozens of web sites, many of them commercial, done under contracts with real deadlines, budgets, staff, etc. I hate editing ugly templates in awkward ugly non-code file formats. There's just so much disgusting stuff out there. So to see Nevow and Stan is a real pleasure. But to try to understand it... argh. Hop onto google and try "nevow documentation" and read through some other people's experiences (e.g. http://www.third-bit.com/pywebblog/archives/cat_twisted.html) The recent "nevow documentation" thread (kicked off by Manlio, at http://twistedmatrix.com/pipermail/twisted-web/2006-May/002652.html) is mainly illustrative for its lack of conclusion. I don't find the suggested page at http://divmod.org/trac/wiki/TitleIndex to have much (if anything) to do with documentation - it seems to be links to code. It's not clear if the wiki is more a help or a hindrance (this is probably too condemning, but after spending more than enough time looking at things that one later discovers are out of date, replaced, deprecated, etc., you start to wonder about everything). Nevow comes with many examples, and that's great. But they're still cryptic. E.g., a grep url\\. */*.py in Nevow's examples directory gets me over 30 hits, but I don't see that class documented anywhere and when I try to use it in my code it doesn't work. E.g., formless is used in many examples, but is apparently a dead end. If you disagree with that, you'll be reinforcing my sentiment that things are a bit of a mess :-). I can give more examples. There's quite a bit that seems very cryptic, and I mean beyond the things described in the already very terse and piecemeal Nevow documentation. Of course it takes time to spell things out, but if people want them I will. I already decided to take the evening off, have a couple of beers, and write this email. Before trying to finish on a politically correct, upbeat, & constructive note, here's a question: How was it that Twisted (or just Nevow?) managed to lose the attentions of Kieran Holland? He (she?) was clearly into Nevow, put a lot of effort into understanding things, and yet headed off to Django, which, reading between the lines, he/she didn't really think was even nearly up to the Twisted/Nevow standard. That's a bad loss. People who can read and write code and who are willing to spend serious amounts of time helping ease others into projects are so valuable. The Addendum of Kieran's Stan page (http://www.kieranholland.com/code/documentation/nevow-stan/) seems to spell things out pretty clearly. It feels like the Nevow (and broader?) community has done itself a serious disservice in losing Kieran's attention (but, see above comments on whether this is perceived as a problem, trajectories, etc). To me, totally on the outside, I do not get the impression that there is a clear design, code, and release trajectory in place. It feels like there is a bunch of very talented designers and coders who are banging out an increasing number of cool projects at high speed, leaving a frustratingly incomplete, misleading, and atrophying breadcrumb trail of documentation behind them. On to another question: do people think there a problem here, or not? The answer may be no. I think that's fine, I'm happy to be told to read the code and to shut up. At least that would be definitive, and not at all unwelcome. But, if you think there is a problem, what can we do? Here are some possible concrete steps that wouldn't take huge amounts of time. 1) Go through all Twisted & friends web sites and wiki and _ruthlessly delete everything_ that mentions obsolete, deprecated, or renamed projects / modules, broken links, example code using such, etc. There are many of these things, and they are highly misleading and frustrating and give a bad overall impression. 2) State, where appropriate, that the source code is currently the best documentation. There's no shame in this, and it's much better than having people run across and waste time on obsolete docs, or spend time looking for things that do not exist. E.g., "At this point the nevow.url class is totally undocumented. See examples/{x,y,z} for example usage and the code and comments in nevow/url.py for the best current documentation." 3) Have someone, preferably an outsider (could be me), write up and maintain a no-punches-pulled overview document about using the various Twisted components. Stick a link to this in a prominent place. http://twistedmatrix.com/trac/wiki/WebDevelopmentWithTwisted is a good start. 4) Have someone go through example code to stick in more comments, get rid of old stuff (no need to rewrite if that takes too long, just throw it away). E.g., the widespread use of formless in Nevow examples. Is it in, is it out? If the latter, get rid of it before more people die. 5) Have someone do some web searches for stuff known to be fixed or out of date, and send mail to the authors asking if they'd please consider updating or otherwise clarifying what they've written, could they add a link or a postscript, to help out future people looking to use Twisted & friends? No need to try to hide the fact that things were a bit of a mess - just aim not to have people get an out-of-date impression or follow links to incorrect / obsolete pages (e.g., links to nevow.com). I could probably come up with more suggestions. Given the inclinations of (I assume) many people still reading this message, no-one is too much inclined towards producing actual documentation (and it may not be the right moment, as above). So it makes sense to focus on easy hits: purging old crud, minimizing wasting the time of people looking to get involved, making it clear what the barriers are (e.g., point people at the source, tell them that at this point the only way to understand is to ask questions (but time is precious) and to read the code). OK, enough words for now. Thanks again for all the help, all the code, etc. Terry
Terry Jones wrote:
Here's a summary of thoughts I've had over the last month or so. Please try to take the following as positive / constructive. Terry, that was a great email. Thanks for the time and consideration.
I'd have to say that I agree with you almost entirely. I tend to run into two different types of responses when I talk to people about developing in Twisted (with regard to the documentation): 1) what they hell? how can you possibly use this? I don't understand what the hell they're talking about! and 2) wow! this is SO much better than what it used to be -- I think I can actually use twisted now. I have been using Nevow since it was first made "public" at pycon 2004. It's the best thing out there for web dev that fits my brain. However, I have had a very difficult time getting dev shops to adopt it. Yet, I have had a few wins lately -- because of several fully developed applications I built for different projects. They were amazed at the applications' architectures, soundness, etc. -- all thanks to the relative logic and soundness of Twisted/Nevow and its developers. Living examples that demonstrate "best practices" may be one of the biggest lacks in the Nevow docs right now. This has come up several times on IRC, as well. What the developers in those shops were able to see for the first time was: * how python classes were used to construct sections of a site (using rend.Page) * how python classes were used to construct sections of a page (using rend.Fragment) * how slots are created and filled * the usefulness of data_*() and render_*() methods * the manipulation of sequences in the XHTML templates Once the developers in the shops saw the whole picture with clear, clean code that did exactly what you would expect it to do, they were sold. They can't wait to do the next few projects using Nevow (instead of Plone). I haven't seen these guys this excited in a long time. It seems there is a need for several tutorials/documentation updates to fill these needs. One of the greatest strengths of Nevow, though, is its flexibility. The combination of Twisted and Nevow almost provide more of a "meta web dev framework" than one of the "competing" web frameworks. And this naturally makes documentation of "the right way to do it" rather difficult. One solution may be a series of tutorials addressing various needs/circumstances while outlining the benefits/reasons for the given approach. I've been thinking of writing a new tutorial lately. Maybe we should crank a few out... hmm, on that thought, maybe there should be a standard format for Nevow tutorials. "Intended Audience", "What this tutorial addresses", "What this tutorial doesn't address", "Components used", etc. Dunno. Just rambling now. Glad you're writing about this :-) d
On Sun, 02 Jul 2006 21:51:57 -0600, Duncan McGreggor wrote:
Terry Jones wrote:
Here's a summary of thoughts I've had over the last month or so. Please try to take the following as positive / constructive. Terry, that was a great email. Thanks for the time and consideration. yep same here :) It seems there is a need for several tutorials/documentation updates to fill these needs. One of the greatest strengths of Nevow, though, is its flexibility. The combination of Twisted and Nevow almost provide more of a "meta web dev framework" than one of the "competing" web frameworks. And this naturally makes documentation of "the right way to do it" rather difficult. One solution may be a series of tutorials addressing various needs/circumstances while outlining the benefits/reasons for the given approach.
I've been thinking of writing a new tutorial lately. Maybe we should crank a few out... hmm, on that thought, maybe there should be a standard format for Nevow tutorials. "Intended Audience", "What this tutorial addresses", "What this tutorial doesn't address", "Components used", etc. Dunno. Just rambling now. Glad you're writing about this :-)
d maybe it'd be a good idea to have write some templates for tutorials and/or DTDs/Stylesheets (that if xml will be used) that way we could also easily generate pdfs,.. i could probably help with that and maybe even a bit with writing documentation though I haven't used nevow much (some years ago I've written a complete working [and deployed] webshop using woven - nevow's predecessor for those who don't know what it is ;) and it worked great - definitely better than mvc (which was even before woven hehe)
I really hope this will result in some good tutorials / examples (some of the woven examples are quite good at the moment as they are (I like the blog / wiki tutorials though I can't get them to work with current svn *and* the code generated by axiomatic is quite different ..) Regards -- -----BEGIN GEEK CODE BLOCK----- GCS/CC/E/M/MU/S d- s: a--- C++++(++) UL++++ P+ L++++ E W+++ N+++ o-- K w-- O M-- V- PS+ PE-- Y++ PGP+++ t+++ 5+ X- R tv b++++ DI- D+ G++ e-->+++++ h-- !r z- ------END GEEK CODE BLOCK------
The others already answered to your email and they basically all said things I agree with, hence I'll concentrate on proposing yet another way of trying to solve this problem. :)
On Mon, 3 Jul 2006 03:15:32 +0200, Terry Jones
But, if you think there is a problem, what can we do? Here are some possible concrete steps that wouldn't take huge amounts of time.
1) Go through all Twisted & friends web sites and wiki and _ruthlessly delete everything_ that mentions obsolete, deprecated, or renamed projects / modules, broken links, example code using such, etc. There are many of these things, and they are highly misleading and frustrating and give a bad overall impression.
I agree on this but only when we will finally have a completed twisted.web2 that will allow us to port Nevow to it. Some effors are already going in this direction with the recent #1252 bug in Divmod trac and the no-more-context-1252 branch. Given the relative simplicity of this newly written code it might be a good way to start looking at Nevow and writing some documentation with developers' aid. I'll have the last exam of this university year on the 5th, after that day I'll probably have plenty of spare time to develop and help documenting Nevow. Not being a: 1) Native english speaker and 2) Good at explaining things clearly I wouldn't be the right person for documenting Nevow in first person while I'd _love_ to help somebody else to write new documentation (by reviewing, answering, completing and whatnot). I also have some applications under the MIT license that show how to use Nevow mechanism outside Divmod infrastructure (which despite being really well written is not always the best solution for any problem, like they know well enough :)) and the code can be taken as an example or best practice or just a starting point.
2) State, where appropriate, that the source code is currently the best documentation. There's no shame in this, and it's much better than having people run across and waste time on obsolete docs, or spend time looking for things that do not exist. E.g., "At this point the nevow.url class is totally undocumented. See examples/{x,y,z} for example usage and the code and comments in nevow/url.py for the best current documentation."
Sometimes the Nevow source code is not really clean and cannot really be understood (the url module is a good example of being mildly obscurish) however it's much better to redirect to it than to provide outdated documentation, this is an unfortunate problem that has plagued nevow since long long ago and nobody bothered fixing because we literally have no time :(.
4) Have someone go through example code to stick in more comments, get rid of old stuff (no need to rewrite if that takes too long, just throw it away). E.g., the widespread use of formless in Nevow examples. Is it in, is it out? If the latter, get rid of it before more people die.
Another thing I agree with. IMHO it's been a while that formless has been more or less discontinued and it should at least be definately separated from nevow if not just dropped. There's also an additional problem to this since currently there are 2 different form libraries one that integrates with athena and another one that does not. formal and liveform. Also in its current shape formless is not even close to being what it was thought for, I know many use formless currently for their needs but IMHO (and I would like some opinions on this of course) having 3 form projects of which 2 (despite being different) solve the same problem is a waste of resources. Anyone that is spending time and effort on fixing/extending formless should move to formal which is considerably easier and cleaner in its design also it is fairly easy to extend. Yes it doesn't still support form customization but adding it is not a 10 month effort because the rendering class is completely separated from the rest and it could be added with an afternoon of work. Allowing this would have _many_ positive effects to Nevow. For example rend.py would finally be cleaned up from the code depending on formless, faster startup for less component registration, less code in rend.py, less code in Nevow, no more CARRYOVER and so on. Thanks for your mail :)
On Mon, 3 Jul 2006 15:28:00 +0200
Valentino Volonghi aka Dialtone
[...] Also in its current shape formless is not even close to being what it was thought for, [...] Anyone that is spending time and effort on fixing/extending formless should move to formal which is considerably easier and cleaner in its design also it is fairly easy to extend [...]
I am using formless in Bic (a weblog engine I'm coding)... does this mean I should move on to formal? I'm not afraid of switching; in fact I prefer to make changes instead of using a piece of code which nobody will take care of... Any explanation is welcome -- I'll search the mailing list archives so maybe even no explanation is a good one if this topic was already addressed. Regards, -- Adrian Perez "Experience is what you get when you don't get what you want" -- (Dan Stanford)
On Mon, 3 Jul 2006 16:27:43 +0200, Adrian Perez
I am using formless in Bic (a weblog engine I'm coding)... does this mean I should move on to formal? I'm not afraid of switching; in fact I prefer to make changes instead of using a piece of code which nobody will take care of...
Yes I think you should. Formless is going away for sure in the future, the only matter is exactly decide when.
Any explanation is welcome -- I'll search the mailing list archives so maybe even no explanation is a good one if this topic was already addressed.
formal is just better at the main goal that both formless and formal do. It still lacks some features but those can be implemented.
Hi Folks, Here's a rough first draft of Techno Turkey's latest adventure. I've tried to capture and integrate both Valentino and Daniel's ideas, but if I've missed or misrepresented anything, please advise. I haven't had a chance yet to finish setting up my own Twisted development environment, so I haven't been able to test it. In preparing to set up my environment, however, I realized that there may be conventions in the community (directory layout and naming, for example) that would be very helpful for turkey's like me. So I've left a hole in the Turkey script for a link to a page (yet to be written) that lay's out best practices for down-loading Twisted/Nevow and setting up a productive development environment. My thought was that we could put this page on the Turkey site, or elsewhere in the Twisted/Nevow docs, then I could simply link to it from whereever appropriate in Turkey scripts. Anybody around to help me write the page? Or, if such a page already exists, please point me to it. Best wishes, Lloyd PS: I've been reviewing current Twisted/Nevow documentation. Much much better than I remember from last year. Nevow tutorial is excellent. So here's... Techno Turkey Dialtone's Resource Magic "It's magic!" said Techno Turkey. "What's magic?" asked Twisted Wizard Daniel. "I just type it in at the top of my browser and... shazamm... back comes just the page I'm looking for -- could be from anywhere in the world. It's like casting a magic spell." "It's far more magical than you think," says Twisted Wizard Daniel. "Just consider it's name: 'Uniform Resource Identifier' -- URI." "I thought it was a URL," said Turkey. "That was before people understood it's full mojo. Back then they called it a 'Uniform Resource Locator.' But then they realized that it could be used to identify anything in cyberspace." "Anything?" asked Turkey. "Sure, anything. We use URI's to identify servers, webpages, web services, images, sound files..." "But the only thing I ever get back are webpages." "That's because that's what static HTTP webservers do. They listen for their designated URI and return the specified HTML web page. But aided by the right software, they can return anything of a digital nature." Twisted Wizard Dialtone happened to be listening in. "Hey, Turkey old bird," said Daniel, "Want to see my Resource magic?" "The bird's computer got trashed by an electrical storm, remember," said Daniel. "No surge protection? No backup?" "Don't rub it in," said Turkey. "Never mind, here's how get back up and running again..." *************** Need link here to step-by-step load and set up instructions for Twisted and Nevow from both Twisted/Nevow sites and Debian. Download Twisted from: http://twistedmatrix.com/products/download Download Nevow from: http://divmod.org/trac/wiki/DivmodNevow#Download Debian users can get the latest alpha, rc, or release by adding the following to their /etc/apt/sources.list: Woody: deb http://twisted.sourceforge.net/debian/woody/ ./ Sid/Sarge: deb http://twisted.sourceforge.net/debian/ ./ Once downloaded: -- how to find Twisted and Nevow source directories -- how to set up development environment -- how to set up working directories -- user source, logs, -- how to set up path variables -- how to test development environment Should be sufficiently detailed that even the slowest turkey on the farm can implement. Anybody up for fleshing in details for this page? I can edit, put it up on the Turkey server, and link it into Turkey adventures. *************** "Now, create a file called ResourceMagic.py in your Twisted source directoy, and enter the following code:" from zope.interface import implements from nevow import inevow ## ## How does a request come to the Resource? ## ## or How to use Nevow without all the fancy automations ## class Root(object): # This is a simple resource, representing the root, or top-most # resource of this site. The inevow.IResource interface # tells us that it must implement two methods: # locateChild and renderHTTP. # locateChild is used to find children of the current resource. It must # return a tuple of (resource, remaining_segments). If no child resource # exists at this location, you may return nevow.rend.NotFound." # For more information see: # http://divmod.org/users/exarkun/nevow-doc/nevow-traversal.html implements(inevow.IResource) def locateChild(self, ctx, segments): # This locateChild is 'stupid' since it can only work if the tree of # resources is static. But it will work for our simple example if segments[0] == '': # If the server is looking for the root resource segments will be ('',) # then renderHTTP will be called on self return self, () elif segments[0] == 'foo': # Now we received a request whose segments had in the first position # the string foo like http://example.org/foo/baz/ -> ('foo', 'baz'). # After the resource has been located we return it with the remaining segments # ('baz') return self.foo, segments[1:] else: return None, () def renderHTTP(self, ctx): # When the server needs to return a response to the request it will call # the renderHTTP method that will return a string of what needs to be sent. def renderHTTP(self, ctx): return """<html><body>Hello, world!<br />""" class Foo(object): implements(inevow.IResource) def locateChild(self, ctx, segments): # segments is the remaining segments returned by the root locateChild # see segments[1:] if segments[0] == 'baz': return self.baz, segment[1:] else: return None, () def renderHTTP(self, ctx): return """<html><body><h1 id="heading">You are in Foo</h1> <a href="./foo/baz" id="baz">baz</a></body></html> """ class Baz(object): implements(inevow.IResource) def locateChild(self, ctx, segments): return None, () def renderHTTP(self, ctx): return '<html><body><h1 id="heading">You are in Baz</h1></body></html>' # We are adding children to the resources. # This could also happen inside the class. root = Root() root.foo = Foo() root.foo.baz = Baz() "Note," said Dialtone, "That this is the IResource interface from Nevow; not the one from twisted.web." "So now what?" said Turkey. "At the end of your ResourceMagic.py source file enter:" from twisted.internet import reactor from nevow import appserver site = appserver.NevowSite(root) reactor.listenTCP(8080, site) "Hey, I recognize the reactor bit from earlier adventures, " said Turkey, "But what's this 'nevow?'" "Nevow is a web-application construction kit written in Python. Check out http://divmod.org/trac/wiki/DivmodNevow. But let's keep our eye on the ball here... "To cast the spell, just save the ResourceMagic.py file and enter at the console prompt:" python ResourceMagic.py "Then point your browser to:" http://localhost:8080/ "Wait," said Turkey, "Before we run it, I'm still confused by this word 'Resource.' When you say, 'Resource,' are you talking about web pages?" "Could be," said Daniel. "But there are many more kinds of Resources than webpages. A Resource is anything that a URI happens to point to -- anything in cyberspace." "Well, at least," said Dialtone, "Anything available through the designated server." "Like my favorite Radiohead MP3?" said Turky. "Anything that gets you through the day," said Daniel. "For more info, check out:" http://en.wikipedia.org/wiki/Resource_(Web) "...or:" http://en.wikipedia.org/wiki/Representational_State_Transfer#Resources "OK," said Turkey, turning to Dialtone. "How's this magic spell of yours supposed to work?" "Well let's run it and see..." Another way to run the code Another way to run Dialtone's code uses the tac file approach. Replace the lines: from twisted.internet import reactor from nevow import appserver site = appserver.NevowSite(root) reactor.listenTCP(8080, site) with from twisted.application import strports, service from nevow import appserver application = service.Application('example') website = appserver.NevowSite(root) webservice = strports.service('8080', website) webservice.setServiceParent(application) Now, at your browser prompt, enter: twistd -noy myscript.tac And point your browser to: http://localhost:8080/ Anticipated Results: You should see a root page with "Hello, world!," and a link to a child page named "foo." Any other url except the one with the child foo will result in a 404. If you go in the foo child you'll see a page with the content "you are in foo" and a link to foo's child page baz. Any other child page will result in a 404. If you go in the baz child you'll see a page with the content "you are in baz." This page has no children and any other url will result in a 404. How does it work? Nevow is fundamentally based on the following request processing schema: arrives request \ --> root resource \ --> root.locateChild(url segments) . child_resource.locateChild(url segments[1:]) . child_resource2.locateChild(url segments[2:]) . so forth until url segments == () ==> FOUND RIGHT RESOURCE ==> RIGHT_RESOURCE.renderHTTP() \ Where to go from here Contrast Dialtone's code with Matt Goodall's Barebnes Dynamic Web Server in Turkey's fourth adventure. Then review: http://divmod.org/users/exarkun/nevow-doc/nevow-intro.html
On Sat, 15 Jul 2006 15:40:24 -0400 (EDT), lloyd@paisite.com wrote:
I haven't had a chance yet to finish setting up my own Twisted development environment, so I haven't been able to test it. In preparing to set up my environment, however, I realized that there may be conventions in the community (directory layout and naming, for example) that would be very helpful for turkey's like me.
There is no such thing for directories AFAIK. It's usually left to the developer and I'd say the follow standard python code organization. The coding standard instead is the Twisted coding style. http://twistedmatrix.com/projects/core/documentation/howto/policy/coding-sta...
So I've left a hole in the Turkey script for a link to a page (yet to be written) that lay's out best practices for down-loading Twisted/Nevow and setting up a productive development environment. My thought was that we could put this page on the Turkey site, or elsewhere in the Twisted/Nevow docs, then I could simply link to it from whereever appropriate in Turkey scripts.
These are the same that you have for any other python project. An alternative for Nevow is using Combinator (explained in the Divmod Combinator page): http://divmod.org/trac/wiki/DivmodCombinator Another alternative for Nevow is using setuptools although I'm unsure how working it is. Yet another for both is using .pth
Anybody around to help me write the page? Or, if such a page already exists, please point me to it.
I hope you like the links :)
Best wishes,
Lloyd
PS: I've been reviewing current Twisted/Nevow documentation. Much much better than I remember from last year. Nevow tutorial is excellent.
So here's... Techno Turkey
Cool :). Looks very good.
Hi Valentino,
On Mon, 17 Jul 2006 12:48:33 -0500, Valentino Volonghi aka Dialtone
On Sat, 15 Jul 2006 15:40:24 -0400 (EDT), lloyd@paisite.com wrote:
I haven't had a chance yet to finish setting up my own Twisted development environment, so I haven't been able to test it. In preparing to set up my environment, however, I realized that there may be conventions in the community (directory layout and naming, for example) that would be very helpful for turkey's like me.
There is no such thing for directories AFAIK. It's usually left to the developer and I'd say the follow standard python code organization. The coding standard instead is the Twisted coding style.
http://twistedmatrix.com/projects/core/documentation/howto/policy/coding-sta...
I'd like to jump in right here and talk abut directory organization for just a second. While you are correct that no specific directory layout conventions have been blessed by the core development team, I think that there is a lot of value in providing one, if only to give really new users some kind of structural anchor point from which to begin learning web-app development with twisted. A good example is actually the layout used by the various Divmod apps. Is the set of directory conventions there the One, True, Code Layout? Surely not, but I think that they are more than adequate for a newbie, and will give them one less thing to think about while getting started. So, that said, I propose we use the following directory layout convention in the Turkey Adventures (with the clear understanding that this convention is not any kind of officially mandated thing): /ProjectName /packagename /static /css /js /images module1.py module2.py ... I think this is plenty simple, and will prevent the reader from being distracted by a bunch choices that are less important than learning the actual framework. Anyway, that's my 2 cents on the issue of directory layout for the examples. I think the new tutorial looks great! L. Daniel Burr
On Mon, 17 Jul 2006 13:08:36 -0500, "L. Daniel Burr"
So, that said, I propose we use the following directory layout convention in the Turkey Adventures (with the clear understanding that this convention is not any kind of officially mandated thing):
Since we are talking about best practices... :)
/ProjectName /packagename /static /css /js /images module1.py module2.py ...
/ProjectName /static /i - images /s - styles /j - javascript /template /doc /database - sql files /bin - various scripts /projectname - python modules /web - for the web stuff /storage - for the data layer /other_eventual_twisted_using_subparts common_python_modules.py
Valentino Volonghi aka Dialtone ha scritto:
[...]
Since we are talking about best practices... :)
[...]
/ProjectName /static /i - images /s - styles /j - javascript
An interesting question is: when one should consider the option of serving static content with a server like lighttpd, and forward with mod_proxy requests for dynamic content to twisted web? static.File is a no match versus lighttpd, but there is the overhead of mod_proxy and one should also consider how to design the site layout at the best. Thanks and regards Manlio Perillo
Hi Manlio,
On Mon, 17 Jul 2006 13:54:18 -0500, Manlio Perillo
Valentino Volonghi aka Dialtone ha scritto:
[...]
Since we are talking about best practices... :)
[...]
/ProjectName /static /i - images /s - styles /j - javascript
An interesting question is: when one should consider the option of serving static content with a server like lighttpd, and forward with mod_proxy requests for dynamic content to twisted web?
For the purposes of the tutorial currently being discussed, I think that this question is premature. A later tutorial, concerning production environments, deployment, etc, would be the perfect place to cover things like using static web servers to offload certain pieces of content from twisted.
static.File is a no match versus lighttpd, but there is the overhead of mod_proxy and one should also consider how to design the site layout at the best.
Agreed, but again, this is a topic for much further down the road. This particular tutorial is much more limited in scope, and more beginner oriented. Also, I'll point out that the directory layouts proposed by myself and by dialtone aren't really affected by the use of static.File versus some other webserver. The layout can remain the same regardless. L. Daniel Burr
On Mon, 17 Jul 2006 16:54:18 -0200, Manlio Perillo
An interesting question is: when one should consider the option of serving static content with a server like lighttpd, and forward with mod_proxy requests for dynamic content to twisted web?
Fairly easy. Just be sure that all the static content is served from a known (or many _known_) urls like: /static/... Thus you can use the following rule: $HTTP["host"] =~ "^(testing.stiq.it|localhost)$" { url.rewrite-once = ( "^/static/.*" => "$0", "^/favicon.ico$" => "/static/$0", "^/(.*)" => "/vhost/http/%0/$1" ) $HTTP["url"] !~ "^/(static/|favicon.ico)" { proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => 8080 ) ) ) } server.document-root = "/usr/local/src/repos/stiq/" } Given the rewrite rule urls are rewritten once and then matched to see if the request should be proxied or served by lighttpd from the document root.
On Mon, 17 Jul 2006 13:08:36 -0500, "L. Daniel Burr"
/ProjectName /packagename /static /css /js /images module1.py module2.py ...
There is actually an under-maintained Axiomatic plugin in Mantissa which helps with this somewhat; it probably doesn't work at the moment, but check out "axiomatic mantissa". It might help with this kind of setup, or at least provide some ideas for tools to expedite the turkey's adventures.
On Mon, 17 Jul 2006 14:58:14 -0400, glyph@divmod.com wrote:
There is actually an under-maintained Axiomatic plugin in Mantissa which helps with this somewhat; it probably doesn't work at the moment, but check out "axiomatic mantissa". It might help with this kind of setup, or at least provide some ideas for tools to expedite the turkey's adventures.
I mean "axiomatic project". I am way too used to typing "axiomatic mantissa" all the time...
Hi Lloyd,
On Sat, 15 Jul 2006 14:40:24 -0500,
So here's... Techno Turkey
Dialtone's Resource Magic
"It's magic!" said Techno Turkey.
"What's magic?" asked Twisted Wizard Daniel.
"I just type it in at the top of my browser and... shazamm... back comes just the page I'm looking for -- could be from anywhere in the world. It's like casting a magic spell."
"It's far more magical than you think," says Twisted Wizard Daniel. "Just consider it's name: 'Uniform Resource Identifier' -- URI."
"I thought it was a URL," said Turkey.
"That was before people understood it's full mojo. Back then they called it a 'Uniform Resource Locator.' But then they realized that it could be used to identify anything in cyberspace."
"Anything?" asked Turkey.
"Sure, anything. We use URI's to identify servers, webpages, web services, images, sound files..."
"But the only thing I ever get back are webpages."
"That's because that's what static HTTP webservers do. They listen for their designated URI and return the specified HTML web page. But aided by the right software, they can return anything of a digital nature."
Twisted Wizard Dialtone happened to be listening in.
"Hey, Turkey old bird," said Daniel, "Want to see my Resource magic?"
Looks like a typo, above. I think it is intended to read "said Dialtone". [snip initial download/install stuff]
Once downloaded:
-- how to find Twisted and Nevow source directories
I'd just make sure twisted and nevow are installed in site-packages, though other options are certainly possible. In the spirit of not distracting the reader from non-essential information, let's just pick one.
-- how to set up development environment
Well, we've got to at least cover setup for Win32 and Linux. I'd argue that OS X and one of the BSDs deserves a little attention too.
-- how to set up working directories -- user source, logs,
See my response to dialtone's recent email on this list, for my suggestion regarding directory layout.
-- how to set up path variables
I suggest using .pth files rather than path variables. Using .pth files should "just work" regardless of your OS, and not require any fiddling around with the environment by hand. So, basically, for a new web project, you'd create your project dirs, and then stick a "myproject.pth" file in the site-packages directory, containing "/path/to/my/project/topmost/folder"
-- how to test development environment
To test that the dev environment is set up correctly, I'd just start up a python interpreter prompt and import twisted, then import nevow, and lastly, import myprojectpackage.
Should be sufficiently detailed that even the slowest turkey on the farm can implement.
Anybody up for fleshing in details for this page? I can edit, put it up on the Turkey server, and link it into Turkey adventures.
***************
"Now, create a file called ResourceMagic.py in your Twisted source directoy, and enter the following code:"
from zope.interface import implements
from nevow import inevow
## ## How does a request come to the Resource? ## ## or How to use Nevow without all the fancy automations ##
class Root(object): # This is a simple resource, representing the root, or top-most
# resource of this site. The inevow.IResource interface # tells us that it must implement two methods: # locateChild and renderHTTP. # locateChild is used to find children of the current resource. It must # return a tuple of (resource, remaining_segments). If no child resource # exists at this location, you may return nevow.rend.NotFound." # For more information see: # http://divmod.org/users/exarkun/nevow-doc/nevow-traversal.html implements(inevow.IResource)
def locateChild(self, ctx, segments): # This locateChild is 'stupid' since it can only work if the tree of # resources is static. But it will work for our simple example if segments[0] == '': # If the server is looking for the root resource segments will be ('',) # then renderHTTP will be called on self return self, () elif segments[0] == 'foo': # Now we received a request whose segments had in the first position # the string foo like http://example.org/foo/baz/ -> ('foo', 'baz'). # After the resource has been located we return it with the remaining segments # ('baz') return self.foo, segments[1:]
else: return None, ()
def renderHTTP(self, ctx): # When the server needs to return a response to the request it will call # the renderHTTP method that will return a string of what needs to be sent. def renderHTTP(self, ctx): return """<html><body>Hello, world!<br />"""
class Foo(object): implements(inevow.IResource)
def locateChild(self, ctx, segments): # segments is the remaining segments returned by the root locateChild # see segments[1:] if segments[0] == 'baz': return self.baz, segment[1:] else: return None, ()
def renderHTTP(self, ctx): return """<html><body><h1 id="heading">You are in Foo</h1> <a href="./foo/baz" id="baz">baz</a></body></html> """
class Baz(object): implements(inevow.IResource) def locateChild(self, ctx, segments): return None, () def renderHTTP(self, ctx): return '<html><body><h1 id="heading">You are in Baz</h1></body></html>'
# We are adding children to the resources. # This could also happen inside the class. root = Root() root.foo = Foo() root.foo.baz = Baz()
"Note," said Dialtone, "That this is the IResource interface from Nevow; not the one from twisted.web."
"So now what?" said Turkey.
"At the end of your ResourceMagic.py source file enter:"
from twisted.internet import reactor from nevow import appserver site = appserver.NevowSite(root) reactor.listenTCP(8080, site)
"Hey, I recognize the reactor bit from earlier adventures, " said Turkey, "But what's this 'nevow?'"
"Nevow is a web-application construction kit written in Python. Check out http://divmod.org/trac/wiki/DivmodNevow. But let's keep our eye on the ball here...
I'd move this section up, and place it just after the ResourceMagic listing. The code uses inevow.IResource, so I think it is best to clear that up right away, rather than having Dialtone mention Nevow, go on to another code snippet and *then* finally explaining what nevow is.
"To cast the spell, just save the ResourceMagic.py file and enter at the console prompt:"
python ResourceMagic.py
"Then point your browser to:"
"Wait," said Turkey, "Before we run it, I'm still confused by this word 'Resource.' When you say, 'Resource,' are you talking about web pages?"
"Could be," said Daniel. "But there are many more kinds of Resources than webpages. A Resource is anything that a URI happens to point to -- anything in cyberspace."
"Well, at least," said Dialtone, "Anything available through the designated server."
"Like my favorite Radiohead MP3?" said Turky.
"Anything that gets you through the day," said Daniel. "For more info, check out:"
http://en.wikipedia.org/wiki/Resource_(Web)
"...or:"
http://en.wikipedia.org/wiki/Representational_State_Transfer#Resources
"OK," said Turkey, turning to Dialtone. "How's this magic spell of yours supposed to work?"
"Well let's run it and see..."
Another way to run the code
Another way to run Dialtone's code uses the tac file approach.
I probably sound like a broken record, but I'd suggest offering one, and only one, way to run the code, so that users aren't fiddling with things that are not central to grasping the concepts being presented. Me, I use .tac files for everything these days, so that's what I would recommend for use within the tutorials. Keep up the great work, it is inspiring to see someone take this on. L. Daniel Burr
Hello, On Sun, July 2, 2006 9:15 pm, Terry Jones wrote:
Here's a summary of thoughts I've had over the last month or so.
As one totally smitten with Twisted but compelled to abandon it due to lack of documentation, I heartily second Terry Jones's critique and proposal. I tried to contribute my small bit toward better Twisted documentation with the Techno Turkey Adventures series... even created a Techno Turkey blog. My assumption was that by starting from a know-nothing stance and asking embarrassingly naive questions I could tease out enough technical direction to build enlightening and useful code experiments designed to accelerate progress up the Twisted learning curve. But several things brought the endeavor to a premature end: 1) My own limited Python skills meant that I couldn't, on my own hook, dig much meaning out of the source code. There are a number of Python (or, perhaps, object) idioms used extenstively in the Twisted corpus that totally baffle me. 2) I never could get a grasp of the big picture -- how all the parts fit together -- or don't. Starting from the Twisted root, the corpus seems to branch out prolifically -- some branches fairly complete and useful, some under active development, and some abandoned. I just couldn't tell which was which. 3) Several people from the Twisted community generously offered guidance and support. Of those, all but one knew little more than I did. These stalworthy folks helped enormously with testing and identification of various copy and code errors, but couldn't lead me toward forward progress. The one person who knew his way through the thicket the way was more than generous with his suggestions, but too time-constrained to give much in the way of big picture. By and large, I had the feeling that the core Twisted developers were moving so fast exploiting Twisted and progeny commercially, that they had little time for documentation or bringing others along behind them. No complaint here; just stating a perception. 4) My own time constraints imposed severe limits on the time I could commit to the projectd. As a result, I just couldn't learn enough fast enough to meet a deadline I had for a web project of my own. The result, I turned to another Python web framework and was able to make significant progress within days -- not because the framework (Karrigell) was better documented, but because it so was far less ambitious -- simple-minded even -- that even a turkey like me could understand it. I would like to return to Twisted. Indeed I would like to pick up the Turkey series again if it is deemed helpful to the community. But to do so I would need one or more talented technical mentors to keep me on a productive path. If not Turkey, then I'm still willing to provide all the help I can, given my limited technical perspective, toward better Twisted documentation. But, again, as a follower, not a leader. Cheers, Terry. I'm pulling for your success. Best wishes, Lloyd R. Prentice
participants (9)
-
Adrian Perez -
Duncan McGreggor -
glyph＠divmod.com -
L. Daniel Burr -
lloyd＠paisite.com -
Manlio Perillo -
Terry Jones -
Thomas Raschbacher -
Valentino Volonghi aka Dialtone