A lot's already gone on in this thread, but I don't think the original questions were answered too well. (Apologies if I've missed a message which did so, there were a lot of them.) Since you asked, I'll leave the filing of the tickets to you :). I appreciate the thorough analysis of the issues that one encounters when trying to write a customized web server using Twisted. If you do go ahead and file these tickets and attach a keyword to them ("tape"?), I'll make sure that somebody has a look at them during the next Twisted sprint in Cambridge. Speaking of which, we should set up another one of those. On 7 Apr, 03:46 pm, jack@chesspark.com wrote:
I just wrote a simple tool to make my AJAX app development cycle better. It's a tiny web server that serves up the current directory along side some reverse proxies. You can find the current code here:
Cool.
It seems like the need for this tool could have been obviated if "twistd web" had better support for composing plugins of its own. I'd encourage you to look at 'twistd web --help' and see if maybe some future release can provide all the functionality you want. For example, we've got "twistd web --path" and "twistd web --wsgi", but no convenient way to combine them to say "My WSGI app is over here, mapped to this URL, but it's got some static content over there, mapped to that URL". If what I'm describing makes sense to you, please go ahead and file an enhancement ticket to add this feature. I'm tempted to dive right in and start describing some implementation ideas, but knowing me this email's going to be long enough already.
* HTTPFactory seems only able to log to files. It takes a path name, not a file like object. This makes it very difficult to log to stdout, as I have to override log() in my own class. This in turns means cherry picking a global out of twisted.web - fun!
This is at _least_ three bugs. HTTPFactory should take a FilePath-like object (not a file-like object, since it may need to control when the log is opened and closed. _openLogFile is specifically documented as being for overriding in subclasses, yet it's private. That doesn't make much sense. Luckily the interface is broken anyway, so when we switch over to using a FilePath we can switch. While we're at it, twisted.python.logfile should really be updated to use FilePath as well, rather than direct os.path operations. The CLF-formatting logic really ought to be separated from the code that decides to actually write it to a file. Off the top of my head it really seems like this should be a method on Request, but a free function would be fine too. In addition, there's this bug: http://twistedmatrix.com/trac/ticket/75 which is maybe too vague to actually be actionable, but I think the idea was to publish Request objects with a custom format string via twisted.python.log.
* log() never gets called when a ReverseProxyResource is used. It appears as though request.finish() is never called, though I wasn't able to fully track this down. I find this behavior very odd.
This one's an existing bug: http://twistedmatrix.com/trac/ticket/2677 Please feel free to attach commentary.
* static.File, when given a directory, creates instances of itself to handle children. This code completely fails for me with when I subclassed static.File. Note that my class only has a few construction parameters, while static.File has 5. This is not documented. It would be nice if the code could detect this case and tell me that I have to overload createSimilarFile.
Again multiple tickets: * In the long term, static.File really ought to compose in a FilePath, not inherit from it. I don't know if it's possible to salvage static.File in its current state; I think we need a new static-content class that generally works the right way. For what it's worth, in your use case, you may want to override createSimilarFile(). In the meanwhile, before we have a new-and- improved File resource, * The parameters to __init__ should obviously be documented. * createSimilarFile should be documented.
* The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go. I find it extremely weird coming from several other web frameworks. There is nothing else but the concept of "hey you! get me child X". This makes it pretty hard to implement anything better than walking the tree since state would have to be collected over the traversal.
In Nevow and web2, locateChild allows you to consume multiple segments at once. The plan has always been to port these to twisted.web, but I can't find a ticket for that plan. We've discovered some problematic issues in the specifics of the way locateChild works; when we design a new one for twisted.web it will probably work subtly differently. But we do need a ticket to discuss that work. I am embarrassed to notice that none of this has been made public yet. So, there's a ticket for general resource lookup improvement. But I think independent of that there's also a need here for an "overlay" resource, which maps specific URIs to different resources, as if they all lived at the root. I'm going to handwave here a bit (don't think too hard about the fact that the first argument below is just a string, or how exactly dispatch works), but for example, let's say you've got a site which has been through a couple of transitions and you want to keep all of your URLs working. currentBlog = AwesomeTwistedBlog() o = Overlay(default=currentBlog) bloggerCompat = BloggerCompatibilityResource(currentBlog) o.mapPath("/2008/*", bloggerCompat) o.mapPath("/2007/*", bloggerCompat) o.mapPath("*.php", WordpressCompatibilityResource(currentBlog)) o.mapPath("/static", FilePath("some-data")) Oops, when I talked about composing plugins above I said I wasn't going to talk about implementation details. I guess I lied. This is what I was talking about; "Overlay()" is the resource which we'd need to implement 'twistd web --wsgi my.application.here --at /apps/my-app --path ./app/my-static-content --at /static'. So again, if this sounds useful, go ahead and file a ticket. Although the one for the command-line stuff should be separate from this substrate, they should link to each other.
* Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now.
The web sucks. There actually *is* a pretty significant semantic distinction between foo and foo/ in a URI. In particular, in order to generate relative links properly you _really_ need to care about that distinction. Even the thing that dotz mentioned, addSlash, is a bit of a misfeature; it's good convenience functionality but the interface to it is magical and unclear. This is the one thing I think isn't really a valid bug unless you have a better idea :).
* ReverseProxyResource returns nothing if you don't give it a '/' at the end. I had to work around this by doing a redirect in the subclass and then returning a new ReverseProxyResource when the '' child is accessed. It seems that if ReverseProxyResource's path is '', it just keels over with no error.
This could do with a bit more detail (mostly about what "keels over" means), but certainly sounds like a valid bug.