Mailman 3 April 2009 - Twisted-web

Is there a way to implement a deferred sequence?
by twisted-web＠udmvt.ru 10 Apr '09

10 Apr '09

Greetings. I want a sequence which is Deferred with two callbacks: one as usual - for the whole result called at the end of the sequence, another - called for every element of the sequence (as they arrive). The number of elements is not known beforehand, so it is not a defer.DeferredList, but the idea is almost the same. To provide some interface to send items to the sequence, fire item callback for every item arrived and have main callback to be called at the end of the sequence. It could be used for example in AJAX web-applications that use Athena to simplify/automate the dynamic updates of some slowly arriving results (while in AJAX-supporting environment) and to gracefully fallback when javascript is not enabled/not desired. It could be deployed as using a slightly modified "sequence"-like renderer with "data" function returning this form of Deferred. The renderer will output some 'live' element immediately and then will append remotely it's contents as items arrive. Or will fallback to the usual builtin "sequence" renderer when AJAX is not desired. I think this approach will make it possible to hide all details into the renderer code. The same could be done to make items replace each other with the last being displayed as the final result... Or maybe it is already done in a different way? Or have I missed something, could anyone point me, please? Thanks for your time. -- Alexey S.

2 1

some questions about twisted.web
by Jack Moffitt 09 Apr '09

09 Apr '09

I just wrote a simple tool to make my AJAX app development cycle better. It's a tiny web server that serves up the current directory along side some reverse proxies. You can find the current code here: http://github.com/metajack/tape/tree/master In writing this I ran into a number of issues with twisted.web. I'm pretty sure some of these are bugs, but I wanted to get some feedback here before I filed them. * HTTPFactory seems only able to log to files. It takes a path name, not a file like object. This makes it very difficult to log to stdout, as I have to override log() in my own class. This in turns means cherry picking a global out of twisted.web - fun! * log() never gets called when a ReverseProxyResource is used. It appears as though request.finish() is never called, though I wasn't able to fully track this down. I find this behavior very odd. * static.File, when given a directory, creates instances of itself to handle children. This code completely fails for me with when I subclassed static.File. Note that my class only has a few construction parameters, while static.File has 5. This is not documented. It would be nice if the code could detect this case and tell me that I have to overload createSimilarFile. * The only way to stick something in the tree at an arbitrary location seems to be to walk the tree to that spot, creating dummy locations as you go. I find it extremely weird coming from several other web frameworks. There is nothing else but the concept of "hey you! get me child X". This makes it pretty hard to implement anything better than walking the tree since state would have to be collected over the traversal. * Related to the above, the handling of foo vs. foo/ is pretty confusing. foo/ is considered the '' child of foo. This is pretty yuck to me. Unfortunately, I don't have any creative suggestions about how to do it better right now. * ReverseProxyResource returns nothing if you don't give it a '/' at the end. I had to work around this by doing a redirect in the subclass and then returning a new ReverseProxyResource when the '' child is accessed. It seems that if ReverseProxyResource's path is '', it just keels over with no error. These combined to make what should have been a few lines of code and an hour into 3-4 hours and almost 150+ lines. Most of the time I spent tracing through the Twisted source trying out figure out where the weird stuff was coming from. And after all that logging still doesn't work right. jack.

11 29

Re: [Twisted-web] some questions about twisted.web
by glyph＠divmod.com 08 Apr '09

08 Apr '09

A lot's already gone on in this thread, but I don't think the original questions were answered too well. (Apologies if I've missed a message which did so, there were a lot of them.) Since you asked, I'll leave the filing of the tickets to you :). I appreciate the thorough analysis of the issues that one encounters when trying to write a customized web server using Twisted. If you do go ahead and file these tickets and attach a keyword to them ("tape"?), I'll make sure that somebody has a look at them during the next Twisted sprint in Cambridge. Speaking of which, we should set up another one of those. On 7 Apr, 03:46 pm, jack(a)chesspark.com wrote: >I just wrote a simple tool to make my AJAX app development cycle >better. It's a tiny web server that serves up the current directory >along side some reverse proxies. You can find the current code here: Cool. >http://github.com/metajack/tape/tree/master It seems like the need for this tool could have been obviated if "twistd web" had better support for composing plugins of its own. I'd encourage you to look at 'twistd web --help' and see if maybe some future release can provide all the functionality you want. For example, we've got "twistd web --path" and "twistd web --wsgi", but no convenient way to combine them to say "My WSGI app is over here, mapped to this URL, but it's got some static content over there, mapped to that URL". If what I'm describing makes sense to you, please go ahead and file an enhancement ticket to add this feature. I'm tempted to dive right in and start describing some implementation ideas, but knowing me this email's going to be long enough already. >* HTTPFactory seems only able to log to files. It takes a path name, >not a file like object. This makes it very difficult to log to >stdout, as I have to override log() in my own class. This in turns >means cherry picking a global out of twisted.web - fun! This is at _least_ three bugs. HTTPFactory should take a FilePath-like object (not a file-like object, since it may need to control when the log is opened and closed. _openLogFile is specifically documented as being for overriding in subclasses, yet it's private. That doesn't make much sense. Luckily the interface is broken anyway, so when we switch over to using a FilePath we can switch. While we're at it, twisted.python.logfile should really be updated to use FilePath as well, rather than direct os.path operations. The CLF-formatting logic really ought to be separated from the code that decides to actually write it to a file. Off the top of my head it really seems like this should be a method on Request, but a free function would be fine too. In addition, there's this bug: http://twistedmatrix.com/trac/ticket/75 which is maybe too vague to actually be actionable, but I think the idea was to publish Request objects with a custom format string via twisted.python.log. >* log() never gets called when a ReverseProxyResource is used. It >appears as though request.finish() is never called, though I wasn't >able to fully track this down. I find this behavior very odd. This one's an existing bug: http://twistedmatrix.com/trac/ticket/2677 Please feel free to attach commentary. >* static.File, when given a directory, creates instances of itself to >handle children. This code completely fails for me with when I >subclassed static.File. Note that my class only has a few >construction parameters, while static.File has 5. This is not >documented. It would be nice if the code could detect this case and >tell me that I have to overload createSimilarFile. Again multiple tickets: * In the long term, static.File really ought to compose in a FilePath, not inherit from it. I don't know if it's possible to salvage static.File in its current state; I think we need a new static-content class that generally works the right way. For what it's worth, in your use case, you may want to override createSimilarFile(). In the meanwhile, before we have a new-and- improved File resource, * The parameters to __init__ should obviously be documented. * createSimilarFile should be documented. >* The only way to stick something in the tree at an arbitrary location >seems to be to walk the tree to that spot, creating dummy locations as >you go. I find it extremely weird coming from several other web >frameworks. There is nothing else but the concept of "hey you! get >me child X". This makes it pretty hard to implement anything better >than walking the tree since state would have to be collected over the >traversal. In Nevow and web2, locateChild allows you to consume multiple segments at once. The plan has always been to port these to twisted.web, but I can't find a ticket for that plan. We've discovered some problematic issues in the specifics of the way locateChild works; when we design a new one for twisted.web it will probably work subtly differently. But we do need a ticket to discuss that work. I am embarrassed to notice that none of this has been made public yet. So, there's a ticket for general resource lookup improvement. But I think independent of that there's also a need here for an "overlay" resource, which maps specific URIs to different resources, as if they all lived at the root. I'm going to handwave here a bit (don't think too hard about the fact that the first argument below is just a string, or how exactly dispatch works), but for example, let's say you've got a site which has been through a couple of transitions and you want to keep all of your URLs working. currentBlog = AwesomeTwistedBlog() o = Overlay(default=currentBlog) bloggerCompat = BloggerCompatibilityResource(currentBlog) o.mapPath("/2008/*", bloggerCompat) o.mapPath("/2007/*", bloggerCompat) o.mapPath("*.php", WordpressCompatibilityResource(currentBlog)) o.mapPath("/static", FilePath("some-data")) Oops, when I talked about composing plugins above I said I wasn't going to talk about implementation details. I guess I lied. This is what I was talking about; "Overlay()" is the resource which we'd need to implement 'twistd web --wsgi my.application.here --at /apps/my-app --path ./app/my-static-content --at /static'. So again, if this sounds useful, go ahead and file a ticket. Although the one for the command-line stuff should be separate from this substrate, they should link to each other. >* Related to the above, the handling of foo vs. foo/ is pretty >confusing. foo/ is considered the '' child of foo. This is pretty >yuck to me. Unfortunately, I don't have any creative suggestions >about how to do it better right now. The web sucks. There actually *is* a pretty significant semantic distinction between foo and foo/ in a URI. In particular, in order to generate relative links properly you _really_ need to care about that distinction. Even the thing that dotz mentioned, addSlash, is a bit of a misfeature; it's good convenience functionality but the interface to it is magical and unclear. This is the one thing I think isn't really a valid bug unless you have a better idea :). >* ReverseProxyResource returns nothing if you don't give it a '/' at >the end. I had to work around this by doing a redirect in the >subclass and then returning a new ReverseProxyResource when the '' >child is accessed. It seems that if ReverseProxyResource's path is >'', it just keels over with no error. This could do with a bit more detail (mostly about what "keels over" means), but certainly sounds like a valid bug.

2 2