Mailman 3 September 2004 - Twisted-web

Implement "Page Not Found "
by Tsai Li Ming 27 Sep '04

27 Sep '04

Dear All How do I implement a 404 Page Not Found using twisted web for those resources that I did not add using putChild? class Server(resource.Resource): def getChild(self, name, request): if name == '': return self return Resource.getChild(self, name, request) Thanks, -Liming

1 0

Nevow 0.3
by Donovan Preston 27 Sep '04

27 Sep '04

Nevow 0.3 is out. Get it from http://nevow.com/releases/0.3.0/ Thanks to everyone for all their hard work over the last 5 months! I'm very proud of this release. dp

1 0

web2
by William Dode 25 Sep '04

25 Sep '04

hi, Is there somewhere a paper on the plan of web2 ? I mean the goal of web2 and the differences with web1. I think it will help us to understand why web2 need nevow. -- William - http://flibuste.net

8 10

web2 cleanup
by Donovan Preston 25 Sep '04

25 Sep '04

Well, I finally got a chance to look at web2 tonight, and as I did I just did whatever random cleanups seemed appropriate. Patch attached, and I will commit unless there is any objection. It's pretty obvious to me that web2 is going to need some pretty major scrubbing before release to remove the stench of web. James, we should coordinate some time on IRC when we can both talk about and work on this. I need to devote some energy to this before Twisted 2.0.

1 0

many virtual hosts -> separate log files
by Bartek Bargiel 25 Sep '04

25 Sep '04

hi there! :) let's assume I'm running a single twisted.web server with many vhosts served. Please tell me how to log requests to separate log files depending on requestHostname detected? regards, -- Bartek Bargiel

1 0

session expiration
by Bartek Bargiel 23 Sep '04

23 Sep '04

Hi! I was looking for a possibility to set session timeout - I found this timeout hardcoded in twisted.web.server module. Shouldn't it be more flexible? ;-) -- Bartek Bargiel

2 1

Re: [Twisted-Python] Fwd: [Twisted-commits] r11721 - revert r11685
by Cory Dodt 20 Sep '04

20 Sep '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Urgh. I usually avoid getting involved in these discussions--not because I don't want to participate but because the ML style does not work very well with my brain. There's all these different parts of this proposal and some are being rejected by some people and modified by other people and it's very hard for me to figure out what the whole picture is. In this case, I have to say something because this is pretty important to the product I'm developing. I just spent 6 weeks doing a conversion to Nevow because Woven isn't being supported any more. I do *not* want to do the same thing with Web. I've made modifications to Twisted trunk that I was hoping to get out of 2.0 for our official release, but if there's not going to be a web1 any more, that puts this project in jeopardy again because it means I may not be able to use 2.0. So here's what I have to say: please *summarize*. I need to get a sense of: 1) Timeline.. when are changes being implemented, in what order, tied to what release etc. 2) How is the svn repo going to change? 3) What is going to be put into what release by whom? 4) What's the impact if you're currently using web1 and you want to be using web2 because 1 isn't supported any more? 5) What's the impact if you stay with web1? If you haven't decided yet, go ahead and decide, but then summarize afterwards. :-) Thanks.. C Itamar Shtull-Trauring wrote: | On Mon, 2004-09-20 at 10:49, Glyph Lefkowitz wrote: | | |>>We already figured out which web server we ("we" being those developers |>>who actually care about the web) are going to be supporting. |>>twisted.web2. web should have a deprecation warning in __init__ for a |>>release, and then should be terminated with extreme prejudice. |> |>I would terminate before providing a deprecation warning. If you want |>to provide a backwards-compatibility release for using web1 with the new |>reactor core, that's fine, but I don't see any reason to include web1 |>with the new download. | | | twisted.web will not be included. There will be two downloads ("Twisted | Web 1 - Deprecated", "Twisted Web 2"). Most people will not download the | first ever. The first will likely not have new releases unless we fine | security issues. Moving it to branch and only doing releases from there | may be fine (if our release infrastructure is up to it!) but I still | think newweb should be twisted.web2. | | | Summary: | | 1. twisted.web2 remains web2. | 2. Move twisted.web (and any other security bugfix only packages, I | guess) to a "deprecated" branch. | | Yes? No? | | If yes, that just leaves the nevow/twisted.web2 merger question, which | should probably be done on twisted-web mailing list. | | | | _______________________________________________ | Twisted-Python mailing list | Twisted-Python(a)twistedmatrix.com | http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBTvtx3A5SrXAiHQcRApINAKCggxacS+kFLmAszBQmrQeuByJHBgCgnyov jBEWFaH7dZ6TZBbtljvVZTg= =tLpe -----END PGP SIGNATURE-----

2 1

Re: [Twisted-web] Re: [Web-SIG] WSGI woes
by Phillip J. Eby 16 Sep '04

16 Sep '04

At 10:41 PM 9/16/04 +0100, Alan Kennedy wrote: >In this way, there could be a middleware component below the >rot13_streamer in the stack that, say, does chunked_transfer encoding and >decoding. It would be the same in form as the above, except that it would FYI, middleware and apps are now banned from dealing in any kind of transfer-encodings, per James' very valuable input on that subject. Like connection properties, these should be the exclusive province of the actual web server. >1. Change the environ entry for 'wsgi.async_input_handler' to be its own >callable that records the callback for the next layer up in the stack, the >rot13_streamer.input_handler. This would lead to the unacceptable situation of every middleware component having to know in principle about extensions. The "Server Extension APIs" section of the PEP demands that any "bypass" API verify replacement for this very reason. >I think that this proposed approach is clean, and not overly complex for >async or blocking programmers to handle. Unless of course they're writing middleware that does something with the input. >But I think we do have to cleanly separate the two. I think there are >problems associated with trying to run *all* components seamlessly across >async or blocking servers. I think that middleware components that are >always going to behave correctly in an async situation will have to be >designed like that from the ground up. It's dangerous to take components >written in a blocking environment and run them in an async environment. It is a non-goal for WSGI to support running multiple requests simultaneously in a single-threaded asynchronous server, so the issue doesn't really come up. A WSGI server *must* allow for the fact that WSGI apps use up a thread while they're running or producing a value: that's the price of being able to run "traditional" web applications under WSGI. >And lastly, if it is desired to spin jobs into a different thread, e.g. >the rot-13 job above, then that should be a middleware concern, not the >WSGI server's. I agree with you -- for *asynchronous* applications. Synchronous web applications are the default case in WSGI and the world in general, so servers *must* use a thread pool to start applications and to run 'next()' calls, if they are asynchronous. But, asynchronous applications wish to yield control, to avoid hogging resources in that thread pool, so they need to delegate the work to their I/O thread, and then yield an empty string to pause output, freeing up that thread for another iterable next(), or application start. Notice, however, that if the server is *synchronous* (e.g. CGI, single-threaded FastCGI containers, mod_python under Apache 1.x, etc., ), then this is a complete waste of time, because you'll only be running one simultaneous request in this process anyway, so you're spinning off a second thread to keep from tying up the first thread, but all the first thread is doing is waiting for the second thread to finish! This is wasteful, to say the least. The only case where pausing output (whether for unrelated network I/O, or because of a need to read from the input stream) is actually useful is when the server is *also* asynchronous -- hence the value of making such pausing an optional extension API. The application can then detect when it's *useful* to pause, and synchronous applications needn't worry about it. Of course, even if the server and application are *both* asynchronous, that's no guarantee that they're using compatible event loops! If you try to run a Twisted app under asyncore or vice versa, you're going to be spinning off an extra thread to run a second event loop, so there's a bit of a trade-off to determining whether your asynchrony is going to actually *gain* anything. But that's a separate question. WSGI will allow you to be asynchronous if you really want to, no matter how bad an idea it might be in some cases. :) >The twisted rot-13 component would then have very thin methods (run from >the server's main thread) which interact with the twisted space i.e. >transferring data and receiving data back through queues, and layer WSGI >semantics on those interactions, i.e. pause_output, yield result, yield >empty_string, etc. You're pretty much describing what I suggested earlier: that async app frameworks like Twisted may want to have a model whereby a generic "thin wrapper" WSGI application object is used to communicate with an application that's written using the underlying framework's async idioms. So, for example, one might perhaps design a Twisted "Transport" that was implemented as a WSGI application. (I don't know if "Transport" is really the correct abstraction to use, I'm just giving an example here.) Anyway, for such a thing to really work, I think you might need server-specific reactor plugins, to integrate Twisted's event loop with that of the server. >When I described your approach as "pulling data up the stack", I saw a >bigger difference between the two approaches. I'm thinking now that there >is little difference between our proposals, except that in mine it's the >bottom component that gets notified of the input by the server, and in >yours it's the top component. Though I suppose having the top component >pulling input from an iterator chain mirrors nicely the situation where >the server pulls output from an iterator chain. Actually, I'm saying you pull data *down* the stack. The bottom-most application iterator calls 'read()' on an input stream provided by a parent middleware component, which then calls read on a higher-level component, and so on. >And my approach basically entails a bunch of nested calls, which might be >less efficient elegant than if, say, generators were used in an input >processing chain. > >You're right again Phillip :-) Not entirely, actually. For my approach to really work, the middleware would have to be guaranteed to return something from read(), as long as the parent's read() returns something. Otherwise, the resumption would block, unless the middleware were much smarter. I've got to think about it some more, because right now I'm still not happy with the specifics of any of the proposals for pausing and resuming output.

1 1

Re: [Twisted-web] Re: [Web-SIG] WSGI woes
by Phillip J. Eby 16 Sep '04

16 Sep '04

At 08:29 PM 9/16/04 +0100, Alan Kennedy wrote: >[Alan Kennedy] > >> In an asynchronous situation, the application cannot simply do a > >> blocking read on the input: that will tie up the server thread. > >[Phillip J. Eby] > > What do you mean by "server thread"? A truly asynchronous server (one > > using "no threads") cannot serve multiple WSGI requests > > simultaneously. In the general case, a WSGI server can only serve as > > many requests simultaneously as it has available threads for. > >Sorry, I should have paid more attention to phrasing in this context. > >By "server thread" I mean the thread of execution that is running the >select/poll operation in the server (which needs at least *one* thread). >If the application did a blocking read of the input running in a simple, >single-threaded asyncore-style server, that single thread would block, >holding up event processing. Right, which is (one reason) why a WSGI server can in the general case only serve as many WSGI requests simultaneously as it has available threads for, although it's possible to improve on that worst-case condition by appropriate use of iterators. >But I don't see the need for pausing logic or queues? Why can't the server >simply call directly into the application, e.g. using a "process_input" >method, in effect saying "you have some input ready". > >And I'm not sure I see the need for the application to check that the >wsgi.input hasn't been replaced: if there were middleware further down >that stack that was intercepting and transforming the input stream, then >*it* should be the one receiving the asynchronous notification from the >server. This lower level component would then read some input, process it, >and then call a "process_input" method on the next component up in the >stack, etc, etc. > >I suppose I'm talking about the server "pushing" the input through the >middleware stack, whereas you're talking about the application at the stop >of the stack "pulling" the data up through the stack. Is that right? That's correct, and that's what I'm trying to avoid if at all possible, because it enormously complicates middleware, to the sole benefit of asynchronous apps -- that mostly aren't going to be portable anyway. So, going by STASCTAP theory (Simple Things Are Simple, Complex Things Are Possible), the pause/resume approach makes asynchronous applications *possible*, while keeping the nominal synchronous cases and middleware *simple*. >And I'd be interested to see how your approach would handle a situation >where there is both streaming input and output. For example, a server that >takes strings of any length, say 10**9 bytes, and .encode('rot13')'s each >byte in turn, before sending it back to the client. Presumably, the function to pause for input needs to take a minimum length, or have some way to communicate available length to the application. I don't pretend to fully understand the needed use cases here, because I have little experience writing web applications that need to wait on other network services (other than databases) while a client is waiting. And if I were writing an asynchronous server, I'd probably at least consider using Greenlets to context-switch blocking operations so that they wouldn't tie up an active thread. Such an approach is conceptually easier to deal with, IMO, than writing everything in continuation-passing style. But I *do* want WSGI to make it *possible* to meet async apps' use cases, which is why I'm seeking input from those that do have the relevant experience. The trade-off is that it shouldn't excessively complicate nominal compliance with WSGI. In particular, I'd prefer that the current "example CGI gateway" in PEP 333 not require any major changes or significant expansion.

1 0

Draft language for WSGI to forbid blocking by middleware
by Phillip J. Eby 16 Sep '04

16 Sep '04

I'm proposing the following language to be added to PEP 333, as a subsection under "Buffering and Streaming", just before the subsection entitled, "The write() callable". It doesn't address pausing or resuming iteration (which I don't have a PEP-able proposal for yet), but it should ensure that middleware doesn't introduce any additional blocking issues: ===excerpt start=== Middleware Handling of Block Boundaries ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to better support asynchronous applications and servers, middleware components **must not** block iteration waiting for multiple values from an application iterable. If the middleware needs to accumulate more data from the application before it can produce any output, it **must** yield an empty string. To put this requirement another way, a middleware component **must yield at least one value** each time its underlying application yields a value. If the middleware cannot yield any other value, it must yield an empty string. This requirement ensures that asynchronous applications and servers can conspire to reduce the number of threads that are required to run a given number of application instances simultaneously. Note also that this requirement means that middleware **must** return an iterable as soon as its underlying application returns an iterable. It is also forbidden for middleware to use the ``write()`` callable to transmit data that is yielded by an underlying application. Middleware may only use their parent server's ``write()`` callable to transmit data that the underlying application sent using a middleware-provided ``write()`` callable. ===excerpt end=== In addition to this insertion, I would modify the 'start_response()' specification to note that HTTP headers should not be sent until the first *non-empty* string is yielded from the iterable. Comments?

1 0