Mailman 3 September 2004 - Twisted-web

Re: [Twisted-web] Re: [Web-SIG] WSGI woes
by Phillip J. Eby 16 Sep '04

16 Sep '04

At 05:45 PM 9/16/04 +0100, Alan Kennedy wrote: >The only thing I disagree on are the names "sleep" and "wake", which IMHO >come with too many semantic hangovers from the threading world. When an >application calls wsgi.sleep(), it's not really sleeping, it's just >declaring that it currently has no output: a call to its iterator will >succeed, but the returned value will be an empty string. > >So basically, WSGI is providing an on/off indicator for every instance of >a middleware stack, which indicates to the server if there is currently >output available. Well, I'm proposing it as an optional extension, not a required feature. And, I think I'd like to streamline it to a single 'wsgi.pause_output' function, e.g.: resume = environ['wsgi.pause_output']() Where 'resume' is then a callback function that can be invoked to resume iteration. This keeps it to a single extension key, helps ensure the correct sequence of actions, and makes it easier to implement in some cases, while not making other cases any harder. >In an asynchronous situation, the application cannot simply do a blocking >read on the input: that will tie up the server thread. What do you mean by "server thread"? A truly asynchronous server (one using "no threads") cannot serve multiple WSGI requests simultaneously. In the general case, a WSGI server can only serve as many requests simultaneously as it has available threads for. However, WSGI applications that use iteration in place of 'write()' can sometimes be run with fewer than one thread per simultaneous request -- that's why iteration is recommended for applications that can be implemented that way. > So we need a way for the application to be notified/called when input > becomes available from the client. > >Perhaps we need to add an environment entry, e.g. "wsgi.input_handler", >which the app uses to pass a callable to the server. This callable would >be called whenever data became available on the input stream. > >So how would that work in the middleware stack? You would have to pass either 'environ' or 'wsgi.input' *into* this input handler request function, so that the server can verify it hasn't been replaced by any middleware. This is the standard way in WSGI of providing enhanced communication facilities that could "bypass" middleware. See: http://www.python.org/peps/pep-0333.html#server-extension-apis So, in principle, if the spec is modified to require middleware to honor child applications' block boundaries, then you could use an extension API to pause iteration until input is available, in much the same way that you would pause iteration for any other reason. Neither of these "pause iteration" solutions are especially elegant, at least from the POV of an async application author. But my objective here is only to make it *possible*, not necessarily pretty. I imagine that if there's actual demand for async apps to run under WSGI, it should be possible to create wrappers to let an application written in Twisted's continuation-passing style be run as a WSGI app. Such a wrapper would basically be just a function returning an iterator, with a bunch of pausing logic and a queue to communicate with the actual asynchronous app. And, such wrappers should only need to be written once for each asynchronous API, which as a practical matter probably means only Twisted, anyway, as (IMO) it has no real competitors in the Python async framework space.

1 0

Re: [Web-SIG] WSGI woes
by Phillip J. Eby 16 Sep '04

16 Sep '04

At 06:48 PM 9/15/04 -0400, Peter Hunt wrote: >It looks like WSGI is not well received over at twisted.web. > >http://twistedmatrix.com/pipermail/twisted-web/2004-September/000644.html Excerpting from that post: """The WSGI spec is unsuitable for use with asynchronous servers and applications. Basically, once the application callable returns, the server (or "gateway" as wsgi calls it) must consider the page finished rendering.""" This is incorrect. Here is a simple WSGI application that demonstrates yielding 50 data blocks for transmission *after* the "application callable returns". def an_application(environ, start_response): start_response("200 OK", [('Content-Type','text/plain')]) for i in range(1,51): yield "Block %d" % i This has been a valid WSGI application since the August 8th posting of the WSGI pre-PEP. It may be, however, that Mr. Preston means that applications which want to use 'write()' or a similar push-oriented approach to produce data cannot do so after the application returns. If so, we should discuss that use case further, preferably on the Web-SIG. >I thought the blocking call was handled by the iterator, but maybe I'm wrong. I'm not sure what you mean, but if you're asking whether the iterable is allowed to create output blocks after the application callable returns, then yes.

3 4

Re: [Twisted-web] Re: [Web-SIG] WSGI woes
by Phillip J. Eby 16 Sep '04

16 Sep '04

At 03:59 PM 9/16/04 +0100, Alan Kennedy wrote: >And, of course, that's what we're really discussing here: server >scheduling, and how servers ensure that application output gets >transmitted to clients with maximum efficiency and timeliness. IMHO, >asynchronous server scheduling algorithms and concerns have no place in >core WSGI, although a well-designed optional extension to support effiency >might have a nice unification effect on python asynchronous server >architectures. Right. I'd encourage people to experiment with async extensions like my sleep/wake idea, and if there's sufficient consensus we could add a "standard extension" to the spec. But I don't want to disturb the write()+iterable model, since that allows middleware to be mostly oblivious to the sync/async issue, and only apps or servers that care have to deal with it. While asynchronous servers are fairly common, most existing asynchronous applications are going to be tied to a particular async server architecture no matter what we do in WSGI.

1 0

Re: [Web-SIG] WSGI - alternate ideas, part II
by Peter Hunt 16 Sep '04

16 Sep '04

I think that the application should be passed a finish() method as a parameter or start_response return value. If the WSGI application is not a generator and returns wsgi.NOT_DONE_YET (similar to Twisted.web's NOT_DONE_YET), it is required to call finish(). Otherwise, the gateway will call finish() after the generator is finished or a string value is returned. That way, one could do all of the deferred calls they want, and simply return NOT_DONE_YET and call finish(). How does that sound? On Wed, 15 Sep 2004 23:27:18 -0400, Phillip J. Eby <pje(a)telecommunity.com> wrote: > At 11:06 PM 9/15/04 -0400, Peter Hunt wrote: > > >Thus, here is my "WSGI-X" proposal. The application will call the > >gateway, opposite of WSGI. For example, a CGI WSGI-X application may > >begin with: > > > >#!/usr/bin/env python > >if __name__ == "__main__": > > from wsgix import cgi > > req = cgi.get_request() > > That's pretty hard to implement correctly in any number of > servers. Really, pretty much every server wants to call the application, > rather than the other way around, because servers want to use their own > event loop. > > > >stdout - the raw, unbuffered direct output stream to the client > > So, header parsing is required? Or are only 'nph-' CGI scripts allowed? > > > >Now we have a basic interface to interact with HTTP. If one wants to > >write an extension to provide services like simplified cookie > >handling, sessions, or buffered headers and content, they write an > >extension function. A simple one for cookies would look like: > > > >def cookie_extension(req): > > if not hasattr(req, "cookie"): > > req.cookie = > > Cookie.SimpleCookie(req.environ.get("HTTP_COOKIE","")) > > Note that this can easily be accomplished in WSGI, by changing > 'hasattr(req,"cookie")' to '"my_extension.cookie" in environ' and > 'req.cookie' to 'environ["my_extension.cookie"]'. > > > >It modifies the request object if it hasn't already been modified. > >This saves us a bit of overhead so we won't need to parse the cookie > >again in case it is called twice (as it will if other extensions > >depend on it). > > Also achievable within 'environ'. > > > >Finish hooks have the same signature and execute when > >the finish() method is called. For example, a buffering extension > >would flush the buffer. Extensions can also add methods to the request > >object, for items such as add_header(). > > Under WSGI, such "finish" hooks can be rendered as a 'close()' method on an > iterable by a piece of middleware. > >

3 2

Adding data to the request
by Syver Enstad 16 Sep '04

16 Sep '04

Is there a supported way for adding data to a twisted.web request object. I am not thinking about the session, because I only want the data to last the current request, the typical use is for a resource to add some data that would be used by a child resource. I know I could use the args dictionary but that seems a little awkward as that is supposed to contain the query string/post for the page.

2 4

Twisted.web2
by James Y Knight 16 Sep '04

16 Sep '04

Itamar and I had dinner today and talked about new-web a bit. Here is what was discussed: 1) We need to keep compatibility with old Nevow resources 2) We ought to keep compatibility with old twisted.web resources 3) The new request API isn't the same as old request API. 4) new-web depends on Twisted 2.0. Nevow will be depending on new-web, so the next Nevow will depend on twisted 2.0. Thus it cannot be released until Twisted 2.0. According to itamar the goal for next Twisted release is around halloween, although I hadn't seen that mentioned anywhere before. ;) 5) Rumors are Anthony is writing a non-sucky HTTP client. (is this true?). J. Simms (slyphon) mentioned something about writing one to me a few weeks ago but I didn't hear anything after. This should be coordinated so the API for the client and server is as similar as possible. Conclusions for action: 1) new-web will be called twisted.web2. I will put what I've got into svn trunk as twisted.web2 tomorrow, assuming no objections. This will allow easy backwards compatibility at the cost of a 2. 2) Nevow's resource traversal API must be moved to twisted.web2. This will probably have to be done by branching Nevow to remove those bits. Therefore, the resource traversal part of nevow should be mostly-frozen while this happens. 3) For distribution: twisted.web2 depends on Nevow (for static directory listings, and error pages), so will be distributed with it. Nevow recommends Twisted but does not require it for simple cases (e.g. flattening to a string from a CGI script), so there can also be a limited-functionality distribution of standalone Nevow. Current status of new-web: the low-level HTTP protocol and header parsing is in passable shape. The high-level Resource API needs to be copied over from Nevow (again), because Nevow's API changed. Backwards compatibility adapters for the t.w.Request/nevow.inevow.IRequest and (possibly) nevow.inevow.IResource should be added. I am not able to do everything myself. I will need help, especially from dp would be helpful for the Nevow transition, to make this happen. If we do it right, twisted.web2 might even be ready in time for the release of Twisted 2.0. James

3 4

Partial freeze of nevow coming up.
by James Y Knight 15 Sep '04

15 Sep '04

Here's the plan. The following modules will be removed from Nevow, and merged/copied/whatever into twisted.web2: appserver.py static.py half of context.py dirlist.py failure.py guard.py static.py vhost.py url.py, perhaps. I want to create a nevow web2 integration branch, and while they are being removed from Nevow, and merged back into web2, I don't want the Nevow copy radically changing cause that just creates extra work. So if there's any changes you want to make to any of those, please do it soon. The rest of Nevow can continue changing during the transition without problem. Sidenote: Divmod needs to approve the relicensing of the code that moves into web2 from nevow under MIT license. Or else move all of Nevow under MIT license. James

2 1

url.URL.fromRequest gives unexpected behaviour
by Matt Goodall 15 Sep '04

15 Sep '04

Hi, URL.fromRequest() currently creates a URL for the path segments that have been resolved so far. That means that creating a URL in locateChild and a rend_* method gives you a different URL. That's unexpected. I think we need more explicit ways of creating a URL to allow for different usecases: URL.fromString(aStr) create a URL from the string. this is already implemented URL.fromRequestURI(aRequest) create a URL that matches the request exactly URL.fromContext(aContext) create a URL for handled path segments. kind of like the url.here overlay but not so lazy. A better name for fromContext might be fromLocation, fromSegment, etc. We can deprecate URL.fromRequest but, until it's removed, make it call fromContext. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt(a)pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer.

2 2

Re: [Twisted-web] separate sites and server
by Bartek Bargiel 13 Sep '04

13 Sep '04

> I have been using twisted.web.proxy to do something similar, but it has > a few flaws: I've already checked out twisted.distrib. Unfortunately it didn't seem to work with Nevow - with or without vhosts. Today I quickly introduced Mary's solution mentioned above. That's what I was looking for! Thanks! :-D Bartek Bargiel

1 0

separate sites and server
by Bartek Bargiel 13 Sep '04

13 Sep '04

Hi! Is there a way to create separate NevowSites and then 'register'/'unregister' them in a running Twisted application? My goal is to make possible to many users to run and restart their sites separately without stopping the whole system (all the apps). Note that virtual hosts mechanism is used to select the desired app. Any experiences or solutions? Regards, -- Bartek Bargiel

3 4