[Twisted-web] Re: [Web-SIG] WSGI woes

[Phillip J. Eby]

At 01:13 AM 9/16/04 -0400, Donovan Preston wrote:

>On Sep 15, 2004, at 7:12 PM, Phillip J. Eby wrote:
>
>>At 06:48 PM 9/15/04 -0400, Peter Hunt wrote:
>>>It looks like WSGI is not well received over at twisted.web.
>>>
>>>http://twistedmatrix.com/pipermail/twisted-web/2004-September/ 000644.html
>>
>>Excerpting from that post:
>>
>>"""The WSGI spec is unsuitable for use with asynchronous servers and
>>applications. Basically, once the application callable returns, the
>>server (or "gateway" as wsgi calls it) must consider the page finished
>>rendering."""
>>
>>This is incorrect.
>
>As I said in my original post, I hadn't mentioned anything about this
>yet because I didn't have a solution or proposal to fix the problem,
>which I maintain remains.

Reading the rest of your post, I see that you are actually addressing the 
issue of asynchronous *applications*, and I have only been addressing 
asynchronous *servers* in the spec to date.  (Technically "half-async" 
servers, since to be properly portable, a WSGI server *must* support 
synchronous applications, and therefore an async WSGI server must have a 
thread pool for running applications, even if it contains only one thread.)

However, I'm not certain that it's actually possible to support *portable* 
asynchronous  applications under WSGI, since such asynchrony requires 
additional support such as an event loop service.  As a practical matter, 
asynchronous applications today require a toolset such as Twisted or 
peak.events in addition to the web server, and I don't really know of a way 
to make such applications portable across web servers, since the web server 
might use a different toolset that insists on having its own event 
loop.  Or it might be like mod_python or CGI, and not really have any 
meaningful way to create an event loop: it could be utterly synchronous in 
nature and impossible to make otherwise.

Thus, as a practical matter, applications that make use of asynchronous I/O 
*may* be effectively outside WSGI's scope, if they have no real chance of 
portability.  As I once said on the Web-SIG, the idea of WSGI is more aimed 
at allowing non-Twisted apps to run under a Twisted web server, than at 
allowing Twisted applications to run under other web servers!  The latter, 
obviously, is much more ambitious than the former.

But I'm happy to nonetheless explore whether there is any way to support 
such applications without unduly complicating middleware.  I don't expect 
it would complicate servers much, but middleware can be quite difficult, 
because middleware currently isn't even required to return when the 
application does!  It's not recommended, but a middleware component can sit 
there and iterate over the return value and call its parent's write() 
method all it wants.  In the presence of this kind of behavior, there isn't 
any real way to guarantee that a thread isn't going to be tied up with 
processing.  But realistically, that's what an async server's thread pool 
is *for*.

Anyway, as you'll see below, WSGI can actually run async apps with minimal 
blocking even without any modifications to the spec, and with 
server-specific extensions you can eliminate *all* the blocking, as long as 
middleware doesn't do anything pathological.  In practice, of course, I 
think the spec *should* be updated so that middleware is prohibited from 
interfering with the control flow, and I'll give some thought as to how 
that should be phrased.


>According to the spec, """The application object must return an
>iterable yielding strings.""" Whether the application callable calls
>write before returning or yields strings to generate content, the
>effect is the same -- there is no way for the application callable to
>say "Wait, hang on a second, I'm not ready to generate more content
>yet. I'll tell you when I am." This means the only way the application
>can pause for network activity is by blocking.

That is correct.  The application must block for such activities.  However, 
as a practical matter, this isn't a problem for e.g. database access, since 
using Twisted's adbapi would still tie up *some* thread with the exact same 
blocking I/O, so there's actually no loss in simply doing unadorned DBAPI 
access from within the application.


>  For example, a page
>which performed an XML-RPC call and transformed the output into HTML
>would be required to perform the XML-RPC call synchronously. Or a page
>which initiated a telnet session and streamed the results into a web
>page would be required to perform reads on the socket synchronously.

Technically, it could perform these tasks asynchronously, as long as the 
data were queued such that the application's return iterable simply 
retrieved results from the queue.  However, this would naturally block 
whenever the client was ready for I/O, but no results were available yet.

However, an asynchronous server isn't going to sit there in a loop calling 
next()!  Presumably, it's going to wait until the previous string gets sent 
to the client, before calling next() again.  And, it's presumably going to 
round-robin the active iterables through the threadpool, so that it doesn't 
keep blocking on iterables that aren't likely to have any data to produce 
as yet.

Yes, this arrangement can still block threads sometimes, if there are only 
a few iterables active and they are waiting for some very slow async 
I/O.  But the frequency of such blockages can be further reduced with a 
couple of extensions.  Suppose there was an 'environ["async.sleep"]' and 
'environ["async.wake"]'.  A call to 'sleep' would mean, "don't bother 
iterating over me again until you get a 'wake' call".

This *still* wouldn't prevent some item of middleware from hogging a thread 
in the threadpool, but I suppose you could actually make the 'sleep' 
function sit in a loop and run active iterables' next() methods until one 
of the suspended iterables in the current thread "wakes", at which point it 
would return control to whatever iterable it was called from.  Or, if you 
want to use Greenlets, you can always return control directly to the 
iterable that needs to "wake up".

Anyway, my point here is that it's possible to get a pretty decent setup 
for async applications, without any need to actually modify the base WSGI 
spec.  And, if you add some optional extensions, you can get an even 
smoother setup for async I/O.

Finally, I'm open to trying to define the 'sleep/wake' facilities as 
"standard options" in WSGI, as well as clarifying the middleware control 
flow to support this better.


>The server or gateway, by calling next(), is assuming that the call
>will yield a string value, and only a string value.

The spec doesn't rule out empty strings, however, which would be the 
natural way to indicate that no data is available.  So, the protocol in an 
async app's iterator would be something like:

      while queue.empty():
          if 'async.wake' in environ:
              someDeferred.addCallback(environ['async.wake'])
              environ['async.sleep']()
              yield ""
              # We should only get to this line once environ['async.wake'] 
has been called
          else:
              yield ""
              # delay an exponentially increasing period if queue is still 
empty

If middleware is required to match the control flow of the application it 
wraps (e.g. write()=>write(), yield=>yield), then this would result in 
complete non-blockingness when the server supports the 'async' extensions.

Of course, a blocking delay *is* required when running in a server that 
doesn't support the async extensions, but that's unavoidable in that 
case.  (Technically, you might be better off just doing synchronous I/O if 
you're being run in a synchronous server, but that's of course optional.)


>"""The application object must return an iterable yielding strings or
>objects implementing the following interface:
>
>def addCallback(callable):
>         '''Add 'callable' to the list of callables to be invoked when a 
> string
>         is available. Callable should take a single argument, which will 
> be a
>string.'''
>
>The application object must invoke the callable passed to addCallback,
>passing a string which will be written to the request.
>"""
>
>This places additional burdens upon implementors of WSGI servers or
>gateways.

And a near-intolerable burden on middleware, which would have to have a way 
to "pass through" this facility.  It would be much better to limit the 
pass-through requirements to covering write and yield, rather than 
requiring middleware to implement addCallback facilities as well.


>Finally, we come to the task of implementing a server or gateway which
>can asynchronously support either asynchronous or blocking
>applications. Since there is no way for the server or gateway to know
>whether the application object it is about to invoke will block,
>starving the main loop and preventing network activity from being
>serviced, it must invoke all applications in a new thread or process.

But *some* thread is going to be working on it, and this is true whether 
you use a thread pool or the server is purely synchronous.  And, because a 
WSGI server *must* support synchronous applications, it *must* have some 
thread available that is amenable to blocking.

Of course "new" threads are not required.  I assume that in the case of 
Twisted, something like reactor.deferToThread() will be used to wrap a WSGI 
application's initial invocation, and each individual 'next()' call.