[Web-SIG] ngx.poll extension (was Re: Are you going to convert Pylons code into Python 3000?)

[Manlio Perillo]

Brian Smith ha scritto:
> Manlio Perillo wrote:
>> Fine with me but there is a *big* problem.
>>
>> WSGI 2.0 "breaks" support for asynchronous applications 
>> (since you can no more send headers in the app iter).
> 
> WSGI 1.0 doesn't guarentee that all asynchronous applications will work
> either, because it allows the WSGI gateway to wait for and buffer all
> the input from the client before even calling the application callable.
> And, it doesn't provide a way to read an indefinite stream of input from
> the client, which is also problematic.
> 
> Anyway, please post a small example of a program that fails to work
> because of these proposed changes for WSGI 2.0.
> 
> Thanks,
> Brian
> 


Attached there are two working examples (I have not committed it yet,
because I'm still testing - there are some problems that I need to solve).


The `curl_client` module is an high level interface to pycurl.

The `nginx-poll-proxy.py` script is an asyncronous WSGI application that
implements an HTTP proxy.

The `nginx-poll-sleep.py` script is a simple asynchronous WSGI
application that get the content of an HTTP resource using poll just to
"sleep" (suspend execution) for a fixed amount of time.


NOTE: I have also added a `ngx.sleep` extension, but I'm going to remove
it since the same behaviour can be obtained with ngx.poll.


An explanation of the interfaces
--------------------------------

The ngx.poll extension is based on the Python stdlib select.poll interface.

There are two constants: `ngx.WSGI_POLLIN` and `ngx.WSGI_POLLOUT`.
These are defined in the WSGI environment, but their value is "know"
(`0x01` and `0x04`) and can be used for bit masking.

The `ngx.connection_wrapper(fd)` function takes as input a file
descriptor (as integer) and returns a Connection wrapper object, to be
used for later operations.


The Connection wrapper object has the following methods:
- fileno():
      return the associated socket descriptor
- register(flags):
      register the connection with the server "reactor";
      flags is a bit mask of ngx.WSGI_POLLIN and ngx.WSGI_POLLOUT
- deregister(flags=None):
      deregister the connection from the server "reactor"
- close:
      close the connection object, deregisterering it from the server
      "reactor" if still active.
      XXX it also can close the socket, but this should be done by the
          client

The last function is `ngx.poll(timeout)`.
When called, the user *should* yield an empty string (yielding a non
empty string will result in an "undefined behaviour").

The WSGI application iteration will be suspended until a connection is
ready for reading or writing, or the timeout expires.

The `ngx.poll` function returns a callable that, when called, returns a
tuple with the connection object "ready" (or None if timedout) and a
flag indicating if the connection is ready for reading or writing.

NOTE: due to the internal architecture of the Nginx event module (it
        have to support several different event systems), mod_wsgi for
        Nginx will only return ngx.WSGI_POLLIN or ngx.WSGI_POLLPUT,
        *never* ngx.WSGI_POLLIN | ngx.WSGI_POLLPUT.

        Also, no error status is reported.



That's all.

An asynchronous application is simply impossible to develope with the
current draft of WSGI 2.0, since I need to send the headers after some
steps in the application iterator.


So, please, don't "ruin" the WSGI specification just to make it more
easy to implement and to use.
For me asynchronous support is very important.


P.S: I have chosen to implement this interface, instead of
       `wsgi.pause_output`, because IMHO it is very easy to implement for
       "normal" servers.

       Moreover it is also more simple to use, with a very "natural"
       interface, and it avoids the use of callbacks and a more strict
       interaction with the server "reactor".


Regards  Manlio Perillo

-------------- next part --------------
A non-text attachment was scrubbed...
Name: curl_client.py
Type: text/x-python
Size: 4582 bytes
Desc: not available
Url : http://mail.python.org/pipermail/web-sig/attachments/20080305/311b4450/attachment.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx-poll-proxy.py
Type: text/x-python
Size: 1410 bytes
Desc: not available
Url : http://mail.python.org/pipermail/web-sig/attachments/20080305/311b4450/attachment-0001.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx-poll-sleep.py
Type: text/x-python
Size: 1628 bytes
Desc: not available
Url : http://mail.python.org/pipermail/web-sig/attachments/20080305/311b4450/attachment-0002.py