[Web-SIG] Use cases for file-like objects (was Re: Bill's comments on WSGI draft 1.4)

[Phillip J. Eby]

At 08:12 PM 9/7/04 +0100, Alan Kennedy wrote:
>[Phillip J. Eby]
> > Instead of using 'fileno' as an extension attribute on the iterable,
> > we'll add a 'wsgi.file_wrapper' key, usable as follows by an
> > application:
> >
> >     return environ['wsgi.file_wrapper'](something,blksize)
> >
> > The 'file_wrapper' may introspect "something" in order to do a
> > fileno() check, or other "I know how to send this kind of object
> > quickly" optimizations.  It must return an iterable, that the
> > application may return back to the server.
>
>[tony at lownds.com]
> > Here's the tail end of the CGI example.
> >
> >     result = application(environ, start_response)
> >     try:
> >         if hasattr(result, 'read'):
> >             result = iter(lambda: result.read(BLOCKSIZE), '')
> >         for data in result:
> >             write(data)
> >     finally:
> >         if hasattr(result,'close'):
> >             result.close()
>
>Since I am just about to implement "wsgi.file_wrapper", I just wanted to 
>check that my understanding of it is correct.

Before you implement it, I should warn you that I'm thinking 'file_wrapper' 
was a bad idea, and that there's a better way to do all this.

As I understand them, the current use cases for file-like objects are:

  1. sendfile(fileno()) for fast file-descriptor copying (Unix-like OSes 
only, and only single-thread synchronous servers like Apache 1.x or CGI)

  2. Convenience in returning an open file or pipe

  3. Convenience in returning a StringIO or other "file-like" object

By the way, as far as I know, none of these use cases are especially common 
in today's existing web frameworks.  Anyway, use cases 2 and 3 can be 
grouped into cases where the object is "large", "small", or "pipe-like":

     "Small" case:

        return [filelike.read()]

     "Large" case:

        return iter(lambda: filelike.read(SIZE), '')

     "Pipe-like" case:

        return iter(filelike.read, '')

These are all very simple, one-line solutions (at least for 2.2+) and have 
the advantage of being explicit, and refusing the temptation to guess.  The 
application is in total control of how the resource will be transmitted.

That leaves only use case 1, which is a fairly limited use case and isn't 
even applicable to most web servers written in Python, as most such servers 
are asynchronous and can't take advantage of the 'sendfile()' system call 
(which Python doesn't expose as an 'os' facility anyway).

Therefore, my current thinking is to relegate use case 1 to a WSGI 
extension, 'wsgi.fd_wrapper', which can used like this (if the application 
is returning an object with a working 'fileno()' method):

     if 'wsgi.fd_wrapper' in environ:
         return environ['wsgi.sendfile'](fd.fileno())
     else:
         # return a normal iterable

In other words, 'wsgi.fd_wrapper' would be sort of like my earlier 
'wsgi.file_wrapper', but it would be *optional* to implement and 
use.  (Meaning it can be relegated to an application note, instead of 
having to be introduced in-line.)

For Alan's attempt to support Jython 2.1, he could write an 'iter' function 
or class and put it in __builtin__, so that programs written to this idiom 
would still work.

After thinking about the 'file_wrapper' idea some more, I'm thinking that 
this way works better for everything but the issue of closing 
files.  However, my example 'file_wrapper' class should maybe be included 
in the PEP under an application note about sending files and file-like objects.