CGI File Uploads and Progress Bars

[Robert Brewer]

Doug Helm wrote:
> I'm writing a CGI to handle very large file uploads.
> I would like to include a progress bar.
> ...I need to know not only the number of
> bytes received, but also the total number of
> incoming bytes.  Here's the heart of the code:
> 
>     while afcommon.True:
>       lstrData = lobjIncomingFile.file.read(afcommon.OneMeg)
>       if not lstrData:
>         break
>       lobjFile.write(lstrData)
>       llngBytes += long(len(lstrData))
>     lobjFile.close()
> 
> Assume that lobjIncomingFile is actually a file-type
> element coming from CGI.FieldStorage.  It's already
> been tested to ensure that it is a file-type element.
> Also, assume that I've already opened a file on the
> server, referred to by lobjFile (so lobjFile is the
> target of the incoming data).

I took a cursory look through the cgi module (and am trying to remember
what we did for CherryPy*). It seems that, at the time you run the above
code, the uploaded file has already been completely read from the client
and placed into a temporary file. That is, lobjIncomingFile.file.read
does not read from the HTTP request body; it reads from a temporary file
instead.

> If this were a client application opening a file,
> I would just do the following:
> 
> import os
> print os.stat('myfile.dat')[6]
> 
> But, of course, this isn't a local file.  In fact,
> it's not really a file at all.

In fact, it is a file, just a temporary one. See
cgi.FieldStorage.makefile().

> So, bottom line: Does anyone know how to get the
> size of the incoming file data without reading the
> whole thing into a string?  Can I do something with
> content_header?

Sure. Subclass cgi.FieldStorage, and override make_file to provide your
own file-like object that you can monitor as its "write" method is
called (see read_binary for the actual upload r/w code). The existing
FieldStorage class places the file size (gleaned from the Content-Length
request header) into self.length.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

* See CherryPy's