[Web-SIG] py3k, cgi, email, and form-data

[Robert Brewer]

There's a major change in functionality in the cgi module between Python
2 and Python 3 which I've just run across: the behavior of
FieldStorage.read_multi, specifically when an HTTP app accepts a file
upload within a multipart/form-data payload.

In Python 2, each part would be read in sequence within its own
FieldStorage instance. This allowed file uploads to be shunted to a
TemporaryFile (via make_file) as needed:

    klass = self.FieldStorageClass or self.__class__
    part = klass(self.fp, {}, ib,
                 environ, keep_blank_values, strict_parsing)
    # Throw first part away
    while not part.done:
        headers = rfc822.Message(self.fp)
        part = klass(self.fp, headers, ib,
                     environ, keep_blank_values, strict_parsing)
        self.list.append(part)

In Python 3 (svn revision 72466), the whole request body is read into
memory first via fp.read(), and then broken into separate parts in a
second step:

    klass = self.FieldStorageClass or self.__class__
    parser = email.parser.FeedParser()
    # Create bogus content-type header for proper multipart parsing
    parser.feed('Content-Type: %s; boundary=%s\r\n\r\n' % (self.type, ib))
    parser.feed(self.fp.read())
    full_msg = parser.close()
    # Get subparts
    msgs = full_msg.get_payload()
    for msg in msgs:
        fp = StringIO(msg.get_payload())
        part = klass(fp, msg, ib, environ, keep_blank_values,
                     strict_parsing)
        self.list.append(part)

This makes the cgi module in Python 3 somewhat crippled for handling
multipart/form-data file uploads of any significant size (and since
the client is the one determining the size, opens a server up for an
unexpected Denial of Service vector).

I *think* the FeedParser is designed to accept incremental writes,
but I haven't yet found a way to do any kind of incremental reads
from it in order to shunt the fp.read out to a tempfile again.
I'm secretly hoping Barry has a one-liner fix for this. ;)


Robert Brewer
fumanchu at aminus.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20090511/0a75edc6/attachment.htm>