<div dir="ltr">That's reasonable. This isn't an issue on Python 2.x because everything is handled as bytes (str on 2.x). It looks like cgi.FieldStorage() only got Unicode support in Python 3.x at all fairly "late" in the game, for 3.2 or 3.3. It was in that commit (<a href="https://github.com/python/cpython/commit/5c23b8e6ea7cdb0002842d16dbce4b4d716dd35a" target="_blank">https://github.com/python/cpy<wbr>thon/commit/5c23b8e6ea7cdb0002<wbr>842d16dbce4b4d716dd35a</a> by Victor Stinner) where the check for "it's a binary file if it has the 'filename' parameter" was added. It wasn't an issue before that because Unicode wasn't really handled.<div><br></div><div>It seems to me this could be added to FieldStorage pretty easily in a backwards-compatible way, by one of the following:</div><div><br></div><div>1) Adding a file_fields keyword argument to FieldStorage.__init__ which would allow you to specify that "these fields are definitely files, please treat them as binary". So in the case of my example on the SO question, you'd do FieldStorage(..., file_fields=('payload',))</div><div><br></div><div>2) Add a .binary_file (or .file_binary) property to FieldStorage which would be like .file but for sure give you a binary file.</div><div><br></div><div>3) Allowing you to specify encoding=None, which would mean don't decode fields to str, return everything as bytes. This might not be so nice, as then other non-file fields would be returned as bytes too, and the caller would have to decode those manually.</div><div><br></div><div>Any thoughts on which of #1-3 might be best?</div><div><br></div><div>I also realized that to work around this, instead of my fairly complicated content-disposition headers fix, I can subclass FieldStorage and just override "filename" with a property, so that FieldStorage.__init__ thinks that field does have a filename:</div><div><br></div><div><div> class MyFieldStorage(cgi.FieldStorage):</div><div> @property</div><div> def filename(self):</div><div> return 'file_name' if <a href="http://self.name">self.name</a> == 'payload' else self._filename</div><div><br></div><div> @filename.setter</div><div> def filename(self, value):</div><div> self._filename = value</div></div><div><br></div><div>Also, on StackOverflow someone responded that you can also work around this by passing errors='surrogateescape' and then file_input.read().encode('utf-8', 'surrogateescape') ... pretty hacky, but at least you can get the bytes back.</div><div><br></div><div><div>-Ben</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 15, 2017 at 12:35 PM, Brett Cannon <span dir="ltr"><<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span class=""><div dir="ltr">On Wed, 15 Feb 2017 at 08:14 Ben Hoyt <<a href="mailto:benhoyt@gmail.com" target="_blank">benhoyt@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_8399943805501588802gmail_msg">I posted this on StackOverflow [1], but I'm posting it here as well, as I believe this is a bug (or at least quirk) in cgi.FieldStorage where you can't access a file upload properly if "filename=" is not present in the MIME part's Content-Disposition header. There are a couple of related bugs open (and closed) on bugs.python.ord, but not quite this issue.<div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Is it legitimate for cgi.FieldStorage to use the presence of "filename=" to determine "this is a binary file" (in which case this is not a bug and my client is just buggy), or is this a bug? I lean towards the latter as the spec indicates that the filename is optional [2].</div></div></blockquote><div><br></div></span><div>Assuming this isn't a recent change in semantics I would say this is now a quick considering how old the module is and people probably rely on its current semantics.</div><div><br></div><div>-Brett</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr" class="m_8399943805501588802gmail_msg"><div class="m_8399943805501588802gmail_msg"><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Copying from my StackOverflow question, including a test/repro case:<div class="m_8399943805501588802gmail_msg"><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"><div class="m_8399943805501588802gmail_msg">When I use `cgi.FieldStorage` to parse a `multipart/form-data` request (or any web framework like Pyramid which uses `cgi.FieldStorage`) I have trouble processing file uploads from certain clients which don't provide a `filename=file.ext` in the part's `Content-Disposition` header.</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">If the `filename=` option is missing, `FieldStorage()` tries to decode the contents of the file as UTF-8 and return a string. And obviously many files are binary and not UTF-8 and as such give bogus results.</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">For example:</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> >>> import cgi</div><div class="m_8399943805501588802gmail_msg"> >>> import io</div><div class="m_8399943805501588802gmail_msg"> >>> body = (b'--KQNTvuH-itP09uVKjjZiegh7\<wbr>r\n' +</div><div class="m_8399943805501588802gmail_msg"> ... b'Content-Disposition: form-data; name=payload\r\n\r\n' +</div><div class="m_8399943805501588802gmail_msg"> ... b'\xff\xd8\xff\xe0\x00\<wbr>x10JFIF')</div><div class="m_8399943805501588802gmail_msg"> >>> env = {</div><div class="m_8399943805501588802gmail_msg"> ... 'REQUEST_METHOD': 'POST',</div><div class="m_8399943805501588802gmail_msg"> ... 'CONTENT_TYPE': 'multipart/form-data; boundary=KQNTvuH-<wbr>itP09uVKjjZiegh7',</div><div class="m_8399943805501588802gmail_msg"> ... 'CONTENT_LENGTH': len(body),</div><div class="m_8399943805501588802gmail_msg"> ... }</div><div class="m_8399943805501588802gmail_msg"> >>> fs = cgi.FieldStorage(fp=io.<wbr>BytesIO(body), environ=env)</div><div class="m_8399943805501588802gmail_msg"> >>> (fs['payload'].filename, fs['payload'].file.read())</div><div class="m_8399943805501588802gmail_msg"> (None, '����\x00\x10JFIF')</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Browsers, and *most* HTTP libraries do include the `filename=` option for file uploads, but I'm currently dealing with a client that doesn't (and omitting the `filename` does seem to be valid according to the spec).</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Currently I'm using a pretty hacky workaround by subclassing `FieldStorage` and replacing the relevant `Content-Disposition` header with one that does have the filename:</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> import cgi</div><div class="m_8399943805501588802gmail_msg"> import os</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> class FileFieldStorage(cgi.<wbr>FieldStorage):</div><div class="m_8399943805501588802gmail_msg"> """To use, subclass FileFieldStorage and override _file_fields with a tuple</div><div class="m_8399943805501588802gmail_msg"> of the names of the file field(s). You can also override _file_name with</div><div class="m_8399943805501588802gmail_msg"> the filename to add.</div><div class="m_8399943805501588802gmail_msg"> """</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> _file_fields = ()</div><div class="m_8399943805501588802gmail_msg"> _file_name = 'file_name'</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> def __init__(self, fp=None, headers=None, outerboundary=b'',</div><div class="m_8399943805501588802gmail_msg"> environ=os.environ, keep_blank_values=0, strict_parsing=0,</div><div class="m_8399943805501588802gmail_msg"> limit=None, encoding='utf-8', errors='replace'):</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> if self._file_fields and headers and headers.get('content-<wbr>disposition'):</div><div class="m_8399943805501588802gmail_msg"> content_disposition = headers['content-disposition']</div><div class="m_8399943805501588802gmail_msg"> key, pdict = cgi.parse_header(content_<wbr>disposition)</div><div class="m_8399943805501588802gmail_msg"> if (key == 'form-data' and pdict.get('name') in self._file_fields and</div><div class="m_8399943805501588802gmail_msg"> 'filename' not in pdict):</div><div class="m_8399943805501588802gmail_msg"> del headers['content-disposition']</div><div class="m_8399943805501588802gmail_msg"> quoted_file_name = self._file_name.replace('"', '\\"')</div><div class="m_8399943805501588802gmail_msg"> headers['content-disposition'] = '{}; filename="{}"'.format(</div><div class="m_8399943805501588802gmail_msg"> content_disposition, quoted_file_name)</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> super().__init__(fp=fp, headers=headers, outerboundary=outerboundary,</div><div class="m_8399943805501588802gmail_msg"> environ=environ, keep_blank_values=keep_blank_<wbr>values,</div><div class="m_8399943805501588802gmail_msg"> strict_parsing=strict_<wbr>parsing, limit=limit,</div><div class="m_8399943805501588802gmail_msg"> encoding=encoding, errors=errors)</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Using the `body` and `env` in my first test, this works now:</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"> >>> class TestFieldStorage(<wbr>FileFieldStorage):</div><div class="m_8399943805501588802gmail_msg"> ... _file_fields = ('payload',)</div><div class="m_8399943805501588802gmail_msg"> >>> fs = TestFieldStorage(fp=io.<wbr>BytesIO(body), environ=env)</div><div class="m_8399943805501588802gmail_msg"> >>> (fs['payload'].filename, fs['payload'].file.read())</div><div class="m_8399943805501588802gmail_msg"> ('file_name', b'\xff\xd8\xff\xe0\x00\<wbr>x10JFIF')</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Is there some way to avoid this hack and tell `FieldStorage` not to decode as UTF-8? It would be nice if you could provide `encoding=None` or something, but it doesn't look like it supports that.</div></div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">Thanks,</div><div class="m_8399943805501588802gmail_msg">Ben.</div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div><div class="m_8399943805501588802gmail_msg">[1] <a href="https://stackoverflow.com/questions/42213318/cgi-fieldstorage-with-multipart-form-data-tries-to-decode-binary-file-as-utf-8-e" class="m_8399943805501588802gmail_msg" target="_blank">https://stackoverflow.com/<wbr>questions/42213318/cgi-<wbr>fieldstorage-with-multipart-<wbr>form-data-tries-to-decode-<wbr>binary-file-as-utf-8-e</a></div><div class="m_8399943805501588802gmail_msg">[2] <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1" class="m_8399943805501588802gmail_msg" target="_blank">https://www.w3.org/<wbr>Protocols/rfc2616/rfc2616-<wbr>sec19.html#sec19.5.1</a></div><div class="m_8399943805501588802gmail_msg"><br class="m_8399943805501588802gmail_msg"></div></div></div></div></div></div></div>
______________________________<wbr>_________________<br class="m_8399943805501588802gmail_msg">
Python-Dev mailing list<br class="m_8399943805501588802gmail_msg">
<a href="mailto:Python-Dev@python.org" class="m_8399943805501588802gmail_msg" target="_blank">Python-Dev@python.org</a><br class="m_8399943805501588802gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer" class="m_8399943805501588802gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-dev</a><br class="m_8399943805501588802gmail_msg">
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/brett%40python.org" rel="noreferrer" class="m_8399943805501588802gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/options/python-dev/<wbr>brett%40python.org</a><br class="m_8399943805501588802gmail_msg">
</blockquote></div></div>
</blockquote></div><br></div>