<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#330033" bgcolor="#ffffff">
On 11/20/2010 10:19 AM, Glenn Linderman wrote:
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
Oops. Yes, that fixes the problem with creation of the temp file,
thanks for catching that. I now get a complete report of the
original error in the temp file (below). I am a bit less confused
now... but it seems that there are still a number of issues. Here
is an enumeration of problems I was hard pressed to make before
you removed my confusion on this issue.<br>
</blockquote>
<br>
Related issues, regarding binary stream requirements for cgi
interface. Perhaps the cgi module should have the API to set binary
mode.<br>
<br>
<a href="http://bugs.python.org/issue1610654">http://bugs.python.org/issue1610654</a><br>
<a href="http://bugs.python.org/issue8077">http://bugs.python.org/issue8077</a><br>
<a href="http://bugs.python.org/issue4953">http://bugs.python.org/issue4953</a><br>
<br>
Sadly, cgi.py input handling seems to depend on the email module,
thought to be fixed for 3.2, but it is not clear if that has been
achieved, or if the surrogate encode workaround is sufficient for
this. More testing needed, but I don't have such a test case
developed yet.<br>
<br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite"> 1.
cgitb should expect to report to a binary stdout, using whatever
encoding (possibly ASCII) that seems appropriate for the output
that in generates.<br>
</blockquote>
<br>
Maybe cgi.py should have an API to set the stdin and stdout to
binary streams. Although cgi.py deals more with stdin than stdout,
cgitb deals more with stdout.<br>
<br>
Created <a href="http://bugs.python.org/issue10479">http://bugs.python.org/issue10479</a><br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite"> <br>
2. Some appropriate documentation or API or both should be
provided to enable a script to set "binary" mode for stdout for
CGI scripts. <a moz-do-not-send="true"
href="http://www.eggheadcafe.com/software/aspnet/36023550/cgi-python-3-write-raw-binary.aspxhttp://www.eggheadcafe.com/software/aspnet/36023550/cgi-python-3-write-raw-binary.aspx">This
link</a> demonstrates the confusion (wish I had found it
earlier) that is encountered by such lack. One must tell msvcrt
the stream is binary (I had figured that out early on), one must
also sidestep the use of the cp1252 default when printing binary,
one must also choose a proper text encoding corresponding to the
HTTP headers sent. My second email in this thread, sent a few
hours after the first, shows a convenient set of cures for all but
msvcrt (as long as only "write" is used for writing. "print"
support could be added, similarly). Likely something along this
line is needed for stdin as well, I haven't yet experimented with
uploading binary content to a CGI.<br>
<br>
One could speculate about having the Python runtime auto-detect
CGI mode, but I don't know of any foolproof technique for that,
and the selection of the "proper" text encoding depends on the
details of the CGI, so having instead an API or two that assists
with doing this sort of thing would be better; the need for
documentation, at least, seems imperative.<br>
</blockquote>
<br>
Created <a href="http://bugs.python.org/issue10480">http://bugs.python.org/issue10480</a><br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite"> <br>
3. subprocess documentation could be improved to point out that
when using subprocess.PIPE to talk to a Python subprocess, that
the communications will be in binary. Again, I don't know of any
way to autodetect the subprocess environment, but if it were
possible to select an appropriate encoding and use it consistently
on both sides of the PIPE, that would be a convenience to its use;
if not possible, documenting the issue, and providing an API to
use to easily select such encodings both in client and server,
would be helpful.<br>
<br>
While the layers are all there, and ".buffer" is documented for
TextIOWrapper, the use of sys.stdout.buffer and the fact that it
has a full set of operations isn't immediately obvious from the
reference material; perhaps it is in a tutorial I haven't found,
but... I was looking, and didn't find it.<br>
<br>
Of course, subprocess may launch non-Python programs; they will
have their own ideas of binary vs text encoding, so it is
important that it is convenient to match them on the Python side.<br>
<br>
It would be nice if subprocess had a mechanism for providing
no-deadlock stdout data to the parent prior to the child
terminating. A CGI implementation via subprocess shouldn't
accumulate all of stdout (or all of stderr, for that matter,
although less important). I don't (yet) know enough about Python
threading to know if this is possible, but it certainly would be
useful.<br>
</blockquote>
<br>
<a href="http://bugs.python.org/issue1048">http://bugs.python.org/issue1048</a>
for subprocess to document that communicate produces byte stream
output.<br>
<br>
<a href="http://bugs.python.org/issue10482">http://bugs.python.org/issue10482</a>
for subprocess enhancements to handle more cases without deadlock.<br>
<br>
Found <a href="http://bugs.python.org/issue4571">http://bugs.python.org/issue4571</a>
which documents how to switch stdin/stdout/stderr to binary mode,
and even back! I couldn't track the documented change to the
actual documentation, though, but I did find it in section 26.1,
under the documentation for the three stdio streams:<br>
<pre><span class="k">def</span> <span class="nf">make_streams_binary</span><span class="p">():</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdin</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span>
</pre>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite"> 4.
http.server has a number of bugs and limitations.<br>
4a. _url_collapse_path_split seems inefficient (although I have to
benchmark it against what I think would be more efficient), and
for its only use within http.server it produces the wrong
information, so the information has to be recombined and resplit
to make it function properly, adding to the perception of
inefficiency.<br>
4b. Detection of "executable" on Windows is simply wrong. Unix
execution bits do not exist.<br>
</blockquote>
<br>
<a href="http://bugs.python.org/issue10483">http://bugs.python.org/issue10483</a>
for 4b.<br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite">
4c. is_cgi doesn't properly handle PATHINFO parts of the path,
this is the other half of 4a. The Python2.x CGIHTTPServer.py had
this right, but the introduction and use of
_url_collapse_path_split broke it.<br>
</blockquote>
<br>
<a href="http://bugs.python.org/issue10484">http://bugs.python.org/issue10484</a>
for 4a and 4c.<br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite">
4d. Searching for a ? to find an explicit query string should use
.find('?') rather than .rfind('?') as there is no prohibition on
using '?' within a query string, AFAIK.<br>
</blockquote>
<br>
<a href="http://bugs.python.org/issue10485">http://bugs.python.org/issue10485</a>
for 4d.<br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite">
4e. doesn't set the REQUEST_URI, HTTP_HOST, or HTTP_PORT
environment variables for the CGI.<br>
</blockquote>
<br>
<a href="http://bugs.python.org/issue10486">http://bugs.python.org/issue10486</a>
for 4e.<br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite">
4f. Should not send the 200 response until it sees if the CGI
sends a Status: header.<br>
</blockquote>
<br>
<a href="http://bugs.python.org/issue10487">http://bugs.python.org/issue10487</a>
for 4f and 4g.<br>
<br>
<blockquote cite="mid:4CE8111F.9060502@g.nevcal.com" type="cite">
4g. Should not buffer all of stdout: subprocess.communicate is
inappropriate for a web server CGI interface. The data should
stream through to avoid consuming inordinate amounts of memory.
The only solution within the current limitations of subprocess is
to abandon stderr, force the CGI to do its own error logging, and
use shutil.copyfileobj to hook up p.stdout to self.wfile once the
Status: message processing has happened.<br>
4h. Doesn't seem to close p.stdin (I'm not sure if that is
necessary, it may happen when p is garbage collected, but effort
was made to close p.stdout and p.stderr, which seem similar.)<br>
</blockquote>
<br>
Discovered that subprocess.communicate closes p.stdin, so it wasn't
needed until I quit using .communicate in my version of the code.<br>
</body>
</html>