[Python-bugs-list] [ python-Bugs-427345 ] CGIHTTPServer fix for Windows

[noreply@sourceforge.net]

Bugs item #427345, was opened at 2001-05-25 13:37
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=427345&group_id=5470

Category: Python Library
Group: Platform-specific
Status: Open
Resolution: None
>Priority: 5
Submitted By: Kevin Altis (kasplat)
>Assigned to: Steve Holden (holdenweb)
Summary: CGIHTTPServer fix for Windows

Initial Comment:
CGIHTTPServer.py in the Python 2.1 library needs two 
changes to the popen2 section in order to support 
binary data and to avoid buffering problems under 
Windows. The complete code block is shown at the end 
of this message. The two changed lines are:
cmdline = "%s -u %s" % (interp, cmdline)
fi, fo = os.popen2(cmdline, 'b')

I've tested this under Windows 2000 Pro and binary 
file uploads now work, however more extensive tests 
should probably be done.

It appears there is another socket-related problem 
and/or end-of-line conversion error that occurs when 
using Internet Explorer 5.x and BASEHTTPServer with 
CGIHTTPServer on the same machine. I'm still 
investigating that issue.

Finally, I have done a small rewrite of 
CGIHTTPServer.py to simplify subclassing it to support 
running CGIs in any or all directories and using other 
file extensions such as .cgi. The maintainer for the 
current file should email me about the changes to see 
whether they want to update the official library file. 
I didn't post it here since it needs more tweaking

ka
---
CGIHTTPServer.p fixes...
elif self.have_popen2:
    # Windows -- use popen2 to create a subprocess
    import shutil
    os.environ.update(env)
    cmdline = scriptfile
    if self.is_python(scriptfile):
        interp = sys.executable
        if interp.lower().endswith("w.exe"):
            # On Windows, use python.exe, not 
python.exe
            interp = interp[:-5] = interp[-4:]
        cmdline = "%s -u %s" % (interp, cmdline)
    if '=' not in query and '"' not in query:
        cmdline = '%s "%s"' % (cmdline, query)
    self.log_error("command: %s", cmdline)
    try:
        nbytes = int(length)
    except:
        nbytes = 0
    fi, fo = os.popen2(cmdline, 'b')
    if self.command.lower() == "post" and nbytes > 0:
        data = self.rfile.read(nbytes)
        fi.write(data)
    fi.close()
    shutil.copyfileobj(fo, self.wfile)
    sts = fo.close()
    if sts:
        self.log_error("CGI script exit status %#x", 
sts)
    else:
        self.log_error("CGI script exited OK")


----------------------------------------------------------------------

>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-08-05 11:00

Message:
Logged In: YES 
user_id=6380

Like 430160, I'm assigning this to Steve Holden who may
check in a fix. Good luck!

----------------------------------------------------------------------

Comment By: Kevin Altis (kasplat)
Date: 2002-08-05 00:07

Message:
Logged In: YES 
user_id=228025

I have a possible fix, at least when the server is running on 
Windows. As suggested previously, the key is to read the 
bytes beyond the content-length, but without blocking. Steve 
Holden suggested I use select, which seems simpler than 
another solution which is to make the socket non-blocking, 
then attempting to read 2 bytes of data within a try/except 
block.

The solution below just loops with select.select

            if self.command.lower() == "post" and nbytes > 0:
                data = self.rfile.read(nbytes)
                fi.write(data)
                # now throw away data past Content-length
                while select.select([self.rfile._sock], [], [], 0)[0] != 
[]:
                    waste = self.rfile._sock.recv(1)
            fi.close()

This is slightly verbose. The two additional lines go below the 
fi.write statement on line 247 of CGIHTTPServer.py, line 248 if 
you count the import select statement that needs to be 
added at the top of the file.

I have only tested this on Windows since the server code is 
Windows-specific. I do not know if a server running on Unix 
suffers this same problem. I verified that POST works 
correctly from the following browsers on Windows 2000: IE 
5.5, Opera 6.0.4, and Netscape 4.7. Only IE sends the extra 
CR/LF, Opera and Netscape do not.

ka

----------------------------------------------------------------------

Comment By: Kevin Altis (kasplat)
Date: 2002-08-03 13:37

Message:
Logged In: YES 
user_id=228025

This bug is related to another bug

https://sourceforge.net/tracker/?
func=detail&aid=430160&group_id=5470&atid=105470

Bob Denny (author of the WebSite server, formerly sold by 
O'Reilly) had this to say about the POST problem:

"This is an old problem, traced back to Netscape and Unix-
based shell CGI scripts. Netscape browsers did this to force 
completion of the POST into shell scripts. The cr/lf is not part 
of the POST payload, and the CGI script that is dealing with 
the POST needs to toss it away. The general solution is, read 
Content-Length bytes, then reas anything else that may be in 
the buffer and junk it. IE is emulating the behavior of 
Netscape."

So, that explains why IE (and possibly other browsers) sticks 
a CR/LF on the end of a POST causing a mismatch with the 
Content-length header. Now the question is where a fix can 
be inserted in the libs?

Looking again at CGIHTTPServer.py, perhaps this problem is 
isolated to Windows, in which case the fix might be in the 
block of code starting on line 218, and more specifically, line 
246

data = self.rfile.read(nbytes)

so at that point, the remaining 2 bytes should be read, but 
tossed. I'll investigate. Steve Holden and I have been emailing 
back and forth on this issue, so if anyone else has other 
suggestions, post here or email one of us.

Thanks,

ka

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2002-07-08 21:27

Message:
Logged In: NO 

hello

----------------------------------------------------------------------

Comment By: Thomas Justin Shaw (justinshaw)
Date: 2002-03-31 03:48

Message:
Logged In: YES 
user_id=135558

To: Matthew King
Thanks for reposting that solution.  That did seem to fix 
the problem.

To: Mark Lutz
Using: Win2000 Pro IE5 python2.2.1c locally
The bug shows up as a popup message but the page loads.
When I tried it accross a lan the page wouldn't even
load (blowing a little demo).  So I guess it is not ONLY a 
local problem.

Thanks for posting.
Justin Shaw

----------------------------------------------------------------------

Comment By: Thomas Justin Shaw (justinshaw)
Date: 2002-03-29 02:36

Message:
Logged In: YES 
user_id=135558

I'm experiencing a problem on windows.
When I execute the scripts as 
'http://localhost:8080/cgi-bin\junk.py' 
the page seems to load ok but I get the popup message:
Internet Explorer cannot open the Internet site 
http://localhost:8080/cgi-bin\junk.py
The connection with the server ws reset

On Netscape no errors but the download also takes forever. 
The slow download seems to be caused by the 
FieldStorage() call.


----------------------------------------------------------------------

Comment By: Matthew King (kyrrigle)
Date: 2002-03-28 16:21

Message:
Logged In: YES 
user_id=247536

this looks like the same issue as #430160.  and here's what 
I just wrote there...

it appears that IE is sending 2 extra bytes ('\r\n') after 
the request data.  and if you don't read those two extra 
bytes off, the window's socket handling gets messed up.

the result is that a partial response is returned and the 
socket closed.  IE tries to recover by re-POST'ing (which 
is behavior specified in the HTTP/1.1 RFC)... only they 
seem to add an embedded NULL the second time through, and 
the original socket problem happens again anyway.

Try reading an extra 2 bytes from the rfile before sending 
your response and the problem should go away.  (you can do 
that by 'self.rfile._rbufsize = content_length + 2' inside 
your do_POST method before reading the content)

not sure what the real fix for this should be?


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-05 11:40

Message:
Logged In: YES 
user_id=6380

Load shedding.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-09 09:14

Message:
Logged In: YES 
user_id=6380

Reopening -- not that I have time for this. :-(

Note that that patch is in socket.py.

If this really is a work-around, I'd like to understand why.

----------------------------------------------------------------------

Comment By: Steve Pike (oztourer)
Date: 2001-08-09 08:27

Message:
Logged In: YES 
user_id=120658

To further elaborate on the problems with POST on Windows 
95: without having reached any real understanding of the 
problem I have found a tolerable workaround. By modifying 
the default _rbufsize for class _fileobject in socket.py I 
can get any POSTs with text length less than _rbufsize to 
work. Here is the mod:

<pre>
class _fileobject:
    def __init__(self, sock, mode, bufsize):
        self._sock = sock
        self._mode = mode
        if bufsize < 0:
            bufsize = 512
        # SP 9 Aug 2001: default _rbufsize is too small, 
crashing CGIHTTPServer on POST's
        # This fix still only allows pages of less than 
given buffer size to be updated,
        # so the real fix has still to be discovered.
        #self._rbufsize = max(1, bufsize)
        self._rbufsize = max(16*1024, bufsize)
        self._wbufsize = bufsize
        self._wbuf = self._rbuf = ""
</pre>

-- StevePike

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-07 15:59

Message:
Logged In: YES 
user_id=6380

I agree that this is a laudable goal, but I don't have the
time to spend to work on this.  Please upload a patch to the
SF patch manager and assign it to me and I'll look at it.

In the mean time, I've applied the two small suggestions and
will close this bug report.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-05-29 12:29

Message:
Logged In: NO 

I want to elaborate on the second-to-last 
paragraph in this report.  CGIHTTPServer 
in 2.1 and 2.0 is apparently broken for 
POST requests (only) to CGI scripts, when
running locally on Windows with an IE client.

The details: there is a problem with 
the combination of a CGIHTTPServer and 
Intenernet Explorer both running locally 
on Windows (with server name "localhost").
In this setup, POST requests to CGI scripts
fail, but GET requests to the exact same 
script work fine.  That is, a form with 
method=GET, or a URL with an appended 
query string, both work ok.

In POST mode, the CGI script gets proper
data in the input stream, and generates 
a correct reply stream (which is in fact 
identical to the output generated for 
quivalent GET requests).  So, somewhere 
between CGIHTTPServer and IE, the data 
seems to be dropped or munged.  The same
thing happens of you force CGIHTTPServer 
to use execfile() to launch the script, 
instead of os.popen2.

I've also observed that endline conventions 
seem to be irrelevant to the problem; using 
\n or \r\n makes no difference in both the 
input and reply streams.  I've also been 
told that the problem may not exist in 
Netscape clients.

Since CGIHTTPServer is a very nice way to test
CGI scripts (all you need is a standard Python
install), this seems important.  It would also 
be nice if the directory assumptions in that 
module were more customizable, but that's just
a wish.

--Mark Lutz


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=427345&group_id=5470