[ python-Bugs-792570 ] SimpleXMLRPCServer cannot handle large requests

[SourceForge.net]

Bugs item #792570, was opened at 2003-08-21 11:37
Message generated for change (Settings changed) made by akuchling
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=792570&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
>Group: Python 2.5
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Marc-André Morissette (morissette)
>Assigned to: A.M. Kuchling (akuchling)
Summary: SimpleXMLRPCServer cannot handle large requests

Initial Comment:
SimpleXMLRPCServer throws a WSAEINTR ioerror on 
large XML-RPC requests.

Under Windows, the socket.read() method cannot seem 
to handle large (tens of megabytes) reads (could be a 
Python specific problem). This means that very large 
XML-RPC requests can cause the exception.

Here is a tentative patch against 2.2.3 to fix the 
problem. It should be easy to port it to 2.3

---
 /cygdrive/c/Python22/Lib/SimpleXMLRPCServer.py      
2003-07-09 14:16:52.000000000 -0400
+++ /cygdrive/z/SimpleXMLRPCServer.py   2003-08-21 
11:01:19.000000000 -0400
@@ -73,6 +73,8 @@
 import SocketServer
 import BaseHTTPServer
 import sys
+import cStringIO

 class SimpleXMLRPCRequestHandler
(BaseHTTPServer.BaseHTTPRequestHandler):
     """Simple XML-RPC request handler class.
@@ -95,7 +97,14 @@

         try:
             # get arguments
-            data = self.rfile.read(int(self.headers["content-
length"]))
+            max_chunk_size = 10000000
+            content_length = int(self.headers["content-
length"])
+            buffer = cStringIO.StringIO()
+            for offset in range(0, content_length, 
max_chunk_size):
+                chunk_size = min(content_length - offset, 
max_chunk_size)
+                buffer.write(self.rfile.read(chunk_size))
+            data = buffer.getvalue()
+            buffer.close()
             params, method = xmlrpclib.loads(data)

             # generate response



----------------------------------------------------------------------

>Comment By: A.M. Kuchling (akuchling)
Date: 2005-12-04 10:38

Message:
Logged In: YES 
user_id=11375

It turns out that on my Mac socket.read() runs into trouble
around 15Mb, so I could reproduce the problem.  Fix committed
in rev 41586; it's the same principle as the suggested
change, but the code is rewritten for better scalability
(appending to a list instead of using a cStringIO).  Thanks!


----------------------------------------------------------------------

Comment By: A.M. Kuchling (akuchling)
Date: 2005-12-04 10:16

Message:
Logged In: YES 
user_id=11375

Why does SimpleXMLRPCServer need to be fixed if
socket.read() is choking?  Shouldn't socket.read() be fixed
instead?

noplay: do you mean you're seeing this bug happen on Linux?
 I don't use Windows, so being able to reproduce the problem
on Linux would be very useful.


----------------------------------------------------------------------

Comment By: julien duponchelle (noplay)
Date: 2005-10-21 05:36

Message:
Logged In: YES 
user_id=446148

I have the same problem with Python2.4 with windows and
linux version.

If XML-RPC server reads to large buffer, it returns only a
part of the buffer.

----------------------------------------------------------------------

Comment By: Reinhold Birkenfeld (birkenfeld)
Date: 2005-06-01 08:26

Message:
Logged In: YES 
user_id=1188172

Marc-Andre, can you still reproduce this with Python 2.4?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=792570&group_id=5470