[Python-Dev] [Python-checkins] commit of r41586 - in python/trunk: Lib/SimpleXMLRPCServer.py Misc/NEWS

[M.-A. Lemburg]

Guido van Rossum wrote:
> Happened to see this commit. What's the magic about 10MB? Is there any
> understanding of what causes it to fail? What is the failure mode?
> Could it just be fragmentation causing the malloc or realloc to fail?
> Should we perhaps use a more conservative buffer size, e.g. 1MB or
> even 8K (the international standard for I/O buffering :-)?

Just as data point: I've been hitting problems much earlier than
with 10MB (unrelated to SimpleXMLRPCServer.py, this is experience
from doing plain socket communication). Even 65535 is too much for
some platforms (AIX at the time). Things got stable at around 64000
bytes.

> --Guido
> 
> On 12/4/05, andrew.kuchling <python-checkins at python.org> wrote:
> 
>>Author: andrew.kuchling
>>Date: Sun Dec  4 16:36:57 2005
>>New Revision: 41586
>>
>>Modified:
>>   python/trunk/Lib/SimpleXMLRPCServer.py
>>   python/trunk/Misc/NEWS
>>Log:
>>[Bug #792570] Under Windows, socket.read() seems to run into trouble when
>>asked to read tens of megabytes of data.  On my Mac, it hits MemoryErrors
>>when reading around 15Mb in one chunk.  The fix is to read the body in several
>>parts, not as one big piece.
>>
>>It would be nice to fix the underlying socket.read() problem, too.
>>
>>2.4 bugfix candidate.
>>
>>
>>Modified: python/trunk/Lib/SimpleXMLRPCServer.py
>>==============================================================================
>>--- python/trunk/Lib/SimpleXMLRPCServer.py      (original)
>>+++ python/trunk/Lib/SimpleXMLRPCServer.py      Sun Dec  4 16:36:57 2005
>>@@ -422,8 +422,19 @@
>>         """
>>
>>         try:
>>-            # get arguments
>>-            data = self.rfile.read(int(self.headers["content-length"]))
>>+            # Get arguments by reading body of request.
>>+            # We read this in chunks to avoid straining
>>+            # socket.read(); around the 10 or 15Mb mark, some platforms
>>+            # begin to have problems (bug #792570).
>>+            max_chunk_size = 10*1024*1024
>>+            size_remaining = int(self.headers["content-length"])
>>+            L = []
>>+            while size_remaining:
>>+                chunk_size = min(size_remaining, max_chunk_size)
>>+                L.append(self.rfile.read(chunk_size))
>>+                size_remaining -= len(L[-1])
>>+            data = ''.join(L)
>>+
>>             # In previous versions of SimpleXMLRPCServer, _dispatch
>>             # could be overridden in this class, instead of in
>>             # SimpleXMLRPCDispatcher. To maintain backwards compatibility,
>>
>>Modified: python/trunk/Misc/NEWS
>>==============================================================================
>>--- python/trunk/Misc/NEWS      (original)
>>+++ python/trunk/Misc/NEWS      Sun Dec  4 16:36:57 2005
>>@@ -451,6 +451,9 @@
>> - Bug #1222790: in SimpleXMLRPCServer, set the reuse-address and close-on-exec
>>   flags on the HTTP listening socket.
>>
>>+- Bug #792570: SimpleXMLRPCServer had problems if the request grew too large.
>>+  Fixed by reading the HTTP body in chunks instead of one big socket.read().
>>+
>> - Bug #1110478: Revert os.environ.update to do putenv again.
>>
>> - Bug #1103844: fix distutils.install.dump_dirs() with negated options.
>>_______________________________________________
>>Python-checkins mailing list
>>Python-checkins at python.org
>>http://mail.python.org/mailman/listinfo/python-checkins
>>
> 
> 
> 
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 06 2005)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::