[Python-bugs-list] [Bug #110674] memory bloat in binary file upload (PR#381)

noreply@sourceforge.net noreply@sourceforge.net
Wed, 27 Sep 2000 03:38:51 -0700


Bug #110674, was updated on 2000-Jul-31 14:13
Here is a current snapshot of the bug.

Project: Python
Category: None
Status: Closed
Resolution: Later
Bug Group: Feature Request
Priority: 4
Summary: memory bloat in binary file upload (PR#381)

Details: Jitterbug-Id: 381
Submitted-By: naris@ensim.com
Date: Mon,  3 Jul 2000 21:29:25 -0400 (EDT)
Version: 1.5.2
OS: RedHat 6.1


read_lines_to_outerboundary chews up memory.
there's a while (1) loop that does a readline(), which
is probably not good, as you could have a binary file that
doesn't have an endline til the end, and thus readline would
read the entire file into memory.  i did the following:
dd'd a 100MB /dev/zero file and used zope to upload it.
it chewed up a large amount of memory.

there seems to be bug reports about text files and windows
systems (and needing to use python -u), but this problem
is unrelated.  i am using a unix browser -> unix web server.



====================================================================
Audit trail:
Tue Jul 11 08:24:22 2000	guido	moved from incoming to open

Follow-Ups:

Date: 2000-Aug-24 12:38
By: naris

Comment:
there's a bug i submitted to zope that deals with a second problem
with read_lines_to_outerboundary, which will also cause a memory
leak.

the read_lines_to_outerboundary has an unnecessary line 
self.lines.append(line).  removing will fix it this second problem.
-------------------------------------------------------

Date: 2000-Sep-25 15:06
By: bwarsaw

Comment:
Agreed that readlines() isn't the best thing to use in the face of large binary uploads.  I've added this feature request to PEP 42 (and am closing this bug report).

Removing self.lines might be more problematic, since although it's not documented as part of the public interface for FieldStorage, it is possible that existing code is using it, and that code would break.
-------------------------------------------------------

Date: 2000-Sep-27 03:38
By: naris

Comment:
i feel the self.lines.append() is is a pretty serious problem...have you got any
second opinions on how to proceed?

if its not part of the public interface for FieldStorage, then wouldn't it be somewhat safe to remove it (you might anger some people, but they could always stick to using the old cgi.py)?

my argument is that if they changed the socket connect/bind functions to reject connect(host,port) in favor of connect((host,port)), then we should certainly be able to remove this self.lines.append() :-)

what do you think?

-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=110674&group_id=5470