What are the limitations of cStringIO.StringIO() ?

[Benjamin Kaplan]

On Wed, Jul 1, 2009 at 7:48 AM, Barak, Ron<Ron.Barak at lsi.com> wrote:
> Hi,
>
> I think I'm up against a limitation in cStringIO.StringIO(), but could not
> find any clues on the web, especially not in
> http://docs.python.org/library/stringio.html.
>
> What I have is the following function:
>
>     def concatenate_files(self,filename_array):
>         import types
>
>         f = cStringIO.StringIO()
>         for filename in filename_array:
>             log_stream = LogStream(filename)
>             if not isinstance(log_stream, types.NoneType):
>                 try:
>                     string_ = log_stream.input_file.read()
>                 except AttributeError, e:
>                         sys.stderr.write("AttributeError: "+str(e)+"\n")
>                         sys.stderr.write("log_stream:
> "+str(log_stream)+"\n")
>             else:
>                 return(None)
>
>             f.write(string_)
>         return (f)
>
> And, when the list of files - in filename_array - is pointing to long/large
> enough files, I get the exception:
>
> Traceback (most recent call last):
>   File "svm_ts_tool_in_progress.py", line 244, in OnSelectCell
>     self.InspectorViewController(row,col)
>   File "svm_ts_tool_in_progress.py", line 1216, in InspectorViewController
>     self.InspectorePaneGetRecords(self.req_table, rec)
>   File "svm_ts_tool_in_progress.py", line 1315, in InspectorePaneGetRecords
>     log_stream =
> ConcatenatedLogStream(self.events_dict[req_table]["db_dict"].keys())
>   File "c:\Documents and
> Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 31, in
> __init__
>     self.input_file = self.concatenate_files(filename_array)
>   File "c:\Documents and
> Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 47, in
> concatenate_files
>     string_ = log_stream.input_file.read()
> MemoryError
>
> Anyone knows whet's the limitation on cStringIO.StringIO() objects ?
> Could you suggest a better way to create a string that contains the
> concatenation of several big files ?
>

MemoryErrors are caused because, for whatever reason, the OS won't
allocate any more memory for the process. For a 32-bit Windows
process, that maximum is 2 GB. No matter what programming language or
library you use, you can't bypass that limit. There is a way to up the
limit to 3GB but I don't know how to do it. The link below has all the
information about Windows memory limits. If you need lots of really
big strings in memory simultaneously, you'll have to get a 64-bit
system with a ton of RAM.

http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx

> Thanks,
> Ron.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>