Splitting strings - by iterators?

[Larry Bates]

Jeremy,

How did you get the string in memory in the first place?
If you read it from a file, perhaps you should change to
reading it from the file a line at the time and use
file.readline as your iterator.

fp=file(inputfile, 'r')
for line in fp:
    ...do your processing...

fp.close()

I don't think I would never read 400,000 lines as a single
string and then split it.  Just a suggestion.

Larry Bates

Jeremy Sanders wrote:
> I have a large string containing lines of text separated by '\n'. I'm
> currently using text.splitlines(True) to break the text into lines, and
> I'm iterating over the resulting list.
> 
> This is very slow (when using 400000 lines!). Other than dumping the
> string to a file, and reading it back using the file iterator, is there a
> way to quickly iterate over the lines?
> 
> I tried using newpos=text.find('\n', pos), and returning the chopped text
> text[pos:newpos+1], but this is much slower than splitlines.
> 
> Any ideas?
> 
> Thanks
> 
> Jeremy
>