Processing a large string

[Peter Otten]

Peter Otten wrote:

> goldtech wrote:

>> Say I have a very big string with a pattern like:
>> 
>> akakksssk3dhdhdhdbddb3dkdkdkddk3dmdmdmd3dkdkdkdk3asnsn.....
>> 
>> I want to split the sting into separate parts on the "3" and process
>> each part separately. I might run into memory limitations if I use
>> "split" and get a big array(?)  I wondered if there's a way I could
>> read (stream?) the string from start to finish and read what's
>> delimited by the "3" into a variable, process the smaller string
>> variable then append/build a new string with the processed data?

> PS: This has come up before, but I couldn't find the relevant threads...

Alex Martelli a looong time ago:

> from __future__ import generators
> 
> def splitby(fileobj, splitter, bufsize=8192):
>     buf = ''
> 
>     while True:
>         try: 
>             item, buf = buf.split(splitter, 1)
>         except ValueError:
>             more = fileobj.read(bufsize)
>             if not more: break
>             buf += more
>         else:
>             yield item + splitter
> 
>     if buf:
>         yield buf

http://mail.python.org/pipermail/python-list/2002-September/770673.html