streaming a file object through re.finditer

[Steven Bethard]

Erick wrote:
> Hello,
> 
> I've been looking for a while for an answer, but so far I haven't been
> able to turn anything up yet. Basically, what I'd like to do is to use
> re.finditer to search a large file (or a file stream), but I haven't
> figured out how to get finditer to work without loading the entire file
> into memory, or just reading one line at a time (or more complicated
> buffering).

Can you use mmap?

http://docs.python.org/lib/module-mmap.html

"You can use mmap objects in most places where strings are expected; for 
example, you can use the re module to search through a memory-mapped file."

Seems applicable, and it should keep your memory use down, but I'm not 
very experienced with it...

Steve