Looking for a buffered/windowed iterator

samwyse samwyse at gmail.com
Mon Oct 12 12:21:30 EDT 2009


I have Python program that lets me interact with a bunch of files.
Unfortunately, the program assumes that the bunch is fairly small, and
I have thousands of files on relatively slow storage.  Just creating a
list of the file names takes several minutes, so I'm planning to
replace the list with an iterator in another thread.  However, each
file requires several seconds to load before I can work with it.  The
current code alleviates this via a thread that loads the next file
while I'm examining the current one.  I'd like my new iterator to
provide a fixed window around the current item.  I've looked at the
pairwise recipe, but it doesn't allow me to peek at items within the
window (to generate thumbnails, etc), and I've looked at arrayterator,
but it divides the stream into small contiguous blocks where crossing
a block boundary is relatively expensive.

Previous discussions in c.l.py (primarily those that propose new
functions to be added to itertools) claim that people do this all the
time, but seem woefully short of actual examples.  Before I possibly
re-invent the wheel(*), could someone point me to some actual code
that approximates what I want to do?  Thanks.

(*) Re-inventing a wheel is generally pretty simple.  But then you
discover that you also need to invert axle grease, a leaf spring
suspension, etc.



More information about the Python-list mailing list