[Tutor] More efficient than glob?

[Doug.Shawhan@gecits.ge.com]

I have a script that parses the contents of logfiles in a directory, puts
the content in a database, then deletes the original logfile. The files are
produced at a rate of about one per minute.

Since there is well over a year's worth of logfiles (> 31536000 files) the
following method:

>>> import glob
>>> files = glob.glob('\\tmp\\*')
 for each in files:
	open each
	add contents of each to database
	close each
	delete each

...would seem to use an undue amount of memory.

Is there a more efficient way of doing this? Glob is pretty awesome for
simplicity, but this method with a huge amount of files seems to gobble up
512 megs of ram in no time! :-)

Thanks!

d