first, second, etc line of text file
jeff at jmcneil.net
Thu Jul 26 01:53:06 CEST 2007
Yup, I actually had log files in mind, too. I process Apache logs from an
8-way cluster every 15 minutes. There are 32,000 sites on said cluster;
that's a lot of log data!
I didn't actually think anyone would go *try* to
open("war_and_peace.txt").readlines().. I just meant it as a generalization
for "really large file." =)
On 7/25/07, Jay Loden <python at jayloden.com> wrote:
> Grant Edwards wrote:
> > On 2007-07-25, Jeff <jeffober at gmail.com> wrote:
> >> That might be a memory problem if you are running multiple processes
> >> regularly, such as on a webserver.
> > I suppose if you did it in parallel 50 processes, you could use
> > up 250MB of RAM. Still not a big deal on many servers. A
> > decent OS will swap regions that aren't being used to disk, so
> > it's likely not to be a problem.
> Or, you might be reading from a text file dramatically larger than a 3MB
> copy of War and Peace. I regularly deal with log files that are often many
> times that, including some that have been well over a GB or more. Trust me,
> you don't want to read in the entire file when it's a 1.5GB text file.
> It's true that many times readlines() will work fine, but there are also
> certainly cases where it's not acceptable for memory and performance
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list