[Tutor] Load Entire File into memory

[Dave Angel]

On 4/11/2013 09:04, Amal Thomas wrote:

> @William:
> Thanks,
>
> My Line size varies from 40 to 550 characters. Please note that text file
> which I have to process is in gigabytes ( approx 50 GB ) . This was the
> code which i used to process line by line without loading into memory.

Now I understand.  Processing line by line is slower because it actually
reads the whole file.  The code you showed earlier:

>    I am currently using this method to load my text file:
> *f = open("output.txt")
> content=io.StringIO(f.read())
> f.close()*
>   But I have found that this method uses 4 times the size of text file.

will only read a tiny portion of the file.  You don't have any loop on
the read() statement, you just read the first buffer full. So naturally
it'll be much faster.

I am of course assuming you don't have a machine with 100+ gig of RAM.


By the way, would you please stop posting html messages to a text
newsgroup?  They come out completely blank on my other newsreader
(except for the Tutor maillist trailer, which IS text), and I end up
having to fire up this one just to read your messages. But even if you
don't care about me, you should realize that messages sent as html as
very frequently garbled when they're interpreted as text.

-- 
DaveA