[Tutor] Load Entire File into memory

[Alan Gauld]

On 04/11/13 16:34, Amal Thomas wrote:
> @Joel: The code runs for weeks..input file which I have to process in
> very huge(in 50 gbs). So its not a matter of hours.its matter of days
> and weeks..

OK, but that's not down to reading the file from disk.
Reading a 50G file will only take a few minutes if you have enough RAM, 
which seems to be the case. If it's taking days/weeks you must be doing 
some incredibly time consuming processing.

It's probably worth putting some more timing statements into your code 
to see where the time is going because it's not the reading from the 
disk that's the problem.

> trying to optimize my code to get the outputs in less time and memory
> efficiently.

Memory efficiency is easy, do it line by line off the disk.
time efficiency is most likely down to your processing algorithm
if we are talking about days.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos