<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Hi Everyone,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> I have a simple program which reads a large file containing few million rows, parses each row (`numpy array`) and converts into an array of doubles (`python array`) and later writes into an `hdf5 file`. I repeat this loop for multiple days. After reading each file, i delete all the objects and call garbage collector. When I run the program, First day is parsed without any error but on the second day i get `MemoryError`. I monitored the memory usage of my program, during first day of parsing, memory usage is around **1.5 GB**. When the first day parsing is finished, memory usage goes down to **50 MB**. Now when 2nd day starts and i try to read the lines from the file I get `MemoryError`. Following is the output of the program.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> source file extracted at C:\rfadump\au\2012.08.07.txt<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> parsing started<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> current time: 2012-09-16 22:40:16.829000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 500000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 1000000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 1500000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 2000000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 2500000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 3000000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 3500000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 4000000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 4500000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> 5000000 lines parsed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> parsing done.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> end time is 2012-09-16 23:34:19.931000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> total time elapsed 0:54:03.102000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> repacking file<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> done<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> > s:\users\aaj\projects\pythonhf\rfadumptohdf.py(132)generateFiles()<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> -> while single_date <= self.end_date:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> (Pdb) c<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> *** 2012-08-08 ***<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> source file extracted at C:\rfadump\au\2012.08.08.txt<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> cought an exception while generating file for day 2012-08-08.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> Traceback (most recent call last):<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> File "rfaDumpToHDF.py", line 175, in generateFile<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> lines = self.rawfile.read().split('|\n')<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> MemoryError<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>I am very sure that windows system task manager shows the memory usage as **50 MB** for this process. It looks like the garbage collector or memory manager for Python is not calculating the free memory correctly. There should be lot of free memory but it thinks there is not enough. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Any idea?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thanks.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span lang=DE-CH style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Alok Jadhav<o:p></o:p></span></p><p class=MsoNormal><b><span lang=DE-CH style='font-size:10.0pt;font-family:"Arial","sans-serif"'>CREDIT SUISSE AG<o:p></o:p></span></b></p><p class=MsoNormal><span lang=DE-CH style='font-size:10.0pt;font-family:"Arial","sans-serif"'>GAT IT Hong Kong, KVAG 67<o:p></o:p></span></p><p class=MsoNormal><span lang=DE-CH style='font-size:10.0pt;font-family:"Arial","sans-serif"'>International Commerce Centre | Hong Kong | Hong Kong<o:p></o:p></span></p><p class=MsoNormal><span lang=DE-CH style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Phone +852 2101 6274 | Mobile +852 9169 7172<o:p></o:p></span></p><p class=MsoNormal><span lang=DE-CH style='font-size:10.0pt;font-family:"Arial","sans-serif"'><a href="mailto:alok.jadhav@credit-suisse.com">alok.jadhav@credit-suisse.com</a> | <a href="http://www.credit-suisse.com/">www.credit-suisse.com</a><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div> <BR/><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:Courier'>==============================================================================<br>Please access the attached hyperlink for an important electronic communications disclaimer:<br><a href="http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html">http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html</a><br>==============================================================================<o:p></o:p></span></p></html> <br>
</body></html>