<div dir="ltr"><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Memory Error while working with pandas dataframe.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Description of Environment Windows 7 python 3.4.2 32-bit version pandas 0.16.0</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">We are running into the error described below. Any help provided will be sincerely appreciated.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">We are able to read a 300MB Csv file into a dataframe using the read_csv function. While working with the dataframe we ran into memory error. We used the pd.Concat function to concatenate two dataframes. So we decided to use chunksize for lazy reading. Chunking returns an object of type TextFileReader.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px"><a href="http://pandas.pydata.org/pandas-docs/stable/io.html#iterating-through-files-chunk-by-chunk" rel="nofollow" style="margin:0px;padding:0px;border:0px;text-decoration:none;color:rgb(12,101,165)">http://pandas.pydata.org/pandas-docs/stable/io.html#iterating-through-files-chunk-by-chunk</a></p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">We are able to iterate over this object once as a debugging measure. The iterator gets exhausted after iterating once. So we are not able to convert the TextFileReader object back into a dataframe, using the pd.concat function.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Error</p><pre class="" style="margin-top:0px;padding:5px;border:0px;font-size:13px;overflow:auto;width:auto;max-height:600px;font-family:Consolas,Menlo,Monaco,'Lucida Console','Liberation Mono','DejaVu Sans Mono','Bitstream Vera Sans Mono','Courier New',monospace,sans-serif;color:rgb(57,51,24);word-wrap:normal;background-color:rgb(238,238,238)"><code style="margin:0px;padding:0px;border:0px;font-family:Consolas,Menlo,Monaco,'Lucida Console','Liberation Mono','DejaVu Sans Mono','Bitstream Vera Sans Mono','Courier New',monospace,sans-serif;white-space:inherit"><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">Traceback</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">(</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">most recent call last</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">):</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">
  </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">File</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">"psindia.py"</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> line </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">60</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">in</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"><</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">module</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">></span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">
    data</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">=</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">pd</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">.</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">concat</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">(</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">tp</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">ignore_index</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">=</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">True</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">)</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">
  </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">File</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">"C:\Python34\lib\site-packages\pandas\tools\merge.py"</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> line </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">754</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">in</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> conca
t
    copy</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">=</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">copy</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">)</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">
  </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">File</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">"C:\Python34\lib\site-packages\pandas\tools\merge.py"</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> line </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">799</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">,</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">in</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> __ini
t__
    </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">raise</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">ValueError</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">(</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(128,0,0)">'All objects passed were None'</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">)</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">
</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">ValueError</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)">:</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(43,145,175)">All</span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,0)"> objects passed were </span><span class="" style="margin:0px;padding:0px;border:0px;color:rgb(0,0,139)">None</span></code></pre><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Thanks for your time.</p><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><font color="#000000">Narendran Elango<br>B.Tech(2014)<br>Department of Computer Engineering<br>National Institute of Technology Karnataka, Surathkal</font><br></div><div dir="ltr"><font color="#000000"><br></font></div></div></div></div>
</div>