<div dir="ltr"><div>As a bit of an aside, I have just discovered that for fixed-width text data, numpy's text readers seems to edge out pandas' read_fwf(), and numpy has the advantage of being able to specify the dtypes ahead of time (seems that the pandas version just won't allow it, which means I end up with float64's and object dtypes instead of float32's and |S12 dtypes where I want them).<br><br></div>Cheers!<br>Ben Root<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 28, 2014 at 4:09 PM, Chris Barker <span dir="ltr"><<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">A few thoughts:<div><br></div><div>1) yes, a faster, more memory efficient text file parser would be great. Yeah, if your workflow relies on parsing lots of huge text files, you probably need another workflow. But it's a really really common thing to nee to do -- why not do it fast?</div><div><br></div><div>2) """<span style="font-family:arial,sans-serif;font-size:12.8000001907349px">you are describing a special case where you know the data size apriori (eg not streaming), dtypes are readily apparent from a small sample case </span></div><span class=""><div style="font-family:arial,sans-serif;font-size:12.8000001907349px">and in general your data is not messy """</div><div style="font-family:arial,sans-serif;font-size:12.8000001907349px"><br></div></span><div style="font-family:arial,sans-serif;font-size:12.8000001907349px">sure -- that's a special case, but it's a really common special case (OK -- without the know your data size ,anyway...)</div><div style="font-family:arial,sans-serif;font-size:12.8000001907349px"><br></div><div style="font-family:arial,sans-serif;font-size:12.8000001907349px">3)</div><div class="gmail_extra"><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Someone also posted some code or the draft thereof for using resizable arrays quite a while ago, which would<br>
reduce the memory overhead for very large arrays.<br></blockquote><div><br></div></span><div>That may have been me -- I have a resizable array class, both pure python and not-quite finished Cython version. In practice, if you add stuff to the array row by row (or item by item), it's no faster than putting it all in a list and then converting to an array -- but it IS more memory efficient, which seems to be the issue here. Let me know if you want it -- I really need to get it up on gitHub one of these days.</div><div><br></div><div>My take: for fast parsing of big files you need:</div><div><br></div><div>To do the parsing/converting in C -- what wrong with good old fscanf, at least for the basic types -- it's pretty darn fast.</div><div><br></div><div>Memory efficiency -- somethign like my growable array is not all that hard to implement and pretty darn quick -- you just do the usual trick_ over allocate a bit of memory, and when it gets full re-allocate a larger chunk. It turns out, at least on the hardware I tested on, that the performance is not very sensitive to how much you over allocate -- if it's tiny (1 element) performance really sucks, but once you get to a 10% or so (maybe less) over-allocation, you don't notice the difference.</div><div><br></div><div>Keep the auto-figuring out of the structure / dtypes separate from the high speed parsing code. I"d say write high speed parsing code first -- that requires specification of the data types and structure, then, if you want, write some nice pure python code that tries to auto-detect all that. If it's a small file, it's fast regardless. if it's a large file, then the overhead of teh fancy parsing will be lost, and you'll want the line by line parsing to be as fast as possible.</div><div><br></div><div>From a quick loo, it seems that the Panda's code is pretty nice -- maybe the 2X memory footprint should be ignored.</div><div><br></div><div>-Chris</div><span class=""><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Cheers,<br>
Derek<br>
<div><div><br>
<br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></span></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R <a href="tel:%28206%29%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>7600 Sand Point Way NE <a href="tel:%28206%29%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>Seattle, WA 98115 <a href="tel:%28206%29%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a>
</font></span></div></div>
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div>