<span></span><div>Thanks for the help, this is considerably faster and easier to read (see below). I changed it to avoid the "break" and I think it makes it easy to understand. I am checking the conditions each time slows it but it is worth it to me at this time. </div>
<div>Thanks again</div><div>Vincent</div><div><br></div><div>def read_data_file(filename):</div><div> reader = csv.reader(open(filename, "U"),delimiter='\t')</div><div><br></div><div> data = []</div>
<div> mask = []</div><div> outliers = []</div><div> modified = []</div><div> </div><div> data_append = data.append</div><div> mask_append = mask.append</div><div> outliers_append = outliers.append</div>
<div> modified_append = modified.append</div><div> </div><div> maskcount = 0</div><div> outliercount = 0</div><div> modifiedcount = 0</div><div> </div><div> for row in reader:</div><div> if '[MASKS]' in row:</div>
<div> maskcount += 1</div><div> if '[OUTLIERS]' in row:</div><div> outliercount += 1</div><div> if '[MODIFIED]' in row:</div><div> modifiedcount += 1</div><div>
if not any((maskcount, outliercount, modifiedcount, not row)):</div><div> data_append(row)</div><div> elif not any((outliercount, modifiedcount, not row)):</div><div> mask_append(row)</div>
<div> elif not any((modifiedcount, not row)):</div><div> outliers_append(row)</div><div> else:</div><div> if row: modified_append(row)</div><div> </div><div> data = data[1:]</div>
<div> mask = mask[3:]</div><div> outliers = outliers[3:]</div><div> modified = modified[3:]</div><div> return [data, mask, outliers, modified]</div><div name="mailplane_signature"> <table><tbody><tr><td width="80">
<img src="http://www.gravatar.com/avatar/226e40fdc55d4597a46279296a616384.png">
</td><td width="10"></td><td width="127" align="center">
<div style="padding-right: 5px; padding-left: 5px;
font-size: 11px; padding-bottom: 5px; color: #666666;
padding-top: 5px">
<p><strong>Vincent Davis<br>
720-301-3003
</strong><br>
<a href="mailto:vincent@vincentdavis.net">vincent@vincentdavis.net</a> </p>
<div style="font-size: 10px">
<a href="http://vincentdavis.net">my blog</a> |
<a href="http://www.linkedin.com/in/vincentdavis">LinkedIn</a></div></div></td></tr><tr></tr></tbody></table></div><br><br><div class="gmail_quote">On Fri, Feb 19, 2010 at 4:36 PM, Jonathan Gardner <span dir="ltr"><<a href="mailto:jgardner@jonathangardner.net">jgardner@jonathangardner.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br><div class="gmail_quote"><div class="im">On Fri, Feb 19, 2010 at 1:58 PM, Vincent Davis <span dir="ltr"><<a href="mailto:vincent@vincentdavis.net" target="_blank">vincent@vincentdavis.net</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex"><div class="im">
In reference to the several comments about "[x for x in read] is basically a copy of the entire list. This isn't necessary." or list(read). I had thought I had a problem with having iterators in the takewhile() statement. I thought I testes and it didn't work. It seems I was wrong. It clearly works. I'll make this change and see if it is any better.</div>
<div>
<div><br></div><div>I actually don't plan to read them all in at once, only as needed, but I do need the whole file in an array to perform some mathematics on them and compare different files. So my interest was in making it faster to open them as needed. I guess part of it is that they are about 5mb so I guess it might be disk speed in part.nks<div>
<div></div><br></div></div></div></blockquote></div><br><br>Record your numbers in an array and then work your magic on them later. Don't store the entire file in memory, though.<br><font color="#888888"><br>-- <br></font><div>
<div></div><div class="h5">Jonathan Gardner<br><a href="mailto:jgardner@jonathangardner.net" target="_blank">jgardner@jonathangardner.net</a><br>
</div></div></blockquote></div><br>