<div class="gmail_quote">On Thu, Jan 27, 2011 at 4:33 PM, Dewald Pieterse <span dir="ltr"><<a href="mailto:dewald.pieterse@gmail.com">dewald.pieterse@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br><br><div class="gmail_quote"><div class="im">On Thu, Jan 27, 2011 at 4:19 PM, Christopher Barker <span dir="ltr"><<a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div>On 1/27/11 1:03 PM, Dewald Pieterse wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
I am processing two csv files against another, my first implementation<br>
used python list of lists and list.append to generate a new list while<br>
looping all the data including the non-relevant data (can't determine<br>
location of specific data element in a list of list). So I re-implented<br>
the exact same code but using numpy.array's (2d arrays) using<br>
numpy.where to prevent looping over an entire dataset needlessly but the<br>
numpy.array based code is about 7.6 times slower?<br>
</blockquote>
<br></div>
Didn't look at your code in any detail, but:<br>
<br>
numpy arrays are not designed to be re-sizable, so numpy.append actually creates a new array, and copies the old to the new, along with the new stuff. It's a convenience function, but it means you are re-allocating and copying all your data with each call.<br>
<br>
python lists, on the other hand, are designed to be re-sizable, so they pre-allocate extra room, so that appending can be fast.<br>
<br>
In general, the recommended solution in this sort of situation is to build up your data in a python list, then convert it to an array.<br>
<br>
If I'm right about what you're doing you could keep the "rows" as numpy arrays, but put them in a list while building it up.<br></blockquote><div> </div></div><div>Thanks Chris, I believe this is the problem then, I can continue to use
the arrays as reference data but build list instead, the only reason I
used the arrays was to be able to use numpy.where, I just use both data
types, best of both worlds. As I already have row arrays I will do a build a list or arrays. <br></div></div></blockquote><div><br>Now my code is nearly 4 times faster than the list of lists implementation! Wonderful, thanks. <br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote"><div></div><div class="im"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
Also, a numpy array of strings isn't necessarily a great dats structure for this kind of data. YOu might want to look at structured arrays.<br></blockquote></div><div><br>Atm, I use :<br><div style="margin-left: 40px;">
comit_eqp_reader = csv.reader(comit_eqp_file, delimiter=',', quotechar='"')<br>
comit_eqp_lt = numpy.array([[col for col in row] for row in comit_eqp_reader])<br></div>to setup the arrays, I will look at using structured arrays<br></div><div><div></div><div class="h5"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
I wrote an appendable numpy array class a while back, to address this. It has some advantages, though, as it it written, not as much as you'd think. It does have some benifits for structured arrays, though.<br>
<br>
<br>
Code enclosed<br>
<br>
-Chris<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div></div><div>
relevant list of list code:<br>
<br>
starttime = time.clock()<br>
#NI_data_list room_eqp_list<br>
NI_data_list_new = []<br>
for NI_row in NI_data_list:<br>
treelevel = NI_row[0]<br>
elevation = NI_row[1]<br>
locater = NI_row[2]<br>
area = NI_row[3]<br>
NIroom = NI_row[4]<br>
#Write appropriate equipment models and drawing into new list<br>
if NIroom != '':<br>
#Write appropriate equipment models and drawing into new list<br>
for row in room_eqp_list:<br>
eqp_room = row[0]<br>
if len(eqp_room) == 5:<br>
eqp_drawing = row[1]<br>
if NIroom == eqp_room:<br>
newrow =<br>
[int(treelevel)+1,elevation,locater,area,NIroom,eqp_drawing]<br>
NI_data_list_new.append(newrow)<br>
#Write appropriate piping info into the new list<br>
for prow in unique_piping_list:<br>
pipe_room = prow[0]<br>
if len(pipe_room) == 5:<br>
pipe_drawing = prow[1]<br>
if pipe_room == NIroom:<br>
piperow =<br>
[int(treelevel)+1,elevation,locater,area,NIroom,pipe_drawing]<br>
NI_data_list_new.append(piperow)<br>
#Write appropriate equipment models and drawing into new list<br>
if (locater != '' and NIroom == ''):<br>
#Write appropriate equipment models and drawing into new list<br>
for row in room_eqp_list:<br>
eqp_locater = row[0]<br>
if len(eqp_locater) == 4:<br>
eqp_drawing = row[1]<br>
if locater == eqp_locater:<br>
newrow =<br>
[int(treelevel)+1,elevation,eqp_locater,area,'',eqp_drawing]<br>
NI_data_list_new.append(newrow)<br>
#Write appropriate piping info into the new list<br>
for prow in unique_piping_list:<br>
pipe_locater = prow[0]<br>
if len(pipe_locater) == 4:<br>
pipe_drawing = prow[1]<br>
if pipe_locater == locater:<br>
piperow =<br>
[int(treelevel)+1,elevation,pipe_locater,area,'',pipe_drawing]<br>
NI_data_list_new.append(piperow)<br>
#Rewrite NI_data to new list<br>
if NIroom == '':<br>
NI_data_list_new.append(NI_row)<br>
<br>
print (time.clock()-starttime)<br>
<br>
<br>
relevant numpy.array code:<br>
<br>
NI_data_write_url = reports_dir + 'NI_data_room2.csv'<br>
NI_data_list_file = open(NI_data_write_url, 'wb')<br>
NI_data_list_writer = csv.writer(NI_data_list_file, delimiter=',',<br>
quotechar='"')<br>
starttime = time.clock()<br>
#NI_data_list room_eqp_list<br>
NI_data_list_new = numpy.array([['TreeDepth', 'Elevation',<br>
'BuildingLocater', 'Area', 'Room', 'Item']])<br>
for NI_row in NI_data_list:<br>
treelevel = NI_row[0]<br>
elevation = NI_row[1]<br>
locater = NI_row[2]<br>
area = NI_row[3]<br>
NIroom = NI_row[4]<br>
#Write appropriate equipment models and drawing into new array<br>
if NIroom != '':<br>
#Write appropriate equipment models and drawing into new array<br>
(rowtest, columntest) = numpy.where(room_eqp_list==NIroom)<br>
for row_iter in rowtest:<br>
eqp_room = room_eqp_list[row_iter,0]<br>
if len(eqp_room) == 5:<br>
eqp_drawing = room_eqp_list[row_iter,1]<br>
if NIroom == eqp_room:<br>
newrow =<br>
numpy.array([[int(treelevel)+1,elevation,locater,area,NIroom,eqp_drawing]])<br>
NI_data_list_new =<br>
numpy.append(NI_data_list_new, newrow, 0)<br>
<br>
#Write appropriate piping info into the new array<br>
(rowtest, columntest) =<br>
numpy.where(unique_room_piping_list==NIroom)<br>
for row_iter in rowtest: #unique_room_piping_list<br>
pipe_room = unique_room_piping_list[row_iter,0]<br>
if len(pipe_room) == 5:<br>
pipe_drawing = unique_room_piping_list[row_iter,1]<br>
if pipe_room == NIroom:<br>
piperow =<br>
numpy.array([[int(treelevel)+1,elevation,locater,area,NIroom,pipe_drawing]])<br>
NI_data_list_new =<br>
numpy.append(NI_data_list_new, piperow, 0)<br>
#Write appropriate equipment models and drawing into new array<br>
if (locater != '' and NIroom == ''):<br>
#Write appropriate equipment models and drawing into new array<br>
(rowtest, columntest) = numpy.where(room_eqp_list==locater)<br>
for row_iter in rowtest:<br>
eqp_locater = room_eqp_list[row_iter,0]<br>
if len(eqp_locater) == 4:<br>
eqp_drawing = room_eqp_list[row_iter,1]<br>
if locater == eqp_locater:<br>
newrow =<br>
numpy.array([[int(treelevel)+1,elevation,eqp_locater,area,'',eqp_drawing]])<br>
NI_data_list_new =<br>
numpy.append(NI_data_list_new, newrow, 0)<br>
#Write appropriate piping info into the new array<br>
(rowtest, columntest) =<br>
numpy.where(unique_room_eqp_list==locater)<br>
for row_iter in rowtest:<br>
pipe_locater = unique_room_piping_list[row_iter,0]<br>
if len(pipe_locater) == 4:<br>
pipe_drawing = unique_room_piping_list[row_iter,1]<br>
if pipe_locater == locater:<br>
piperow =<br>
numpy.array([[int(treelevel)+1,elevation,pipe_locater,area,'',pipe_drawing]])<br>
NI_data_list_new =<br>
numpy.append(NI_data_list_new, piperow, 0)<br>
#Rewrite NI_data to new list<br>
if NIroom == '':<br>
NI_data_list_new = numpy.append(NI_data_list_new,[NI_row],0)<br>
<br>
print (time.clock()-starttime)<br>
<br>
<br>
some relevant output<br>
<br>
>>> print NI_data_list_new<br>
[['TreeDepth' 'Elevation' 'BuildingLocater' 'Area' 'Room' 'Item']<br>
['0' '' '1000' '' '' '']<br>
['1' '' '1000' '' '' 'docname Rev 0']<br>
...,<br>
['5' '6' '1164' '4' '' 'eqp11 RB, R. surname, 24-NOV-08']<br>
['4' '6' '1164' '4' '' 'anotherdoc Rev A']<br>
['0' '' '' '' '' '']]<br>
<br>
<br>
Is numpy.append so slow? or is the culprit numpy.where?<br>
<br>
Dewald Pieterse<br>
<br>
"A democracy is nothing more than mob rule, where fifty-one percent of<br>
the people take away the rights of the other forty-nine." ~ Thomas Jefferson<br>
<br>
<br>
<br></div></div>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote>
<br>
<br>
-- <br>
Christopher Barker, Ph.D.<br>
Oceanographer<br>
<br>
Emergency Response Division<br>
NOAA/NOS/OR&R (206) 526-6959 voice<br>
7600 Sand Point Way NE (206) 526-6329 fax<br>
Seattle, WA 98115 (206) 526-6317 main reception<br>
<br>
<a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a><br>
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div></div></div><div><div></div><div class="h5"><br><br clear="all"><br>-- <br>Dewald Pieterse<br><br>"A democracy is nothing more than mob rule, where fifty-one percent of the people take away the rights of the other forty-nine." ~ Thomas Jefferson<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Dewald Pieterse<br><br>"A democracy is nothing more than mob rule, where fifty-one percent of the people take away the rights of the other forty-nine." ~ Thomas Jefferson<br>