<div><br><div class="gmail_quote"><div dir="auto">On Sat, 7 Oct 2017 at 16.59, Nicholas Nadeau <<a href="mailto:nicholas.nadeau@gmail.com">nicholas.nadeau@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi Andrea!<div><br></div><div>Checkout the following SO answers for similar contexts:</div><div>- <a href="https://stackoverflow.com/questions/22108488/are-list-comprehensions-and-functional-functions-faster-than-for-loops" target="_blank">https://stackoverflow.com/questions/22108488/are-list-comprehensions-and-functional-functions-faster-than-for-loops</a></div><div>- <a href="https://stackoverflow.com/questions/30245397/why-is-list-comprehension-so-faster" target="_blank">https://stackoverflow.com/questions/30245397/why-is-list-comprehension-so-faster</a></div><div><br></div><div>To better visualize the issue, I made a iPython gist (simplifying the code a bit): <a href="https://gist.github.com/nnadeau/3deb6f18d028009a4495590cfbbfaa40" target="_blank">https://gist.github.com/nnadeau/3deb6f18d028009a4495590cfbbfaa40</a></div><div><br></div><div>From a quick view of the disassembled code (I'm not an expert, so correct me if I'm wrong), list comprehension has much less overhead compared to iterating/looping through the pre-allocated data and building/storing each slice.</div><div></div></div></blockquote><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Thank you Nicholas, I suspected that the approach of using list comprehensions was close to unbeatable, thanks for the analysis!</div><div dir="auto"><br></div><div dir="auto">Andrea.</div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br></div><div>Cheers,</div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="m_-8051758525003978217gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><span></span><div><span style="font-family:Arial">-- </span><br>
<font face="Arial">Nicholas Nadeau, P.Eng., AVS</font><br></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">On 7 October 2017 at 05:56, Andrea Gavana <span><<a href="mailto:andrea.gavana@gmail.com" target="_blank">andrea.gavana@gmail.com</a>></span> wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Apologies, correct timeit code this time (I had gotten the wrong shape for the output matrix in the loop case):<div><br></div><div><span><div>if __name__ == '__main__':</div><div><br></div><div> repeat = 1000</div><div> items = [Item('item_%d'%(i+1)) for i in xrange(500)]</div><div><br></div></span><span><div> output = numpy.asarray([item.do_something() for item in items]).T</div></span><span><div> statements = ['''</div><div> output = numpy.asarray([item.do_something() for item in items]).T</div><div> ''',</div><div> '''</div></span><div> output = numpy.empty((8, 500))</div><span><div> for i, item in enumerate(items):</div></span><div> output[:, i] = item.do_something()</div><span><div> ''']</div><div><br></div><div> methods = ['List Comprehension', 'Empty plus Loop '] </div><div> setup = 'from __main__ import numpy, items'</div><div><br></div><div> for stmnt, method in zip(statements, methods):</div><div><br></div><div> elapsed = timeit.repeat(stmnt, setup=setup, number=1, repeat=repeat)</div><div> minv, maxv, meanv = min(elapsed), max(elapsed), numpy.mean(elapsed)</div><div> elapsed.sort()</div><div> best_of_3 = numpy.mean(elapsed[0:3])</div><div> result = numpy.asarray((minv, maxv, meanv, best_of_3))*repeat</div><div><br></div><div> print method, ': MIN: %0.2f ms , MAX: %0.2f ms , MEAN: %0.2f ms , BEST OF 3: %0.2f ms'%tuple(result.tolist())</div></span></div><div><br></div><div><br></div><div>Results are the same as before...</div><div><br></div><div><br></div></div><div class="m_-8051758525003978217HOEnZb"><div class="m_-8051758525003978217h5"><div class="gmail_extra"><br><div class="gmail_quote">On 7 October 2017 at 11:52, Andrea Gavana <span><<a href="mailto:andrea.gavana@gmail.com" target="_blank">andrea.gavana@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi All,<div><br></div><div> I have this little snippet of code:</div><div><br></div><div><div>import timeit</div><div>import numpy</div><div><br></div><div>class Item(object):</div><div><br></div><div> def __init__(self, name):</div><div><br></div><div> <a href="http://self.name" target="_blank">self.name</a> = name</div><div> self.values = numpy.random.rand(8, 1)</div><div><br></div><div> def do_something(self):</div><div><br></div><div> sv = self.values.sum(axis=0)</div><div> array = numpy.empty((8, ))</div><div> f = numpy.dot(0.5*numpy.ones((8, )), self.values)[0]</div><div> array.fill(f)</div><div> return array</div></div><div><br></div><div><br></div><div>In my real application, the method do_something does a bit more than that, but I believe the snippet is enough to start playing with it. What I have is a list of (on average) 500-1,000 classes Item, and I am trying to retrieve the output of do_something for each of them in a single, big 2D numpy array.</div><div><br></div><div>My current approach is to use list comprehension like this:</div><div><br></div><div>output = numpy.asarray([item.do_something() for item in items]).T<br></div><div><br></div><div>(Note: I need the transposed of that 2D array, always).</div><div><br></div><div>But then I though: why not preallocating the output array and make a simple loop:</div><div><br></div><div><div>output = numpy.empty((500, 8))</div><div>for i, item in enumerate(items):</div><div> output[i, :] = item.do_something()</div></div><div><br></div><div><br></div><div>I was expecting this version to be marginally faster - as the previous one has to call asarray and then transpose the matrix, but I was in for a surprise:</div><div><br></div><div><div>if __name__ == '__main__':</div><div><br></div><div> repeat = 1000</div><div> items = [Item('item_%d'%(i+1)) for i in xrange(500)]</div><div><br></div><div> statements = ['''</div><div> output = numpy.asarray([item.do_something() for item in items]).T</div><div> ''',</div><div> '''</div><div> output = numpy.empty((500, 8))</div><div> for i, item in enumerate(items):</div><div> output[i, :] = item.do_something()</div><div> ''']</div><div><br></div><div> methods = ['List Comprehension', 'Empty plus Loop '] </div><div> setup = 'from __main__ import numpy, items'</div><div><br></div><div> for stmnt, method in zip(statements, methods):</div><div><br></div><div> elapsed = timeit.repeat(stmnt, setup=setup, number=1, repeat=repeat)</div><div> minv, maxv, meanv = min(elapsed), max(elapsed), numpy.mean(elapsed)</div><div> elapsed.sort()</div><div> best_of_3 = numpy.mean(elapsed[0:3])</div><div> result = numpy.asarray((minv, maxv, meanv, best_of_3))*repeat</div><div><br></div><div> print method, ': MIN: %0.2f ms , MAX: %0.2f ms , MEAN: %0.2f ms , BEST OF 3: %0.2f ms'%tuple(result.tolist())</div><div><br></div></div><div><br></div><div>I get this:</div><div><br></div><div><div>List Comprehension : MIN: 7.32 ms , MAX: 9.13 ms , MEAN: 7.85 ms , BEST OF 3: 7.33 ms</div><div>Empty plus Loop : MIN: 7.99 ms , MAX: 9.57 ms , MEAN: 8.31 ms , BEST OF 3: 8.01 ms</div><div><br></div><div><br></div></div><div>Now, I know that list comprehensions are renowned for being insanely fast, but I though that doing asarray plus transpose would by far defeat their advantage, especially since the list comprehension is used to call a method, not to do some simple arithmetic inside it...</div><div><br></div><div>I guess I am missing something obvious here... oh, and if anyone has suggestions about how to improve my crappy code (performance wise), please feel free to add your thoughts.</div><div><br></div><div>Thank you.</div><span class="m_-8051758525003978217m_2801045735208817108HOEnZb"><font color="#888888"><div><br></div><div>Andrea.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></font></span></div>
</blockquote></div><br></div>
</div></div><br></blockquote></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote></div></div>