<div dir="ltr">Hi Nathaniel,<div><br></div><div>Thanks for the suggestion. I did profile the program before, just not using Python.</div><div><br></div><div>But following your suggestion, I used %prun. Here's (part of) the output (when I use savez):</div>
<div> </div><div> 195503 function calls in 4.466 seconds</div><div><br></div><div> Ordered by: internal time</div><div><br></div><div> ncalls tottime percall cumtime percall filename:lineno(function)</div><div> 2 2.284 1.142 2.284 1.142 {method 'close' of '_io.BufferedWriter' objects}</div>
<div> 1 0.918 0.918 0.918 0.918 {built-in method remove}</div><div> 48841 0.568 0.000 0.568 0.000 {method 'write' of '_io.BufferedWriter' objects}</div><div> 48829 0.379 0.000 0.379 0.000 {built-in method crc32}</div>
<div> 48830 0.148 0.000 0.148 0.000 {method 'read' of '_io.BufferedReader' objects}</div><div> 1 0.090 0.090 0.993 0.993 zipfile.py:1315(write)</div><div> 1 0.072 0.072 0.072 0.072 {method 'tostring' of 'numpy.ndarray' objects}</div>
<div> 48848 0.005 0.000 0.005 0.000 {built-in method len}</div><div> 1 0.001 0.001 0.270 0.270 format.py:362(write_array)</div><div> 3 0.000 0.000 0.000 0.000 {built-in method open}</div>
<div> 1 0.000 0.000 4.466 4.466 npyio.py:560(_savez)</div><div> 2 0.000 0.000 0.000 0.000 zipfile.py:1459(close)</div><div> 1 0.000 0.000 4.466 4.466 {built-in method exec}</div>
<div><br></div><div>Here's the output when I use save to save to a npy file:</div><div><br></div><div><div> 39 function calls in 0.266 seconds</div><div><br></div><div> Ordered by: internal time</div><div><br></div>
<div> ncalls tottime percall cumtime percall filename:lineno(function)</div><div> 4 0.196 0.049 0.196 0.049 {method 'write' of '_io.BufferedWriter' objects}</div><div> 1 0.069 0.069 0.069 0.069 {method 'tostring' of 'numpy.ndarray' objects}</div>
<div> 1 0.001 0.001 0.266 0.266 format.py:362(write_array)</div><div> 1 0.000 0.000 0.000 0.000 {built-in method open}</div><div> 1 0.000 0.000 0.266 0.266 npyio.py:406(save)</div>
<div> 1 0.000 0.000 0.000 0.000 format.py:261(write_array_header_1_0)</div><div> 1 0.000 0.000 0.000 0.000 {method 'close' of '_io.BufferedWriter' objects}</div><div> 1 0.000 0.000 0.266 0.266 {built-in method exec}</div>
<div> 1 0.000 0.000 0.000 0.000 format.py:154(magic)</div><div> 1 0.000 0.000 0.000 0.000 format.py:233(header_data_from_array_1_0)</div><div> 1 0.000 0.000 0.266 0.266 <string>:1(<module>)</div>
<div> 1 0.000 0.000 0.000 0.000 numeric.py:462(asanyarray)</div><div> 1 0.000 0.000 0.000 0.000 py3k.py:28(asbytes)</div></div><div><br></div><div>The calls to close and the built-in method remove seem to be the responsible for the inefficiency of the Numpy implementation (compared to the Julia package that I mentioned before). This was tested using Python 3.4 and Numpy 1.8.1.</div>
<div>However if I do the tests with Python 3.3.5 and Numpy 1.8.0, savez becomes much faster, so I think there is something wrong with this combination Python 3.4/Numpy 1.8.1. </div><div>Also, if I use Python 2.4 and Numpy 1.2 (from my school's cluster) I get that np.save takes about 3.5 seconds and np.savez takes about 7 seconds, so all these timings seem to be hugely dependent on the system/version (maybe this explain David Palao's results?). </div>
<div><br></div><div>However, they all point out that a significant amount of time is spent computing the crc32. Notice that prun reports that it takes 0.379 second to compute the crc32 of an array that takes 0.2 seconds to save to a npy file. I believe this is too much! And it get worse if you try to save bigger arrays. </div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Apr 17, 2014 at 5:23 AM, Nathaniel Smith <span dir="ltr"><<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=""><p dir="ltr">On 17 Apr 2014 01:57, "onefire" <<a href="mailto:onefire.myself@gmail.com" target="_blank">onefire.myself@gmail.com</a>> wrote:<br>
><br>
> What I cannot understand is why savez takes more than 10 times longer than saving the data to a npy file. The only reason that I could come up with was the computation of the crc32. </p>
</div><p dir="ltr">We can all make guesses but the solution is just to profile it :-). %prun in ipython (and then if you need more granularity installing line_profiler is useful).</p><span class="HOEnZb"><font color="#888888">
<p dir="ltr">-n</p>
</font></span><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div>