vstack and hstack performance penalty
When using vstack or hstack for large arrays, are there any performance penalties eg. takes longer time-wise or makes a copy of an array during operation ?
On Fri, 2014-01-24 at 06:13 -0800, Dinesh Vadhia wrote:
When using vstack or hstack for large arrays, are there any performance penalties eg. takes longer time-wise or makes a copy of an array during operation ?
No, they all use concatenate. There are only constant overheads on top of the necessary data copying. Though performance may vary because of memory order, etc. - Sebastian
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
If A is very large and B is very small then np.concatenate(A, B) will copy B's data over to A which would take less time than the other way around - is that so? Does 'memory order' mean that it depends on sufficient contiguous memory being available for B otherwise it will be fragmented or something else?
On Fri, Jan 24, 2014 at 4:01 PM, Dinesh Vadhia <dineshbvadhia@hotmail.com> wrote:
If A is very large and B is very small then np.concatenate(A, B) will copy B's data over to A which would take less time than the other way around -
is
that so?
No, neither array is modified in-place. A new array is created and both A and B are copied into it. The order is largely unimportant.
Does 'memory order' mean that it depends on sufficient contiguous memory being available for B otherwise it will be fragmented or something else?
No, the output is never fragmented. numpy arrays may be strided, but never fragmented arbitrarily to fit into a fragmented address space. http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#internal-memor... The issue is what axis the concatenation happens on. If it's the first axis (and both inputs are contiguous), then it only takes two memcpy() calls to copy the data, one for each input, because the regions where they go into the output are juxtaposed. If you concatenate on one of the other axes, though, then the memory regions for A and B will be interleaved and you have to do 2*N memory copies (N being some number depending on the shape). -- Robert Kern
participants (3)
-
Dinesh Vadhia -
Robert Kern -
Sebastian Berg