Hi, I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm having a problem that did not exist before:
import Numeric import numarray Numeric.__version__ '24.0' numarray.__version__ '1.4.1' na=numarray.zeros((3,),'f') num=Numeric.zeros((3,),'f') na[...] = num Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: array sizes must be consistent.
but...
na=numarray.zeros((3,),'i') num=Numeric.zeros((3,),'i') na[...] = num na array([0, 0, 0])
????? However, using Numeric 23.8 and numarray 1.3.3:
import Numeric import numarray Numeric.__version__ '23.8' numarray.__version__ '1.3.3' na=numarray.zeros((3,),'f') num=Numeric.zeros((3,),'f') na[...] = num na array([ 0., 0., 0.], type=Float32)
I don't know if the problem is coming from Numeric or numarray, but it only happens when one mix both:
Numeric.__version__ '24.0' numarray.__version__ '1.4.1' na=numarray.zeros((3,),'f') na2=numarray.zeros((3,),'f') na2[...] = na na2 array([ 0., 0., 0.], type=Float32) num=Numeric.zeros((3,),'f') num2=Numeric.zeros((3,),'f') num2[...] = num num array([ 0., 0., 0.],'f')
Perhaps the new array protocol is involved here? --
0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"
I added support for the newcore "array interface" to the numarray C-API and this exception is now gone. The fix/enhancement is checked into CVS. Kick it around some... Todd Francesc Altet wrote:
Hi,
I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm having a problem that did not exist before:
import Numeric import numarray Numeric.__version__
'24.0'
numarray.__version__
'1.4.1'
na=numarray.zeros((3,),'f') num=Numeric.zeros((3,),'f') na[...] = num
Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: array sizes must be consistent.
but...
na=numarray.zeros((3,),'i') num=Numeric.zeros((3,),'i') na[...] = num na
array([0, 0, 0])
?????
However, using Numeric 23.8 and numarray 1.3.3:
import Numeric import numarray Numeric.__version__
'23.8'
numarray.__version__
'1.3.3'
na=numarray.zeros((3,),'f') num=Numeric.zeros((3,),'f') na[...] = num na
array([ 0., 0., 0.], type=Float32)
I don't know if the problem is coming from Numeric or numarray, but it only happens when one mix both:
Numeric.__version__
'24.0'
numarray.__version__
'1.4.1'
na=numarray.zeros((3,),'f') na2=numarray.zeros((3,),'f') na2[...] = na na2
array([ 0., 0., 0.], type=Float32)
num=Numeric.zeros((3,),'f') num2=Numeric.zeros((3,),'f') num2[...] = num num
array([ 0., 0., 0.],'f')
Perhaps the new array protocol is involved here?
El dc 02 de 11 del 2005 a les 17:16 -0500, en/na Todd Miller va escriure:
I added support for the newcore "array interface" to the numarray C-API and this exception is now gone. The fix/enhancement is checked into CVS. Kick it around some...
Yes. This issue has been solved:
numarray.__version__ '1.4.2' Numeric.__version__ '24.0' num=Numeric.zeros((3,),'f') na=numarray.ones((3,),'f') na[...] = num na array([ 0., 0., 0.], type=Float32)
Now, I've taken the opportunity and tested the Numeric<-->numarray conversion again:
t1_1=timeit.Timer("num=Numeric.array(na,copy=1)","import Numeric;import numarray; na=numarray.arange(10)") t1_1.repeat(3,10000) [0.25493311882019043, 0.25352001190185547, 0.25211095809936523]
and this figures compared with numarray 1.4.1:
t1.repeat(3,10000) [0.59375977516174316, 0.57908082008361816, 0.56574010848999023]
shows that you have accelerated the conversion numarray-->Numeric more than 2x. Now, the Numeric-->numarray:
t3_1=timeit.Timer("na=numarray.array(num,copy=1)","import Numeric;import numarray; num=Numeric.arange(10)") t3_1.repeat(3,10000) [0.40311408042907715, 0.40207314491271973, 0.39954400062561035]
while the 1.4.1 performance was:
t3.repeat(3,10000) [1.3475611209869385, 1.3277668952941895, 1.3417830467224121]
which is a factor 3x of improvement, and almost the same than using the buffer interface. Very nice job, Todd! Now, let's test the conversion for large objects, with data copy and without data copy. For numarray-->Numeric:
t2=timeit.Timer("num=Numeric.array(na,copy=0)","import Numeric;import numarray; na=numarray.arange(100000)") t2.repeat(3,100) [0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516] t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import Numeric;import numarray; na=numarray.arange(100000)") t2_copy.repeat(3,100) [0.50105500221252441, 0.49400091171264648, 0.49266600608825684]
So it seems like if the data copy is taking place. For Numeric-->numarray:
t4=timeit.Timer("na=numarray.array(num,copy=0)","import Numeric;import numarray; num=Numeric.arange(100000)") t4.repeat(3,100) [0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891] t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import Numeric;importnumarray; num=Numeric.arange(100000)") t4_copy.repeat(3,100) [0.0063569545745849609, 0.004302978515625, 0.0042738914489746094]
Ooops! the times for conversion with copy and without copy are more or less the same. Perhaps the copy is not done? It seems so:
num=Numeric.arange(10) na=numarray.array(num,copy=0) na_copy=numarray.array(num,copy=1) na.info() ... data pointer: 0x08312280 (DEBUG ONLY) na_copy.info() ... data pointer: 0x08312280 (DEBUG ONLY)
i.e. the same data pointer. So it seems that the Numeric-->numarray is not copying the data even in the case that we are asking to do that. Other minor things (maybe this is just because you are in the middle of refactoring):
na._data array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
while I would expect:
na._data <memory at 0x081e3618 with size:0x00000028 held by object 0x401f06a0 aliasing object 0x00000000>
Also:
na Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 930, in __repr__ return array_repr(self) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 1660, in array_repr lst = arrayprint.array2string( File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 188, in array2string separator, prefix) File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 140, in _array2string data = _gen.ravel(a) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 918, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers.
Cheers, --
0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"
Francesc Altet wrote:
Now, let's test the conversion for large objects, with data copy and without data copy.
I didn't consider that case but was thinking the semantics of the array interface would be "buffer-like" and therefore non-copying. So at the moment numarray *never* copies and completely ignores the non-sequence array() parameters for the "array interface" case.
For numarray-->Numeric:
t2=timeit.Timer("num=Numeric.array(na,copy=0)","import
Numeric;import numarray; na=numarray.arange(100000)")
t2.repeat(3,100)
[0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516]
t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import
Numeric;import numarray; na=numarray.arange(100000)")
t2_copy.repeat(3,100)
[0.50105500221252441, 0.49400091171264648, 0.49266600608825684]
So it seems like if the data copy is taking place.
I agree and was to be able to verify this by making buffer()s out of the resulting arrays to examine their data addresses.
For Numeric-->numarray:
t4=timeit.Timer("na=numarray.array(num,copy=0)","import
Numeric;import numarray; num=Numeric.arange(100000)")
t4.repeat(3,100)
[0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891]
t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import
Numeric;importnumarray; num=Numeric.arange(100000)")
t4_copy.repeat(3,100)
[0.0063569545745849609, 0.004302978515625, 0.0042738914489746094]
Ooops! the times for conversion with copy and without copy are more or less the same. Perhaps the copy is not done? It seems so:
num=Numeric.arange(10) na=numarray.array(num,copy=0) na_copy=numarray.array(num,copy=1) na.info()
... data pointer: 0x08312280 (DEBUG ONLY)
na_copy.info()
... data pointer: 0x08312280 (DEBUG ONLY)
i.e. the same data pointer. So it seems that the Numeric-->numarray is not copying the data even in the case that we are asking to do that.
Yeah, the copy=X flag is totally ignored at the moment.
Other minor things (maybe this is just because you are in the middle of refactoring):
na._data
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
while I would expect:
na._data
<memory at 0x081e3618 with size:0x00000028 held by object 0x401f06a0 aliasing object 0x00000000>
Also:
This is actually correct I think. Since a Numeric array supports the buffer protocol, it can be used as a buffer. The "memory" output shown above is the output from one particular kind of buffer; i.e. it's the repr of numarray's memory object.
na
Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 930, in __repr__ return array_repr(self) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 1660, in array_repr lst = arrayprint.array2string( File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 188, in array2string separator, prefix) File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 140, in _array2string data = _gen.ravel(a) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 918, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers.
For some reason Numeric was returning None for __array_strides__ and hence this message. I fixed array() so that strides are not set when __array_strides__ returns None. With the added complexity of handling the shape, type, and copy parameters in array(), I'm now seeing performance like this from CVS: numarray-->Numeric array_if: [0.40103912353515625, 0.41119098663330078, 0.41148614883422852] numarray-->Numeric fromstring: [0.24139499664306641, 0.27034401893615723, 0.21657609939575195] Numeric-->numarray array_if: [0.81009912490844727, 0.79906082153320312, 0.74358081817626953] Numeric-->numarray buffer_if: [0.27320504188537598, 0.22041201591491699, 0.22667884826660156] Todd
participants (2)
-
Francesc Altet -
Todd Miller