[Numpy-discussion] portability issue

[David M. Cooke]

<pbtransfert at freesurf.fr> writes:

> hi !
>
> i try to transfer a pickle which contains numeric array, from a 64-bits
> system to a 32-bits system. it seems to fail due to bad (or lack of)
> conversion... more precisely, here is what i do on the 64-bits system :
>
> import Numeric,cPickle
> a=Numeric.array([1,2,3])
> f=open('test.pickle64','w')
> cPickle.dump(a,f)
> f.close()
>
> and here is what i try to do on the 32-bits system :
>
> import Numeric,cPickle
> f=open('test.pickle64','r')
> a=cPickle.load(f)
> f.close()
>
> and here is the log of the load :
>
>     a=cPickle.load(f)
>   File "/usr/lib/python2.3/site-packages/Numeric/Numeric.py", line 539, in
> array_constructor
>     x.shape = shape
> ValueError: ('total size of new array must be unchanged', <function
> array_constructor at 0x40a1002c>, ((3,), 'l',
> '\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00',True))
>
>
> Is there something to do to solve this difficulty ?

Specify the integer type with the number of bits.
Numeric.array([1,2,3]) will create an array with a typecode of 'l'
(Numeric.Int), which is the type that can hold Python ints (= C
longs). On your 64-bit system, it's a 64-bit integer; on the 64-bit,
it's a 32-bit integer. So, on the 32-bit system, when reading the
pickle, it sees an array of type 'l', but there is too much data to
fill the array it expects.

The solution is to explicitly create your array using a typecode that
gives the size of the integer. Either:

a = Numeric.array([1,2,3], Numeric.Int32)

or

a = Numeric.array([1,2,3], Numeric.Int64)

I haven't checked this, but I would think that using Int32 is better
if all your numbers will fit in that. Using 64-bit integers would mean
the 32-bit machine would have to use 'long long' types to do its math,
which would be slower, while using 32-bit integers would mean the
64-bit machine would use 'int', which would still be fast for it.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca