[Numpy-discussion] cPickle/unPickle across archs

[James Mazer]

Hi,

I've got a some Numeric arrays that were created without
an explicit byte size in the initial declaration and pickled.
Something like this:

   >>> cPickle.write(array(ones((3,3,)), 'f'), open('foo.pic', 'w'))

as opposed to:

   >>> cPickle.write(array(ones((3,3,)), Float32), open('foo.pic', 'w'))

This works as long as the word size doesn't change between the
reading and writing machines.

The data were generated under a 32bit linux kernel and now I'm trying
to read them under a 64bit kernel, so the word size has changed and
Numeric assumes that the 'f' type is the NATIVE float and 'l' type is
the NATIVE long) and dies miserable when the native types don't match
the actual types (which defeats the whole point of pickling, to some
extent -- I thought that cPickle.save/load were "ensured" to be
invertable...)

I've got terrabytes of data that need to be read by both 32bit and
64bit machines (and it's not really feasible to scan all the files
into new structures with explict types on a 32bit machine). Anybody
have hints for addressing this problem?  I found similar questions,
but no answers, so I'm not completely alone iwth this problem.

Thanks,
/jamie


-- 
James Mazer
Department of Neurobiology
Yale School of Medicine
phone: 203-737-5853
fax: 203-785-5263