[Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines

[Reckoner]

FYI, I uploaded the two files in question to the numpy ticket

    http://projects.scipy.org/numpy/ticket/1284

Thanks!


On Wed, Nov 4, 2009 at 3:56 PM, Bruce Southey <bsouthey at gmail.com> wrote:
> On Wed, Nov 4, 2009 at 8:06 AM, Reckoner <reckoner at gmail.com> wrote:
>> Here's an example:
>>
>> On winxp 64-bit:
>>
>> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
>> win32
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import numpy
>>>>> import cPickle
>>>>> a = numpy.eye(10)
>>>>> cPickle.dump(a,open('from32bitxp.pkl','w'))
>>>>> import numpy.core.multiarray
>>>>> numpy.__version__
>> '1.0.4'
>>>>>
>>
>> On linux 64 bit:
>>
>> Python 2.5.4 (r254:67916, Feb  5 2009, 19:52:35)
>> [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import numpy
>>>>> import cPickle
>>>>> cPickle.load(open('from32bitxp.pkl'))
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>> ImportError: No module named multiarray
>>>>> numpy.__version__
>> '1.2.1'
>>>>> import numpy.core.multiarray
>>>>>
>>
>> Note that I transfer the from32bitxp file from the winxp32 machine to
>> the linux host. Also, I've tried this with version 1.3 on winxp and
>> get the same problem on the linux host.
>>
>> Here's more interesting info:
>>
>> On linux:
>>
>>>>> a = numpy.eye(10)
>>>>> cPickle.dump(a,open('from64bitLinux.pkl','w'))
>>
>> upon transferring the file to winxp 32 and on winxp32:
>>
>>>>> cPickle.load(open('from64bitLinux.pkl'))
>>
>> See? No problem going from linux to winxp32; but problems going the other way.
>>
>> Please let me know if you need more info on this.
>>
>> Any help appreciated.
>>
>> On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey <bsouthey at gmail.com> wrote:
>>> On Mon, Nov 2, 2009 at 6:31 PM, Reckoner <reckoner at gmail.com> wrote:
>>>> thanks for the suggestion! I will look into it. The other thing is
>>>> that the numpy arrays in question are actually embedded in another
>>>> object. When I convert the numpy arrays into plain lists, and then
>>>> cPickle them, there is no problem with any of the larger objects. That
>>>> is the way we are currently working around this issue.
>>>>
>>>> Thanks again.
>>>>
>>>> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>>>>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner <reckoner at gmail.com> wrote:
>>>>>> Anybody have any ideas here?
>>>>>>
>>>>>> Otherwise, I'm thinking this should be posted to the numpy bugs list.
>>>>>> What's the best way to report a bug of this kind?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner <reckoner at gmail.com> wrote:
>>>>>>>> Robert Kern wrote:
>>>>>>>> You can import numpy.core.multiarray on both machines?
>>>>>>>
>>>>>>> Yes. For each machine separately, you can cPickle files with numpy
>>>>>>> arrays without problems loading/dumping. The problem comes from
>>>>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying
>>>>>>> to load them. Transferring cPickle'd files that do *not* have numpy
>>>>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine
>>>>>>> back and forth between the two machines. In fact, we currently get
>>>>>>> around this problem by converting the numpy arrays to lists,
>>>>>>> transferring them, and then re-numpy-ing them on the respective hosts
>>>>>>>
>>>>>>> thanks.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner <reckoner at gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> % python -c 'import numpy.core.multiarray'
>>>>>>>>
>>>>>>>> works just fine, but when I try to load a file that I have transferred
>>>>>>>> from another machine running Windows to one running Linux, I get:
>>>>>>>>
>>>>>>>> %  python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))'
>>>>>>>>
>>>>>>>> Traceback (most recent call last):
>>>>>>>>  File "<string>", line 1, in <module>
>>>>>>>> ImportError: No module named multiarray
>>>>>>>>
>>>>>>>> otherwise, cPickle works normally when transferring files that *do*
>>>>>>>> not contain numpy arrays.
>>>>>>>>
>>>>>>>> I am using version 1.2 on both machines. It's not so easy for me to
>>>>>>>> change versions, by the way, since this is the version that my working
>>>>>>>> group has decided on to standardize on for this effort.
>>>>>>>>
>>>>>>>>
>>>>>>>> Any help appreciated.
>>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>
>>>>> Have you have tried the other Cookbook approaches:
>>>>> http://www.scipy.org/Cookbook/InputOutput
>>>>> Like using numpy's own array io functions - load/save(z)?
>>>>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has
>>>>> different numpy versions)
>>>>>
>>>>> Bruce
>>>>> _______________________________________________
>>>
>>> Can you provide you provide a small self-contained example of the
>>> problem including object creation especially as your example does not
>>> import numpy?
>>>
>>> Really you have to start at the beginning (like pickling and
>>> transferring numpy arrays) and then increase the complexity to include
>>> the object.
>>>
>>>
>>> Bruce
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> Hi,
> I did not see the file 'from32bitxp.pkl'. It would really help if you
> can provide the full example that includes the creation of the
> complete object so at least people could try doing the same process
> with the object that you are using.
>
> Bruce
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>