[Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines

[Bruce Southey]

On Wed, Nov 4, 2009 at 9:31 PM, Reckoner <reckoner at gmail.com> wrote:
> FYI, I uploaded the two files in question to the numpy ticket
>
>    http://projects.scipy.org/numpy/ticket/1284
>
> Thanks!
>
>
> On Wed, Nov 4, 2009 at 3:56 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>> On Wed, Nov 4, 2009 at 8:06 AM, Reckoner <reckoner at gmail.com> wrote:
>>> Here's an example:
>>>
>>> On winxp 64-bit:
>>>
>>> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
>>> win32
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> import numpy
>>>>>> import cPickle
>>>>>> a = numpy.eye(10)
>>>>>> cPickle.dump(a,open('from32bitxp.pkl','w'))
>>>>>> import numpy.core.multiarray
>>>>>> numpy.__version__
>>> '1.0.4'
>>>>>>
>>>
>>> On linux 64 bit:
>>>
>>> Python 2.5.4 (r254:67916, Feb  5 2009, 19:52:35)
>>> [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> import numpy
>>>>>> import cPickle
>>>>>> cPickle.load(open('from32bitxp.pkl'))
>>> Traceback (most recent call last):
>>>  File "<stdin>", line 1, in <module>
>>> ImportError: No module named multiarray
>>>>>> numpy.__version__
>>> '1.2.1'
>>>>>> import numpy.core.multiarray
>>>>>>
>>>
>>> Note that I transfer the from32bitxp file from the winxp32 machine to
>>> the linux host. Also, I've tried this with version 1.3 on winxp and
>>> get the same problem on the linux host.
>>>
>>> Here's more interesting info:
>>>
>>> On linux:
>>>
>>>>>> a = numpy.eye(10)
>>>>>> cPickle.dump(a,open('from64bitLinux.pkl','w'))
>>>
>>> upon transferring the file to winxp 32 and on winxp32:
>>>
>>>>>> cPickle.load(open('from64bitLinux.pkl'))
>>>
>>> See? No problem going from linux to winxp32; but problems going the other way.
>>>
>>> Please let me know if you need more info on this.
>>>
>>> Any help appreciated.
>>>
>>> On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey <bsouthey at gmail.com> wrote:
>>>> On Mon, Nov 2, 2009 at 6:31 PM, Reckoner <reckoner at gmail.com> wrote:
>>>>> thanks for the suggestion! I will look into it. The other thing is
>>>>> that the numpy arrays in question are actually embedded in another
>>>>> object. When I convert the numpy arrays into plain lists, and then
>>>>> cPickle them, there is no problem with any of the larger objects. That
>>>>> is the way we are currently working around this issue.
>>>>>
>>>>> Thanks again.
>>>>>
>>>>> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>>>>>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner <reckoner at gmail.com> wrote:
>>>>>>> Anybody have any ideas here?
>>>>>>>
>>>>>>> Otherwise, I'm thinking this should be posted to the numpy bugs list.
>>>>>>> What's the best way to report a bug of this kind?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner <reckoner at gmail.com> wrote:
>>>>>>>>> Robert Kern wrote:
>>>>>>>>> You can import numpy.core.multiarray on both machines?
>>>>>>>>
>>>>>>>> Yes. For each machine separately, you can cPickle files with numpy
>>>>>>>> arrays without problems loading/dumping. The problem comes from
>>>>>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying
>>>>>>>> to load them. Transferring cPickle'd files that do *not* have numpy
>>>>>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine
>>>>>>>> back and forth between the two machines. In fact, we currently get
>>>>>>>> around this problem by converting the numpy arrays to lists,
>>>>>>>> transferring them, and then re-numpy-ing them on the respective hosts
>>>>>>>>
>>>>>>>> thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner <reckoner at gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> % python -c 'import numpy.core.multiarray'
>>>>>>>>>
>>>>>>>>> works just fine, but when I try to load a file that I have transferred
>>>>>>>>> from another machine running Windows to one running Linux, I get:
>>>>>>>>>
>>>>>>>>> %  python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))'
>>>>>>>>>
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>  File "<string>", line 1, in <module>
>>>>>>>>> ImportError: No module named multiarray
>>>>>>>>>
>>>>>>>>> otherwise, cPickle works normally when transferring files that *do*
>>>>>>>>> not contain numpy arrays.
>>>>>>>>>
>>>>>>>>> I am using version 1.2 on both machines. It's not so easy for me to
>>>>>>>>> change versions, by the way, since this is the version that my working
>>>>>>>>> group has decided on to standardize on for this effort.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help appreciated.
>>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> NumPy-Discussion mailing list
>>>>>>> NumPy-Discussion at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>>>
>>>>>>
>>>>>> Have you have tried the other Cookbook approaches:
>>>>>> http://www.scipy.org/Cookbook/InputOutput
>>>>>> Like using numpy's own array io functions - load/save(z)?
>>>>>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has
>>>>>> different numpy versions)
>>>>>>
>>>>>> Bruce
>>>>>> _______________________________________________
>>>>
>>>> Can you provide you provide a small self-contained example of the
>>>> problem including object creation especially as your example does not
>>>> import numpy?
>>>>
>>>> Really you have to start at the beginning (like pickling and
>>>> transferring numpy arrays) and then increase the complexity to include
>>>> the object.
>>>>
>>>>
>>>> Bruce
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>> Hi,
>> I did not see the file 'from32bitxp.pkl'. It would really help if you
>> can provide the full example that includes the creation of the
>> complete object so at least people could try doing the same process
>> with the object that you are using.
>>
>> Bruce
>> _______________________________________________

Hi
I read both of those files on a 32bit Windows vista system running
Python 2.6.4 and numpy 1.3. Perhaps no surprise as it appears that
both files are identical.

Bruce