A am having some trouble when pickling numpy arrays. Basically I use one python script to create the array and then pickle it. When I load the pickled array using a different python script it appears to load fine. When I try to perform a matrix multiply on the array with a vector (using dot) python crashes with a segmentation violation. The array contains 64bit floating point numbers and has a shape of (47, 16384) and the vector has a shape of (16384,1). Other shapes I have tested also have this problem. A workaround is to create a new array that is initialized with the data from the first. For example: a = array(pickle.loads(buffer)) After calling this command the script runs fine. Here is some information on my system setup: OS: Macos X Leopard 64bit Intel numpy.version.version = 1.0.5.dev4624 Below is a simple script that reproduces this error on my system. I would like to get this problem fixed so that I don't have to use this work around every time I load a pickled array. Is this a known bug? I could not find it in the bug tracker. Thanks in advance for the help. Dave import pickle from numpy import array,dot,ones,random # Create the arrays a = random.normal(size=(47,128*128)) b = ones((128*128,2)) c = ones((128*128,1)) # Test dot print "Computing dot product on original arrays..." print dot(a,b).shape # WORKS print dot(a,c).shape # WORKS print "Pickle and unpickel the array with workaround." buffer = pickle.dumps(a) a = array(pickle.loads(buffer)) # Test dot print "Computing dot product with workaround..." print dot(a,b).shape # WORKS print dot(a,c).shape # WORKS print "Pickle and unpickel the array without workaround." a = pickle.loads(buffer) # Test dot print "Computing dot product without workaround..." print dot(a,b).shape # WORKS print dot(a,c).shape # CRASH
Hi David On Thu, Jan 24, 2008 at 01:18:45PM -0700, David Bolme wrote:
A am having some trouble when pickling numpy arrays. Basically I use one python script to create the array and then pickle it. When I load the pickled array using a different python script it appears to load fine. When I try to perform a matrix multiply on the array with a vector (using dot) python crashes with a segmentation violation.
Please see if http://projects.scipy.org/scipy/numpy/ticket/551 matches your problem, and add your comments there. Regards Stéfan
to, 2008-01-24 kello 13:18 -0700, David Bolme kirjoitti:
A am having some trouble when pickling numpy arrays. Basically I use one python script to create the array and then pickle it. When I load the pickled array using a different python script it appears to load fine. When I try to perform a matrix multiply on the array with a vector (using dot) python crashes with a segmentation violation. The array contains 64bit floating point numbers and has a shape of (47, 16384) and the vector has a shape of (16384,1). Other shapes I have tested also have this problem. A workaround is to create a new array that is initialized with the data from the first. For example:
This probably is the same problem as bug #551, http://projects.scipy.org/scipy/numpy/ticket/551 Summary: The cause is not known at this point, but so far appears to have something to do with SSE2 optimized ATLAS library. Are you using Atlas library on Mac OS X, and if yes, which version? Do you have SSE2 optimizations enabled? What happens if you try to use a non-SSE2-optimized Atlas, eg. by pointing LD_LIBRARY_PATH (I'm not sure this works on Mac, but I assume so...) to non-optimized Atlas libraries: LD_LIBRARY_PATH=/usr/lib/atlas python crasher.py -- Pauli Virtanen
I think you are right. This does seem to be the same bug as 551. I will try a non optimized ATLAS to see if that helps.
I have added a valgrind report to bug 551. The report indicates a problem with uninitialized values. The segfault does seem to be related to certain configurations of atlas. I can confirm that I had this same problem occurs with the Ubuntu 7.04 installed scipy with SSE2 optimized ATLAS. The valgrind output is from a run where the code did not crash but valgrind still detected many errors. On Jan 26, 2008, at 3:01 PM, David Bolme wrote:
I think you are right. This does seem to be the same bug as 551. I will try a non optimized ATLAS to see if that helps. _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
participants (3)
-
David Bolme
-
Pauli Virtanen
-
Stefan van der Walt