is this an error when shuffle doubles&looses items on a 2-d array? :
r=arange(20) rr=ziparrays(r,r) #numpy.vstack(*args).transpose() rr array([[ 0, 0], [ 1, 1], [ 2, 2], [ 3, 3], [ 4, 4], [ 5, 5], [ 6, 6], [ 7, 7], [ 8, 8], [ 9, 9], [10, 10], [11, 11], [12, 12], [13, 13], [14, 14], [15, 15], [16, 16], [17, 17], [18, 18], [19, 19]]) numpy.random.shuffle(rr) rr array([[ 0, 0], [ 0, 0], [ 2, 2], [ 2, 2], [ 3, 3], [ 3, 3], [ 5, 5], [ 2, 2], [ 0, 0], [ 8, 8], [10, 10], [ 1, 1], [10, 10], [11, 11], [12, 12], [ 2, 2], [10, 10], [15, 15], [ 6, 6], [16, 16]])
on 1-d arrays it seems to be consistent as expected:
numpy.random.shuffle(r) r array([ 6, 1, 3, 17, 0, 11, 15, 7, 14, 10, 9, 2, 8, 12, 5, 4, 19, 16, 13, 18])
Robert
On Wed, 22 Nov 2006, Robert apparently wrote:
numpy.random.shuffle(rr)
The docstring is incomplete. From the NumPy Book: shuffle (sequence) Randomly permute the items of any sequence. If sequence is an array, then it must be 1-d.
rr=N.array(zip(range(20),range(20))) N.random.shuffle(rr.flat) rr array([[ 0, 16], [10, 1], [19, 13], [18, 14], [13, 11], [12, 4], [19, 9], [18, 16], [14, 7], [ 5, 17], [ 2, 3], [ 9, 7], [12, 3], [ 1, 4], [ 8, 17], [ 6, 6], [11, 0], [ 5, 15], [10, 8], [ 2, 15]])
What surprised me more is that you cannot shuffle the rows with Python's random.shuffle either: the exact same problem you noted crops up! (Works fine with a list of tuples, of course.) But you could do:
rr=N.array(zip(range(20),range(20))) pid = N.random.permutation(rr.shape[0]) rr=rr[pid] rr array([[ 9, 9], [ 2, 2], [ 3, 3], [15, 15], [ 4, 4], [11, 11], [18, 18], [17, 17], [ 5, 5], [12, 12], [13, 13], [ 1, 1], [ 8, 8], [14, 14], [16, 16], [19, 19], [ 7, 7], [ 6, 6], [10, 10], [ 0, 0]])
hth, Alan Isaac
Alan G Isaac wrote:
rr=N.array(zip(range(20),range(20))) pid = N.random.permutation(rr.shape[0]) rr=rr[pid] rr array([[ 9, 9], [ 2, 2], [ 3, 3], [15, 15], [ 4, 4], [11, 11], [18, 18], [17, 17], [ 5, 5], [12, 12], [13, 13], [ 1, 1], [ 8, 8], [14, 14], [16, 16], [19, 19], [ 7, 7], [ 6, 6], [10, 10], [ 0, 0]])
This is really a thinko on my part. I copied the algorithm from Python's random module. At the core of it is a set of swaps: x[i], x[j] = x[j], x[i] With the kind of sequences that the stdlib random module is expecting, that makes perfect sense. However, with N-dim arrays (N > 1), x[i] is a *view* into the array. By the time that x[j] = x[i] gets executed, x[i] = x[j] has already executed and the underlying memory that x[i] points to has been modified. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
This is really a thinko on my part.
What, exactly, is a thinko?
I copied the algorithm from Python's random module. At the core of it is a set of swaps:
x[i], x[j] = x[j], x[i]
With the kind of sequences that the stdlib random module is expecting, that makes perfect sense. However, with N-dim arrays (N > 1), x[i] is a *view* into the array. By the time that x[j] = x[i] gets executed, x[i] = x[j] has already executed and the underlying memory that x[i] points to has been modified.
wouldn't something like: temp = x[i].copy() x[i], x[j] = x[j], temp work? In any case, it should raise an error if ndim > 1, rather than giving a wrong result. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Christopher Barker wrote:
Robert Kern wrote:
This is really a thinko on my part.
What, exactly, is a thinko?
Like a typo except that the fault lies with the brain, not the fingers.
I copied the algorithm from Python's random module. At the core of it is a set of swaps:
x[i], x[j] = x[j], x[i]
With the kind of sequences that the stdlib random module is expecting, that makes perfect sense. However, with N-dim arrays (N > 1), x[i] is a *view* into the array. By the time that x[j] = x[i] gets executed, x[i] = x[j] has already executed and the underlying memory that x[i] points to has been modified.
wouldn't something like:
temp = x[i].copy() x[i], x[j] = x[j], temp
work?
In any case, it should raise an error if ndim > 1, rather than giving a wrong result.
The method is intended to work on sequences in general, not just numpy arrays, so I can't really use .copy() or even test for ndim > 1. One possibility is to check if the object is an ndarray (or subclass) and use .copy() if so; otherwise, use the current implementation and hope that you didn't pass it a Numeric or numarray array (or some other view-based object). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
Christopher Barker wrote:
Robert Kern wrote:
[SNIP]
I copied the algorithm from Python's random module. At the core of it is a set of swaps:
x[i], x[j] = x[j], x[i]
With the kind of sequences that the stdlib random module is expecting, that makes perfect sense. However, with N-dim arrays (N > 1), x[i] is a *view* into the array. By the time that x[j] = x[i] gets executed, x[i] = x[j] has already executed and the underlying memory that x[i] points to has been modified.
wouldn't something like:
temp = x[i].copy() x[i], x[j] = x[j], temp
work?
In any case, it should raise an error if ndim > 1, rather than giving a wrong result.
The method is intended to work on sequences in general, not just numpy arrays, so I can't really use .copy() or even test for ndim > 1.
One possibility is to check if the object is an ndarray (or subclass) and use .copy() if so; otherwise, use the current implementation and hope that you didn't pass it a Numeric or numarray array (or some other view-based object).
I think I would invert this test and instead check if the object is a Python list and *not* copy in that case. Otherwise, use copy.copy to copy the object whatever it is. This looks like it would be more robust in that it would work in all sensible case, and just be a tad slower in some of them. Another possible refinement / complication would be to special case 1D arrays so that they run fastish. A third possibility involves rewriting this in this form: indices = arange(len(x)) _shuffle_core(indices) # This just does what current shuffle now does x[:] = take(x, indices, 0) Of course, all of these are pretty inefficient in the common case of a 1D array. -tim
Tim Hochberg wrote:
Robert Kern wrote:
One possibility is to check if the object is an ndarray (or subclass) and use .copy() if so; otherwise, use the current implementation and hope that you didn't pass it a Numeric or numarray array (or some other view-based object).
I think I would invert this test and instead check if the object is a Python list and *not* copy in that case. Otherwise, use copy.copy to copy the object whatever it is. This looks like it would be more robust in that it would work in all sensible case, and just be a tad slower in some of them.
I don't want to assume that the only two sequence types are lists and arrays. The problem with using copy.copy() on non-arrays is that it, well, makes copies of the elements. The objects in the shuffled sequence are not the same objects before and after the shuffling. I consider that to be a violation of the spec. Views are rare outside of numpy/Numeric/numarray, partially because Guido considers them to be evil. I'm beginning to see why.
Another possible refinement / complication would be to special case 1D arrays so that they run fastish.
A third possibility involves rewriting this in this form:
indices = arange(len(x)) _shuffle_core(indices) # This just does what current shuffle now does x[:] = take(x, indices, 0)
That's problematic since the elements all turn into numpy scalar objects: In [1]: from numpy import * In [2]: a = range(9,-1,-1) In [3]: idx = arange(len(a)) In [4]: a[:] = take(a, idx, 0) In [5]: a Out[5]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] In [6]: type(a[0]) Out[6]: <type 'numpy.int32'> -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
Tim Hochberg wrote:
Robert Kern wrote:
One possibility is to check if the object is an ndarray (or subclass) and use .copy() if so; otherwise, use the current implementation and hope that you didn't pass it a Numeric or numarray array (or some other view-based object).
I think I would invert this test and instead check if the object is a Python list and *not* copy in that case. Otherwise, use copy.copy to copy the object whatever it is. This looks like it would be more robust in that it would work in all sensible case, and just be a tad slower in some of them.
I don't want to assume that the only two sequence types are lists and arrays. The problem with using copy.copy() on non-arrays is that it, well, makes copies of the elements. The objects in the shuffled sequence are not the same objects before and after the shuffling. I consider that to be a violation of the spec.
OK, makes sense.
Views are rare outside of numpy/Numeric/numarray, partially because Guido considers them to be evil. I'm beginning to see why.
They are enough rope to shoot yourself in the foot and that's for sure.
Another possible refinement / complication would be to special case 1D arrays so that they run fastish.
A third possibility involves rewriting this in this form:
indices = arange(len(x)) _shuffle_core(indices) # This just does what current shuffle now does x[:] = take(x, indices, 0)
That's problematic since the elements all turn into numpy scalar objects:
[SNIP] I believe that this one is fixable: def shuffle_take(x): indices = np.arange(len(x)) np.random.shuffle(indices) source = np.asarray(x, dtype=object) x[:] = source[indices] return x a = range(9,-1,-1) shuffle_take(a) print a print type(a[0]), a[0] b = np.arange(10).reshape(5,2) print b shuffle_take(b) print b ====> [0, 8, 1, 4, 9, 5, 7, 3, 6, 2] <type 'int'> 0 [[0 1] [2 3] [4 5] [6 7] [8 9]] [[8 9] [6 7] [0 1] [2 3] [4 5]] I think that in real life you'd probably want only convert to an array of type object if 'x' is not an array to begin with, but I'm keeping it simple for right now. -tim
Robert Kern wrote:
I don't want to assume that the only two sequence types are lists and arrays.
Does numpy.random.shuffle really have to work on any sequence? and without making a copy? I'm not so sure -- having num* functions operate on any sequence has been a design goal of Numeric from the beginning, but I've never thought it that important. However, it is pretty consistent across numpy, so we should probably support it here. How about two cases: 1) sequences that support copy() 2) sequences that don't -- in that case, it is assumed that indexing has copy semantics. It truth, this means: 1) numpy arrays 2) standard python mutable sequences But it does satisfy the duck typing approach. Maybe we could specifically check for copy vs. view semantics, but that seems like overkill. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Christopher Barker wrote:
Robert Kern wrote:
I don't want to assume that the only two sequence types are lists and arrays.
Does numpy.random.shuffle really have to work on any sequence? and without making a copy? I'm not so sure -- having num* functions operate on any sequence has been a design goal of Numeric from the beginning, but I've never thought it that important.
I think it is for this one function. I want numpy.random to be able to have all of the functionality of the stdlib's random module (although not as a drop-in replacement). The reason is that random number generation is something that should be completely and centrally controllable. I should be able to write my application such that it seeds the PRNG state *once* and uses that state everywhere. If I need to keep instances of both numpy.random.RandomState and random.Random around, then something's wrong.
However, it is pretty consistent across numpy, so we should probably support it here.
How about two cases:
1) sequences that support copy()
It's not the container sequences would support a .copy() method, but the indexed items.
2) sequences that don't -- in that case, it is assumed that indexing has copy semantics.
It truth, this means: 1) numpy arrays 2) standard python mutable sequences
But it does satisfy the duck typing approach.
Maybe we could specifically check for copy vs. view semantics, but that seems like overkill.
It's also probably impossible. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
Tim Hochberg wrote:
Robert Kern wrote:
One possibility is to check if the object is an ndarray (or subclass) and use .copy() if so; otherwise, use the current implementation and hope that you didn't pass it a Numeric or numarray array (or some other view-based object).
I think I would invert this test and instead check if the object is a Python list and *not* copy in that case. Otherwise, use copy.copy to copy the object whatever it is. This looks like it would be more robust in that it would work in all sensible case, and just be a tad slower in some of them.
I don't want to assume that the only two sequence types are lists and arrays. The problem with using copy.copy() on non-arrays is that it, well, makes copies of the elements. The objects in the shuffled sequence are not the same objects before and after the shuffling. I consider that to be a violation of the spec.
Views are rare outside of numpy/Numeric/numarray, partially because Guido considers them to be evil. I'm beginning to see why.
Another possible refinement / complication would be to special case 1D arrays so that they run fastish.
A third possibility involves rewriting this in this form:
indices = arange(len(x)) _shuffle_core(indices) # This just does what current shuffle now does x[:] = take(x, indices, 0)
That's problematic since the elements all turn into numpy scalar objects:
In [1]: from numpy import *
In [2]: a = range(9,-1,-1)
In [3]: idx = arange(len(a))
In [4]: a[:] = take(a, idx, 0)
In [5]: a Out[5]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
In [6]: type(a[0]) Out[6]: <type 'numpy.int32'>
a[:]=take(asarray(a,object),idx,0) ? works also correct with ndarray's even if I didn't dig the reason why... all element will be probably re-casted twice. Think the take-method on shuffled indizes is basically right and natural for a numpy-shuffler. The example is just possibly another vote against the default behavior of letting numpy.scalar types out of arrays, which are set up with a "harmless" type.
array([1,2,3],float) array([ 1., 2., 3.]) type(_[0]) <type 'numpy.float64'>
is just ill as it I think. In (Guido's) Python objects should probably come out of collections best as typy as they went in. Currently numpy-scalars will just "infect" the whole app almost like a virus (and kill performance and pickle's etc.) Of course views are essential for an efficient array type, but type-altering possibly not. For rare cases for generalized algs (I need to think hard to find even an example), where the array-interface is needed on elements (and a array(obj) cast is too uncomfortable), there could be still the different possibilty:
array([1,2,3],numpy.float64)
then its natural that numpy.float64, numpy.int32.... come out, as the programmer would even expect it so. Thus maybe for array types: * float!=numpy.float64 (but common base class (or 'float' itself) maybe) * int !=numpy.intXX * complex !=numpy.complex128 * default array type is (python.)float * default array type from list of ints is (python.)int * default array type from list of complex is (python.)complex * default array type of other lists is always <object> currently this is also problematic:
array([1,2,"3",[]]) array(['1', '2', '3', '[]'], dtype='|S4')
and even
array([1,2,"3ef",'wefwfewoiwjefo iwjef']) array(['1', '2', '3ef', 'wefwfewoiwjefo iwjef'], dtype='|S20') _[0]='woeifjwo woie pwioef wliuefh lwieufh wleifuh welfiu ' _ array(['woeifjwo woie pwioef', '2', '3ef', 'wefwfewoiwjefo iwjef'], dtype='|S20')
is rarely what a Pythoneer would expect. Guess fix string arrays should only be created explicitely Robert
participants (5)
-
Alan G Isaac -
Christopher Barker -
Robert -
Robert Kern -
Tim Hochberg