Quick array value assignment based on common values
Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul
Are they numbered like that? If so you can index into the first array by the second one. x[y[:,0], 1] if you can't get them into an indexable format, I think it's going to be slow no matter how you do it. On Wed, Aug 4, 2010 at 4:59 PM, <PHobson@geosyntec.com> wrote:
Hey folks,
I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]]
Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]]
Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]]
My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately.
Many thanks, -paul
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
John, Thanks for the quick reply. Unfortunately, no, they're not indexed like that. The first columns are actually floating-point date numbers from matplotlib.dates.date2num. Looks like this is just going to be painful... Thanks for the tip though. That'll definitely be useful elsewhere. -paul From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of John Salvatier Sent: Wednesday, August 04, 2010 5:34 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Quick array value assignment based on common values Are they numbered like that? If so you can index into the first array by the second one. x[y[:,0], 1] if you can't get them into an indexable format, I think it's going to be slow no matter how you do it. On Wed, Aug 4, 2010 at 4:59 PM, <PHobson@geosyntec.com<mailto:PHobson@geosyntec.com>> wrote: Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org<mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z:
First use mask = (x[:,0] == y[:,0]) # integers or mask = np.abs(x[:,0] - y[:,0]) < eps # floats and then y[mask,1] = x[mask,1] Sturla
On Wed, Aug 4, 2010 at 6:59 PM, <PHobson@geosyntec.com> wrote:
Hey folks,
I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]]
Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]]
Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]]
My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately.
Many thanks, -paul
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
My simplest approach would be: y[x[:0]-1] = x # Providing the arrays are nicely ordered and 1st column x is all integer. -- Gökhan
On Wed, Aug 4, 2010 at 8:00 PM, Gökhan Sever <gokhansever@gmail.com> wrote:
On Wed, Aug 4, 2010 at 6:59 PM, <PHobson@geosyntec.com> wrote:
Hey folks,
I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]]
Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]]
Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]]
My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately.
Many thanks, -paul
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
My simplest approach would be:
y[x[:0]-1] = x
# Providing the arrays are nicely ordered and 1st column x is all integer.
-- Gökhan
With the forgotten comma ;) y[x[:,0]-1] = x -- Gökhan
How exactly are you looping? That sounds absurdly slow. What you need is a fast dictionary. On Wed, Aug 4, 2010 at 6:00 PM, Gökhan Sever <gokhansever@gmail.com> wrote:
On Wed, Aug 4, 2010 at 6:59 PM, <PHobson@geosyntec.com> wrote:
Hey folks,
I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]]
Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]]
Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]]
My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately.
Many thanks, -paul
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
My simplest approach would be:
y[x[:0]-1] = x
# Providing the arrays are nicely ordered and 1st column x is all integer.
-- Gökhan
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Perhaps try the following: 1) sort x by x[:,0] 2) sort y by y[:,0] 3) loop through both at the same time building an array of indexes A that tells you the index of y[i,0] in x or just building a new array z with the value if you don't need them in order 4) if you do need them in order, unsort A by the sorting used to sort y and then index into x using the unsorted A. use http://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html#numpy... On Wed, Aug 4, 2010 at 6:09 PM, John Salvatier <jsalvati@u.washington.edu>wrote:
How exactly are you looping? That sounds absurdly slow.
What you need is a fast dictionary.
On Wed, Aug 4, 2010 at 6:00 PM, Gökhan Sever <gokhansever@gmail.com>wrote:
On Wed, Aug 4, 2010 at 6:59 PM, <PHobson@geosyntec.com> wrote:
Hey folks,
I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]]
Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]]
Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]]
My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately.
Many thanks, -paul
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
My simplest approach would be:
y[x[:0]-1] = x
# Providing the arrays are nicely ordered and 1st column x is all integer.
-- Gökhan
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
I've deleted the code b/c it was absurdly slow. It was pretty brute-force. -Looped through each row (r) of y -check to see where y[r,0] - x[:,0] < eps (call that row r_hit) -set y[r,1] = x[r_hit,1] There was kind of a short fuse on this, and I was already reading the data from a text file. So I just wrote all of the continuous dates to a file, threw it in a spreadsheet, brought in the other data and did a lookup function that about destroyed my machine. Probably would have been faster to let the looping run [hangs head in shame]. In the future, I'll definitely try the solutions you've outlined. Thanks again! -paul From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of John Salvatier Sent: Wednesday, August 04, 2010 6:23 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Quick array value assignment based on common values Perhaps try the following: 1) sort x by x[:,0] 2) sort y by y[:,0] 3) loop through both at the same time building an array of indexes A that tells you the index of y[i,0] in x or just building a new array z with the value if you don't need them in order 4) if you do need them in order, unsort A by the sorting used to sort y and then index into x using the unsorted A. use http://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html#numpy... On Wed, Aug 4, 2010 at 6:09 PM, John Salvatier <jsalvati@u.washington.edu<mailto:jsalvati@u.washington.edu>> wrote: How exactly are you looping? That sounds absurdly slow. What you need is a fast dictionary. On Wed, Aug 4, 2010 at 6:00 PM, Gökhan Sever <gokhansever@gmail.com<mailto:gokhansever@gmail.com>> wrote: On Wed, Aug 4, 2010 at 6:59 PM, <PHobson@geosyntec.com<mailto:PHobson@geosyntec.com>> wrote: Hey folks, I've one array, x, that you could define as follows: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [8, 0.00], [9, 2.75]] Then my second array, y, is: [[1, 0.00], [2, 0.00], [3, 0.00], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 0.00], [10,0.00]] Is there a concise, Numpythonic way to copy the values of x[:,1] over to y[:,1] where x[:,0] = y[:,0]? Resulting in, z: [[1, 2.25], [2, 2.50], [3, 2.25], [4, 0.00], [5, 0.00], [6, 0.00], [7, 0.00], [8, 0.00], [9, 2.75], [10,0.00]] My current task has len(x) = 25000 and len(y) = 350000 and looping through is quite slow unfortunately. Many thanks, -paul _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org<mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion My simplest approach would be: y[x[:0]-1] = x # Providing the arrays are nicely ordered and 1st column x is all integer. -- Gökhan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org<mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (4)
-
Gökhan Sever -
John Salvatier -
PHobson＠Geosyntec.com -
Sturla Molden