numpy sort is not working

Hi everyone,
I have a numpy array of dimensions
allRics.shape
(583760, 1)
To sort the array, I set the dtype of the array as follows:
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value' ,np.float)]
allRics.dtype
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])
I checked and the endianness in dtype is correct.
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on 'time'. I do the following.
x=np.sort(allRics,order='time')
x[17330:17350]['time']
array([[ 61184.4 ],
[ 61188.51 ],
[ 61188.979],
[ 61188.979],
[ 61189.989],
[ 61191.66 ],
[ 61194.35 ],
[ 61194.35 ],
[ 61198.79 ],
[ 61198.145],
[ 36126.217],
[ 36126.217],
[ 36126.218],
[ 36126.218],
[ 36126.219],
[ 36126.271],
[ 36126.271],
[ 36126.271],
[ 36126.293],
[ 36126.293]])
Time column doesn't change its order. Could someone please advise what is missing here? Is this related to the bug
http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).
Regards,
Alok
Alok Jadhav
CREDIT SUISSE AG
GAT IT Hong Kong, KVAG 67
International Commerce Centre | Hong Kong | Hong Kong
Phone +852 2101 6274 | Mobile +852 9169 7172
alok.jadhav@credit-suisse.com | www.credit-suisse.com http://www.credit-suisse.com/
=============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================

Hey Alok,
This is worth taking a look. What version of NumPy are you using?
It is not related directly to the issue you referenced as that was an endian-ness issue and your data is native-order.
Your example seems to work for me (with a simulated case on 1.6.1)
Best,
-Travis
On Sep 10, 2012, at 10:46 PM, Jadhav, Alok wrote:
Hi everyone,
I have a numpy array of dimensions
allRics.shape
(583760, 1)
To sort the array, I set the dtype of the array as follows:
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value',np.float)]
allRics.dtype
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])
I checked and the endianness in dtype is correct.
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on ‘time’. I do the following.
x=np.sort(allRics,order='time') x[17330:17350]['time']
array([[ 61184.4 ], [ 61188.51 ], [ 61188.979], [ 61188.979], [ 61189.989], [ 61191.66 ], [ 61194.35 ], [ 61194.35 ], [ 61198.79 ], [ 61198.145], [ 36126.217], [ 36126.217], [ 36126.218], [ 36126.218], [ 36126.219], [ 36126.271], [ 36126.271], [ 36126.271], [ 36126.293], [ 36126.293]])
Time column doesn’t change its order. Could someone please advise what is missing here? Is this related to the bug http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).
Regards, Alok
Alok Jadhav CREDIT SUISSE AG GAT IT Hong Kong, KVAG 67 International Commerce Centre | Hong Kong | Hong Kong Phone +852 2101 6274 | Mobile +852 9169 7172 alok.jadhav@credit-suisse.com | www.credit-suisse.com
============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ==============================================================================
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi Travis,
Very Strange. I am on version 1.6.2 L
What could I be missing. I started using numpy quite recently. Is there a way to share the data with you?
Regards,
Alok Jadhav
GAT IT Hong Kong
+852 2101 6274 (*852 6274)
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Travis Oliphant Sent: Tuesday, September 11, 2012 11:58 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy sort is not working
Hey Alok,
This is worth taking a look. What version of NumPy are you using?
It is not related directly to the issue you referenced as that was an endian-ness issue and your data is native-order.
Your example seems to work for me (with a simulated case on 1.6.1)
Best,
-Travis
On Sep 10, 2012, at 10:46 PM, Jadhav, Alok wrote:
Hi everyone,
I have a numpy array of dimensions
allRics.shape
(583760, 1)
To sort the array, I set the dtype of the array as follows:
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value' ,np.float)]
allRics.dtype
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])
I checked and the endianness in dtype is correct.
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on 'time'. I do the following.
x=np.sort(allRics,order='time')
x[17330:17350]['time']
array([[ 61184.4 ],
[ 61188.51 ],
[ 61188.979],
[ 61188.979],
[ 61189.989],
[ 61191.66 ],
[ 61194.35 ],
[ 61194.35 ],
[ 61198.79 ],
[ 61198.145],
[ 36126.217],
[ 36126.217],
[ 36126.218],
[ 36126.218],
[ 36126.219],
[ 36126.271],
[ 36126.271],
[ 36126.271],
[ 36126.293],
[ 36126.293]])
Time column doesn't change its order. Could someone please advise what is missing here? Is this related to the bug
http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).
Regards,
Alok
Alok Jadhav
CREDIT SUISSE AG
GAT IT Hong Kong, KVAG 67
International Commerce Centre | Hong Kong | Hong Kong
Phone +852 2101 6274 | Mobile +852 9169 7172
alok.jadhav@credit-suisse.com | www.credit-suisse.com http://www.credit-suisse.com/
======================================================================== ====== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ======================================================================== ======
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
=============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================

On Tue, Sep 11, 2012 at 12:52 AM, Jadhav, Alok < alok.jadhav@credit-suisse.com> wrote:
Hi Travis,****
Very Strange. I am on version 1.6.2 L****
What could I be missing. I started using numpy quite recently. Is there a way to share the data with you?
Hi Alok,
Typically, a self-contained example which reproduces the error and generates its own data is more useful / valuable than sharing datasets. if you could come up with this, that'd be great!
Be Well Anthony
Regards,****
Alok Jadhav****
GAT IT Hong Kong****
+852 2101 6274 (*852 6274)****
*From:* numpy-discussion-bounces@scipy.org [mailto: numpy-discussion-bounces@scipy.org] *On Behalf Of *Travis Oliphant *Sent:* Tuesday, September 11, 2012 11:58 AM *To:* Discussion of Numerical Python *Subject:* Re: [Numpy-discussion] numpy sort is not working****
Hey Alok, ****
This is worth taking a look. What version of NumPy are you using? ****
It is not related directly to the issue you referenced as that was an endian-ness issue and your data is native-order. ****
Your example seems to work for me (with a simulated case on 1.6.1)****
Best,****
-Travis****
On Sep 10, 2012, at 10:46 PM, Jadhav, Alok wrote:****
Hi everyone,****
I have a numpy array of dimensions****
allRics.shape
(583760, 1) ****
To sort the array, I set the dtype of the array as follows:****
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np. float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value',np. float)] ****
allRics.dtype****
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])****
I checked and the endianness in dtype is correct.****
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on ‘time’. I do the following.****
x=np.sort(allRics,order='time')****
x[17330:17350]['time']****
array([[ 61184.4 ],****
[ 61188.51 ],**** [ 61188.979],**** [ 61188.979],**** [ 61189.989],**** [ 61191.66 ],**** [ 61194.35 ],**** [ 61194.35 ],**** [ 61198.79 ],**** [ 61198.145],**** [ 36126.217],**** [ 36126.217],**** [ 36126.218],**** [ 36126.218],**** [ 36126.219],**** [ 36126.271],**** [ 36126.271],**** [ 36126.271],**** [ 36126.293],**** [ 36126.293]])****
Time column doesn’t change its order. Could someone please advise what is missing here? Is this related to the bug****
http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).****
Regards,****
Alok****
Alok Jadhav****
*CREDIT SUISSE AG*****
GAT IT Hong Kong, KVAG 67****
International Commerce Centre | Hong Kong | Hong Kong****
Phone +852 2101 6274 | Mobile +852 9169 7172****
alok.jadhav@credit-suisse.com | www.credit-suisse.com****
============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion****
**
============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Anthony, Travis,
I understand how an example that generates the data would be useful but it will be difficult to provide this in my case for following reasons
- Data is read from hdf5 files (>50 MB)
- Attaching the code which is used to generate the data. It may provide some more light.
- numpy.sort() example on the site for small array works fine. So my guess is that only my data is having issue.
- Fyi, I am on a windows 7 machine with python 2.6 and numpy 1.6.2
for j in range(0,len(rics)):
ric=rics[j]
trd=h5.getTrades(ric)
idx=np.ones(len(trd))*j
opened=np.zeros(len(trd))
time=trd[0:,0]
trdp1=trd[0:,1]
trdp0=np.insert(trd[1:,1], 0, basePx[ric])
dt=trd[0:,0]- np.insert(trd[0:-1,0], 0, trd[0,0])
value=trd[0:,1]*trd[0:,2]
ricData=np.array([idx,opened,time,trdp1,trdp0,dt,value])
ricData=np.transpose(ricData)
if allRics is None:
allRics=ricData
else:
allRics=np.vstack((allRics, ricData))
allRics.dtype=[('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value' ,np.float)]
allRics=np.sort(allRics,order='time') # This doesn't work
Please notice that I am using vstack to generate the array.
I just found out that I am able to sort numpy array before I set the dtype in following crude way
allRics=allRics[allRics[:,2].argsort()] # This works
I am able to continue with my code right now but not sure why structured array could not be sorted.
Alok Jadhav
GAT IT Hong Kong
+852 2101 6274 (*852 6274)
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Anthony Scopatz Sent: Tuesday, September 11, 2012 2:04 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy sort is not working
On Tue, Sep 11, 2012 at 12:52 AM, Jadhav, Alok alok.jadhav@credit-suisse.com wrote:
Hi Travis,
Very Strange. I am on version 1.6.2 L
What could I be missing. I started using numpy quite recently. Is there a way to share the data with you?
Hi Alok,
Typically, a self-contained example which reproduces the error and generates its own data is more useful / valuable than sharing datasets. if you could come up with this, that'd be great!
Be Well
Anthony
Regards,
Alok Jadhav
GAT IT Hong Kong
+852 2101 6274 tel:%2B852%202101%206274 (*852 6274)
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Travis Oliphant Sent: Tuesday, September 11, 2012 11:58 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy sort is not working
Hey Alok,
This is worth taking a look. What version of NumPy are you using?
It is not related directly to the issue you referenced as that was an endian-ness issue and your data is native-order.
Your example seems to work for me (with a simulated case on 1.6.1)
Best,
-Travis
On Sep 10, 2012, at 10:46 PM, Jadhav, Alok wrote:
Hi everyone,
I have a numpy array of dimensions
>>> allRics.shape (583760, 1)
To sort the array, I set the dtype of the array as follows:
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value' ,np.float)]
>>> allRics.dtype
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])
I checked and the endianness in dtype is correct.
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on 'time'. I do the following.
>>> x=np.sort(allRics,order='time')
>>> x[17330:17350]['time']
array([[ 61184.4 ],
[ 61188.51 ],
[ 61188.979],
[ 61188.979],
[ 61189.989],
[ 61191.66 ],
[ 61194.35 ],
[ 61194.35 ],
[ 61198.79 ],
[ 61198.145],
[ 36126.217],
[ 36126.217],
[ 36126.218],
[ 36126.218],
[ 36126.219],
[ 36126.271],
[ 36126.271],
[ 36126.271],
[ 36126.293],
[ 36126.293]])
Time column doesn't change its order. Could someone please advise what is missing here? Is this related to the bug
http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).
Regards,
Alok
Alok Jadhav
CREDIT SUISSE AG
GAT IT Hong Kong, KVAG 67
International Commerce Centre | Hong Kong | Hong Kong
Phone +852 2101 6274 tel:%2B852%202101%206274 | Mobile +852 9169 7172 tel:%2B852%209169%207172
alok.jadhav@credit-suisse.com | www.credit-suisse.com http://www.credit-suisse.com/
======================================================================== ====== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ======================================================================== ======
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
======================================================================== ====== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ======================================================================== ======
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
=============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================

I am wondering of this has to do with the size of the array.
It looks like the array is sorted --- but in chunks.
-- Travis Oliphant (on a mobile) 512-826-7480
On Sep 10, 2012, at 10:46 PM, "Jadhav, Alok" alok.jadhav@credit-suisse.com wrote:
Hi everyone,
I have a numpy array of dimensions
allRics.shape
(583760, 1)
To sort the array, I set the dtype of the array as follows:
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value',np.float)]
allRics.dtype
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])
I checked and the endianness in dtype is correct.
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on ‘time’. I do the following.
x=np.sort(allRics,order='time') x[17330:17350]['time']
array([[ 61184.4 ], [ 61188.51 ], [ 61188.979], [ 61188.979], [ 61189.989], [ 61191.66 ], [ 61194.35 ], [ 61194.35 ], [ 61198.79 ], [ 61198.145], [ 36126.217], [ 36126.217], [ 36126.218], [ 36126.218], [ 36126.219], [ 36126.271], [ 36126.271], [ 36126.271], [ 36126.293], [ 36126.293]])
Time column doesn’t change its order. Could someone please advise what is missing here? Is this related to the bug http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).
Regards, Alok
Alok Jadhav CREDIT SUISSE AG GAT IT Hong Kong, KVAG 67 International Commerce Centre | Hong Kong | Hong Kong Phone +852 2101 6274 | Mobile +852 9169 7172 alok.jadhav@credit-suisse.com | www.credit-suisse.com
============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ==============================================================================
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

The sorted array you see is the same as original array. I have replied to code to generate the below data. (done in a loop. Each loop generates sorted numpy arrayas it reads from file) and combines all arrays into a single numpy array. I need to sort the final array into a single sorted array. It could be because of array size. It maybe silently failing somewhere? I don’t see any error, but output is not sorted. Array sorting works fine if the array is not structured.
Alok Jadhav
GAT IT Hong Kong
+852 2101 6274 (*852 6274)
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Travis Oliphant Sent: Tuesday, September 11, 2012 10:07 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy sort is not working
I am wondering of this has to do with the size of the array.
It looks like the array is sorted --- but in chunks.
--
Travis Oliphant
(on a mobile)
512-826-7480
On Sep 10, 2012, at 10:46 PM, "Jadhav, Alok" alok.jadhav@credit-suisse.com wrote:
Hi everyone,
I have a numpy array of dimensions
>>> allRics.shape (583760, 1)
To sort the array, I set the dtype of the array as follows:
allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value',np.float)]
>>> allRics.dtype
dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')])
I checked and the endianness in dtype is correct.
When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on ‘time’. I do the following.
>>> x=np.sort(allRics,order='time')
>>> x[17330:17350]['time']
array([[ 61184.4 ],
[ 61188.51 ],
[ 61188.979],
[ 61188.979],
[ 61189.989],
[ 61191.66 ],
[ 61194.35 ],
[ 61194.35 ],
[ 61198.79 ],
[ 61198.145],
[ 36126.217],
[ 36126.217],
[ 36126.218],
[ 36126.218],
[ 36126.219],
[ 36126.271],
[ 36126.271],
[ 36126.271],
[ 36126.293],
[ 36126.293]])
Time column doesn’t change its order. Could someone please advise what is missing here? Is this related to the bug
http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010).
Regards,
Alok
Alok Jadhav
CREDIT SUISSE AG
GAT IT Hong Kong, KVAG 67
International Commerce Centre | Hong Kong | Hong Kong
Phone +852 2101 6274 | Mobile +852 9169 7172
alok.jadhav@credit-suisse.com | www.credit-suisse.com http://www.credit-suisse.com/
============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ==============================================================================
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
=============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
participants (3)
-
Anthony Scopatz
-
Jadhav, Alok
-
Travis Oliphant