Hello, I am really liking Numpy a lot. It is wonderful to be able to do the things that it does in a language as friendly as Python, and with the performance Numpy delivers over standard Python. Thanks. I am having a problem with creation of Numpy arrays with my generated dtypes. I am creating a dataset of a weeks worth of financial instruments data, in order to explore and test relationships with various Technical Analysis functions. My basic data is simply a list (or tuple) of lists (or tuples). ((startdate, bidopen, bidhigh, bidlow, bidclose, askopen, askhigh, asklow, askclose), ...) Nothing unusual. However I am creating arrays which have many, many more columns to allow storing the data generated from applying the functions to the original data. I have created two functions. One to dynamically create the dtype based on data I want to create for the exploration. And another to create the array and populate it with the initial data from a database. Code slightly modified, not tested. #examples taFunctions = (smva, wmva) inputColumns = (bidclose, ohlcavg) def createDType(): """Will create a dtype based on the pattern for naming and the parameters of those items being stored in the array. """ dttypes = [('startdate','object'), ('bidopen','f8'), ('bidhigh','f8'), ('bidlow','f8'), ('bidclose','f8'), ('askopen','f8'), ('askhigh','f8'), ('asklow','f8'), ('askclose','f8'), ('ocavg','f8'), ('hlavg','f8'), ('ohlavg','f8'), ('ohlcavg','f8'), ('direction','i1'), ('volatility', 'f8'), ('spread', 'f8'), ('pivot', 'S4')] for f in taFunctions: for i in inputColumns: dttypes.append((f+"-"+i,'f8')) dtminute = np.dtype(dttypes) return dtminute, dttypes def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data) return np.array(wdata,dtype=dtminute) The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer. However changing the getArray() function to this works just fine. def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() arrayLength = len(weekData) lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') listLength = len(lst) weekArray = np.zeros(arrayLength, dtype=dtminute) for i in range(arrayLength): for j in range(listLength): if j < 9: weekArray[i][j] = weekData[i][j] else: weekArray[i][j] = lst[j] return weekArray After I finally worked out getArray number two I am back in business writing the rest of my app. But I banged my head on version number one for quite some time trying to figure out what I am doing wrong. I still don't know. I find no errors in my data length or types. I would thing that either would cause version two to fail also. In help in understanding is greatly appreciated. This is using Numpy 1.4.1. Thanks. Jimmie
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchin <jlhouchin@gmail.com> wrote:
Hello, I am really liking Numpy a lot. It is wonderful to be able to do the things that it does in a language as friendly as Python, and with the performance Numpy delivers over standard Python. Thanks.
I am having a problem with creation of Numpy arrays with my generated dtypes. I am creating a dataset of a weeks worth of financial instruments data, in order to explore and test relationships with various Technical Analysis functions.
My basic data is simply a list (or tuple) of lists (or tuples). ((startdate, bidopen, bidhigh, bidlow, bidclose, askopen, askhigh, asklow, askclose), ...)
Nothing unusual. However I am creating arrays which have many, many more columns to allow storing the data generated from applying the functions to the original data.
I have created two functions. One to dynamically create the dtype based on data I want to create for the exploration. And another to create the array and populate it with the initial data from a database.
Code slightly modified, not tested.
#examples taFunctions = (smva, wmva) inputColumns = (bidclose, ohlcavg)
def createDType(): """Will create a dtype based on the pattern for naming and the parameters of those items being stored in the array. """ dttypes = [('startdate','object'), ('bidopen','f8'), ('bidhigh','f8'), ('bidlow','f8'), ('bidclose','f8'), ('askopen','f8'), ('askhigh','f8'), ('asklow','f8'), ('askclose','f8'), ('ocavg','f8'), ('hlavg','f8'), ('ohlavg','f8'), ('ohlcavg','f8'), ('direction','i1'), ('volatility', 'f8'), ('spread', 'f8'), ('pivot', 'S4')] for f in taFunctions: for i in inputColumns: dttypes.append((f+"-"+i,'f8')) dtminute = np.dtype(dttypes) return dtminute, dttypes
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example Josef
However changing the getArray() function to this works just fine.
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() arrayLength = len(weekData) lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') listLength = len(lst) weekArray = np.zeros(arrayLength, dtype=dtminute) for i in range(arrayLength): for j in range(listLength): if j < 9: weekArray[i][j] = weekData[i][j] else: weekArray[i][j] = lst[j] return weekArray
After I finally worked out getArray number two I am back in business writing the rest of my app. But I banged my head on version number one for quite some time trying to figure out what I am doing wrong. I still don't know.
I find no errors in my data length or types. I would thing that either would cause version two to fail also.
In help in understanding is greatly appreciated.
This is using Numpy 1.4.1.
Thanks.
Jimmie
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On 5/15/2010 6:30 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchin<jlhouchin@gmail.com> wrote:
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
Hello Josef, Wrapping data, tuple(list(m)+lst[9:]) works. Thanks. For some reason I was under the impression that numpy accepted either lists or tuples as long as the shape of the structure, and the data types was the same as the dtype array structure that it is filling. Is there a particular reason this is not so? Again, thanks. I can now get rid of my moderately less elegant, but working second version. Jimmie
On Sat, May 15, 2010 at 9:27 AM, Jimmie Houchin <jlhouchin@gmail.com> wrote:
On 5/15/2010 6:30 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchin<jlhouchin@gmail.com> wrote:
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
Hello Josef,
Wrapping data, tuple(list(m)+lst[9:]) works.
Thanks.
For some reason I was under the impression that numpy accepted either lists or tuples as long as the shape of the structure, and the data types was the same as the dtype array structure that it is filling. Is there a particular reason this is not so?
the tuple (row) is one element of the structured array. It's possible to have an n-dimensional structured array where each element is a tuple. So, I guess, numpy needs the distinction between list and tuples to know what is an element. That's from hitting at this very often, I never looked at the numpy internals for this. Josef
Again, thanks. I can now get rid of my moderately less elegant, but working second version.
Jimmie
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
the tuple (row) is one element of the structured array. It's possible to have an n-dimensional structured array where each element is a tuple.
Also just was looking at this and while you can't do this anarray = np.array([1,2,3], dtype = [('num', int)]) you can anarray = np.array([(1,),(2,),(3,)], dtype = [('num', int)]) Vincent On Sat, May 15, 2010 at 7:37 AM, <josef.pktd@gmail.com> wrote:
On Sat, May 15, 2010 at 9:27 AM, Jimmie Houchin <jlhouchin@gmail.com> wrote:
On 5/15/2010 6:30 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchin<jlhouchin@gmail.com> wrote:
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails
with:
ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
Hello Josef,
Wrapping data, tuple(list(m)+lst[9:]) works.
Thanks.
For some reason I was under the impression that numpy accepted either lists or tuples as long as the shape of the structure, and the data types was the same as the dtype array structure that it is filling. Is there a particular reason this is not so?
the tuple (row) is one element of the structured array. It's possible to have an n-dimensional structured array where each element is a tuple.
So, I guess, numpy needs the distinction between list and tuples to know what is an element. That's from hitting at this very often, I never looked at the numpy internals for this.
Josef
Again, thanks. I can now get rid of my moderately less elegant, but working second version.
Jimmie
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
*Vincent Davis 720-301-3003 * vincent@vincentdavis.net my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
On Sat, May 15, 2010 at 08:37, <josef.pktd@gmail.com> wrote:
So, I guess, numpy needs the distinction between list and tuples to know what is an element. That's from hitting at this very often, I never looked at the numpy internals for this.
This is correct. There is only so much mind-reading that numpy.array() can do. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
participants (4)
-
Jimmie Houchin -
josef.pktd＠gmail.com -
Robert Kern -
Vincent Davis