Hello, I am really liking Numpy a lot. It is wonderful to be able to do the things that it does in a language as friendly as Python, and with the performance Numpy delivers over standard Python. Thanks.
I am having a problem with creation of Numpy arrays with my generated dtypes. I am creating a dataset of a weeks worth of financial instruments data, in order to explore and test relationships with various Technical Analysis functions.
My basic data is simply a list (or tuple) of lists (or tuples). ((startdate, bidopen, bidhigh, bidlow, bidclose, askopen, askhigh, asklow, askclose), ...)
Nothing unusual. However I am creating arrays which have many, many more columns to allow storing the data generated from applying the functions to the original data.
I have created two functions. One to dynamically create the dtype based on data I want to create for the exploration. And another to create the array and populate it with the initial data from a database.
Code slightly modified, not tested.
#examples taFunctions = (smva, wmva) inputColumns = (bidclose, ohlcavg)
def createDType(): """Will create a dtype based on the pattern for naming and the parameters of those items being stored in the array. """ dttypes = [('startdate','object'), ('bidopen','f8'), ('bidhigh','f8'), ('bidlow','f8'), ('bidclose','f8'), ('askopen','f8'), ('askhigh','f8'), ('asklow','f8'), ('askclose','f8'), ('ocavg','f8'), ('hlavg','f8'), ('ohlavg','f8'), ('ohlcavg','f8'), ('direction','i1'), ('volatility', 'f8'), ('spread', 'f8'), ('pivot', 'S4')] for f in taFunctions: for i in inputColumns: dttypes.append((f+"-"+i,'f8')) dtminute = np.dtype(dttypes) return dtminute, dttypes
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data) return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
However changing the getArray() function to this works just fine.
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() arrayLength = len(weekData) lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') listLength = len(lst) weekArray = np.zeros(arrayLength, dtype=dtminute) for i in range(arrayLength): for j in range(listLength): if j < 9: weekArray[i][j] = weekData[i][j] else: weekArray[i][j] = lst[j] return weekArray
After I finally worked out getArray number two I am back in business writing the rest of my app. But I banged my head on version number one for quite some time trying to figure out what I am doing wrong. I still don't know.
I find no errors in my data length or types. I would thing that either would cause version two to fail also.
In help in understanding is greatly appreciated.
This is using Numpy 1.4.1.
Thanks.
Jimmie
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchin jlhouchin@gmail.com wrote:
Hello, I am really liking Numpy a lot. It is wonderful to be able to do the things that it does in a language as friendly as Python, and with the performance Numpy delivers over standard Python. Thanks.
I am having a problem with creation of Numpy arrays with my generated dtypes. I am creating a dataset of a weeks worth of financial instruments data, in order to explore and test relationships with various Technical Analysis functions.
My basic data is simply a list (or tuple) of lists (or tuples). ((startdate, bidopen, bidhigh, bidlow, bidclose, askopen, askhigh, asklow, askclose), ...)
Nothing unusual. However I am creating arrays which have many, many more columns to allow storing the data generated from applying the functions to the original data.
I have created two functions. One to dynamically create the dtype based on data I want to create for the exploration. And another to create the array and populate it with the initial data from a database.
Code slightly modified, not tested.
#examples taFunctions = (smva, wmva) inputColumns = (bidclose, ohlcavg)
def createDType(): """Will create a dtype based on the pattern for naming and the parameters of those items being stored in the array. """ dttypes = [('startdate','object'), ('bidopen','f8'), ('bidhigh','f8'), ('bidlow','f8'), ('bidclose','f8'), ('askopen','f8'), ('askhigh','f8'), ('asklow','f8'), ('askclose','f8'), ('ocavg','f8'), ('hlavg','f8'), ('ohlavg','f8'), ('ohlcavg','f8'), ('direction','i1'), ('volatility', 'f8'), ('spread', 'f8'), ('pivot', 'S4')] for f in taFunctions: for i in inputColumns: dttypes.append((f+"-"+i,'f8')) dtminute = np.dtype(dttypes) return dtminute, dttypes
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
However changing the getArray() function to this works just fine.
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() arrayLength = len(weekData) lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') listLength = len(lst) weekArray = np.zeros(arrayLength, dtype=dtminute) for i in range(arrayLength): for j in range(listLength): if j < 9: weekArray[i][j] = weekData[i][j] else: weekArray[i][j] = lst[j] return weekArray
After I finally worked out getArray number two I am back in business writing the rest of my app. But I banged my head on version number one for quite some time trying to figure out what I am doing wrong. I still don't know.
I find no errors in my data length or types. I would thing that either would cause version two to fail also.
In help in understanding is greatly appreciated.
This is using Numpy 1.4.1.
Thanks.
Jimmie
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On 5/15/2010 6:30 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchinjlhouchin@gmail.com wrote:
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
Hello Josef,
Wrapping data, tuple(list(m)+lst[9:]) works.
Thanks.
For some reason I was under the impression that numpy accepted either lists or tuples as long as the shape of the structure, and the data types was the same as the dtype array structure that it is filling. Is there a particular reason this is not so?
Again, thanks. I can now get rid of my moderately less elegant, but working second version.
Jimmie
On Sat, May 15, 2010 at 9:27 AM, Jimmie Houchin jlhouchin@gmail.com wrote:
On 5/15/2010 6:30 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchinjlhouchin@gmail.com wrote:
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)
The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
Hello Josef,
Wrapping data, tuple(list(m)+lst[9:]) works.
Thanks.
For some reason I was under the impression that numpy accepted either lists or tuples as long as the shape of the structure, and the data types was the same as the dtype array structure that it is filling. Is there a particular reason this is not so?
the tuple (row) is one element of the structured array. It's possible to have an n-dimensional structured array where each element is a tuple.
So, I guess, numpy needs the distinction between list and tuples to know what is an element. That's from hitting at this very often, I never looked at the numpy internals for this.
Josef
Again, thanks. I can now get rid of my moderately less elegant, but working second version.
Jimmie
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
the tuple (row) is one element of the structured array. It's possible to have an n-dimensional structured array where each element is a tuple.
Also just was looking at this and while you can't do this anarray = np.array([1,2,3], dtype = [('num', int)]) you can anarray = np.array([(1,),(2,),(3,)], dtype = [('num', int)])
Vincent
On Sat, May 15, 2010 at 7:37 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 9:27 AM, Jimmie Houchin jlhouchin@gmail.com wrote:
On 5/15/2010 6:30 AM, josef.pktd@gmail.com wrote:
On Sat, May 15, 2010 at 12:24 AM, Jimmie Houchinjlhouchin@gmail.com
wrote:
def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data)
I think "data" here should be a tuple, i.e. tuple(data) structured arrays expect tuples for each element/row
If this is not it, then you could provide a mini example of wdata with just a few rows.
return np.array(wdata,dtype=dtminute)The createDType() function works fine. The getArray() function fails
with:
ValueError: Setting void-array with object members using buffer.
cryptic exceptions messages in array construction usually means there is some structure in the argument data that numpy doesn't understand, I usually work with trial and error for a specific example
Josef
Hello Josef,
Wrapping data, tuple(list(m)+lst[9:]) works.
Thanks.
For some reason I was under the impression that numpy accepted either lists or tuples as long as the shape of the structure, and the data types was the same as the dtype array structure that it is filling. Is there a particular reason this is not so?
the tuple (row) is one element of the structured array. It's possible to have an n-dimensional structured array where each element is a tuple.
So, I guess, numpy needs the distinction between list and tuples to know what is an element. That's from hitting at this very often, I never looked at the numpy internals for this.
Josef
Again, thanks. I can now get rid of my moderately less elegant, but working second version.
Jimmie
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
*Vincent Davis 720-301-3003 * vincent@vincentdavis.net my blog http://vincentdavis.net | LinkedInhttp://www.linkedin.com/in/vincentdavis
On Sat, May 15, 2010 at 08:37, josef.pktd@gmail.com wrote:
So, I guess, numpy needs the distinction between list and tuples to know what is an element. That's from hitting at this very often, I never looked at the numpy internals for this.
This is correct. There is only so much mind-reading that numpy.array() can do.
participants (4)
-
Jimmie Houchin -
josef.pktd＠gmail.com -
Robert Kern -
Vincent Davis