List with numpy semantics
Hello,
I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array.
My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs.
Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe?
Best,
-Nikolaus
On 31 October 2010 17:10, Nikolaus Rath Nikolaus@rath.org wrote:
Hello,
I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array.
My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs.
Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe?
You could try a record array with a clever dtype, maybe?
Gerrit.
Gerrit Holl gerrit.holl@gmail.com writes:
On 31 October 2010 17:10, Nikolaus Rath Nikolaus@rath.org wrote:
Hello,
I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array.
My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs.
Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe?
You could try a record array with a clever dtype, maybe?
It seems that this requires more cleverness than I have... Could you give me an example? How do I replace l in the following code with a record array?
l = list() l.append(np.arange(3)) l.append(np.arange(42)) l.append(np.arange(9))
for i in range(len(l)): l[i] += 32
Thanks,
-Nikolaus
On Tue, Nov 2, 2010 at 10:02 PM, Nikolaus Rath Nikolaus@rath.org wrote:
Gerrit Holl gerrit.holl@gmail.com writes:
On 31 October 2010 17:10, Nikolaus Rath Nikolaus@rath.org wrote:
Hello,
I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array.
My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs.
Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe?
You could try a record array with a clever dtype, maybe?
It seems that this requires more cleverness than I have... Could you give me an example? How do I replace l in the following code with a record array?
l = list() l.append(np.arange(3)) l.append(np.arange(42)) l.append(np.arange(9))
for i in range(len(l)): l[i] += 32
Depending on how you want to use it, it might be more convenient to use masked arrays or fill with nan (like pandas and larry) to get a rectangular array. it might be more convenient for some things, but if the sizes differ a lot then it might not be more efficient.
Josef
Thanks,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Nov 2, 2010 at 10:21 PM, josef.pktd@gmail.com wrote:
On Tue, Nov 2, 2010 at 10:02 PM, Nikolaus Rath Nikolaus@rath.org wrote:
Gerrit Holl gerrit.holl@gmail.com writes:
On 31 October 2010 17:10, Nikolaus Rath Nikolaus@rath.org wrote:
Hello,
I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array.
My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs.
Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe?
You could try a record array with a clever dtype, maybe?
It seems that this requires more cleverness than I have... Could you give me an example? How do I replace l in the following code with a record array?
l = list() l.append(np.arange(3)) l.append(np.arange(42)) l.append(np.arange(9))
for i in range(len(l)): l[i] += 32
Depending on how you want to use it, it might be more convenient to use masked arrays or fill with nan (like pandas and larry) to get a rectangular array. it might be more convenient for some things, but if the sizes differ a lot then it might not be more efficient.
another option I sometimes use (e.g. for unbalanced panel data), is to just stack them on top of each other into one long 1d array, and keep track which is which, e.g. keeping the (start-end) indices or using an indicator array. For example, with an integer label array np.bincount is very fast to work with it. This is mainly an advantage if there are many short arrays and many operations have to applied to all of them.
Josef
Josef
Thanks,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Nov 2, 2010 at 10:31 PM, josef.pktd@gmail.com wrote:
On Tue, Nov 2, 2010 at 10:21 PM, josef.pktd@gmail.com wrote:
On Tue, Nov 2, 2010 at 10:02 PM, Nikolaus Rath Nikolaus@rath.org wrote:
Gerrit Holl gerrit.holl@gmail.com writes:
On 31 October 2010 17:10, Nikolaus Rath Nikolaus@rath.org wrote:
Hello,
I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array.
My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs.
Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe?
You could try a record array with a clever dtype, maybe?
It seems that this requires more cleverness than I have... Could you give me an example? How do I replace l in the following code with a record array?
l = list() l.append(np.arange(3)) l.append(np.arange(42)) l.append(np.arange(9))
for i in range(len(l)): l[i] += 32
Depending on how you want to use it, it might be more convenient to use masked arrays or fill with nan (like pandas and larry) to get a rectangular array. it might be more convenient for some things, but if the sizes differ a lot then it might not be more efficient.
another option I sometimes use (e.g. for unbalanced panel data), is to just stack them on top of each other into one long 1d array, and keep track which is which, e.g. keeping the (start-end) indices or using an indicator array. For example, with an integer label array np.bincount is very fast to work with it. This is mainly an advantage if there are many short arrays and many operations have to applied to all of them.
And the third option, also often used with panel data, is to stack them sparse, like an unbalanced kronecker product, on top but next to each other and fill the empty space with zeros.
Josef (typing is faster than thinking)
Josef
Josef
Thanks,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (3)
-
Gerrit Holl -
josef.pktd＠gmail.com -
Nikolaus Rath