On Wed, 2014-01-22 at 07:58 +0100, Dr. Leo wrote:

Hi,

thanks. Both recarray and itertools.chain work just fine in the example case.

However, the real purpose of this is to read strings from a large xml file into a pandas DataFrame. But fromiter cannot create arrays of dtype 'object'. Fixed length strings may be worth trying. But as the xml schema does not guarantee a max. length, and pandas generally uses 'object' arrays for strings, I see no better way than creating the array through list comprehensions and turn it into a DataFrame.

If your datatype is object, I doubt that using an intermediate list is a real overhead, since the list will use much less memory then the string objects anyway.

- Sebastian

Maybe a variable length string/unicode type would help in the long term.

Leo

...

I would like to write something like:

In [25]: iterable=((i, i**2) for i in range(10))

In [26]: a=np.fromiter(iterable, int32)

ValueError Traceback (most recent call last) <ipython-input-26-5bcc2e94dbca> in <module>() ----> 1 a=np.fromiter(iterable, int32)

ValueError: setting an array element with a sequence.

Is there an efficient way to do this?

Perhaps you could just utilize structured arrays ( http://docs.scipy.org/doc/numpy/user/basics.rec.html), like: iterable= ((i, i**2) for i in range(10)) a= np.fromiter(iterable, [('a', int32), ('b', int32)], 10) a.view(int32).reshape(-1, 2)

You could use itertools:

...

...

...

from itertools import chain g = ((i, i**2) for i in range(10)) import numpy numpy.fromiter(chain.from_iterable(g), numpy.int32).reshape(-1, 2)

---

NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion