[Numpy-discussion] fromiter cannot create array of object - was: Creating an ndarray from an iterable, over sequences

Dr. Leo fhaxbox66 at googlemail.com
Wed Jan 22 01:58:27 EST 2014


thanks. Both recarray and itertools.chain work just fine in the example

However, the real purpose of this is to read strings from a large xml
file into a pandas DataFrame. But fromiter cannot create arrays of dtype
'object'. Fixed length strings may be worth trying. But as the xml
schema does not guarantee a max. length, and pandas generally uses
'object' arrays for strings, I see no better way than creating the array
through list comprehensions and turn it into a DataFrame.

Maybe a variable length string/unicode type would help in the long term.


> I would like to write something like:
> In [25]: iterable=((i, i**2) for i in range(10))
> In [26]: a=np.fromiter(iterable, int32)
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call
> last)
> <ipython-input-26-5bcc2e94dbca> in <module>()
> ----> 1 a=np.fromiter(iterable, int32)
> ValueError: setting an array element with a sequence.
> Is there an efficient way to do this?
Perhaps you could just utilize structured arrays (
http://docs.scipy.org/doc/numpy/user/basics.rec.html), like:
iterable= ((i, i**2) for i in range(10))
a= np.fromiter(iterable, [('a', int32), ('b', int32)], 10)
a.view(int32).reshape(-1, 2)

You could use itertools:

>>> from itertools import chain
>>> g = ((i, i**2) for i in range(10))
>>> import numpy
>>> numpy.fromiter(chain.from_iterable(g), numpy.int32).reshape(-1, 2)

More information about the NumPy-Discussion mailing list