[Numpy-discussion] load from text files Pull Request Review

Christopher Jordan-Squire cjordan1 at uw.edu
Tue Sep 6 10:36:40 EDT 2011

On Tue, Sep 6, 2011 at 9:32 AM, Derek Homeier
<derek at astro.physik.uni-goettingen.de> wrote:
> On 02.09.2011, at 11:45PM, Christopher Jordan-Squire wrote:
>>>> and unfortunately it's for 1D-arrays only).
>>> That's not bad for this use -- make a row a struct dtype, and you've got
>>> a 1-d array anyway -- you can optionally convert to a 2-d array after
>>> the fact.
>>> I don't know why I didn't think of using fromiter() when I build
>>> accumulator.  Though what I did is a bit more flexible -- you can add
>>> stuff later on, too, you don't need to do it allat once.
>> I'm unsure how to use fromiter for missing data. It sounds like a
>> potential solution when no data is missing, though.
> Strange I haven't thought about it before either; I guess for record arrays it
> comes more natural to view them as a collection of 1D arrays.
> However, you'd need to construct a list or something of ncolumn iterators from
> the input - should not be too hard; but then
> how do you feed the ncolumn fromiter() instances synchronously from that??
> As far as I can see there is no way to make them read one item at a time,
> row by row. Then there are additional complications with multi-D dtypes,
> and in your case, especially datetime instances, but the problem that all columns
> have to be read in in parallel really seems to be the showstopper here.
> Of course for "flat" 2D arrays of data (all the same dtype) this would work with
> simply reshaping the array - that's probably even the most common use case for
> loadtxt, but that method lacks way too much generality for my taste.
> Back to accumulator, I suppose.

Yes, I believe the thinking was that if your data is all one dtype
that's simple enough to figure out, and there are other method for
reading in such an array to produce a 2-d array. This is strictly for
structured arrays currently, though I suppose that could change.


> Cheers,
>                                                Derek
> --
> ----------------------------------------------------------------
> Derek Homeier          Centre de Recherche Astrophysique de Lyon
> ENS Lyon                                      46, Allée d'Italie
> 69364 Lyon Cedex 07, France                  +33 1133 47272-8894
> ----------------------------------------------------------------
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

More information about the NumPy-Discussion mailing list