[Numpy-discussion] genfromtxt universal newline support
Derek Homeier
derek at astro.physik.uni-goettingen.de
Mon Jun 30 16:58:37 EDT 2014
On 30 Jun 2014, at 04:56 pm, Nathaniel Smith <njs at pobox.com> wrote:
>> A real need, which had also been discussed at length, is a truly performant text IO
>> function (i.e. one using a compiled ASCII number parser, and optimally also a more
>> memory-efficient one), but unfortunately all people interested in implementing this
>> seem to have drifted away (not excluding myself from this)…
>
> It's possible we could steal some code from Pandas for this. IIRC they
> have C/Cython text parsing routines. (It's also an interesting
> question whether they've fixed the unicode/binary issues, might be
> worth checking before rewriting from scratch...)
Good point, last time I was playing with Pandas it was not any faster, but now a 10x
speedup speaks for itself. Their C engine does not support generic whitespace separators,
but that could probably be addressed in a numpy implementation.
Derek
More information about the NumPy-Discussion
mailing list