March 29, 2019
12:12 a.m.
On 29 Mar 2019, at 12:38 am, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Mar 29, 2019 at 12:34 AM Nicholas Alexeev <naalexeev@alaska.edu> wrote: I noticed that the genfromtxt() function is slow and it consumes a large amount of memory. I am trying to improve it. Does anyone have any tips? Thanks.
If your data can be read by pandas.read_csv, that's a lot faster
Astropy also has a similar implementation of an ASCII file parser written in C - fastbasic.py, cparser.pyx and src/tokenizer.* here: https://github.com/astropy/astropy/tree/master/astropy/io/ascii