I noticed that the genfromtxt() function is slow and it consumes a large amount of memory. I am trying to improve it. Does anyone have any tips? Thanks.
On Fri, Mar 29, 2019 at 12:34 AM Nicholas Alexeev <naalexeev@alaska.edu> wrote:
I noticed that the genfromtxt() function is slow and it consumes a large amount of memory. I am trying to improve it. Does anyone have any tips? Thanks.
If your data can be read by pandas.read_csv, that's a lot faster
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On 29 Mar 2019, at 12:38 am, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Mar 29, 2019 at 12:34 AM Nicholas Alexeev <naalexeev@alaska.edu> wrote: I noticed that the genfromtxt() function is slow and it consumes a large amount of memory. I am trying to improve it. Does anyone have any tips? Thanks.
If your data can be read by pandas.read_csv, that's a lot faster
Astropy also has a similar implementation of an ASCII file parser written in C - fastbasic.py, cparser.pyx and src/tokenizer.* here: https://github.com/astropy/astropy/tree/master/astropy/io/ascii
participants (3)
-
Derek Homeier -
Nicholas Alexeev -
Ralf Gommers