[AstroPy] Question re. astropy.table

Aldcroft, Thomas aldcroft at head.cfa.harvard.edu
Thu Feb 25 18:59:56 EST 2016

Hi Brian,

If your input file an ASCII data table then please see:

There is a code snippet there that should work to read in a large table in


On Thu, Feb 25, 2016 at 2:56 PM, Brian York <york at stsci.edu> wrote:

> Greetings,
> I have a number of situations where I'm reading data from a (rather large)
> table in order to add that data into a separate numpy array (sample code
> below):
>   t = Table.read(name, format)
>   x_locations = t['X']
>   y_locations = t['Y']
>   fluxes = t['FLUX']
>   image_data[y_locations, x_locations] += fluxes
> Now, this works well for relatively short tables (~100,000 rows), but
> sometimes I end up with considerably longer tables (~5-10 million rows),
> and there tends to be a fairly high memory overhead in loading the table
> in those circumstances (especially given that the table has more than just
> the three columns -- the above code is an example, not exactly what I'm
> doing).
> Is there any way to load a table N rows at a time? Or some other way to
> reduce the memory footprint?
> Thank you,
>         -Brian York
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20160225/91436601/attachment.html>

More information about the AstroPy mailing list