[AstroPy] Question re. astropy.table

[Aldcroft, Thomas]

Hi Brian,

If your input file an ASCII data table then please see:
   https://github.com/astropy/astropy/issues/3334

There is a code snippet there that should work to read in a large table in
chunks.

Best,
Tom

On Thu, Feb 25, 2016 at 2:56 PM, Brian York <york at stsci.edu> wrote:

> Greetings,
>
> I have a number of situations where I'm reading data from a (rather large)
> table in order to add that data into a separate numpy array (sample code
> below):
>
>   t = Table.read(name, format)
>   x_locations = t['X']
>   y_locations = t['Y']
>   fluxes = t['FLUX']
>
>   image_data[y_locations, x_locations] += fluxes
>
> Now, this works well for relatively short tables (~100,000 rows), but
> sometimes I end up with considerably longer tables (~5-10 million rows),
> and there tends to be a fairly high memory overhead in loading the table
> in those circumstances (especially given that the table has more than just
> the three columns -- the above code is an example, not exactly what I'm
> doing).
>
> Is there any way to load a table N rows at a time? Or some other way to
> reduce the memory footprint?
>
> Thank you,
>         -Brian York
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20160225/91436601/attachment.html>