[AstroPy] Question re. astropy.table

[Timothy Pickering]

for a use-case like this, it might be worth looking into using pandas: 

http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking

tim

> On Feb 25, 2016, at 2:56 PM, Brian York <york at stsci.edu> wrote:
> 
> Greetings,
> 
> I have a number of situations where I'm reading data from a (rather large)
> table in order to add that data into a separate numpy array (sample code
> below):
> 
>  t = Table.read(name, format)
>  x_locations = t['X']
>  y_locations = t['Y']
>  fluxes = t['FLUX']
> 
>  image_data[y_locations, x_locations] += fluxes
> 
> Now, this works well for relatively short tables (~100,000 rows), but
> sometimes I end up with considerably longer tables (~5-10 million rows),
> and there tends to be a fairly high memory overhead in loading the table
> in those circumstances (especially given that the table has more than just
> the three columns -- the above code is an example, not exactly what I'm
> doing).
> 
> Is there any way to load a table N rows at a time? Or some other way to
> reduce the memory footprint?
> 
> Thank you,
> 	-Brian York
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy

—
+-------------------------------------------------------------------+
| T. E. Pickering, Ph.D.      |   Space Telescope Science Institute |
| Systems Software Engineer   |                 3700 San Martin Dr. |
| +1-520-305-9823             |           Baltimore, MD, 21218, USA |
| te.pickering at gmail.com      |                 pickering at stsci.edu |
+-------------------------------------------------------------------+