[AstroPy] Question re. astropy.table

[Perry Greenfield]

It may be possible to speed it up a lot if you can access the underlying table as a numpy recarray and take advantage of memory mapping as well. The downside of doing that is that you have to deal with logical column translations yourself (i.e., testing for ’T’ or ‘F’), and having to scale results if any columns use BSCALE or BZERO. But for what you show below, that probably isn’t the case. I wish I had time to try this out myself, but it would work something like this (untested).

nprarr = tablehdu.data.view(np.recarray)

Then you would access whatever columns you want. If it is really large, you may want to read in column data in subsets (e.g., only a few hundred thousand rows at a time) to copy to an in memory array for those data. Perhaps tomorrow I can give this a try (particularly if you can point me to an example file).

Perry

On Feb 25, 2016, at 2:56 PM, Brian York <york at stsci.edu> wrote:

> Greetings,
> 
> I have a number of situations where I'm reading data from a (rather large)
> table in order to add that data into a separate numpy array (sample code
> below):
> 
>  t = Table.read(name, format)
>  x_locations = t['X']
>  y_locations = t['Y']
>  fluxes = t['FLUX']
> 
>  image_data[y_locations, x_locations] += fluxes
> 
> Now, this works well for relatively short tables (~100,000 rows), but
> sometimes I end up with considerably longer tables (~5-10 million rows),
> and there tends to be a fairly high memory overhead in loading the table
> in those circumstances (especially given that the table has more than just
> the three columns -- the above code is an example, not exactly what I'm
> doing).
> 
> Is there any way to load a table N rows at a time? Or some other way to
> reduce the memory footprint?
> 
> Thank you,
> 	-Brian York
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy