[AstroPy] PyFITS: Appending tables with variable length array...?
taro at ap.smu.ca
Mon May 9 16:40:41 EDT 2011
On Mon, May 9, 2011 at 11:38 AM, Erik Bray <embray at stsci.edu> wrote:
> Hi Taro,
> On 05/06/2011 07:05 PM, Taro Sato wrote:
>> On Fri, May 6, 2011 at 5:56 PM, Erik Bray<embray at stsci.edu> wrote:
>>> Here's a workaround that worked for me:
>>> for idx in range(len(t1.columns)):
>>> col = t1.columns[idx]
>>> if col.format == 'P':
>>> hdu.columns[idx].array = np.resize(col.array, nr)
>>> hdu.data._convert[idx] = hdu.columns[idx].array
>>> hdu.data.field(idx)[nr1:] = t2.data.field(idx)
>> Thanks very much for your response, Erik. Unfortunately it doesn't
>> work for me quite yet...
>> I see what you are doing, but for some reason the column doesn't
>> remain a _VLF object after setting _convert. (Before setting _convert
>> I confirm it's still a _VLF.) For me, right after _convert setting it
>> becomes a flat np.array of int32 and it complains about me trying to
>> set an array element with a sequence.
>> Any idea how to work around this?
> Sorry I couldn't get back to you sooner. I see what you're saying; I might
> have made a mistake in writing down the sample code. There are maybe two
> things to try here:
> Where it assigns to hdu.data._convert[idx], instead try:
> hdu.data._convert[idx] = None
> Or if that doesn't work you can manually wrap the array in a _VLF object:
> hdu.data._convert[idx] = pyfits.core._VLF(hdu.columns[idx].array)
> I created a ticket for this bug here:
> Thanks, and I hope that works.
Erik -- Thanks for your response (and absolutely no need for apology
since I wasn't expecting a tech support anything like that...). I
think I managed to get it to work for me, but I need to do a tiny bit
more of dirty hacking of the private parts... I love/hate that Python
doesn't hide anything... So indecent...
I needed to assign _VLF to _convert[idx], but unfortunately that
somehow messed up the structure of _VLF by flattening the original
_VLF array (into an array of int, float, etc.). Within PyFITS it
appears _VLF is always assumed to be an array of arrays (i.e., 2D) so
I had to manually maintain it that way. Here's the code
#### BEGIN ####
import pyfits as pf
t1 = pf.open(fits1)
t2 = pf.open(fits2)
nr1 = t1.data.shape
nr2 = t2.data.shape
nr = nr1 + nr2
hdu = pf.new_table(t1.columns, nrows=nr)
for idx in range(len(t1.columns)):
col = t1.columns[idx]
if col.format == 'P':
orig = t1.data.field(idx)[:nr1]
hdu.columns[idx].array = np.resize(col.array, nr)
hdu.data._convert[idx] = pf.core._VLF(hdu.columns[idx].array)
hdu.data._convert[idx][:nr1] = orig
hdu.data.field(idx)[nr1:] = t2.data.field(idx)
#### END ####
For now it's working for me. This post is just for completeness.
Hope it gets implemented more elegantly in PyFITS.
More information about the AstroPy