[AstroPy] ValueError in vstacking tables

Tue Jun 23 10:12:15 EDT 2015

Hi Thomas, 

Excellent!  Thanks a lot.  Really appreciate it.  

Cheers, Brian

---------------------------------------
Brian Cherinka, Ph.D
Dunlap Institute for Astronomy & Astrophysics
University of Toronto
Toronto, ON, Canada M5S 3H4 
phone: 416-978-7299
email: cherinka at di.utoronto.ca
----------------------------------------

On Jun 23, 2015, at 10:10 AM, Thomas Robitaille <thomas.robitaille at gmail.com> wrote:

> Hi Brian,
> 
> Thanks, this is helpful, and I managed to boil this down to a simple
> failing example that I've reported here:
> 
> https://github.com/astropy/astropy/issues/3877
> 
> It seems to be due to the vector columns. We'll look into it!
> 
> Cheers,
> Tom
> 
> 
> On 23 June 2015 at 14:58, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>> Hi Thomas,
>> 
>> In [187]: t1.dtype
>> Out[187]: dtype([('plate', '>i8'), ('ifudsgn', 'S5'), ('plateifu', 'S9'),
>> ('mangaid', 'S4'), ('versdrp2', 'S30'), ('versdrp3', 'S30'), ('verscore',
>> 'S30'), ('versutil', 'S30'), ('platetyp', 'S30'), ('srvymode', 'S30'),
>> ('objRA', '>f8'), ('objDEC', '>f8'), ('objGLON', '>f8'), ('objGLAT', '>f8'),
>> ('ebvgal', '>f8'), ('nexp', '>i8'), ('exptime', '>f8'), ('drp3qual', '>i8'),
>> ('bluesn2', '>f8'), ('redsn2', '>f8'), ('harname', 'S30'), ('frlplug',
>> '>i8'), ('cartid', 'S30'), ('designid', '>i8'), ('cenRA', '>f8'), ('cenDEC',
>> '>f8'), ('airmsmin', '>f8'), ('airmsmed', '>f8'), ('airmsmax', '>f8'),
>> ('seemin', '>f8'), ('seemed', '>f8'), ('seemax', '>f8'), ('transmin',
>> '>f8'), ('transmed', '>f8'), ('transmax', '>f8'), ('mjdmin', '>i8'),
>> ('mjdmed', '>i8'), ('mjdmax', '>i8'), ('ufwhm', '>f8'), ('gfwhm', '>f8'),
>> ('rfwhm', '>f8'), ('ifwhm', '>f8'), ('zfwhm', '>f8'), ('mngtarg1', '>i8'),
>> ('mngtarg2', '>i8'), ('mngtarg3', '>i8'), ('catidnum', '>i8'), ('plttarg',
>> 'S30')])
>> 
>> In [188]: t2.dtype
>> Out[188]: dtype([('manga_tileid', '<i8'), ('ifu_ra', '<f8'), ('ifu_dec',
>> '<f8'), ('target_ra', '<f8'), ('target_dec', '<f8'), ('iauname', 'S19'),
>> ('ifudesignsize', '<i8'), ('ifutargetsize', '<i8'), ('ifudesignwrongsize',
>> '<i8'), ('field', '<i8'), ('run', '<i8'), ('nsa_version', 'S6'), ('nsa_id',
>> '<i8'), ('nsa_id100', '<i8'), ('nsa_redshift', '<f8'), ('nsa_zdist', '<f8'),
>> ('nsa_absmag', '<f8', (7,)), ('nsa_absmag_el', '<f8', (7,)),
>> ('nsa_amivar_el', '<f8', (7,)), ('nsa_mstar', '<f8'), ('nsa_mstar_el',
>> '<f8'), ('nsa_ba', '<f8'), ('nsa_phi', '<f8'), ('nsa_extinction', '<f8',
>> (7,)), ('nsa_petro_th50', '<f8'), ('nsa_petro_th50_el', '<f8'),
>> ('nsa_petroflux', '<f8', (7,)), ('nsa_petroflux_ivar', '<f8', (7,)),
>> ('nsa_petroflux_el', '<f8', (7,)), ('nsa_petroflux_el_ivar', '<f8', (7,)),
>> ('nsa_sersic_ba', '<f8'), ('nsa_sersic_n', '<f8'), ('nsa_sersic_phi',
>> '<f8'), ('nsa_sersic_th50', '<f8'), ('nsa_sersicflux', '<f8', (7,)),
>> ('nsa_sersicflux_ivar', '<f8', (7,))])
>> 
>> t3 = hstack([t1,t2],join_type='outer’)  completes with no errors, however
>> 
>> t3[0] gives
>> 
>> <repr(<astropy.table.row.Row at 0x108af0b90>) failed: ValueError: setting an
>> array element with a sequence.>
>> 
>> then vstacking t3 with t4 completes with no errors, but gives the same error
>> when row indexing.
>> 
>> Cheers, Brian
>> 
>> ---------------------------------------
>> Brian Cherinka, Ph.D
>> Dunlap Institute for Astronomy & Astrophysics
>> University of Toronto
>> Toronto, ON, Canada M5S 3H4
>> phone: 416-978-7299
>> email: cherinka at di.utoronto.ca
>> ----------------------------------------
>> 
>> On Jun 23, 2015, at 9:53 AM, Thomas Robitaille <thomas.robitaille at gmail.com>
>> wrote:
>> 
>> Hi Brian,
>> 
>> On 23 June 2015 at 14:49, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>> 
>> Hi Thomas,
>> 
>> Could you let us know what table1.dtype and table3.dtype are equal to,
>> and how many rows are in each table?
>> 
>> 
>> The two tables, t1 and t2, that I am hstacking have 48 columns, and 36
>> columns, respectively, each with 29 rows.  The dtype of those columns are a
>> mix of things.  Mostly float and string scalars.  I do have several columns
>> of float arrays.
>> 
>> 
>> Just to clarify, could you copy and paste the exact output from
>> table1.dtype and table3.dtype here so that we can try and reproduce
>> the exact issue? (it may be a corner case due to the type of one of
>> the columns).
>> 
>> Thanks!
>> Tom
>> 
>> After hstacking, the resulting table (84 columns, 29 rows) is combined as
>> expected, although I can no longer index the rows.  Then I vstack it with
>> another table 3, which has the same number of columns, and same dtype.  I
>> can’t row index that table either.
>> 
>> 
>> 
>> 
>> Cheers, Brian
>> 
>> 
>> ---------------------------------------
>> Brian Cherinka, Ph.D
>> Dunlap Institute for Astronomy & Astrophysics
>> University of Toronto
>> Toronto, ON, Canada M5S 3H4
>> phone: 416-978-7299
>> email: cherinka at di.utoronto.ca
>> ----------------------------------------
>> 
>> On Jun 19, 2015, at 8:43 AM, Thomas Robitaille <thomas.robitaille at gmail.com>
>> wrote:
>> 
>> Hi Brian,
>> 
>> Could you let us know what table1.dtype and table3.dtype are equal to,
>> and how many rows are in each table?
>> 
>> Thanks!
>> Tom
>> 
>> 
>> On 18 June 2015 at 16:34, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>> 
>> Anymore thoughts on this?  My table length issue wasn’t the problem.  I’m
>> stilling having a problem with my resulting hstacks and vstacks.  The stacks
>> happen without error.  I just lose the ability to index by row.
>> 
>> Cheers, Brian
>> 
>> ---------------------------------------
>> Brian Cherinka, Ph.D
>> Dunlap Institute for Astronomy & Astrophysics
>> University of Toronto
>> Toronto, ON, Canada M5S 3H4
>> phone: 416-978-7299
>> email: cherinka at di.utoronto.ca
>> ----------------------------------------
>> 
>> On Jun 17, 2015, at 3:25 PM, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>> 
>> Hi Thomas,
>> 
>> Your example does indeed work for me.  I cannot yet guess why it doesn’t
>> work in my full example.  However it’s not quite a complete reconstruction
>> of what I’m doing.  Technically, I’m hstacking two tables together, and then
>> vstacking that result with another table.  Something like this, which works
>> just fine.
>> 
>> In [382]: t1 = Table([[1], [2]], names=['a', 'b'])
>> 
>> In [383]: t2 = Table([[3], [4]], names=['c','d'])
>> 
>> In [384]: t3 = hstack([t1,t2],join_type='outer')
>> 
>> In [386]: t4 = Table([[5], [6]], names=['a', 'b'])
>> 
>> In [387]: t34 = vstack([t3,t4],join_type='outer')
>> 
>> In [388]: t34
>> Out[388]:
>> <Table masked=True length=2>
>> a     b     c     d
>> int64 int64 int64 int64
>> ----- ----- ----- -----
>>  1     2     3     4
>>  5     6    --    --
>> 
>> In [389]: t34[0]
>> Out[389]:
>> <Row 0 of table
>> values=(1, 2, 3, 4)
>> dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8'), ('d', '<i8')]>
>> 
>> In my full example, I did find though that I can row index the individual
>> tables t1 and t2, but cannot row index table t3 after hstacking.  I get the
>> same ValueError.  When I attempt to vstack with table t4, that error
>> propagates forward.  I can row index t4 but not t34.  Looking at it a bit
>> more, it might be that my rows in the initial tables t1, t2 do not match up
>> with one another, and I appear to have one extra row in table t2, than I
>> should.   I’ll dig a bit more.
>> 
>> Cheers, Brian
>> 
>> ---------------------------------------
>> Brian Cherinka, Ph.D
>> Dunlap Institute for Astronomy & Astrophysics
>> University of Toronto
>> Toronto, ON, Canada M5S 3H4
>> phone: 416-978-7299
>> email: cherinka at di.utoronto.ca
>> ----------------------------------------
>> 
>> On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas
>> <aldcroft at head.cfa.harvard.edu> wrote:
>> 
>> Hi Brian,
>> 
>> Below is what I think is a minimal example of what you are describing.  Can
>> you look at this and first reproduce that this works for you, and second,
>> can you guess what might be different in your full example vs. my minimal
>> example?
>> 
>> In [19]: t1 = Table([[1], [2]], names=['a', 'b'])
>> 
>> In [20]: t2 = Table([[3], [4]], names=['b', 'c'])
>> 
>> In [23]: t12 = vstack([t1, t2], join_type='outer')
>> 
>> In [24]: t12
>> Out[24]:
>> <Table masked=True length=2>
>> a     b     c
>> int64 int64 int64
>> ----- ----- -----
>>  1     2    --
>> --     3     4
>> 
>> In [25]: t12[0]
>> Out[25]:
>> <Row 0 of table
>> values=(1, 2, --)
>> dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')]>
>> 
>> 
>> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka <cherinka at di.utoronto.ca>
>> wrote:
>> 
>> 
>> Hi,
>> 
>> I’m trying to vstack (vertically stack) two Astropy tables with different
>> numbers of columns.  The tables appear to stack correctly without error.
>> However, upon attempting to index a row in the table, it throws me this
>> error
>> 
>> <repr(<astropy.table.row.Row at 0x10c72ed50>) failed: ValueError: setting
>> an array element with a sequence.>
>> 
>> I’m not trying to do anything fancy with mixed columns, or Quantities.
>> Just stackin’ two good ol’ natural Tables together.     One table has 84
>> columns, and another has 57, which are all a part of the 84 in the first
>> table.  At first glance I would guess it’s a mismatch in the dtypes of a
>> particular column that I’m trying to merge.  However, the columns all have
>> the same dtype, format, and they are in the same order.  I get no messages
>> regarding any kind of merge errors.
>> 
>> Any ideas on this?
>> 
>> Thanks, Brian
>> 
>> P.S. Full print of traceback below
>> 
>> table3 = vstack([table1, table3],join_type=‘outer’)
>> print table3[0]
>> 
>> ---------------------------------------------------------------------------
>> ValueError                                Traceback (most recent call
>> last)
>> <ipython-input-278-a0c75e8d3ef3> in <module>()
>> ----> 1 print tmp3[0]
>> 
>> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in
>> __repr__(self)
>>  176     def __repr__(self):
>>  177         return "<{3} {0} of table\n values={1!r}\n
>> dtype={2}>".format(
>> --> 178             self.index, self.as_void(), self.dtype,
>> self.__class__.__name__)
>>  179
>>  180
>> 
>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in
>> __repr__(self)
>> 5699             return self._data.__repr__()
>> 5700         m = tuple(m)
>> -> 5701         if not any(m):
>> 5702             return self._data.__repr__()
>> 5703         p = masked_print_option
>> 
>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in
>> __call__(self, a, *args, **params)
>> 6089         method = getattr(MaskedArray, method_name, None)
>> 6090         if method is not None:
>> -> 6091             return method(MaskedArray(a), *args, **params)
>> 6092         # Still here ? OK, let's call the corresponding np
>> function
>> 6093         method = getattr(np, method_name)
>> 
>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in
>> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask,
>> hard_mask, shrink, **options)
>> 2649         """
>> 2650         # Process data............
>> -> 2651         _data = np.array(data, dtype=dtype, copy=copy, subok=True,
>> ndmin=ndmin)
>> 2652         _baseclass = getattr(data, '_baseclass', type(_data))
>> 2653         # Check that we're not erasing the mask..........
>> 
>> ValueError: setting an array element with a sequence.
>> 
>> 
>> 
>> 
>> ---------------------------------------
>> Brian Cherinka, Ph.D
>> Dunlap Institute for Astronomy & Astrophysics
>> University of Toronto
>> Toronto, ON, Canada M5S 3H4
>> phone: 416-978-7299
>> email: cherinka at di.utoronto.ca
>> ----------------------------------------
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy