From demitri.muna at gmail.com Mon Jun 1 12:06:51 2015 From: demitri.muna at gmail.com (Demitri Muna) Date: Mon, 1 Jun 2015 12:06:51 -0400 Subject: [AstroPy] Grid lines on FITS images Message-ID: Hi, I'm looking for advice on creating a simple plot. First, I want to take a FITS image and plot it (AplPY seems like the appropriate tool). I'd like the area of the plot to be configurable, e.g. extend beyond the bounds of the image by an amount of my choosing. I then have a polygon/region whose points are defined in world coordinates. I want to draw this outline on top of the image. That's all. I saw that AplPY can draw DS9 regions ("fig,show_regions()"), and it's not a big deal to convert my points into a DS9 region file format. My question is whether the lines drawn will properly follow a path on the great circle; it's not apparent how the lines are being drawn when I look at the source code. I'm familiar with PyAST and know that it will draw the grid lines correctly, but it doesn't draw images, so a mixture of these two packages seems ideal. I'm kind of hoping that someone has done something almost exactly like this and has simple sample code - that would be greatly appreciated! Cheers, Demitri _________________________________________ Demitri Muna Department of Astronomy La Ohio State University http://trillianverse.org http://scicoder.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Mon Jun 1 12:20:11 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 01 Jun 2015 18:20:11 +0200 Subject: [AstroPy] Grid lines on FITS images In-Reply-To: References: Message-ID: <556C863B.7070403@gmail.com> Hi Demitri, I think that WCSAxes (http://wcsaxes.readthedocs.org) will do what you need? Let me know if you have any issues using it. Cheers, Tom Demitri Muna wrote: > Hi, > > I'm looking for advice on creating a simple plot. First, I want to take > a FITS image and plot it (AplPY seems like the appropriate tool). I'd > like the area of the plot to be configurable, e.g. extend beyond the > bounds of the image by an amount of my choosing. I then have a > polygon/region whose points are defined in world coordinates. I want to > draw this outline on top of the image. That's all. > > I saw that AplPY can draw DS9 regions ("fig,show_regions()"), and it's > not a big deal to convert my points into a DS9 region file format. My > question is whether the lines drawn will properly follow a path on the > great circle; it's not apparent how the lines are being drawn when I > look at the source code. > > I'm familiar with PyAST and know that it will draw the grid lines > correctly, but it doesn't draw images, so a mixture of these two > packages seems ideal. I'm kind of hoping that someone has done something > almost exactly like this and has simple sample code - that would be > greatly appreciated! > > Cheers, > Demitri > _________________________________________ > Demitri Muna > > Department of Astronomy > La Ohio State University > > http://trillianverse.org > http://scicoder.org > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From demitri.muna at gmail.com Wed Jun 3 10:20:37 2015 From: demitri.muna at gmail.com (Demitri Muna) Date: Wed, 3 Jun 2015 10:20:37 -0400 Subject: [AstroPy] Grid lines on FITS images In-Reply-To: <556C863B.7070403@gmail.com> References: <556C863B.7070403@gmail.com> Message-ID: Hi, On Jun 1, 2015, at 12:20 PM, Thomas Robitaille wrote: > I think that WCSAxes (http://wcsaxes.readthedocs.org ) will do what you > need? Let me know if you have any issues using it. Thanks Tom. I went through the documentation/tutorial and created a sample script. It doesn't do exactly what I'm looking for - as far as I can tell. I'm using matplotlib.patches.Polygon to define my polygon, supplying a world transform based on the FITS header. I don't know how the underlying drawing mechanism works. If it is taking the world coordinates that I apply, e.g. the four corners of a polygon, converting them to pixel coordinates, and then drawing straight lines on the canvas between them, then they are not proper representations as they are not following great circles. If the transform is applied to each pixel drawn, then it might be. My (pure) guess is that it's doing the former. (I would like to test this, but that will have to wait until after SciCoder.) I think in the end PyAST will do what I need, but I may still use parts of WSCAxes. Thanks, Demitri _________________________________________ Demitri Muna Department of Astronomy Ohio State University http://trillianverse.org http://scicoder.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Wed Jun 3 10:34:12 2015 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 3 Jun 2015 10:34:12 -0400 Subject: [AstroPy] Grid lines on FITS images In-Reply-To: References: <556C863B.7070403@gmail.com> Message-ID: <556F1064.4080804@stsci.edu> On 06/03/2015 10:20 AM, Demitri Muna wrote: > Hi, > > On Jun 1, 2015, at 12:20 PM, Thomas Robitaille > > wrote: > >> I think that WCSAxes (http://wcsaxes.readthedocs.org >> ) will do what you >> need? Let me know if you have any issues using it. > > Thanks Tom. I went through the documentation/tutorial and created a > sample script. It doesn't do exactly what I'm looking for - as far as > I can tell. I'm using matplotlib.patches.Polygon to define my polygon, > supplying a world transform based on the FITS header. > > I don't know how the underlying drawing mechanism works. If it is > taking the world coordinates that I apply, e.g. the four corners of a > polygon, converting them to pixel coordinates, and then drawing > straight lines on the canvas between them, then they are not proper > representations as they are not following great circles. If the > transform is applied to each pixel drawn, then it might be. My (pure) > guess is that it's doing the former. (I would like to test this, but > that will have to wait until after SciCoder.) Can?t speak specifically for wcsaxes, but for ?raw? matplotlib it would do the former. There is, however, a quick-and-dirty way to interpolate your polygon using a private method intended for the built-in geo projections in matplotlib. If |poly| is your |Polygon| object and |N| is the number of line segments to create for each existing line segment: | poly._path = poly._path.interpolated(N) | On-the-fly interpolation for projections has been on the TODO list for matplotlib for many years. It just hasn?t got done (and will probably require a C++ implementation to be performant enough). Mike > > I think in the end PyAST will do what I need, but I may still use > parts of WSCAxes. > > Thanks, > Demitri > > _________________________________________ > Demitri Muna > > Department of Astronomy > Ohio State University > > http://trillianverse.org > http://scicoder.org > > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fb55 at nyu.edu Wed Jun 3 11:16:44 2015 From: fb55 at nyu.edu (federica) Date: Wed, 3 Jun 2015 09:16:44 -0600 Subject: [AstroPy] Grid lines on FITS images In-Reply-To: References: Message-ID: hi dimitri, how are you? if you find a solution, it would be really useful to me. would you mind sharing it? fed On Mon, Jun 1, 2015 at 10:06 AM, Demitri Muna wrote: > Hi, > > I'm looking for advice on creating a simple plot. First, I want to take a > FITS image and plot it (AplPY seems like the appropriate tool). I'd like > the area of the plot to be configurable, e.g. extend beyond the bounds of > the image by an amount of my choosing. I then have a polygon/region whose > points are defined in world coordinates. I want to draw this outline on top > of the image. That's all. > > I saw that AplPY can draw DS9 regions ("fig,show_regions()"), and it's not > a big deal to convert my points into a DS9 region file format. My question > is whether the lines drawn will properly follow a path on the great circle; > it's not apparent how the lines are being drawn when I look at the source > code. > > I'm familiar with PyAST and know that it will draw the grid lines > correctly, but it doesn't draw images, so a mixture of these two packages > seems ideal. I'm kind of hoping that someone has done something almost > exactly like this and has simple sample code - that would be greatly > appreciated! > > Cheers, > Demitri > _________________________________________ > Demitri Muna > > Department of Astronomy > La Ohio State University > > http://trillianverse.org > http://scicoder.org > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- _____________________________________________ dr. federica bianco James Arthur Fellow NYU-CCPP 4 Washington Pl #545 New York, NY 10003 http://cosmo.nyu.edu/~fb55/ ___________________________________________ //\\ //||\\ Please consider the environment before printing this e-mail email checking schedule: 9-5 on the hour. otherwise sporadically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From demitri.muna at gmail.com Wed Jun 3 17:48:46 2015 From: demitri.muna at gmail.com (Demitri Muna) Date: Wed, 3 Jun 2015 17:48:46 -0400 Subject: [AstroPy] Astropy and large VOTable files In-Reply-To: <40C5B651-04C8-4E78-ABD3-F68E8A3195C3@yale.edu> References: <555A28DB.5000207@stsci.edu> <40C5B651-04C8-4E78-ABD3-F68E8A3195C3@yale.edu> Message-ID: <5C57E548-CC98-4229-AD14-8AFE810305D0@gmail.com> On May 18, 2015, at 2:09 PM, Andrew Hearin wrote: > Being able to read large data in chunks, make cuts on the chunks, and return a table of rows that pass the cuts is a pretty common data mining task that I think would be good to include in Astropy. I?m happy to (re-)raise a GitHub issue for this purpose, and contribute some code, but first: Jennifer, this is the functionality you are describing, right? If so: Mike, do you see any fundamental obstacles with this? If you are working with very large FITS files, I recommend using Erin Sheldon's fitsio package. The underlying C (cfitsio) library is able to directly read slices without reading the whole file into memory. As for XML... I wouldn't recommend XML for crazy large files. It wasn't designed for that. Cheers, Demitri _________________________________________ Demitri Muna Department of Astronomy Le Ohio State University http://trillianverse.org http://scicoder.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From demitri.muna at gmail.com Wed Jun 3 17:54:44 2015 From: demitri.muna at gmail.com (Demitri Muna) Date: Wed, 3 Jun 2015 17:54:44 -0400 Subject: [AstroPy] Grid lines on FITS images In-Reply-To: <556F1064.4080804@stsci.edu> References: <556C863B.7070403@gmail.com> <556F1064.4080804@stsci.edu> Message-ID: Hi Mike, On Jun 3, 2015, at 10:34 AM, Michael Droettboom wrote: > On-the-fly interpolation for projections has been on the TODO list for matplotlib for many years. It just hasn?t got done (and will probably require a C++ implementation to be performant enough). Thanks for the tip. I think ideally one would not need to specify the number of line segments, but as you point out this is a private method. It sounds like PyAST will do this properly now, but I'm still wrapping my mind around how it interacts with matplotlib. Cheers, Demitri _________________________________________ Demitri Muna Department of Astronomy Le Ohio State University http://trillianverse.org http://scicoder.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ghang.naoc at gmail.com Thu Jun 4 09:49:47 2015 From: ghang.naoc at gmail.com (gonghang.naoc) Date: Thu, 4 Jun 2015 21:49:47 +0800 Subject: [AstroPy] scatter plot+colorbar Message-ID: Hi all, I have a three column data set, x, y, and colorvalue. How to show the colorvalue for every point? I tried the code below, but it does not work. The colorbar appears, but all points have the same color. ############# import numpy as np import pylab as pl x,y,colorvlaue=np.loadtxt(dataset',usecols=(0,1,2),delimiter=",",unpack=True) pts=pl.scatter(x,y,s=0.1,c=colorvalue,cmap=pl.cm.jet) cb = pl.colorbar(pts) cmin,cmax = cb.get_clim() ticks = np.linspace(cmin,cmax,10) cb.set_ticks(ticks) pl.show() ################ Could anybody specify the error in the code above? Thank you. Hang -------------- next part -------------- An HTML attachment was scrubbed... URL: From ejensen1 at swarthmore.edu Thu Jun 4 10:17:58 2015 From: ejensen1 at swarthmore.edu (Eric L. N. Jensen) Date: Thu, 4 Jun 2015 10:17:58 -0400 Subject: [AstroPy] scatter plot+colorbar In-Reply-To: References: Message-ID: Hi Hang, What is the range of values of your ?colorvalue? column? You may need to renormalize them to match the range expected by the ?c? argument to scatter (which I think is 0 to 1). I think you can do this with the ?norm? argument to scatter, but I don?t know the syntax. Hope this helps - Eric On Jun 4, 2015, at 9:49 AM, gonghang.naoc wrote: > Hi all, > I have a three column data set, x, y, and colorvalue. How to show the colorvalue for every point? > > I tried the code below, but it does not work. The colorbar appears, but all points have the same color. > > ############# > import numpy as np > import pylab as pl > x,y,colorvlaue=np.loadtxt(dataset',usecols=(0,1,2),delimiter=",",unpack=True) > pts=pl.scatter(x,y,s=0.1,c=colorvalue,cmap=pl.cm.jet) > cb = pl.colorbar(pts) > > cmin,cmax = cb.get_clim() > ticks = np.linspace(cmin,cmax,10) > cb.set_ticks(ticks) > > pl.show() > ################ > Could anybody specify the error in the code above? > > Thank you. > Hang > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ghang.naoc at gmail.com Thu Jun 4 10:43:19 2015 From: ghang.naoc at gmail.com (gonghang.naoc) Date: Thu, 4 Jun 2015 22:43:19 +0800 Subject: [AstroPy] scatter plot+colorbar In-Reply-To: References: Message-ID: Thanks, Eric. I find the norm method here, but unfortunately it does not work. http://matplotlib.1069221.n5.nabble.com/cmap-normalization-question-td16253.html nm = matplotlib.colors.Normalize(desired_min, desired_max) plt.scatter(x1,y1,c=myarr1,cmap=plt.get_cmap("gist_heat"),norm=nm) On Thu, Jun 4, 2015 at 10:17 PM, Eric L. N. Jensen wrote: > Hi Hang, > > What is the range of values of your ?colorvalue? column? You may need to > renormalize them to match the range expected by the ?c? argument to scatter > (which I think is 0 to 1). I think you can do this with the ?norm? > argument to scatter, but I don?t know the syntax. > > Hope this helps - > > Eric > > > > On Jun 4, 2015, at 9:49 AM, gonghang.naoc wrote: > > Hi all, > I have a three column data set, x, y, and colorvalue. How to show the > colorvalue for every point? > > I tried the code below, but it does not work. The colorbar appears, but > all points have the same color. > > ############# > import numpy as np > import pylab as pl > > x,y,colorvlaue=np.loadtxt(dataset',usecols=(0,1,2),delimiter=",",unpack=True) > pts=pl.scatter(x,y,s=0.1,c=colorvalue,cmap=pl.cm.jet) > cb = pl.colorbar(pts) > > cmin,cmax = cb.get_clim() > ticks = np.linspace(cmin,cmax,10) > cb.set_ticks(ticks) > > pl.show() > ################ > Could anybody specify the error in the code above? > > Thank you. > Hang > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Thu Jun 4 11:19:38 2015 From: mdroe at stsci.edu (Michael Droettboom) Date: Thu, 4 Jun 2015 11:19:38 -0400 Subject: [AstroPy] Astropy and large VOTable files In-Reply-To: <5C57E548-CC98-4229-AD14-8AFE810305D0@gmail.com> References: <555A28DB.5000207@stsci.edu> <40C5B651-04C8-4E78-ABD3-F68E8A3195C3@yale.edu> <5C57E548-CC98-4229-AD14-8AFE810305D0@gmail.com> Message-ID: <55706C8A.6030900@stsci.edu> On 06/03/2015 05:48 PM, Demitri Muna wrote: > > On May 18, 2015, at 2:09 PM, Andrew Hearin > wrote: > >> Being able to read large data in chunks, make cuts on the chunks, and >> return a table of rows that pass the cuts is a pretty common data >> mining task that I think would be good to include in Astropy. I?m >> happy to (re-)raise a GitHub issue for this purpose, and contribute >> some code, but first: Jennifer, this is the functionality you are >> describing, right? If so: Mike, do you see any fundamental obstacles >> with this? > > If you are working with very large FITS files, I recommend using Erin > Sheldon's fitsio package. The underlying C (cfitsio) library is able > to directly read slices without reading the whole file into memory. Just for completeness, this is also true of astropy.io.fits. > As for XML... I wouldn't recommend XML for crazy large files. It > wasn't designed for that. Ditto. Mike > > Cheers, > Demitri > > _________________________________________ > Demitri Muna > > Department of Astronomy > Le Ohio State University > > http://trillianverse.org > http://scicoder.org > > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ghang.naoc at gmail.com Tue Jun 9 10:44:13 2015 From: ghang.naoc at gmail.com (gonghang.naoc) Date: Tue, 9 Jun 2015 22:44:13 +0800 Subject: [AstroPy] ffa in python Message-ID: Hi all, Is there any python resource about fast folding algorithm? I have a series of arrival times for a lot of photons. I need to do some periodical test, but do not know how to start. Thank you. best Hang -------------- next part -------------- An HTML attachment was scrubbed... URL: From archibald at astron.nl Tue Jun 9 10:49:04 2015 From: archibald at astron.nl (Anne Archibald) Date: Tue, 09 Jun 2015 14:49:04 +0000 Subject: [AstroPy] ffa in python In-Reply-To: References: Message-ID: I don't think there's a general FFA algorithm anywhere, though Vlad Kondratiev wrote a tool for applying the FFA to radio data, and I have some python code lying around somewhere. If you're searching for periodicities, the standard procedure is to search an FFT with incoherent harmonic summing; there are some pulsar searching tools that will do it (PRESTO, for example) quickly and efficiently. If you only have a handful of photons, or a modest number of periods to search, there are direct approaches that are more sensitive. If you have really a long time series, there is a trick where you do an FFT on the pairwise differences of photon arrival times. In short, there are a whole range of procedures for searching for periodicity in data. Can you be a little more specific about the problem you are trying to solve? Anne On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc wrote: > Hi all, > Is there any python resource about fast folding algorithm? I have a > series of arrival times for a lot of photons. I need to do some periodical > test, but do not know how to start. > Thank you. > best > Hang > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ghang.naoc at gmail.com Tue Jun 9 11:06:33 2015 From: ghang.naoc at gmail.com (gonghang.naoc) Date: Tue, 9 Jun 2015 23:06:33 +0800 Subject: [AstroPy] ffa in python In-Reply-To: References: Message-ID: Thank you. A pulsar guy recommends ffa to me. I need to find short period signal. The photon counts are poor in fact, but I hope I can find 10*P or 100*P at least if I can not find P. Another problem is even there is a short period signal, it may not necessarily be the strongest signal. It is also a problem? So do you think FFT is better? When is ffa better? Hang On Tue, Jun 9, 2015 at 10:49 PM, Anne Archibald wrote: > I don't think there's a general FFA algorithm anywhere, though Vlad > Kondratiev wrote a tool for applying the FFA to radio data, and I have some > python code lying around somewhere. If you're searching for periodicities, > the standard procedure is to search an FFT with incoherent harmonic > summing; there are some pulsar searching tools that will do it (PRESTO, for > example) quickly and efficiently. If you only have a handful of photons, or > a modest number of periods to search, there are direct approaches that are > more sensitive. If you have really a long time series, there is a trick > where you do an FFT on the pairwise differences of photon arrival times. > > In short, there are a whole range of procedures for searching for > periodicity in data. Can you be a little more specific about the problem > you are trying to solve? > > Anne > > On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc wrote: > >> Hi all, >> Is there any python resource about fast folding algorithm? I have a >> series of arrival times for a lot of photons. I need to do some periodical >> test, but do not know how to start. >> Thank you. >> best >> Hang >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rays at blue-cove.com Tue Jun 9 11:18:01 2015 From: rays at blue-cove.com (R Schumacher) Date: Tue, 09 Jun 2015 08:18:01 -0700 Subject: [AstroPy] ffa in python In-Reply-To: References: Message-ID: <201506091518.t59FIG7B027076@blue-cove.com> https://github.com/petigura/FFA/blob/master/FFA.py At 07:49 AM 6/9/2015, Anne Archibald wrote: >I don't think there's a general FFA algorithm >anywhere, though Vlad Kondratiev wrote a tool >for applying the FFA to radio data, and I have >some python code lying around somewhere. If >you're searching for periodicities, the standard >procedure is to search an FFT with incoherent >harmonic summing; there are some pulsar >searching tools that will do it (PRESTO, for >example) quickly and efficiently. If you only >have a handful of photons, or a modest number of >periods to search, there are direct approaches >that are more sensitive. If you have really a >long time series, there is a trick where you do >an FFT on the pairwise differences of photon arrival times.? > >In short, there are a whole range of procedures >for searching for periodicity in data. Can you >be a little more specific about the problem you are trying to solve? > >Anne > >On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc ><ghang.naoc at gmail.com> wrote: >Hi all, >Is there any python resource about fast folding >algorithm?? I have a series of arrival times >for a lot of photons. I need to do some >periodical test, but do not know how to start. >Thank you. >best >Hang > >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.org >http://mail.scipy.org/mailman/listinfo/astropy > >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.org >http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ghang.naoc at gmail.com Tue Jun 9 11:33:16 2015 From: ghang.naoc at gmail.com (gonghang.naoc) Date: Tue, 9 Jun 2015 23:33:16 +0800 Subject: [AstroPy] ffa in python In-Reply-To: <201506091518.t59FIG7B027076@blue-cove.com> References: <201506091518.t59FIG7B027076@blue-cove.com> Message-ID: Good ! Thanks ! Any comment about the comparison between ffa and fft? Hang On Tue, Jun 9, 2015 at 11:18 PM, R Schumacher wrote: > https://github.com/petigura/FFA/blob/master/FFA.py > > > > At 07:49 AM 6/9/2015, Anne Archibald wrote: > > I don't think there's a general FFA algorithm anywhere, though Vlad > Kondratiev wrote a tool for applying the FFA to radio data, and I have some > python code lying around somewhere. If you're searching for periodicities, > the standard procedure is to search an FFT with incoherent harmonic > summing; there are some pulsar searching tools that will do it (PRESTO, for > example) quickly and efficiently. If you only have a handful of photons, or > a modest number of periods to search, there are direct approaches that are > more sensitive. If you have really a long time series, there is a trick > where you do an FFT on the pairwise differences of photon arrival times.? > > In short, there are a whole range of procedures for searching for > periodicity in data. Can you be a little more specific about the problem > you are trying to solve? > > Anne > > On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc wrote: > Hi all, > Is there any python resource about fast folding algorithm?? I have a > series of arrival times for a lot of photons. I need to do some periodical > test, but do not know how to start. > Thank you. > best > Hang > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rays at blue-cove.com Tue Jun 9 12:21:06 2015 From: rays at blue-cove.com (R Schumacher) Date: Tue, 09 Jun 2015 09:21:06 -0700 Subject: [AstroPy] ffa in python In-Reply-To: References: <201506091518.t59FIG7B027076@blue-cove.com> Message-ID: <201506091621.t59GLLfl017029@blue-cove.com> I've only a cursory knowledge of FFA, but FFT assumes sinusoidal data (or sums of) and so may not best represent periodic pulses in, say, single ended data acquisition, or swept signals etc FFA is supposed to look for regularity in pulses http://en.wikipedia.org/wiki/Talk%3AFast_folding_algorithm Being an amateur astronomer I searched your use case https://books.google.com/books?id=OZ8tdN6qJcsC&pg=PA151&lpg=PA151&dq=fast+folding+algorithm&source=bl&ots=mwe0QvaHU5&sig=Xg1Kwad46yC_BMxyKu_GR37LaFA&hl=en&sa=X&ei=EhF3VYudBojiUeiogvAH&ved=0CDAQ6AEwBg - Ray At 08:33 AM 6/9/2015, you wrote: >Good ! Thanks ! >Any comment about the comparison between ffa and fft? >Hang > >On Tue, Jun 9, 2015 at 11:18 PM, R Schumacher ><rays at blue-cove.com> wrote: >https://github.com/petigura/FFA/blob/master/FFA.py > > > >At 07:49 AM 6/9/2015, Anne Archibald wrote: >>I don't think there's a general FFA algorithm >>anywhere, though Vlad Kondratiev wrote a tool >>for applying the FFA to radio data, and I have >>some python code lying around somewhere. If >>you're searching for periodicities, the >>standard procedure is to search an FFT with >>incoherent harmonic summing; there are some >>pulsar searching tools that will do it (PRESTO, >>for example) quickly and efficiently. If you >>only have a handful of photons, or a modest >>number of periods to search, there are direct >>approaches that are more sensitive. If you have >>really a long time series, there is a trick >>where you do an FFT on the pairwise differences of photon arrival times.?? >> >>In short, there are a whole range of procedures >>for searching for periodicity in data. Can you >>be a little more specific about the problem you are trying to solve? >> >>Anne >> >>On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc >><ghang.naoc at gmail.com> wrote: >>Hi all, >>Is there any python resource about fast folding >>algorithm???? I have a series of arrival times >>for a lot of photons. I need to do some >>periodical test, but do not know how to start. >>Thank you. >>best >>Hang >>_______________________________________________ >>AstroPy mailing list >>AstroPy at scipy.org >>http://mail.scipy.org/mailman/listinfo/astropy >> >>_______________________________________________ >>AstroPy mailing list >>AstroPy at scipy.org >>http://mail.scipy.org/mailman/listinfo/astropy > >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.org >http://mail.scipy.org/mailman/listinfo/astropy > > >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.org >http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter at newton.cx Tue Jun 9 12:25:31 2015 From: peter at newton.cx (Peter Williams) Date: Tue, 09 Jun 2015 12:25:31 -0400 Subject: [AstroPy] ffa in python In-Reply-To: <201506091621.t59GLLfl017029@blue-cove.com> References: <201506091518.t59FIG7B027076@blue-cove.com> <201506091621.t59GLLfl017029@blue-cove.com> Message-ID: <1433867131.3474.65.camel@newton.cx> To search for periodicity in the arrival times of discrete events, rather than continuously sampled data, the Rayleigh test is a good place to start: http://adsabs.harvard.edu/abs/1994MNRAS.268..709B http://www.astro.ubc.ca/people/jvw/ASTROSTATS/Answers/Chap9/chapter%209 %20rayleigh%20test.pdf Peter On Tue, 2015-06-09 at 09:21 -0700, R Schumacher wrote: > I've only a cursory knowledge of FFA, but FFT assumes sinusoidal data > (or sums of) and so may not best represent periodic pulses in, say, > single ended data acquisition, or swept signals etc > > FFA is supposed to look for regularity in pulses > http://en.wikipedia.org/wiki/Talk%3AFast_folding_algorithm > Being an amateur astronomer I searched your use case > https://books.google.com/books?id=OZ8tdN6qJcsC&pg=PA151&lpg=PA151&dq= > fast+folding+algorithm&source=bl&ots=mwe0QvaHU5&sig=Xg1Kwad46yC_BMxyK > u_GR37LaFA&hl=en&sa=X&ei=EhF3VYudBojiUeiogvAH&ved=0CDAQ6AEwBg > > > - Ray > > > At 08:33 AM 6/9/2015, you wrote: > > Good ! Thanks ! > > Any comment about the comparison between ffa and fft? > > Hang > > > > On Tue, Jun 9, 2015 at 11:18 PM, R Schumacher > > wrote: > > https://github.com/petigura/FFA/blob/master/FFA.py > > > > > > > > At 07:49 AM 6/9/2015, Anne Archibald wrote: > > > I don't think there's a general FFA algorithm anywhere, though > > > Vlad Kondratiev wrote a tool for applying the FFA to radio data, > > > and I have some python code lying around somewhere. If you're > > > searching for periodicities, the standard procedure is to search > > > an FFT with incoherent harmonic summing; there are some pulsar > > > searching tools that will do it (PRESTO, for example) quickly and > > > efficiently. If you only have a handful of photons, or a modest > > > number of periods to search, there are direct approaches that are > > > more sensitive. If you have really a long time series, there is a > > > trick where you do an FFT on the pairwise differences of photon > > > arrival times.?? > > > > > > In short, there are a whole range of procedures for searching for > > > periodicity in data. Can you be a little more specific about the > > > problem you are trying to solve? > > > > > > Anne > > > > > > On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc < > > > ghang.naoc at gmail.com> wrote: > > > Hi all, > > > Is there any python resource about fast folding algorithm???? I > > > have a series of arrival times for a lot of photons. I need to do > > > some periodical test, but do not know how to start. > > > Thank you. > > > best > > > Hang > > > _______________________________________________ > > > AstroPy mailing list > > > AstroPy at scipy.org > > > http://mail.scipy.org/mailman/listinfo/astropy > > > > > > _______________________________________________ > > > AstroPy mailing list > > > AstroPy at scipy.org > > > http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > > AstroPy mailing list > > AstroPy at scipy.org > > http://mail.scipy.org/mailman/listinfo/astropy > > > > > > _______________________________________________ > > AstroPy mailing list > > AstroPy at scipy.org > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From archibald at astron.nl Tue Jun 9 17:23:36 2015 From: archibald at astron.nl (Anne Archibald) Date: Tue, 09 Jun 2015 21:23:36 +0000 Subject: [AstroPy] ffa in python In-Reply-To: <1433867131.3474.65.camel@newton.cx> References: <201506091518.t59FIG7B027076@blue-cove.com> <201506091621.t59GLLfl017029@blue-cove.com> <1433867131.3474.65.camel@newton.cx> Message-ID: This is why I asked about your problem parameters. If you have a modest number of photons, and a modest range of periods/other parameters to search, then you can just fold your photons with every trial period, and the question is then whether a distribution is uniform. The Rayleigh test is sensitive to sinusoidal variations but doesn't take advantage of higher harmonics; the H test takes advantage of the total power in all significant harmonics; the Kuiper test is sensitive to something else, maybe roughly the peak height. If you have a lot of periods to search, then folding your photons at every single period gets really expensive. Then you want some kind of faster algorithm. If you take an FFT of your time series, then the most naive approach - take the power in each Fourier bin and look for big values - is equivalent to the Rayleigh test. Tools like accelsearch incoherently add multiple harmonics, making them closer to the H test in principle. An FFA gives you the folded profile at each trial period; you can then use whatever method you like to evaluate the profile: taking the squared norm gives you the H test again, taking the height of the highest peak gives you a coherent sum of harmonics, which is more sensitive to sharply-peaked profiles. If you have really a lot of periods to search - like, years worth of photon-a-day data and you're searching for sub-second periods - then you need to discard some coherency to reduce your period search space; here you can take an FFT of all pairwise differences less than some coherency time and then apply (say) coherent harmonic summing. If you have other parameters to search, say the period changes with time, you can do some tricks like folding short segments, then shifting these and combining. So the problem depends a lot on what parameters you are searching, how many photons you have, and what kind of ultimate profile you're expecting (sharply-peaked, sinusoidal, multi-peaked, other). Anne P.S. the H test and incoherent summing of harmonic power have together found pulsations from a *lot* of pulsars. -A On Tue, Jun 9, 2015 at 6:25 PM Peter Williams wrote: > To search for periodicity in the arrival times of discrete events, > rather than continuously sampled data, the Rayleigh test is a good > place to start: > > http://adsabs.harvard.edu/abs/1994MNRAS.268..709B > http://www.astro.ubc.ca/people/jvw/ASTROSTATS/Answers/Chap9/chapter%209 > %20rayleigh%20test.pdf > > > Peter > > On Tue, 2015-06-09 at 09:21 -0700, R Schumacher wrote: > > I've only a cursory knowledge of FFA, but FFT assumes sinusoidal data > > (or sums of) and so may not best represent periodic pulses in, say, > > single ended data acquisition, or swept signals etc > > > > FFA is supposed to look for regularity in pulses > > http://en.wikipedia.org/wiki/Talk%3AFast_folding_algorithm > > Being an amateur astronomer I searched your use case > > https://books.google.com/books?id=OZ8tdN6qJcsC&pg=PA151&lpg=PA151&dq= > > fast+folding+algorithm&source=bl&ots=mwe0QvaHU5&sig=Xg1Kwad46yC_BMxyK > > u_GR37LaFA&hl=en&sa=X&ei=EhF3VYudBojiUeiogvAH&ved=0CDAQ6AEwBg > > > > > > - Ray > > > > > > At 08:33 AM 6/9/2015, you wrote: > > > Good ! Thanks ! > > > Any comment about the comparison between ffa and fft? > > > Hang > > > > > > On Tue, Jun 9, 2015 at 11:18 PM, R Schumacher > > > wrote: > > > https://github.com/petigura/FFA/blob/master/FFA.py > > > > > > > > > > > > At 07:49 AM 6/9/2015, Anne Archibald wrote: > > > > I don't think there's a general FFA algorithm anywhere, though > > > > Vlad Kondratiev wrote a tool for applying the FFA to radio data, > > > > and I have some python code lying around somewhere. If you're > > > > searching for periodicities, the standard procedure is to search > > > > an FFT with incoherent harmonic summing; there are some pulsar > > > > searching tools that will do it (PRESTO, for example) quickly and > > > > efficiently. If you only have a handful of photons, or a modest > > > > number of periods to search, there are direct approaches that are > > > > more sensitive. If you have really a long time series, there is a > > > > trick where you do an FFT on the pairwise differences of photon > > > > arrival times.?? > > > > > > > > In short, there are a whole range of procedures for searching for > > > > periodicity in data. Can you be a little more specific about the > > > > problem you are trying to solve? > > > > > > > > Anne > > > > > > > > On Tue, Jun 9, 2015 at 4:44 PM gonghang.naoc < > > > > ghang.naoc at gmail.com> wrote: > > > > Hi all, > > > > Is there any python resource about fast folding algorithm???? I > > > > have a series of arrival times for a lot of photons. I need to do > > > > some periodical test, but do not know how to start. > > > > Thank you. > > > > best > > > > Hang > > > > _______________________________________________ > > > > AstroPy mailing list > > > > AstroPy at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/astropy > > > > > > > > _______________________________________________ > > > > AstroPy mailing list > > > > AstroPy at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > > > AstroPy mailing list > > > AstroPy at scipy.org > > > http://mail.scipy.org/mailman/listinfo/astropy > > > > > > > > > _______________________________________________ > > > AstroPy mailing list > > > AstroPy at scipy.org > > _______________________________________________ > > AstroPy mailing list > > AstroPy at scipy.org > > http://mail.scipy.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonian.7 at buckeyemail.osu.edu Tue Jun 16 18:07:58 2015 From: simonian.7 at buckeyemail.osu.edu (Gregory Simonian) Date: Tue, 16 Jun 2015 18:07:58 -0400 Subject: [AstroPy] Changing dtype when copying masked table throws TypeError Message-ID: <55809E3E.6000602@buckeyemail.osu.edu> Hi all, I have a Table with several columns which were read in as strings because they had a combination of numeric and non-numeric characters. After cleaning out the non-numeric characters, I'd like to convert the dtype of the column from string to int. Looking at the astropy documentation for modifying tables [0], under the caveats section it states: "The data type for a column cannot be changed in place. In order to do this you must make a copy of the table with the column type changed appropriately." So I try making a new table (the table is small, so I don't mind the temporary memory hit), but I run into the error: "TypeError: Cannot set fill value of string with array of dtype int64" When checking the fill value of the column (which is masked from the reader), it's 'N/A', which makes sense that it wouldn't work. But when I change it to '0', I still get the same problem. I managed to find a minimal working example as: import numpy as np t = Table({'a': ['1']}, masked=True) t['a'].set_fill_value('0') u = Table(t, names=['a'], dtype=[np.int]) > TypeError: Cannot set fill value of string with array of dtype int64 When debugging the exception, it seems like the Table constructor doesn't convert the fill value of masked tables when generating a new table, which seems to defeat the ability to change the dtype of masked columns. I was wondering if other people have run by this, and if there's a more straightforward way of changing the dtype of a column. This is with Astropy 1.0.3 on Python 2.7.10 through Anaconda on CentOS 6.6. When checking to see if this was reported, I ran into issue #3809 [1], which might end up being the ultimate solution to my problem. It seems like a workaround would be: temp_column = np.ma.asanyarray(t['a'], np.int) del(t['a']) t['a'] = temp_column Thanks! Gregory Simonian [0]http://docs.astropy.org/en/stable/table/modify_table.html#caveats [1] https://github.com/astropy/astropy/issues/3809 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From cherinka at di.utoronto.ca Wed Jun 17 14:08:59 2015 From: cherinka at di.utoronto.ca (Brian Cherinka) Date: Wed, 17 Jun 2015 14:08:59 -0400 Subject: [AstroPy] ValueError in vstacking tables Message-ID: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> Hi, I?m trying to vstack (vertically stack) two Astropy tables with different numbers of columns. The tables appear to stack correctly without error. However, upon attempting to index a row in the table, it throws me this error ) failed: ValueError: setting an array element with a sequence.> I?m not trying to do anything fancy with mixed columns, or Quantities. Just stackin? two good ol? natural Tables together. One table has 84 columns, and another has 57, which are all a part of the 84 in the first table. At first glance I would guess it?s a mismatch in the dtypes of a particular column that I?m trying to merge. However, the columns all have the same dtype, format, and they are in the same order. I get no messages regarding any kind of merge errors. Any ideas on this? Thanks, Brian P.S. Full print of traceback below table3 = vstack([table1, table3],join_type=?outer?) print table3[0] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () ----> 1 print tmp3[0] /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in __repr__(self) 176 def __repr__(self): 177 return "<{3} {0} of table\n values={1!r}\n dtype={2}>".format( --> 178 self.index, self.as_void(), self.dtype, self.__class__.__name__) 179 180 /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __repr__(self) 5699 return self._data.__repr__() 5700 m = tuple(m) -> 5701 if not any(m): 5702 return self._data.__repr__() 5703 p = masked_print_option /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __call__(self, a, *args, **params) 6089 method = getattr(MaskedArray, method_name, None) 6090 if method is not None: -> 6091 return method(MaskedArray(a), *args, **params) 6092 # Still here ? OK, let's call the corresponding np function 6093 method = getattr(np, method_name) /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, hard_mask, shrink, **options) 2649 """ 2650 # Process data............ -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, ndmin=ndmin) 2652 _baseclass = getattr(data, '_baseclass', type(_data)) 2653 # Check that we're not erasing the mask.......... ValueError: setting an array element with a sequence. --------------------------------------- Brian Cherinka, Ph.D Dunlap Institute for Astronomy & Astrophysics University of Toronto Toronto, ON, Canada M5S 3H4 phone: 416-978-7299 email: cherinka at di.utoronto.ca ---------------------------------------- From aldcroft at head.cfa.harvard.edu Wed Jun 17 14:20:48 2015 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Wed, 17 Jun 2015 14:20:48 -0400 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> Message-ID: Hi Brian, Below is what I think is a minimal example of what you are describing. Can you look at this and first reproduce that this works for you, and second, can you guess what might be different in your full example vs. my minimal example? In [19]: t1 = Table([[1], [2]], names=['a', 'b']) In [20]: t2 = Table([[3], [4]], names=['b', 'c']) In [23]: t12 = vstack([t1, t2], join_type='outer') In [24]: t12 Out[24]: a b c int64 int64 int64 ----- ----- ----- 1 2 -- -- 3 4 In [25]: t12[0] Out[25]: On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka wrote: > Hi, > > I?m trying to vstack (vertically stack) two Astropy tables with different > numbers of columns. The tables appear to stack correctly without error. > However, upon attempting to index a row in the table, it throws me this > error > > ) failed: ValueError: setting > an array element with a sequence.> > > I?m not trying to do anything fancy with mixed columns, or Quantities. > Just stackin? two good ol? natural Tables together. One table has 84 > columns, and another has 57, which are all a part of the 84 in the first > table. At first glance I would guess it?s a mismatch in the dtypes of a > particular column that I?m trying to merge. However, the columns all have > the same dtype, format, and they are in the same order. I get no messages > regarding any kind of merge errors. > > Any ideas on this? > > Thanks, Brian > > P.S. Full print of traceback below > > table3 = vstack([table1, table3],join_type=?outer?) > print table3[0] > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > in () > ----> 1 print tmp3[0] > > /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in > __repr__(self) > 176 def __repr__(self): > 177 return "<{3} {0} of table\n values={1!r}\n > dtype={2}>".format( > --> 178 self.index, self.as_void(), self.dtype, > self.__class__.__name__) > 179 > 180 > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in > __repr__(self) > 5699 return self._data.__repr__() > 5700 m = tuple(m) > -> 5701 if not any(m): > 5702 return self._data.__repr__() > 5703 p = masked_print_option > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in > __call__(self, a, *args, **params) > 6089 method = getattr(MaskedArray, method_name, None) > 6090 if method is not None: > -> 6091 return method(MaskedArray(a), *args, **params) > 6092 # Still here ? OK, let's call the corresponding np function > 6093 method = getattr(np, method_name) > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in > __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, > hard_mask, shrink, **options) > 2649 """ > 2650 # Process data............ > -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, > ndmin=ndmin) > 2652 _baseclass = getattr(data, '_baseclass', type(_data)) > 2653 # Check that we're not erasing the mask.......... > > ValueError: setting an array element with a sequence. > > > > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cherinka at di.utoronto.ca Wed Jun 17 15:25:22 2015 From: cherinka at di.utoronto.ca (Brian Cherinka) Date: Wed, 17 Jun 2015 15:25:22 -0400 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> Message-ID: <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> Hi Thomas, Your example does indeed work for me. I cannot yet guess why it doesn?t work in my full example. However it?s not quite a complete reconstruction of what I?m doing. Technically, I?m hstacking two tables together, and then vstacking that result with another table. Something like this, which works just fine. In [382]: t1 = Table([[1], [2]], names=['a', 'b']) In [383]: t2 = Table([[3], [4]], names=['c','d']) In [384]: t3 = hstack([t1,t2],join_type='outer') In [386]: t4 = Table([[5], [6]], names=['a', 'b']) In [387]: t34 = vstack([t3,t4],join_type='outer') In [388]: t34 Out[388]:
a b c d int64 int64 int64 int64 ----- ----- ----- ----- 1 2 3 4 5 6 -- -- In [389]: t34[0] Out[389]: In my full example, I did find though that I can row index the individual tables t1 and t2, but cannot row index table t3 after hstacking. I get the same ValueError. When I attempt to vstack with table t4, that error propagates forward. I can row index t4 but not t34. Looking at it a bit more, it might be that my rows in the initial tables t1, t2 do not match up with one another, and I appear to have one extra row in table t2, than I should. I?ll dig a bit more. Cheers, Brian --------------------------------------- Brian Cherinka, Ph.D Dunlap Institute for Astronomy & Astrophysics University of Toronto Toronto, ON, Canada M5S 3H4 phone: 416-978-7299 email: cherinka at di.utoronto.ca ---------------------------------------- On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas wrote: > Hi Brian, > > Below is what I think is a minimal example of what you are describing. Can you look at this and first reproduce that this works for you, and second, can you guess what might be different in your full example vs. my minimal example? > > In [19]: t1 = Table([[1], [2]], names=['a', 'b']) > > In [20]: t2 = Table([[3], [4]], names=['b', 'c']) > > In [23]: t12 = vstack([t1, t2], join_type='outer') > > In [24]: t12 > Out[24]: >
> a b c > int64 int64 int64 > ----- ----- ----- > 1 2 -- > -- 3 4 > > In [25]: t12[0] > Out[25]: > values=(1, 2, --) > dtype=[('a', ' > > > On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka wrote: > Hi, > > I?m trying to vstack (vertically stack) two Astropy tables with different numbers of columns. The tables appear to stack correctly without error. However, upon attempting to index a row in the table, it throws me this error > > ) failed: ValueError: setting an array element with a sequence.> > > I?m not trying to do anything fancy with mixed columns, or Quantities. Just stackin? two good ol? natural Tables together. One table has 84 columns, and another has 57, which are all a part of the 84 in the first table. At first glance I would guess it?s a mismatch in the dtypes of a particular column that I?m trying to merge. However, the columns all have the same dtype, format, and they are in the same order. I get no messages regarding any kind of merge errors. > > Any ideas on this? > > Thanks, Brian > > P.S. Full print of traceback below > > table3 = vstack([table1, table3],join_type=?outer?) > print table3[0] > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > in () > ----> 1 print tmp3[0] > > /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in __repr__(self) > 176 def __repr__(self): > 177 return "<{3} {0} of table\n values={1!r}\n dtype={2}>".format( > --> 178 self.index, self.as_void(), self.dtype, self.__class__.__name__) > 179 > 180 > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __repr__(self) > 5699 return self._data.__repr__() > 5700 m = tuple(m) > -> 5701 if not any(m): > 5702 return self._data.__repr__() > 5703 p = masked_print_option > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __call__(self, a, *args, **params) > 6089 method = getattr(MaskedArray, method_name, None) > 6090 if method is not None: > -> 6091 return method(MaskedArray(a), *args, **params) > 6092 # Still here ? OK, let's call the corresponding np function > 6093 method = getattr(np, method_name) > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, hard_mask, shrink, **options) > 2649 """ > 2650 # Process data............ > -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, ndmin=ndmin) > 2652 _baseclass = getattr(data, '_baseclass', type(_data)) > 2653 # Check that we're not erasing the mask.......... > > ValueError: setting an array element with a sequence. > > > > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From cherinka at di.utoronto.ca Thu Jun 18 11:34:11 2015 From: cherinka at di.utoronto.ca (Brian Cherinka) Date: Thu, 18 Jun 2015 11:34:11 -0400 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> Message-ID: <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Anymore thoughts on this? My table length issue wasn?t the problem. I?m stilling having a problem with my resulting hstacks and vstacks. The stacks happen without error. I just lose the ability to index by row. Cheers, Brian --------------------------------------- Brian Cherinka, Ph.D Dunlap Institute for Astronomy & Astrophysics University of Toronto Toronto, ON, Canada M5S 3H4 phone: 416-978-7299 email: cherinka at di.utoronto.ca ---------------------------------------- On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: > Hi Thomas, > > Your example does indeed work for me. I cannot yet guess why it doesn?t work in my full example. However it?s not quite a complete reconstruction of what I?m doing. Technically, I?m hstacking two tables together, and then vstacking that result with another table. Something like this, which works just fine. > > In [382]: t1 = Table([[1], [2]], names=['a', 'b']) > > In [383]: t2 = Table([[3], [4]], names=['c','d']) > > In [384]: t3 = hstack([t1,t2],join_type='outer') > > In [386]: t4 = Table([[5], [6]], names=['a', 'b']) > > In [387]: t34 = vstack([t3,t4],join_type='outer') > > In [388]: t34 > Out[388]: >
> a b c d > int64 int64 int64 int64 > ----- ----- ----- ----- > 1 2 3 4 > 5 6 -- -- > > In [389]: t34[0] > Out[389]: > values=(1, 2, 3, 4) > dtype=[('a', ' > > In my full example, I did find though that I can row index the individual tables t1 and t2, but cannot row index table t3 after hstacking. I get the same ValueError. When I attempt to vstack with table t4, that error propagates forward. I can row index t4 but not t34. Looking at it a bit more, it might be that my rows in the initial tables t1, t2 do not match up with one another, and I appear to have one extra row in table t2, than I should. I?ll dig a bit more. > > Cheers, Brian > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas wrote: > >> Hi Brian, >> >> Below is what I think is a minimal example of what you are describing. Can you look at this and first reproduce that this works for you, and second, can you guess what might be different in your full example vs. my minimal example? >> >> In [19]: t1 = Table([[1], [2]], names=['a', 'b']) >> >> In [20]: t2 = Table([[3], [4]], names=['b', 'c']) >> >> In [23]: t12 = vstack([t1, t2], join_type='outer') >> >> In [24]: t12 >> Out[24]: >>
>> a b c >> int64 int64 int64 >> ----- ----- ----- >> 1 2 -- >> -- 3 4 >> >> In [25]: t12[0] >> Out[25]: >> > values=(1, 2, --) >> dtype=[('a', ' >> >> >> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka wrote: >> Hi, >> >> I?m trying to vstack (vertically stack) two Astropy tables with different numbers of columns. The tables appear to stack correctly without error. However, upon attempting to index a row in the table, it throws me this error >> >> ) failed: ValueError: setting an array element with a sequence.> >> >> I?m not trying to do anything fancy with mixed columns, or Quantities. Just stackin? two good ol? natural Tables together. One table has 84 columns, and another has 57, which are all a part of the 84 in the first table. At first glance I would guess it?s a mismatch in the dtypes of a particular column that I?m trying to merge. However, the columns all have the same dtype, format, and they are in the same order. I get no messages regarding any kind of merge errors. >> >> Any ideas on this? >> >> Thanks, Brian >> >> P.S. Full print of traceback below >> >> table3 = vstack([table1, table3],join_type=?outer?) >> print table3[0] >> --------------------------------------------------------------------------- >> ValueError Traceback (most recent call last) >> in () >> ----> 1 print tmp3[0] >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in __repr__(self) >> 176 def __repr__(self): >> 177 return "<{3} {0} of table\n values={1!r}\n dtype={2}>".format( >> --> 178 self.index, self.as_void(), self.dtype, self.__class__.__name__) >> 179 >> 180 >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __repr__(self) >> 5699 return self._data.__repr__() >> 5700 m = tuple(m) >> -> 5701 if not any(m): >> 5702 return self._data.__repr__() >> 5703 p = masked_print_option >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __call__(self, a, *args, **params) >> 6089 method = getattr(MaskedArray, method_name, None) >> 6090 if method is not None: >> -> 6091 return method(MaskedArray(a), *args, **params) >> 6092 # Still here ? OK, let's call the corresponding np function >> 6093 method = getattr(np, method_name) >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, hard_mask, shrink, **options) >> 2649 """ >> 2650 # Process data............ >> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, ndmin=ndmin) >> 2652 _baseclass = getattr(data, '_baseclass', type(_data)) >> 2653 # Check that we're not erasing the mask.......... >> >> ValueError: setting an array element with a sequence. >> >> >> >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Fri Jun 19 08:43:48 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 19 Jun 2015 13:43:48 +0100 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Message-ID: Hi Brian, Could you let us know what table1.dtype and table3.dtype are equal to, and how many rows are in each table? Thanks! Tom On 18 June 2015 at 16:34, Brian Cherinka wrote: > Anymore thoughts on this? My table length issue wasn?t the problem. I?m > stilling having a problem with my resulting hstacks and vstacks. The stacks > happen without error. I just lose the ability to index by row. > > Cheers, Brian > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: > > Hi Thomas, > > Your example does indeed work for me. I cannot yet guess why it doesn?t > work in my full example. However it?s not quite a complete reconstruction > of what I?m doing. Technically, I?m hstacking two tables together, and then > vstacking that result with another table. Something like this, which works > just fine. > > In [382]: t1 = Table([[1], [2]], names=['a', 'b']) > > In [383]: t2 = Table([[3], [4]], names=['c','d']) > > In [384]: t3 = hstack([t1,t2],join_type='outer') > > In [386]: t4 = Table([[5], [6]], names=['a', 'b']) > > In [387]: t34 = vstack([t3,t4],join_type='outer') > > In [388]: t34 > Out[388]: >
> a b c d > int64 int64 int64 int64 > ----- ----- ----- ----- > 1 2 3 4 > 5 6 -- -- > > In [389]: t34[0] > Out[389]: > values=(1, 2, 3, 4) > dtype=[('a', ' > > In my full example, I did find though that I can row index the individual > tables t1 and t2, but cannot row index table t3 after hstacking. I get the > same ValueError. When I attempt to vstack with table t4, that error > propagates forward. I can row index t4 but not t34. Looking at it a bit > more, it might be that my rows in the initial tables t1, t2 do not match up > with one another, and I appear to have one extra row in table t2, than I > should. I?ll dig a bit more. > > Cheers, Brian > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas > wrote: > > Hi Brian, > > Below is what I think is a minimal example of what you are describing. Can > you look at this and first reproduce that this works for you, and second, > can you guess what might be different in your full example vs. my minimal > example? > > In [19]: t1 = Table([[1], [2]], names=['a', 'b']) > > In [20]: t2 = Table([[3], [4]], names=['b', 'c']) > > In [23]: t12 = vstack([t1, t2], join_type='outer') > > In [24]: t12 > Out[24]: >
> a b c > int64 int64 int64 > ----- ----- ----- > 1 2 -- > -- 3 4 > > In [25]: t12[0] > Out[25]: > values=(1, 2, --) > dtype=[('a', ' > > > On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka > wrote: >> >> Hi, >> >> I?m trying to vstack (vertically stack) two Astropy tables with different >> numbers of columns. The tables appear to stack correctly without error. >> However, upon attempting to index a row in the table, it throws me this >> error >> >> ) failed: ValueError: setting >> an array element with a sequence.> >> >> I?m not trying to do anything fancy with mixed columns, or Quantities. >> Just stackin? two good ol? natural Tables together. One table has 84 >> columns, and another has 57, which are all a part of the 84 in the first >> table. At first glance I would guess it?s a mismatch in the dtypes of a >> particular column that I?m trying to merge. However, the columns all have >> the same dtype, format, and they are in the same order. I get no messages >> regarding any kind of merge errors. >> >> Any ideas on this? >> >> Thanks, Brian >> >> P.S. Full print of traceback below >> >> table3 = vstack([table1, table3],join_type=?outer?) >> print table3[0] >> >> --------------------------------------------------------------------------- >> ValueError Traceback (most recent call >> last) >> in () >> ----> 1 print tmp3[0] >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in >> __repr__(self) >> 176 def __repr__(self): >> 177 return "<{3} {0} of table\n values={1!r}\n >> dtype={2}>".format( >> --> 178 self.index, self.as_void(), self.dtype, >> self.__class__.__name__) >> 179 >> 180 >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >> __repr__(self) >> 5699 return self._data.__repr__() >> 5700 m = tuple(m) >> -> 5701 if not any(m): >> 5702 return self._data.__repr__() >> 5703 p = masked_print_option >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >> __call__(self, a, *args, **params) >> 6089 method = getattr(MaskedArray, method_name, None) >> 6090 if method is not None: >> -> 6091 return method(MaskedArray(a), *args, **params) >> 6092 # Still here ? OK, let's call the corresponding np >> function >> 6093 method = getattr(np, method_name) >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, >> hard_mask, shrink, **options) >> 2649 """ >> 2650 # Process data............ >> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, >> ndmin=ndmin) >> 2652 _baseclass = getattr(data, '_baseclass', type(_data)) >> 2653 # Check that we're not erasing the mask.......... >> >> ValueError: setting an array element with a sequence. >> >> >> >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From cherinka at di.utoronto.ca Tue Jun 23 09:49:48 2015 From: cherinka at di.utoronto.ca (Brian Cherinka) Date: Tue, 23 Jun 2015 09:49:48 -0400 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Message-ID: Hi Thomas, > Could you let us know what table1.dtype and table3.dtype are equal to, > and how many rows are in each table? The two tables, t1 and t2, that I am hstacking have 48 columns, and 36 columns, respectively, each with 29 rows. The dtype of those columns are a mix of things. Mostly float and string scalars. I do have several columns of float arrays. After hstacking, the resulting table (84 columns, 29 rows) is combined as expected, although I can no longer index the rows. Then I vstack it with another table 3, which has the same number of columns, and same dtype. I can?t row index that table either. Cheers, Brian --------------------------------------- Brian Cherinka, Ph.D Dunlap Institute for Astronomy & Astrophysics University of Toronto Toronto, ON, Canada M5S 3H4 phone: 416-978-7299 email: cherinka at di.utoronto.ca ---------------------------------------- On Jun 19, 2015, at 8:43 AM, Thomas Robitaille wrote: > Hi Brian, > > Could you let us know what table1.dtype and table3.dtype are equal to, > and how many rows are in each table? > > Thanks! > Tom > > > On 18 June 2015 at 16:34, Brian Cherinka wrote: >> Anymore thoughts on this? My table length issue wasn?t the problem. I?m >> stilling having a problem with my resulting hstacks and vstacks. The stacks >> happen without error. I just lose the ability to index by row. >> >> Cheers, Brian >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: >> >> Hi Thomas, >> >> Your example does indeed work for me. I cannot yet guess why it doesn?t >> work in my full example. However it?s not quite a complete reconstruction >> of what I?m doing. Technically, I?m hstacking two tables together, and then >> vstacking that result with another table. Something like this, which works >> just fine. >> >> In [382]: t1 = Table([[1], [2]], names=['a', 'b']) >> >> In [383]: t2 = Table([[3], [4]], names=['c','d']) >> >> In [384]: t3 = hstack([t1,t2],join_type='outer') >> >> In [386]: t4 = Table([[5], [6]], names=['a', 'b']) >> >> In [387]: t34 = vstack([t3,t4],join_type='outer') >> >> In [388]: t34 >> Out[388]: >>
>> a b c d >> int64 int64 int64 int64 >> ----- ----- ----- ----- >> 1 2 3 4 >> 5 6 -- -- >> >> In [389]: t34[0] >> Out[389]: >> > values=(1, 2, 3, 4) >> dtype=[('a', ' >> >> In my full example, I did find though that I can row index the individual >> tables t1 and t2, but cannot row index table t3 after hstacking. I get the >> same ValueError. When I attempt to vstack with table t4, that error >> propagates forward. I can row index t4 but not t34. Looking at it a bit >> more, it might be that my rows in the initial tables t1, t2 do not match up >> with one another, and I appear to have one extra row in table t2, than I >> should. I?ll dig a bit more. >> >> Cheers, Brian >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas >> wrote: >> >> Hi Brian, >> >> Below is what I think is a minimal example of what you are describing. Can >> you look at this and first reproduce that this works for you, and second, >> can you guess what might be different in your full example vs. my minimal >> example? >> >> In [19]: t1 = Table([[1], [2]], names=['a', 'b']) >> >> In [20]: t2 = Table([[3], [4]], names=['b', 'c']) >> >> In [23]: t12 = vstack([t1, t2], join_type='outer') >> >> In [24]: t12 >> Out[24]: >>
>> a b c >> int64 int64 int64 >> ----- ----- ----- >> 1 2 -- >> -- 3 4 >> >> In [25]: t12[0] >> Out[25]: >> > values=(1, 2, --) >> dtype=[('a', ' >> >> >> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka >> wrote: >>> >>> Hi, >>> >>> I?m trying to vstack (vertically stack) two Astropy tables with different >>> numbers of columns. The tables appear to stack correctly without error. >>> However, upon attempting to index a row in the table, it throws me this >>> error >>> >>> ) failed: ValueError: setting >>> an array element with a sequence.> >>> >>> I?m not trying to do anything fancy with mixed columns, or Quantities. >>> Just stackin? two good ol? natural Tables together. One table has 84 >>> columns, and another has 57, which are all a part of the 84 in the first >>> table. At first glance I would guess it?s a mismatch in the dtypes of a >>> particular column that I?m trying to merge. However, the columns all have >>> the same dtype, format, and they are in the same order. I get no messages >>> regarding any kind of merge errors. >>> >>> Any ideas on this? >>> >>> Thanks, Brian >>> >>> P.S. Full print of traceback below >>> >>> table3 = vstack([table1, table3],join_type=?outer?) >>> print table3[0] >>> >>> --------------------------------------------------------------------------- >>> ValueError Traceback (most recent call >>> last) >>> in () >>> ----> 1 print tmp3[0] >>> >>> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in >>> __repr__(self) >>> 176 def __repr__(self): >>> 177 return "<{3} {0} of table\n values={1!r}\n >>> dtype={2}>".format( >>> --> 178 self.index, self.as_void(), self.dtype, >>> self.__class__.__name__) >>> 179 >>> 180 >>> >>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>> __repr__(self) >>> 5699 return self._data.__repr__() >>> 5700 m = tuple(m) >>> -> 5701 if not any(m): >>> 5702 return self._data.__repr__() >>> 5703 p = masked_print_option >>> >>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>> __call__(self, a, *args, **params) >>> 6089 method = getattr(MaskedArray, method_name, None) >>> 6090 if method is not None: >>> -> 6091 return method(MaskedArray(a), *args, **params) >>> 6092 # Still here ? OK, let's call the corresponding np >>> function >>> 6093 method = getattr(np, method_name) >>> >>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, >>> hard_mask, shrink, **options) >>> 2649 """ >>> 2650 # Process data............ >>> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, >>> ndmin=ndmin) >>> 2652 _baseclass = getattr(data, '_baseclass', type(_data)) >>> 2653 # Check that we're not erasing the mask.......... >>> >>> ValueError: setting an array element with a sequence. >>> >>> >>> >>> >>> --------------------------------------- >>> Brian Cherinka, Ph.D >>> Dunlap Institute for Astronomy & Astrophysics >>> University of Toronto >>> Toronto, ON, Canada M5S 3H4 >>> phone: 416-978-7299 >>> email: cherinka at di.utoronto.ca >>> ---------------------------------------- >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From thomas.robitaille at gmail.com Tue Jun 23 09:53:36 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 23 Jun 2015 14:53:36 +0100 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Message-ID: Hi Brian, On 23 June 2015 at 14:49, Brian Cherinka wrote: > Hi Thomas, > >> Could you let us know what table1.dtype and table3.dtype are equal to, >> and how many rows are in each table? > > The two tables, t1 and t2, that I am hstacking have 48 columns, and 36 columns, respectively, each with 29 rows. The dtype of those columns are a mix of things. Mostly float and string scalars. I do have several columns of float arrays. Just to clarify, could you copy and paste the exact output from table1.dtype and table3.dtype here so that we can try and reproduce the exact issue? (it may be a corner case due to the type of one of the columns). Thanks! Tom > After hstacking, the resulting table (84 columns, 29 rows) is combined as expected, although I can no longer index the rows. Then I vstack it with another table 3, which has the same number of columns, and same dtype. I can?t row index that table either. > > Cheers, Brian > > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 19, 2015, at 8:43 AM, Thomas Robitaille wrote: > >> Hi Brian, >> >> Could you let us know what table1.dtype and table3.dtype are equal to, >> and how many rows are in each table? >> >> Thanks! >> Tom >> >> >> On 18 June 2015 at 16:34, Brian Cherinka wrote: >>> Anymore thoughts on this? My table length issue wasn?t the problem. I?m >>> stilling having a problem with my resulting hstacks and vstacks. The stacks >>> happen without error. I just lose the ability to index by row. >>> >>> Cheers, Brian >>> >>> --------------------------------------- >>> Brian Cherinka, Ph.D >>> Dunlap Institute for Astronomy & Astrophysics >>> University of Toronto >>> Toronto, ON, Canada M5S 3H4 >>> phone: 416-978-7299 >>> email: cherinka at di.utoronto.ca >>> ---------------------------------------- >>> >>> On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: >>> >>> Hi Thomas, >>> >>> Your example does indeed work for me. I cannot yet guess why it doesn?t >>> work in my full example. However it?s not quite a complete reconstruction >>> of what I?m doing. Technically, I?m hstacking two tables together, and then >>> vstacking that result with another table. Something like this, which works >>> just fine. >>> >>> In [382]: t1 = Table([[1], [2]], names=['a', 'b']) >>> >>> In [383]: t2 = Table([[3], [4]], names=['c','d']) >>> >>> In [384]: t3 = hstack([t1,t2],join_type='outer') >>> >>> In [386]: t4 = Table([[5], [6]], names=['a', 'b']) >>> >>> In [387]: t34 = vstack([t3,t4],join_type='outer') >>> >>> In [388]: t34 >>> Out[388]: >>>
>>> a b c d >>> int64 int64 int64 int64 >>> ----- ----- ----- ----- >>> 1 2 3 4 >>> 5 6 -- -- >>> >>> In [389]: t34[0] >>> Out[389]: >>> >> values=(1, 2, 3, 4) >>> dtype=[('a', ' >>> >>> In my full example, I did find though that I can row index the individual >>> tables t1 and t2, but cannot row index table t3 after hstacking. I get the >>> same ValueError. When I attempt to vstack with table t4, that error >>> propagates forward. I can row index t4 but not t34. Looking at it a bit >>> more, it might be that my rows in the initial tables t1, t2 do not match up >>> with one another, and I appear to have one extra row in table t2, than I >>> should. I?ll dig a bit more. >>> >>> Cheers, Brian >>> >>> --------------------------------------- >>> Brian Cherinka, Ph.D >>> Dunlap Institute for Astronomy & Astrophysics >>> University of Toronto >>> Toronto, ON, Canada M5S 3H4 >>> phone: 416-978-7299 >>> email: cherinka at di.utoronto.ca >>> ---------------------------------------- >>> >>> On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas >>> wrote: >>> >>> Hi Brian, >>> >>> Below is what I think is a minimal example of what you are describing. Can >>> you look at this and first reproduce that this works for you, and second, >>> can you guess what might be different in your full example vs. my minimal >>> example? >>> >>> In [19]: t1 = Table([[1], [2]], names=['a', 'b']) >>> >>> In [20]: t2 = Table([[3], [4]], names=['b', 'c']) >>> >>> In [23]: t12 = vstack([t1, t2], join_type='outer') >>> >>> In [24]: t12 >>> Out[24]: >>>
>>> a b c >>> int64 int64 int64 >>> ----- ----- ----- >>> 1 2 -- >>> -- 3 4 >>> >>> In [25]: t12[0] >>> Out[25]: >>> >> values=(1, 2, --) >>> dtype=[('a', ' >>> >>> >>> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka >>> wrote: >>>> >>>> Hi, >>>> >>>> I?m trying to vstack (vertically stack) two Astropy tables with different >>>> numbers of columns. The tables appear to stack correctly without error. >>>> However, upon attempting to index a row in the table, it throws me this >>>> error >>>> >>>> ) failed: ValueError: setting >>>> an array element with a sequence.> >>>> >>>> I?m not trying to do anything fancy with mixed columns, or Quantities. >>>> Just stackin? two good ol? natural Tables together. One table has 84 >>>> columns, and another has 57, which are all a part of the 84 in the first >>>> table. At first glance I would guess it?s a mismatch in the dtypes of a >>>> particular column that I?m trying to merge. However, the columns all have >>>> the same dtype, format, and they are in the same order. I get no messages >>>> regarding any kind of merge errors. >>>> >>>> Any ideas on this? >>>> >>>> Thanks, Brian >>>> >>>> P.S. Full print of traceback below >>>> >>>> table3 = vstack([table1, table3],join_type=?outer?) >>>> print table3[0] >>>> >>>> --------------------------------------------------------------------------- >>>> ValueError Traceback (most recent call >>>> last) >>>> in () >>>> ----> 1 print tmp3[0] >>>> >>>> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in >>>> __repr__(self) >>>> 176 def __repr__(self): >>>> 177 return "<{3} {0} of table\n values={1!r}\n >>>> dtype={2}>".format( >>>> --> 178 self.index, self.as_void(), self.dtype, >>>> self.__class__.__name__) >>>> 179 >>>> 180 >>>> >>>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>>> __repr__(self) >>>> 5699 return self._data.__repr__() >>>> 5700 m = tuple(m) >>>> -> 5701 if not any(m): >>>> 5702 return self._data.__repr__() >>>> 5703 p = masked_print_option >>>> >>>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>>> __call__(self, a, *args, **params) >>>> 6089 method = getattr(MaskedArray, method_name, None) >>>> 6090 if method is not None: >>>> -> 6091 return method(MaskedArray(a), *args, **params) >>>> 6092 # Still here ? OK, let's call the corresponding np >>>> function >>>> 6093 method = getattr(np, method_name) >>>> >>>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>>> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, >>>> hard_mask, shrink, **options) >>>> 2649 """ >>>> 2650 # Process data............ >>>> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, >>>> ndmin=ndmin) >>>> 2652 _baseclass = getattr(data, '_baseclass', type(_data)) >>>> 2653 # Check that we're not erasing the mask.......... >>>> >>>> ValueError: setting an array element with a sequence. >>>> >>>> >>>> >>>> >>>> --------------------------------------- >>>> Brian Cherinka, Ph.D >>>> Dunlap Institute for Astronomy & Astrophysics >>>> University of Toronto >>>> Toronto, ON, Canada M5S 3H4 >>>> phone: 416-978-7299 >>>> email: cherinka at di.utoronto.ca >>>> ---------------------------------------- >>>> >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/astropy >>> >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >>> >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >>> >>> >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >>> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From cherinka at di.utoronto.ca Tue Jun 23 09:58:27 2015 From: cherinka at di.utoronto.ca (Brian Cherinka) Date: Tue, 23 Jun 2015 09:58:27 -0400 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Message-ID: Hi Thomas, In [187]: t1.dtype Out[187]: dtype([('plate', '>i8'), ('ifudsgn', 'S5'), ('plateifu', 'S9'), ('mangaid', 'S4'), ('versdrp2', 'S30'), ('versdrp3', 'S30'), ('verscore', 'S30'), ('versutil', 'S30'), ('platetyp', 'S30'), ('srvymode', 'S30'), ('objRA', '>f8'), ('objDEC', '>f8'), ('objGLON', '>f8'), ('objGLAT', '>f8'), ('ebvgal', '>f8'), ('nexp', '>i8'), ('exptime', '>f8'), ('drp3qual', '>i8'), ('bluesn2', '>f8'), ('redsn2', '>f8'), ('harname', 'S30'), ('frlplug', '>i8'), ('cartid', 'S30'), ('designid', '>i8'), ('cenRA', '>f8'), ('cenDEC', '>f8'), ('airmsmin', '>f8'), ('airmsmed', '>f8'), ('airmsmax', '>f8'), ('seemin', '>f8'), ('seemed', '>f8'), ('seemax', '>f8'), ('transmin', '>f8'), ('transmed', '>f8'), ('transmax', '>f8'), ('mjdmin', '>i8'), ('mjdmed', '>i8'), ('mjdmax', '>i8'), ('ufwhm', '>f8'), ('gfwhm', '>f8'), ('rfwhm', '>f8'), ('ifwhm', '>f8'), ('zfwhm', '>f8'), ('mngtarg1', '>i8'), ('mngtarg2', '>i8'), ('mngtarg3', '>i8'), ('catidnum', '>i8'), ('plttarg', 'S30')]) In [188]: t2.dtype Out[188]: dtype([('manga_tileid', ') failed: ValueError: setting an array element with a sequence.> then vstacking t3 with t4 completes with no errors, but gives the same error when row indexing. Cheers, Brian --------------------------------------- Brian Cherinka, Ph.D Dunlap Institute for Astronomy & Astrophysics University of Toronto Toronto, ON, Canada M5S 3H4 phone: 416-978-7299 email: cherinka at di.utoronto.ca ---------------------------------------- On Jun 23, 2015, at 9:53 AM, Thomas Robitaille wrote: > Hi Brian, > > On 23 June 2015 at 14:49, Brian Cherinka wrote: >> Hi Thomas, >> >>> Could you let us know what table1.dtype and table3.dtype are equal to, >>> and how many rows are in each table? >> >> The two tables, t1 and t2, that I am hstacking have 48 columns, and 36 columns, respectively, each with 29 rows. The dtype of those columns are a mix of things. Mostly float and string scalars. I do have several columns of float arrays. > > Just to clarify, could you copy and paste the exact output from > table1.dtype and table3.dtype here so that we can try and reproduce > the exact issue? (it may be a corner case due to the type of one of > the columns). > > Thanks! > Tom > >> After hstacking, the resulting table (84 columns, 29 rows) is combined as expected, although I can no longer index the rows. Then I vstack it with another table 3, which has the same number of columns, and same dtype. I can?t row index that table either. > > >> >> Cheers, Brian >> >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 19, 2015, at 8:43 AM, Thomas Robitaille wrote: >> >>> Hi Brian, >>> >>> Could you let us know what table1.dtype and table3.dtype are equal to, >>> and how many rows are in each table? >>> >>> Thanks! >>> Tom >>> >>> >>> On 18 June 2015 at 16:34, Brian Cherinka wrote: >>>> Anymore thoughts on this? My table length issue wasn?t the problem. I?m >>>> stilling having a problem with my resulting hstacks and vstacks. The stacks >>>> happen without error. I just lose the ability to index by row. >>>> >>>> Cheers, Brian >>>> >>>> --------------------------------------- >>>> Brian Cherinka, Ph.D >>>> Dunlap Institute for Astronomy & Astrophysics >>>> University of Toronto >>>> Toronto, ON, Canada M5S 3H4 >>>> phone: 416-978-7299 >>>> email: cherinka at di.utoronto.ca >>>> ---------------------------------------- >>>> >>>> On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: >>>> >>>> Hi Thomas, >>>> >>>> Your example does indeed work for me. I cannot yet guess why it doesn?t >>>> work in my full example. However it?s not quite a complete reconstruction >>>> of what I?m doing. Technically, I?m hstacking two tables together, and then >>>> vstacking that result with another table. Something like this, which works >>>> just fine. >>>> >>>> In [382]: t1 = Table([[1], [2]], names=['a', 'b']) >>>> >>>> In [383]: t2 = Table([[3], [4]], names=['c','d']) >>>> >>>> In [384]: t3 = hstack([t1,t2],join_type='outer') >>>> >>>> In [386]: t4 = Table([[5], [6]], names=['a', 'b']) >>>> >>>> In [387]: t34 = vstack([t3,t4],join_type='outer') >>>> >>>> In [388]: t34 >>>> Out[388]: >>>>
>>>> a b c d >>>> int64 int64 int64 int64 >>>> ----- ----- ----- ----- >>>> 1 2 3 4 >>>> 5 6 -- -- >>>> >>>> In [389]: t34[0] >>>> Out[389]: >>>> >>> values=(1, 2, 3, 4) >>>> dtype=[('a', ' >>>> >>>> In my full example, I did find though that I can row index the individual >>>> tables t1 and t2, but cannot row index table t3 after hstacking. I get the >>>> same ValueError. When I attempt to vstack with table t4, that error >>>> propagates forward. I can row index t4 but not t34. Looking at it a bit >>>> more, it might be that my rows in the initial tables t1, t2 do not match up >>>> with one another, and I appear to have one extra row in table t2, than I >>>> should. I?ll dig a bit more. >>>> >>>> Cheers, Brian >>>> >>>> --------------------------------------- >>>> Brian Cherinka, Ph.D >>>> Dunlap Institute for Astronomy & Astrophysics >>>> University of Toronto >>>> Toronto, ON, Canada M5S 3H4 >>>> phone: 416-978-7299 >>>> email: cherinka at di.utoronto.ca >>>> ---------------------------------------- >>>> >>>> On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas >>>> wrote: >>>> >>>> Hi Brian, >>>> >>>> Below is what I think is a minimal example of what you are describing. Can >>>> you look at this and first reproduce that this works for you, and second, >>>> can you guess what might be different in your full example vs. my minimal >>>> example? >>>> >>>> In [19]: t1 = Table([[1], [2]], names=['a', 'b']) >>>> >>>> In [20]: t2 = Table([[3], [4]], names=['b', 'c']) >>>> >>>> In [23]: t12 = vstack([t1, t2], join_type='outer') >>>> >>>> In [24]: t12 >>>> Out[24]: >>>>
>>>> a b c >>>> int64 int64 int64 >>>> ----- ----- ----- >>>> 1 2 -- >>>> -- 3 4 >>>> >>>> In [25]: t12[0] >>>> Out[25]: >>>> >>> values=(1, 2, --) >>>> dtype=[('a', ' >>>> >>>> >>>> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I?m trying to vstack (vertically stack) two Astropy tables with different >>>>> numbers of columns. The tables appear to stack correctly without error. >>>>> However, upon attempting to index a row in the table, it throws me this >>>>> error >>>>> >>>>> ) failed: ValueError: setting >>>>> an array element with a sequence.> >>>>> >>>>> I?m not trying to do anything fancy with mixed columns, or Quantities. >>>>> Just stackin? two good ol? natural Tables together. One table has 84 >>>>> columns, and another has 57, which are all a part of the 84 in the first >>>>> table. At first glance I would guess it?s a mismatch in the dtypes of a >>>>> particular column that I?m trying to merge. However, the columns all have >>>>> the same dtype, format, and they are in the same order. I get no messages >>>>> regarding any kind of merge errors. >>>>> >>>>> Any ideas on this? >>>>> >>>>> Thanks, Brian >>>>> >>>>> P.S. Full print of traceback below >>>>> >>>>> table3 = vstack([table1, table3],join_type=?outer?) >>>>> print table3[0] >>>>> >>>>> --------------------------------------------------------------------------- >>>>> ValueError Traceback (most recent call >>>>> last) >>>>> in () >>>>> ----> 1 print tmp3[0] >>>>> >>>>> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in >>>>> __repr__(self) >>>>> 176 def __repr__(self): >>>>> 177 return "<{3} {0} of table\n values={1!r}\n >>>>> dtype={2}>".format( >>>>> --> 178 self.index, self.as_void(), self.dtype, >>>>> self.__class__.__name__) >>>>> 179 >>>>> 180 >>>>> >>>>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>>>> __repr__(self) >>>>> 5699 return self._data.__repr__() >>>>> 5700 m = tuple(m) >>>>> -> 5701 if not any(m): >>>>> 5702 return self._data.__repr__() >>>>> 5703 p = masked_print_option >>>>> >>>>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>>>> __call__(self, a, *args, **params) >>>>> 6089 method = getattr(MaskedArray, method_name, None) >>>>> 6090 if method is not None: >>>>> -> 6091 return method(MaskedArray(a), *args, **params) >>>>> 6092 # Still here ? OK, let's call the corresponding np >>>>> function >>>>> 6093 method = getattr(np, method_name) >>>>> >>>>> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >>>>> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, >>>>> hard_mask, shrink, **options) >>>>> 2649 """ >>>>> 2650 # Process data............ >>>>> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, >>>>> ndmin=ndmin) >>>>> 2652 _baseclass = getattr(data, '_baseclass', type(_data)) >>>>> 2653 # Check that we're not erasing the mask.......... >>>>> >>>>> ValueError: setting an array element with a sequence. >>>>> >>>>> >>>>> >>>>> >>>>> --------------------------------------- >>>>> Brian Cherinka, Ph.D >>>>> Dunlap Institute for Astronomy & Astrophysics >>>>> University of Toronto >>>>> Toronto, ON, Canada M5S 3H4 >>>>> phone: 416-978-7299 >>>>> email: cherinka at di.utoronto.ca >>>>> ---------------------------------------- >>>>> >>>>> _______________________________________________ >>>>> AstroPy mailing list >>>>> AstroPy at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> >>>> >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> >>>> >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> >>>> >>>> >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Tue Jun 23 10:10:22 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 23 Jun 2015 15:10:22 +0100 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Message-ID: Hi Brian, Thanks, this is helpful, and I managed to boil this down to a simple failing example that I've reported here: https://github.com/astropy/astropy/issues/3877 It seems to be due to the vector columns. We'll look into it! Cheers, Tom On 23 June 2015 at 14:58, Brian Cherinka wrote: > Hi Thomas, > > In [187]: t1.dtype > Out[187]: dtype([('plate', '>i8'), ('ifudsgn', 'S5'), ('plateifu', 'S9'), > ('mangaid', 'S4'), ('versdrp2', 'S30'), ('versdrp3', 'S30'), ('verscore', > 'S30'), ('versutil', 'S30'), ('platetyp', 'S30'), ('srvymode', 'S30'), > ('objRA', '>f8'), ('objDEC', '>f8'), ('objGLON', '>f8'), ('objGLAT', '>f8'), > ('ebvgal', '>f8'), ('nexp', '>i8'), ('exptime', '>f8'), ('drp3qual', '>i8'), > ('bluesn2', '>f8'), ('redsn2', '>f8'), ('harname', 'S30'), ('frlplug', > '>i8'), ('cartid', 'S30'), ('designid', '>i8'), ('cenRA', '>f8'), ('cenDEC', > '>f8'), ('airmsmin', '>f8'), ('airmsmed', '>f8'), ('airmsmax', '>f8'), > ('seemin', '>f8'), ('seemed', '>f8'), ('seemax', '>f8'), ('transmin', > '>f8'), ('transmed', '>f8'), ('transmax', '>f8'), ('mjdmin', '>i8'), > ('mjdmed', '>i8'), ('mjdmax', '>i8'), ('ufwhm', '>f8'), ('gfwhm', '>f8'), > ('rfwhm', '>f8'), ('ifwhm', '>f8'), ('zfwhm', '>f8'), ('mngtarg1', '>i8'), > ('mngtarg2', '>i8'), ('mngtarg3', '>i8'), ('catidnum', '>i8'), ('plttarg', > 'S30')]) > > In [188]: t2.dtype > Out[188]: dtype([('manga_tileid', ' ' ('ifudesignsize', ' ' ' ('nsa_absmag', ' ('nsa_amivar_el', ' ' (7,)), ('nsa_petro_th50', ' ('nsa_petroflux', ' ('nsa_petroflux_el', ' ('nsa_sersic_ba', ' ' ('nsa_sersicflux_ivar', ' > t3 = hstack([t1,t2],join_type='outer?) completes with no errors, however > > t3[0] gives > > ) failed: ValueError: setting an > array element with a sequence.> > > then vstacking t3 with t4 completes with no errors, but gives the same error > when row indexing. > > Cheers, Brian > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 23, 2015, at 9:53 AM, Thomas Robitaille > wrote: > > Hi Brian, > > On 23 June 2015 at 14:49, Brian Cherinka wrote: > > Hi Thomas, > > Could you let us know what table1.dtype and table3.dtype are equal to, > and how many rows are in each table? > > > The two tables, t1 and t2, that I am hstacking have 48 columns, and 36 > columns, respectively, each with 29 rows. The dtype of those columns are a > mix of things. Mostly float and string scalars. I do have several columns > of float arrays. > > > Just to clarify, could you copy and paste the exact output from > table1.dtype and table3.dtype here so that we can try and reproduce > the exact issue? (it may be a corner case due to the type of one of > the columns). > > Thanks! > Tom > > After hstacking, the resulting table (84 columns, 29 rows) is combined as > expected, although I can no longer index the rows. Then I vstack it with > another table 3, which has the same number of columns, and same dtype. I > can?t row index that table either. > > > > > Cheers, Brian > > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 19, 2015, at 8:43 AM, Thomas Robitaille > wrote: > > Hi Brian, > > Could you let us know what table1.dtype and table3.dtype are equal to, > and how many rows are in each table? > > Thanks! > Tom > > > On 18 June 2015 at 16:34, Brian Cherinka wrote: > > Anymore thoughts on this? My table length issue wasn?t the problem. I?m > stilling having a problem with my resulting hstacks and vstacks. The stacks > happen without error. I just lose the ability to index by row. > > Cheers, Brian > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: > > Hi Thomas, > > Your example does indeed work for me. I cannot yet guess why it doesn?t > work in my full example. However it?s not quite a complete reconstruction > of what I?m doing. Technically, I?m hstacking two tables together, and then > vstacking that result with another table. Something like this, which works > just fine. > > In [382]: t1 = Table([[1], [2]], names=['a', 'b']) > > In [383]: t2 = Table([[3], [4]], names=['c','d']) > > In [384]: t3 = hstack([t1,t2],join_type='outer') > > In [386]: t4 = Table([[5], [6]], names=['a', 'b']) > > In [387]: t34 = vstack([t3,t4],join_type='outer') > > In [388]: t34 > Out[388]: >
> a b c d > int64 int64 int64 int64 > ----- ----- ----- ----- > 1 2 3 4 > 5 6 -- -- > > In [389]: t34[0] > Out[389]: > values=(1, 2, 3, 4) > dtype=[('a', ' > > In my full example, I did find though that I can row index the individual > tables t1 and t2, but cannot row index table t3 after hstacking. I get the > same ValueError. When I attempt to vstack with table t4, that error > propagates forward. I can row index t4 but not t34. Looking at it a bit > more, it might be that my rows in the initial tables t1, t2 do not match up > with one another, and I appear to have one extra row in table t2, than I > should. I?ll dig a bit more. > > Cheers, Brian > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas > wrote: > > Hi Brian, > > Below is what I think is a minimal example of what you are describing. Can > you look at this and first reproduce that this works for you, and second, > can you guess what might be different in your full example vs. my minimal > example? > > In [19]: t1 = Table([[1], [2]], names=['a', 'b']) > > In [20]: t2 = Table([[3], [4]], names=['b', 'c']) > > In [23]: t12 = vstack([t1, t2], join_type='outer') > > In [24]: t12 > Out[24]: >
> a b c > int64 int64 int64 > ----- ----- ----- > 1 2 -- > -- 3 4 > > In [25]: t12[0] > Out[25]: > values=(1, 2, --) > dtype=[('a', ' > > > On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka > wrote: > > > Hi, > > I?m trying to vstack (vertically stack) two Astropy tables with different > numbers of columns. The tables appear to stack correctly without error. > However, upon attempting to index a row in the table, it throws me this > error > > ) failed: ValueError: setting > an array element with a sequence.> > > I?m not trying to do anything fancy with mixed columns, or Quantities. > Just stackin? two good ol? natural Tables together. One table has 84 > columns, and another has 57, which are all a part of the 84 in the first > table. At first glance I would guess it?s a mismatch in the dtypes of a > particular column that I?m trying to merge. However, the columns all have > the same dtype, format, and they are in the same order. I get no messages > regarding any kind of merge errors. > > Any ideas on this? > > Thanks, Brian > > P.S. Full print of traceback below > > table3 = vstack([table1, table3],join_type=?outer?) > print table3[0] > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > in () > ----> 1 print tmp3[0] > > /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in > __repr__(self) > 176 def __repr__(self): > 177 return "<{3} {0} of table\n values={1!r}\n > dtype={2}>".format( > --> 178 self.index, self.as_void(), self.dtype, > self.__class__.__name__) > 179 > 180 > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in > __repr__(self) > 5699 return self._data.__repr__() > 5700 m = tuple(m) > -> 5701 if not any(m): > 5702 return self._data.__repr__() > 5703 p = masked_print_option > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in > __call__(self, a, *args, **params) > 6089 method = getattr(MaskedArray, method_name, None) > 6090 if method is not None: > -> 6091 return method(MaskedArray(a), *args, **params) > 6092 # Still here ? OK, let's call the corresponding np > function > 6093 method = getattr(np, method_name) > > /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in > __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, > hard_mask, shrink, **options) > 2649 """ > 2650 # Process data............ > -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, > ndmin=ndmin) > 2652 _baseclass = getattr(data, '_baseclass', type(_data)) > 2653 # Check that we're not erasing the mask.......... > > ValueError: setting an array element with a sequence. > > > > > --------------------------------------- > Brian Cherinka, Ph.D > Dunlap Institute for Astronomy & Astrophysics > University of Toronto > Toronto, ON, Canada M5S 3H4 > phone: 416-978-7299 > email: cherinka at di.utoronto.ca > ---------------------------------------- > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From cherinka at di.utoronto.ca Tue Jun 23 10:12:15 2015 From: cherinka at di.utoronto.ca (Brian Cherinka) Date: Tue, 23 Jun 2015 10:12:15 -0400 Subject: [AstroPy] ValueError in vstacking tables In-Reply-To: References: <46F38639-106E-4EF2-8449-6D6A949D851A@di.utoronto.ca> <751D45BA-F658-40DF-9F42-1386499AD2B5@di.utoronto.ca> <70485B93-CED7-4C4D-8618-84EA06D8C483@di.utoronto.ca> Message-ID: <43427FA1-884D-4D70-BF21-EC7C0E3E97C5@di.utoronto.ca> Hi Thomas, Excellent! Thanks a lot. Really appreciate it. Cheers, Brian --------------------------------------- Brian Cherinka, Ph.D Dunlap Institute for Astronomy & Astrophysics University of Toronto Toronto, ON, Canada M5S 3H4 phone: 416-978-7299 email: cherinka at di.utoronto.ca ---------------------------------------- On Jun 23, 2015, at 10:10 AM, Thomas Robitaille wrote: > Hi Brian, > > Thanks, this is helpful, and I managed to boil this down to a simple > failing example that I've reported here: > > https://github.com/astropy/astropy/issues/3877 > > It seems to be due to the vector columns. We'll look into it! > > Cheers, > Tom > > > On 23 June 2015 at 14:58, Brian Cherinka wrote: >> Hi Thomas, >> >> In [187]: t1.dtype >> Out[187]: dtype([('plate', '>i8'), ('ifudsgn', 'S5'), ('plateifu', 'S9'), >> ('mangaid', 'S4'), ('versdrp2', 'S30'), ('versdrp3', 'S30'), ('verscore', >> 'S30'), ('versutil', 'S30'), ('platetyp', 'S30'), ('srvymode', 'S30'), >> ('objRA', '>f8'), ('objDEC', '>f8'), ('objGLON', '>f8'), ('objGLAT', '>f8'), >> ('ebvgal', '>f8'), ('nexp', '>i8'), ('exptime', '>f8'), ('drp3qual', '>i8'), >> ('bluesn2', '>f8'), ('redsn2', '>f8'), ('harname', 'S30'), ('frlplug', >> '>i8'), ('cartid', 'S30'), ('designid', '>i8'), ('cenRA', '>f8'), ('cenDEC', >> '>f8'), ('airmsmin', '>f8'), ('airmsmed', '>f8'), ('airmsmax', '>f8'), >> ('seemin', '>f8'), ('seemed', '>f8'), ('seemax', '>f8'), ('transmin', >> '>f8'), ('transmed', '>f8'), ('transmax', '>f8'), ('mjdmin', '>i8'), >> ('mjdmed', '>i8'), ('mjdmax', '>i8'), ('ufwhm', '>f8'), ('gfwhm', '>f8'), >> ('rfwhm', '>f8'), ('ifwhm', '>f8'), ('zfwhm', '>f8'), ('mngtarg1', '>i8'), >> ('mngtarg2', '>i8'), ('mngtarg3', '>i8'), ('catidnum', '>i8'), ('plttarg', >> 'S30')]) >> >> In [188]: t2.dtype >> Out[188]: dtype([('manga_tileid', '> '> ('ifudesignsize', '> '> '> ('nsa_absmag', '> ('nsa_amivar_el', '> '> (7,)), ('nsa_petro_th50', '> ('nsa_petroflux', '> ('nsa_petroflux_el', '> ('nsa_sersic_ba', '> '> ('nsa_sersicflux_ivar', '> >> t3 = hstack([t1,t2],join_type='outer?) completes with no errors, however >> >> t3[0] gives >> >> ) failed: ValueError: setting an >> array element with a sequence.> >> >> then vstacking t3 with t4 completes with no errors, but gives the same error >> when row indexing. >> >> Cheers, Brian >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 23, 2015, at 9:53 AM, Thomas Robitaille >> wrote: >> >> Hi Brian, >> >> On 23 June 2015 at 14:49, Brian Cherinka wrote: >> >> Hi Thomas, >> >> Could you let us know what table1.dtype and table3.dtype are equal to, >> and how many rows are in each table? >> >> >> The two tables, t1 and t2, that I am hstacking have 48 columns, and 36 >> columns, respectively, each with 29 rows. The dtype of those columns are a >> mix of things. Mostly float and string scalars. I do have several columns >> of float arrays. >> >> >> Just to clarify, could you copy and paste the exact output from >> table1.dtype and table3.dtype here so that we can try and reproduce >> the exact issue? (it may be a corner case due to the type of one of >> the columns). >> >> Thanks! >> Tom >> >> After hstacking, the resulting table (84 columns, 29 rows) is combined as >> expected, although I can no longer index the rows. Then I vstack it with >> another table 3, which has the same number of columns, and same dtype. I >> can?t row index that table either. >> >> >> >> >> Cheers, Brian >> >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 19, 2015, at 8:43 AM, Thomas Robitaille >> wrote: >> >> Hi Brian, >> >> Could you let us know what table1.dtype and table3.dtype are equal to, >> and how many rows are in each table? >> >> Thanks! >> Tom >> >> >> On 18 June 2015 at 16:34, Brian Cherinka wrote: >> >> Anymore thoughts on this? My table length issue wasn?t the problem. I?m >> stilling having a problem with my resulting hstacks and vstacks. The stacks >> happen without error. I just lose the ability to index by row. >> >> Cheers, Brian >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 17, 2015, at 3:25 PM, Brian Cherinka wrote: >> >> Hi Thomas, >> >> Your example does indeed work for me. I cannot yet guess why it doesn?t >> work in my full example. However it?s not quite a complete reconstruction >> of what I?m doing. Technically, I?m hstacking two tables together, and then >> vstacking that result with another table. Something like this, which works >> just fine. >> >> In [382]: t1 = Table([[1], [2]], names=['a', 'b']) >> >> In [383]: t2 = Table([[3], [4]], names=['c','d']) >> >> In [384]: t3 = hstack([t1,t2],join_type='outer') >> >> In [386]: t4 = Table([[5], [6]], names=['a', 'b']) >> >> In [387]: t34 = vstack([t3,t4],join_type='outer') >> >> In [388]: t34 >> Out[388]: >>
>> a b c d >> int64 int64 int64 int64 >> ----- ----- ----- ----- >> 1 2 3 4 >> 5 6 -- -- >> >> In [389]: t34[0] >> Out[389]: >> > values=(1, 2, 3, 4) >> dtype=[('a', ' >> >> In my full example, I did find though that I can row index the individual >> tables t1 and t2, but cannot row index table t3 after hstacking. I get the >> same ValueError. When I attempt to vstack with table t4, that error >> propagates forward. I can row index t4 but not t34. Looking at it a bit >> more, it might be that my rows in the initial tables t1, t2 do not match up >> with one another, and I appear to have one extra row in table t2, than I >> should. I?ll dig a bit more. >> >> Cheers, Brian >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas >> wrote: >> >> Hi Brian, >> >> Below is what I think is a minimal example of what you are describing. Can >> you look at this and first reproduce that this works for you, and second, >> can you guess what might be different in your full example vs. my minimal >> example? >> >> In [19]: t1 = Table([[1], [2]], names=['a', 'b']) >> >> In [20]: t2 = Table([[3], [4]], names=['b', 'c']) >> >> In [23]: t12 = vstack([t1, t2], join_type='outer') >> >> In [24]: t12 >> Out[24]: >>
>> a b c >> int64 int64 int64 >> ----- ----- ----- >> 1 2 -- >> -- 3 4 >> >> In [25]: t12[0] >> Out[25]: >> > values=(1, 2, --) >> dtype=[('a', ' >> >> >> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka >> wrote: >> >> >> Hi, >> >> I?m trying to vstack (vertically stack) two Astropy tables with different >> numbers of columns. The tables appear to stack correctly without error. >> However, upon attempting to index a row in the table, it throws me this >> error >> >> ) failed: ValueError: setting >> an array element with a sequence.> >> >> I?m not trying to do anything fancy with mixed columns, or Quantities. >> Just stackin? two good ol? natural Tables together. One table has 84 >> columns, and another has 57, which are all a part of the 84 in the first >> table. At first glance I would guess it?s a mismatch in the dtypes of a >> particular column that I?m trying to merge. However, the columns all have >> the same dtype, format, and they are in the same order. I get no messages >> regarding any kind of merge errors. >> >> Any ideas on this? >> >> Thanks, Brian >> >> P.S. Full print of traceback below >> >> table3 = vstack([table1, table3],join_type=?outer?) >> print table3[0] >> >> --------------------------------------------------------------------------- >> ValueError Traceback (most recent call >> last) >> in () >> ----> 1 print tmp3[0] >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in >> __repr__(self) >> 176 def __repr__(self): >> 177 return "<{3} {0} of table\n values={1!r}\n >> dtype={2}>".format( >> --> 178 self.index, self.as_void(), self.dtype, >> self.__class__.__name__) >> 179 >> 180 >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >> __repr__(self) >> 5699 return self._data.__repr__() >> 5700 m = tuple(m) >> -> 5701 if not any(m): >> 5702 return self._data.__repr__() >> 5703 p = masked_print_option >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >> __call__(self, a, *args, **params) >> 6089 method = getattr(MaskedArray, method_name, None) >> 6090 if method is not None: >> -> 6091 return method(MaskedArray(a), *args, **params) >> 6092 # Still here ? OK, let's call the corresponding np >> function >> 6093 method = getattr(np, method_name) >> >> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in >> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, >> hard_mask, shrink, **options) >> 2649 """ >> 2650 # Process data............ >> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True, >> ndmin=ndmin) >> 2652 _baseclass = getattr(data, '_baseclass', type(_data)) >> 2653 # Check that we're not erasing the mask.......... >> >> ValueError: setting an array element with a sequence. >> >> >> >> >> --------------------------------------- >> Brian Cherinka, Ph.D >> Dunlap Institute for Astronomy & Astrophysics >> University of Toronto >> Toronto, ON, Canada M5S 3H4 >> phone: 416-978-7299 >> email: cherinka at di.utoronto.ca >> ---------------------------------------- >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From vvinuv at gmail.com Thu Jun 25 13:06:16 2015 From: vvinuv at gmail.com (Vinu Vikram) Date: Thu, 25 Jun 2015 12:06:16 -0500 Subject: [AstroPy] How to change fits table header to another name Message-ID: Hi All I would like to change a fits binary table column name to another name. E.g. if the fits table has the header column RA and I need to change that to RA_1. Thanks Vinu From yannick.roehlly at gmail.com Thu Jun 25 14:07:51 2015 From: yannick.roehlly at gmail.com (Yannick Roehlly) Date: Thu, 25 Jun 2015 20:07:51 +0200 Subject: [AstroPy] How to change fits table header to another name In-Reply-To: References: Message-ID: Hi Vinu, The easiest way is to use Table: from astropy.table import Table t = Table.read("foo.fits") t['RA'].name = 'RA_1' t.write("bar.fits") But if your FITS file has other extensions you may loose them when saving back the table. In that case, you must use io.fits but you must know the position of the column you want to rename: from astropy.io import fits hdu_list = fits.open("foo.fits") hdu_list[1].columns[1].name = "RA_1" # The column index starts at 0 hdu_list.writeto("bar.fits") Regards, Yannick 2015-06-25 19:06 GMT+02:00 Vinu Vikram : > Hi All > I would like to change a fits binary table column name to another > name. E.g. if the fits table has the header column RA and I need to > change that to RA_1. > Thanks > Vinu > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From embray at stsci.edu Thu Jun 25 14:53:33 2015 From: embray at stsci.edu (Erik Bray) Date: Thu, 25 Jun 2015 14:53:33 -0400 Subject: [AstroPy] How to change fits table header to another name In-Reply-To: References: Message-ID: <558C4E2D.4080708@stsci.edu> On 06/25/2015 02:07 PM, Yannick Roehlly wrote: > Hi Vinu, > > The easiest way is to use Table: > > from astropy.table import Table > t = Table.read("foo.fits") > t['RA'].name = 'RA_1' > t.write("bar.fits") > > But if your FITS file has other extensions you may loose them when saving back > the table. In that case, you must use io.fits but you must know the position of > the column you want to rename: > > from astropy.io import fits > hdu_list = fits.open("foo.fits") > hdu_list[1].columns[1].name = "RA_1" # The column index starts at 0 > hdu_list.writeto("bar.fits") By the way, in the current dev version, columns['RA'].name = 'RA_1' will work. I don't think it works in any released version though. Erik > > 2015-06-25 19:06 GMT+02:00 Vinu Vikram >: > > Hi All > I would like to change a fits binary table column name to another > name. E.g. if the fits table has the header column RA and I need to > change that to RA_1. > Thanks > Vinu > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From vvinuv at gmail.com Thu Jun 25 15:20:17 2015 From: vvinuv at gmail.com (Vinu Vikram) Date: Thu, 25 Jun 2015 14:20:17 -0500 Subject: [AstroPy] How to change fits table header to another name In-Reply-To: <558C4E2D.4080708@stsci.edu> References: <558C4E2D.4080708@stsci.edu> Message-ID: Thank you Yannick and Erik! It works now, vinu On 25 June 2015 at 13:53, Erik Bray wrote: > On 06/25/2015 02:07 PM, Yannick Roehlly wrote: >> Hi Vinu, >> >> The easiest way is to use Table: >> >> from astropy.table import Table >> t = Table.read("foo.fits") >> t['RA'].name = 'RA_1' >> t.write("bar.fits") >> >> But if your FITS file has other extensions you may loose them when saving back >> the table. In that case, you must use io.fits but you must know the position of >> the column you want to rename: >> >> from astropy.io import fits >> hdu_list = fits.open("foo.fits") >> hdu_list[1].columns[1].name = "RA_1" # The column index starts at 0 >> hdu_list.writeto("bar.fits") > > By the way, in the current dev version, columns['RA'].name = 'RA_1' will work. > I don't think it works in any released version though. > > Erik > >> >> 2015-06-25 19:06 GMT+02:00 Vinu Vikram >: >> >> Hi All >> I would like to change a fits binary table column name to another >> name. E.g. if the fits table has the header column RA and I need to >> change that to RA_1. >> Thanks >> Vinu >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- Vinu Vikraman http://www.sas.upenn.edu/~vinu/ From yannick.roehlly at gmail.com Thu Jun 25 16:15:46 2015 From: yannick.roehlly at gmail.com (Yannick Roehlly) Date: Thu, 25 Jun 2015 22:15:46 +0200 Subject: [AstroPy] How to change fits table header to another name In-Reply-To: <558C4E2D.4080708@stsci.edu> References: <558C4E2D.4080708@stsci.edu> Message-ID: <1797598.LYncYu6oVW@tardis> Le jeudi 25 juin 2015, 14:53:33 Erik Bray a ?crit : > > from astropy.io import fits > > hdu_list = fits.open("foo.fits") > > hdu_list[1].columns[1].name = "RA_1" # The column index starts at 0 > > hdu_list.writeto("bar.fits") > > By the way, in the current dev version, columns['RA'].name = 'RA_1' will > work. I don't think it works in any released version though. Hi Erik, Thanks, that's very useful. Yannick -- Everything should be made as simple as possible, but not simpler. -- Albert Einstein From stsci.perry at gmail.com Mon Jun 29 12:50:38 2015 From: stsci.perry at gmail.com (Perry Greenfield) Date: Mon, 29 Jun 2015 12:50:38 -0400 Subject: [AstroPy] sprint for astropy at scipy? Message-ID: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Who plans to be around for part or all of the sprints? I?ll be there Saturday. From sosey at stsci.edu Mon Jun 29 12:52:32 2015 From: sosey at stsci.edu (Megan Sosey) Date: Mon, 29 Jun 2015 16:52:32 +0000 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Message-ID: I will be there saturday and sunday, Megan -----Original Message----- From: Perry Greenfield Reply-To: Astronomical Python mailing list Date: Monday, June 29, 2015 at 12:50 PM To: Astronomical Python mailing list Subject: [AstroPy] sprint for astropy at scipy? >Who plans to be around for part or all of the sprints? I?ll be there >Saturday. > > >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.org >http://mail.scipy.org/mailman/listinfo/astropy From te.pickering at gmail.com Mon Jun 29 12:53:39 2015 From: te.pickering at gmail.com (Timothy Pickering) Date: Mon, 29 Jun 2015 12:53:39 -0400 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Message-ID: <92366CDF-125F-47DF-A459-E44C7958E100@gmail.com> > Who plans to be around for part or all of the sprints? I?ll be there Saturday. i?ll be there all day saturday and sunday? tim ? +-------------------------------------------------------------------+ | T. E. Pickering, Ph.D. | Space Telescope Science Institute | | Systems Software Engineer | 3700 San Martin Dr. | | +1-520-305-9823 | Baltimore, MD, 21218, USA | | te.pickering at gmail.com | pickering at stsci.edu | +-------------------------------------------------------------------+ From kevin.gullikson at gmail.com Mon Jun 29 13:00:02 2015 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Mon, 29 Jun 2015 12:00:02 -0500 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <92366CDF-125F-47DF-A459-E44C7958E100@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> <92366CDF-125F-47DF-A459-E44C7958E100@gmail.com> Message-ID: I don't have plans yet, but live in Austin so could make it. Kevin Gullikson PhD Candidate University of Texas Astronomy RLM 15.310E On Mon, Jun 29, 2015 at 11:53 AM, Timothy Pickering wrote: > > > Who plans to be around for part or all of the sprints? I?ll be there > Saturday. > > i?ll be there all day saturday and sunday? > > tim > > ? > +-------------------------------------------------------------------+ > | T. E. Pickering, Ph.D. | Space Telescope Science Institute | > | Systems Software Engineer | 3700 San Martin Dr. | > | +1-520-305-9823 | Baltimore, MD, 21218, USA | > | te.pickering at gmail.com | pickering at stsci.edu | > +-------------------------------------------------------------------+ > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From embray at stsci.edu Mon Jun 29 13:03:40 2015 From: embray at stsci.edu (Erik Bray) Date: Mon, 29 Jun 2015 13:03:40 -0400 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Message-ID: <55917A6C.6030600@stsci.edu> On 06/29/2015 12:50 PM, Perry Greenfield wrote: > Who plans to be around for part or all of the sprints? I?ll be there Saturday. What days are that? I might try to participate remotely. Or might not. Depends on when it is :) Erik From stsci.perry at gmail.com Mon Jun 29 13:06:50 2015 From: stsci.perry at gmail.com (Perry Greenfield) Date: Mon, 29 Jun 2015 13:06:50 -0400 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <55917A6C.6030600@stsci.edu> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> <55917A6C.6030600@stsci.edu> Message-ID: <8247A397-8C22-45CB-BB9A-5CF0EE5B5DBC@gmail.com> Saturday and Sunday a week from this weekend (11th and 12th I think). By experience, there seem to be a lot fewer people around Sunday for whatever reason. Perry On Jun 29, 2015, at 1:03 PM, Erik Bray wrote: > On 06/29/2015 12:50 PM, Perry Greenfield wrote: >> Who plans to be around for part or all of the sprints? I?ll be there Saturday. > > What days are that? I might try to participate remotely. Or might not. > Depends on when it is :) > > Erik > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From lbradley at stsci.edu Mon Jun 29 13:13:16 2015 From: lbradley at stsci.edu (Larry Bradley) Date: Mon, 29 Jun 2015 13:13:16 -0400 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Message-ID: I'll be there Saturday, but leaving early afternoon. On Mon, Jun 29, 2015 at 12:50 PM, Perry Greenfield wrote: > Who plans to be around for part or all of the sprints? I?ll be there Saturday. From mcraig at mnstate.edu Mon Jun 29 15:03:27 2015 From: mcraig at mnstate.edu (Matthew Craig) Date: Mon, 29 Jun 2015 19:03:27 +0000 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Message-ID: I'm around Saturday and half of Sunday. Matt Craig --- Professor, Department off Physics and Astronomy Minnesota State University Moorhead Hagen 307F Schedule: http://physics.mnstate.edu/craig On June 29, 2015 at 12:50:38 PM EDT, Perry Greenfield wrote: Who plans to be around for part or all of the sprints? I?ll be there Saturday. _______________________________________________ AstroPy mailing list AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From bushouse at stsci.edu Tue Jun 30 08:37:23 2015 From: bushouse at stsci.edu (Howard Bushouse) Date: Tue, 30 Jun 2015 08:37:23 -0400 Subject: [AstroPy] sprint for astropy at scipy? In-Reply-To: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> References: <99A9B149-013C-461E-AB6B-74878EA7CF9E@gmail.com> Message-ID: <55928D83.7070106@stsci.edu> No, I'll be leaving town Saturday morning. -Howard On 06/29/2015 12:50 PM, Perry Greenfield wrote: > Who plans to be around for part or all of the sprints? I?ll be there Saturday. > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy