From thoger.emil at gmail.com Tue Dec 11 09:03:04 2012 From: thoger.emil at gmail.com (=?ISO-8859-1?Q?Th=F8ger_Rivera-Thorsen?=) Date: Tue, 11 Dec 2012 15:03:04 +0100 Subject: [AstroPy] Lists of stellar and galactic emission lines? Message-ID: <50C73D18.8070404@gmail.com> Hello list; This is not stricxtly Python-related, but I am working on some spectroscopy (Python) code, and would like to know if there are any good, comprehensive-but-not-too-comprehensive lists of the strongest/most common emission (and secondarily absorption) lines in stellar and galactic spectra? I have looked at a couple; the ones in Astropysics and VPfit. The former is good but a bit short, the latter is (a bit too) long and comprehensive but has very few lines longer than ~3000 ?. What do you people out there use in your daily work, and is it accessible for inclusion in a Python module? Cheers; T. Rivera-Thorsen University of Stockholm Department of Astronomy From crawford at saao.ac.za Tue Dec 11 09:11:29 2012 From: crawford at saao.ac.za (Steve Crawford) Date: Tue, 11 Dec 2012 16:11:29 +0200 (SAST) Subject: [AstroPy] Lists of stellar and galactic emission lines? In-Reply-To: <50C73D18.8070404@gmail.com> Message-ID: <3b19dcb5-8179-4fc2-9ecc-2a08512d0b58@mailcape> The SDSS line list is pretty good starting point: http://www.sdss.org/dr5/algorithms/linestable.html From jturner at gemini.edu Tue Dec 11 11:32:27 2012 From: jturner at gemini.edu (James Turner) Date: Tue, 11 Dec 2012 13:32:27 -0300 Subject: [AstroPy] Lists of stellar and galactic emission lines? In-Reply-To: <50C73D18.8070404@gmail.com> References: <50C73D18.8070404@gmail.com> Message-ID: <50C7601B.1030402@gemini.edu> > What do you people out there use in your daily work, and is it > accessible for inclusion in a Python module? Good idea. I usually refer to a list compiled by a colleague in the first instance but I'm just noticing some, erm, anomalies WRT the SDSS lists (not just air to vacuum). James. From tim.jenness at gmail.com Tue Dec 11 11:48:52 2012 From: tim.jenness at gmail.com (Tim Jenness) Date: Tue, 11 Dec 2012 09:48:52 -0700 Subject: [AstroPy] Lists of stellar and galactic emission lines? In-Reply-To: <50C73D18.8070404@gmail.com> References: <50C73D18.8070404@gmail.com> Message-ID: Depending on your wavelength of interest there is splatalogue.net and that looks pretty easy to connect to from python: http://vilhelmp.blogspot.com/2011/10/querying-splataloguenet-with-python.html On Tue, Dec 11, 2012 at 7:03 AM, Th?ger Rivera-Thorsen < thoger.emil at gmail.com> wrote: > Hello list; > > This is not stricxtly Python-related, but I am working on some > spectroscopy (Python) code, and would like to know if there are any > good, comprehensive-but-not-too-comprehensive lists of the > strongest/most common emission (and secondarily absorption) lines in > stellar and galactic spectra? > > I have looked at a couple; the ones in Astropysics and VPfit. The former > is good but a bit short, the latter is (a bit too) long and > comprehensive but has very few lines longer than ~3000 ?. > > What do you people out there use in your daily work, and is it > accessible for inclusion in a Python module? > > Cheers; > > T. Rivera-Thorsen > University of Stockholm > Department of Astronomy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stsci.perry at gmail.com Thu Dec 13 07:57:32 2012 From: stsci.perry at gmail.com (Perry Greenfield) Date: Thu, 13 Dec 2012 07:57:32 -0500 Subject: [AstroPy] positions available at STScI Message-ID: The Science Software Branch at the Space Telescope Science Institute has openings for developers. If you are interested please apply through this link: https://rn11.ultipro.com/SPA1004/JobBoard/JobDetails.aspx?__ID=*41BCE81D823F3F8C The relevant section is the one labeled "Science Data Calibration and Data Analysis" Although the position is advertised as a junior position, candidates with more experience may be considered for higher level positions. From wkerzendorf at gmail.com Fri Dec 14 15:04:21 2012 From: wkerzendorf at gmail.com (Wolfgang Kerzendorf) Date: Fri, 14 Dec 2012 15:04:21 -0500 Subject: [AstroPy] anyone using Pandas? Message-ID: Hey guys, I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? Cheers Wolfgang From ogk at astro.caltech.edu Fri Dec 14 15:46:07 2012 From: ogk at astro.caltech.edu (Oliver King) Date: Fri, 14 Dec 2012 12:46:07 -0800 Subject: [AstroPy] anyone using Pandas? In-Reply-To: References: Message-ID: I'm by no means a Pandas expert, but I usually merge dataframe objects with pandas.concat([df1,df2,df3],join='outer') - is this not what you're looking for? Cheers, Oliver On Dec 14, 2012, at 12:04 PM, Wolfgang Kerzendorf wrote: > Hey guys, > > I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? > > Cheers > Wolfgang > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy ------ Oliver King Senior Scientist Astronomy Department, Caltech +1-626-395-4197 From wtylergee at gmail.com Fri Dec 14 15:48:35 2012 From: wtylergee at gmail.com (Tyler) Date: Fri, 14 Dec 2012 10:48:35 -1000 Subject: [AstroPy] anyone using Pandas? In-Reply-To: References: Message-ID: What is it you are looking to do? I'm by no means an expert either but have been playing around with Pandas for some astronomy stuff. Aloha, On Fri, Dec 14, 2012 at 10:04 AM, Wolfgang Kerzendorf wrote: > Hey guys, > > I'm trying to play around with pandas. Currently I have a look at join and > it always seems to copy the data. I believe I want to use advanced > indexing, but am not quite sure how to do that: any Pandas experts here? > > Cheers > Wolfgang > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- ~Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Fri Dec 14 17:52:06 2012 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Fri, 14 Dec 2012 17:52:06 -0500 Subject: [AstroPy] Vizier Web API Message-ID: Does anyone know how use the web API to do cone searches on specific catalogs using Vizier? I'm looking at [1] which says I should be able to use "-out.form" to select the output format, but it always returns the results as an HTML-formatted page. I just need any machine readable format like CSV, XML, whatever. For instance: http://vizier.cfa.harvard.edu/viz-bin/VizieR?-source=I%2F284%2Fout&-out.form=plain&-c=50.00+50.00 If I load that, then conveniently there is a submit button to the lower left that then returns what I want. Hmm. Thanks, Tom [1]: http://vizier.cfa.harvard.edu/doc/asu-summary.htx From trive at astro.su.se Fri Dec 14 20:15:54 2012 From: trive at astro.su.se (=?ISO-8859-1?Q?Th=F8ger_Rivera-Thorsen?=) Date: Sat, 15 Dec 2012 02:15:54 +0100 Subject: [AstroPy] anyone using Pandas? In-Reply-To: References: Message-ID: <50CBCF4A.2020201@astro.su.se> Like Tyler said, can you be a bit more specific about what you want to obtain? What is your starting point, and where do you want to go from there? I've been looking quite a bit into multiindexing lately, and it is very handy but it does have some caveats. On 12/14/2012 09:04 PM, Wolfgang Kerzendorf wrote: > Hey guys, > > I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? > > Cheers > Wolfgang > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From wkerzendorf at gmail.com Fri Dec 14 23:52:40 2012 From: wkerzendorf at gmail.com (Wolfgang Kerzendorf) Date: Fri, 14 Dec 2012 23:52:40 -0500 Subject: [AstroPy] anyone using Pandas? In-Reply-To: <50CBCF4A.2020201@astro.su.se> References: <50CBCF4A.2020201@astro.su.se> Message-ID: <999E60D0-296D-4EED-8220-C4C73863E1F8@gmail.com> Sorry before this didn't go to the list: Hey guys, I'm slowly understanding better what pandas is about. The object I'm representing in a pandas data frame is an atomic database. Each line in there has atomic_number, ion_number, level_number, energy, g. So I have created a Pandas dataFrame and then set the index to atomic_number, ion_number, level_number. Now I want to make a new DataFrame where atomic_number in (6, 7, 8, 9) - but it is an index. how do I do that? Cheers W On 2012-12-14, at 8:15 PM, Th?ger Rivera-Thorsen wrote: > Like Tyler said, can you be a bit more specific about what you want to > obtain? > What is your starting point, and where do you want to go from there? > > I've been looking quite a bit into multiindexing lately, and it is very > handy but it does have some caveats. > > > > > > > On 12/14/2012 09:04 PM, Wolfgang Kerzendorf wrote: >> Hey guys, >> >> I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? >> >> Cheers >> Wolfgang >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From thomas.boch at astro.unistra.fr Sat Dec 15 02:43:07 2012 From: thomas.boch at astro.unistra.fr (Thomas Boch) Date: Sat, 15 Dec 2012 08:43:07 +0100 Subject: [AstroPy] Vizier Web API In-Reply-To: References: Message-ID: <50CC2A0B.5000602@astro.unistra.fr> Hi Tom, http://vizier.cfa.harvard.edu/viz-bin/asu-tsv?-source=I%2F284%2Fout&-c=50.00+50.00 should return data in TSV. http://vizier.cfa.harvard.edu/viz-bin/votable?-source=I%2F284%2Fout&-c=50.00+50.00 will return data in VOTable. Cheers, Thomas Le 14/12/12 23:52, Tom Aldcroft a ?crit : > Does anyone know how use the web API to do cone searches on specific > catalogs using Vizier? I'm looking at [1] which says I should be able > to use "-out.form" to select the output format, but it always returns > the results as an HTML-formatted page. I just need any machine > readable format like CSV, XML, whatever. > > For instance: > > http://vizier.cfa.harvard.edu/viz-bin/VizieR?-source=I%2F284%2Fout&-out.form=plain&-c=50.00+50.00 > > If I load that, then conveniently there is a submit button to the > lower left that then returns what I want. Hmm. > > Thanks, > Tom > > [1]: http://vizier.cfa.harvard.edu/doc/asu-summary.htx > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From trive at astro.su.se Sat Dec 15 06:44:18 2012 From: trive at astro.su.se (=?ISO-8859-1?Q?Th=F8ger_Rivera-Thorsen?=) Date: Sat, 15 Dec 2012 12:44:18 +0100 Subject: [AstroPy] anyone using Pandas? In-Reply-To: <999E60D0-296D-4EED-8220-C4C73863E1F8@gmail.com> References: <50CBCF4A.2020201@astro.su.se> <999E60D0-296D-4EED-8220-C4C73863E1F8@gmail.com> Message-ID: <50CC6292.1030707@astro.su.se> Hi; It still depends. Your MultiIndex can be made from lists, tuples or arrays; I usually use arrays since I'm familiar with them from using NumPy. For a 3-level index, you need to have 3 1D-arrays of equal length, that civer alle the atomic nr. - Ion - Level combinations you want covered. Toy example; 3 atoms with three levels each, atoms denoted by number and levels by letters. Let's say the two first atoms have 3, and your third atom five levels. Then your arrays should be: atoms = sp.array(['1', '1', '1', '2', '2', '2', '3', '3', '3', '3', '3']) levels = sp.array(['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'd', 'e']) idx = pd.MultiIndex.from_arrays([atoms, levels], names=['atoms', 'levels']) Your data should then be a 11x2 array with energy and g as the columns. You then build the array as: MyDF = pd.DataFrame(dataarray, index=idx, columns=['Nrg_lvl', 'g.']) That gives: In [22]: print MyDF Nrg_lvl g. atoms levels 1 a 0.052846 0.533835 b 0.185949 0.064069 c 0.384630 0.646803 2 a 0.835958 0.392594 b 0.016399 0.165862 c 0.300874 0.975590 3 a 0.124640 0.815488 b 0.590613 0.749555 c 0.284481 0.299149 d 0.104408 0.723406 e 0.733087 0.730055 (Here I have just used random numbers for the data). From here, you can do all the Pandas goodness you want: In [23]: print MyDF.xs('b', level=1) Nrg_lvl g. atoms 1 0.185949 0.064069 2 0.016399 0.165862 3 0.590613 0.749555 Once you have the basic structure built, it is easy to extend either by concatenating or by using the set_value function. However, most of these operations are returning a modified copy rather than in-place operations, so better build as much of the DataFrame as possible in one go, if you're concerned about memory overhead. ?andas doesn't seem to be designed to gradually build its data structures. Cheers; Th?ger On 12/15/2012 05:52 AM, Wolfgang Kerzendorf wrote: > Sorry before this didn't go to the list: > > Hey guys, > > I'm slowly understanding better what pandas is about. The object I'm representing in a pandas data frame is an atomic database. > Each line in there has atomic_number, ion_number, level_number, energy, g. So I have created a Pandas dataFrame and then set the index to atomic_number, ion_number, level_number. Now I want to make a new DataFrame where atomic_number in (6, 7, 8, 9) - but it is an index. how do I do that? > > Cheers > W > On 2012-12-14, at 8:15 PM, Th?ger Rivera-Thorsen wrote: > >> Like Tyler said, can you be a bit more specific about what you want to >> obtain? >> What is your starting point, and where do you want to go from there? >> >> I've been looking quite a bit into multiindexing lately, and it is very >> handy but it does have some caveats. >> >> >> >> >> >> >> On 12/14/2012 09:04 PM, Wolfgang Kerzendorf wrote: >>> Hey guys, >>> >>> I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? >>> >>> Cheers >>> Wolfgang >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From jerome_caron_astro at ymail.com Sun Dec 16 16:00:14 2012 From: jerome_caron_astro at ymail.com (Jerome Caron) Date: Sun, 16 Dec 2012 21:00:14 +0000 (GMT) Subject: [AstroPy] VO table from skybot cannot be read by vo.table Message-ID: <1355691614.69237.YahooMailNeo@web171401.mail.ir2.yahoo.com> Dear list I have a problem with vo.table and SkyBOT. I make a SkyBoT request and I get the following xml file ? SkyBoT - List of the targets located in the FOV No solar system object was found in the requested FOV: RA DEC: 04:26:22.91 +44:45:13.30 EPOCH: 2012-12-16 18:42:54.72 I have a function that uses vo.table from astropy, that usually works fine. But this time it crashes with the following messages: ? ?[0;31mERROR?[0m: IndexError: No table found in VOTABLE file. [astropy.io.vo.tree] Traceback (most recent call last): ? File "X:\imagesheq5\2012-12-16\pav-18\04_check_objects_v10.py", line 159, in ??? cat = astrometry.read_votable(Folder+ '\\skybot.xml', text_display=False) ? File "C:\aspylib\aspylib\astrometry\query2.py", line 34, in read_votable ??? votable = parse_single_table(filename, pedantic=False) ? File "C:\Python27\lib\site-packages\astropy-0.1-py2.7-win32.egg\astropy\io\vo\table.py", line 126, in parse_single_table ??? return votable.get_first_table() ? File "C:\Python27\lib\site-packages\astropy-0.1-py2.7-win32.egg\astropy\io\vo\tree.py", line 2970, in get_first_table ??? raise IndexError("No table found in VOTABLE file.") IndexError: No table found in VOTABLE file. Appuyez sur une touche pour continuer... ? Do you have any idea ? Jerome Caron -------------- next part -------------- An HTML attachment was scrubbed... URL: From jerome_caron_astro at ymail.com Mon Dec 17 15:22:36 2012 From: jerome_caron_astro at ymail.com (Jerome Caron) Date: Mon, 17 Dec 2012 20:22:36 +0000 (GMT) Subject: [AstroPy] VO table from skybot cannot be read by vo.table In-Reply-To: <1355691614.69237.YahooMailNeo@web171401.mail.ir2.yahoo.com> References: <1355691614.69237.YahooMailNeo@web171401.mail.ir2.yahoo.com> Message-ID: <1355775756.58468.YahooMailNeo@web171404.mail.ir2.yahoo.com> I don't know if the file I get from SkyBoT is an incorrect VO table file, or if there is a problem with the function astropy.io.vo.table.parse_single_table. Can anybody guide me?? Many thanks for your help Clear skies Jerome Caron ________________________________ De?: Jerome Caron ??: "astropy at scipy.org" Cc?: francois Kugel Envoy? le : Dimanche 16 d?cembre 2012 22h00 Objet?: [AstroPy] VO table from skybot cannot be read by vo.table Dear list I have a problem with vo.table and SkyBOT. I make a SkyBoT request and I get the following xml file SkyBoT - List of the targets located in the FOV No solar system object was found in the requested FOV: RA DEC: 04:26:22.91 +44:45:13.30 EPOCH: 2012-12-16 18:42:54.72 I have a function that uses vo.table from astropy, that usually works fine. But this time it crashes with the following messages: ?[0;31mERROR?[0m: IndexError: No table found in VOTABLE file. [astropy.io.vo.tree] Traceback (most recent call last): ? File "X:\imagesheq5\2012-12-16\pav-18\04_check_objects_v10.py", line 159, in ??? cat = astrometry.read_votable(Folder+ '\\skybot.xml', text_display=False) ? File "C:\aspylib\aspylib\astrometry\query2.py", line 34, in read_votable ??? votable = parse_single_table(filename, pedantic=False) ? File "C:\Python27\lib\site-packages\astropy-0.1-py2.7-win32.egg\astropy\io\vo\table.py", line 126, in parse_single_table ??? return votable.get_first_table() ? File "C:\Python27\lib\site-packages\astropy-0.1-py2.7-win32.egg\astropy\io\vo\tree.py", line 2970, in get_first_table ??? raise IndexError("No table found in VOTABLE file.") IndexError: No table found in VOTABLE file. Appuyez sur une touche pour continuer... Do you have any idea ? Jerome Caron _______________________________________________ AstroPy mailing list AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Mon Dec 17 16:19:24 2012 From: mdroe at stsci.edu (Michael Droettboom) Date: Mon, 17 Dec 2012 16:19:24 -0500 Subject: [AstroPy] VO table from skybot cannot be read by vo.table In-Reply-To: <1355775756.58468.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <1355691614.69237.YahooMailNeo@web171401.mail.ir2.yahoo.com> <1355775756.58468.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: <50CF8C5C.1080601@stsci.edu> The result doesn't contain any tables, so parse_single_table rightfully raises an exception. If you want to access the contents of the file (which contain an error message etc.), so can use vo.table.parse. Cheers, Mike On 12/17/2012 03:22 PM, Jerome Caron wrote: > I don't know if the file I get from SkyBoT is an incorrect VO table > file, or if there is a problem with the function > astropy.io.vo.table.parse_single_table. > Can anybody guide me ? > Many thanks for your help > Clear skies > Jerome Caron > > *De :* Jerome Caron > *? :* "astropy at scipy.org" > *Cc :* francois Kugel > *Envoy? le :* Dimanche 16 d?cembre 2012 22h00 > *Objet :* [AstroPy] VO table from skybot cannot be read by vo.table > > Dear list > I have a problem with vo.table and SkyBOT. > I make a SkyBoT request and I get the following xml file > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns:vot="http://www.ivoa.net/xml/VOTable/v1.2" > xsi:schemaLocation="http://www.ivoa.net/xml/VOTable/v1.2 > http://www.ivoa.net/xml/VOTable/VOTable-1.2.xsd" > xmlns:stc="http://www.ivoa.net/xml/STC"> > > value="1.1623100147908+0.78109932048627,rm=848.52813742386"/> > > > SkyBoT - List > of the targets located in the FOV > No solar system object was > found in the requested FOV: RA DEC: 04:26:22.91 +44:45:13.30 EPOCH: > 2012-12-16 18:42:54.72 > > > I have a function that uses vo.table from astropy, that usually works > fine. But this time it crashes with the following messages: > ?[0;31mERROR?[0m: IndexError: No table found in VOTABLE file. > [astropy.io.vo.tree] > Traceback (most recent call last): > File "X:\imagesheq5\2012-12-16\pav-18\04_check_objects_v10.py", line > 159, in > cat = astrometry.read_votable(Folder+ '\\skybot.xml', > text_display=False) > File "C:\aspylib\aspylib\astrometry\query2.py", line 34, in read_votable > votable = parse_single_table(filename, pedantic=False) > File > "C:\Python27\lib\site-packages\astropy-0.1-py2.7-win32.egg\astropy\io\vo\table.py", > line 126, in parse_single_table > return votable.get_first_table() > File > "C:\Python27\lib\site-packages\astropy-0.1-py2.7-win32.egg\astropy\io\vo\tree.py", > line 2970, in get_first_table > raise IndexError("No table found in VOTABLE file.") > IndexError: No table found in VOTABLE file. > Appuyez sur une touche pour continuer... > Do you have any idea ? > Jerome Caron > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From trive at astro.su.se Mon Dec 17 17:49:36 2012 From: trive at astro.su.se (=?windows-1252?Q?Th=F8ger_Rivera-Thorsen?=) Date: Mon, 17 Dec 2012 23:49:36 +0100 Subject: [AstroPy] anyone using Pandas? In-Reply-To: References: <50CBCF4A.2020201@astro.su.se> <999E60D0-296D-4EED-8220-C4C73863E1F8@gmail.com> <50CC6292.1030707@astro.su.se> <5DFFE289-582A-4C31-9837-8CA8DC61CB0D@gmail.com> <50CF43D2.9050202@astro.su.se> Message-ID: <50CFA180.7030203@astro.su.se> On 12/17/2012 08:48 PM, Wolfgang Kerzendorf wrote: > Hi Thoger, > > Thanks, I maybe didn't phrase my question properly. I already did know how to do a single slice, what I wanted to know is to slice with multiple atom numbers > > I tried df.xs([1,3], level=0) which doesn't do it (same with xs). For me, with the dataframe from earlier, MyDF.ix[['1', '3']]['Nrg_lvl'] works just fine. Note, that there's a subtle difference: MyDF.ix[['1', '3']] will slice the two row'wise indices '1' and '3', while MyDF.ix[('1', '3')] will slice the row-wise index '1', column-wise index '3'. > In numpy I can slice with multiple indices by doing array[[1,3,4,5]] - I would like to do the same with Pandas however just using the first index. Here's how I do it now - but maybe there's a better way: > > levels_atom_filter = atom_data.levels_data['atomic_number'].isin([1,3,4,5]) > levels_data = atom_data.levels_data[levels_atom_filter] > > > As for the second question, I now figured that group by can do that. > > Cheers > Wolfgang > On 2012-12-17, at 11:09 AM, Th?ger Rivera-Thorsen wrote: > >> As for the first question: You simply use >> >> MyDF.xs('1') >> >> (level=0 is the default). That, or >> >> MyDF.ix['1'] >> >> if you want to also be able to change/set values (the .xs method only lets you read values). >> >> As for the second question, the .ix[] and .xs() methods slices index-wise, while column selection can be done with a simple [] like if you were slicing a Python list or 1D NumPy array: >> >> In [46]: MyDF.ix['1']['Nrg_lvl'] >> >> Out[46]: >> >> levels >> >> a 0.922009 >> >> b 0.575255 >> >> c 0.341259 >> >> Name: Nrg_lvl >> >> In [47]: MyDF.xs('1')['Nrg_lvl'] >> >> Out[47]: >> >> levels >> >> a 0.922009 >> >> b 0.575255 >> >> c 0.341259 >> >> Name: Nrg_lvl >> >> You can access the index[ by the .index parameter: >> >> MyDF.index >> >> Out[73]: >> >> MultiIndex >> >> [(1, a), (1, b), (1, c), (2, a), (2, b), (2, c), (3, a), (3, b), (3, c), (3, d), (3, e)] >> >> As you can see, the MultiIndex is essentially a list of tuples, so you can cycle through the list and >> for each tuple select the relevant entry and use for whatever you want. >> >> In general, I'll recommend that you take some time doing what I have done: Read the Pandas documentation - it's quite long, but it's full of examples and very comprehensive, and you can read only the chapters that seem to be relevant for what you are doing right now. Have an iPython interface of your choice open while you do it, and run the examples and maybe try out different variations of them - it gives a feeling of how it works far beyond what any e-mail explanations can provide. Also, I have learned a lot from simply using IPython's Tab-autocompletion - it tells what methods are available for an object and gives an idea about which ones could be interesting. >> Of course, there'll still be questions to ask, but you'll get many questions answered quickly that way. >> >> But before doing that, I think this video (if you haven't already watched it) is a quite thorough introduction without being too long and tehnical: >> >> http://www.youtube.com/watch?v=MxRMXhjXZos >> >> >> Cheers >> Th?ger >> >> >> On 12/16/2012 11:06 PM, Wolfgang Kerzendorf wrote: >>> Hi Thoger, >>> >>> Thanks for your suggestions. I'm using multi indexed levels now - there's still a couple of questions I have: >>> >>> How can I filter by atoms - in your example only have a data frame with 1s in them? >>> >>> How can I loop through one of the index - so get 1 and the a list of (0.052846 0.533835 0.185949, ?) >>> >>> Thanks for helping, >>> Wolfgang >>> >>> >>> How can I get back all the >>> On 2012-12-15, at 6:44 AM, Th?ger Rivera-Thorsen wrote: >>> >>>> Hi; >>>> >>>> It still depends. Your MultiIndex can be made from lists, tuples or >>>> arrays; I usually use arrays since I'm familiar with them from using >>>> NumPy. For a 3-level index, you need to have 3 1D-arrays of equal >>>> length, that civer alle the atomic nr. - Ion - Level combinations you >>>> want covered. Toy example; 3 atoms with three levels each, atoms denoted >>>> by number and levels by letters. Let's say the two first atoms have 3, >>>> and your third atom five levels. Then your arrays should be: >>>> atoms = sp.array(['1', '1', '1', '2', '2', '2', '3', '3', '3', '3', '3']) >>>> levels = sp.array(['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'd', 'e']) >>>> >>>> idx = pd.MultiIndex.from_arrays([atoms, levels], names=['atoms', 'levels']) >>>> >>>> Your data should then be a 11x2 array with energy and g as the columns. >>>> You then build the array as: >>>> >>>> MyDF = pd.DataFrame(dataarray, index=idx, columns=['Nrg_lvl', 'g.']) >>>> >>>> >>>> That gives: >>>> >>>> In [22]: print MyDF >>>> >>>> Nrg_lvl g. >>>> >>>> atoms levels >>>> >>>> 1 a 0.052846 0.533835 >>>> >>>> b 0.185949 0.064069 >>>> >>>> c 0.384630 0.646803 >>>> >>>> 2 a 0.835958 0.392594 >>>> >>>> b 0.016399 0.165862 >>>> >>>> c 0.300874 0.975590 >>>> >>>> 3 a 0.124640 0.815488 >>>> >>>> b 0.590613 0.749555 >>>> >>>> c 0.284481 0.299149 >>>> >>>> d 0.104408 0.723406 >>>> >>>> e 0.733087 0.730055 >>>> >>>> >>>> (Here I have just used random numbers for the data). >>>> >>>> From here, you can do all the Pandas goodness you want: >>>> >>>> In [23]: print MyDF.xs('b', level=1) >>>> >>>> Nrg_lvl g. >>>> >>>> atoms >>>> >>>> 1 0.185949 0.064069 >>>> >>>> 2 0.016399 0.165862 >>>> >>>> 3 0.590613 0.749555 >>>> >>>> >>>> Once you have the basic structure built, it is easy to extend either by >>>> concatenating or by using the set_value function. However, most of these >>>> operations are returning a modified copy rather than in-place >>>> operations, so better build as much of the DataFrame as possible in one >>>> go, if you're concerned about memory overhead. ?andas doesn't seem to be >>>> designed to gradually build its data structures. >>>> >>>> Cheers; >>>> >>>> Th?ger >>>> >>>> >>>> On 12/15/2012 05:52 AM, Wolfgang Kerzendorf wrote: >>>>> Sorry before this didn't go to the list: >>>>> >>>>> Hey guys, >>>>> >>>>> I'm slowly understanding better what pandas is about. The object I'm representing in a pandas data frame is an atomic database. >>>>> Each line in there has atomic_number, ion_number, level_number, energy, g. So I have created a Pandas dataFrame and then set the index to atomic_number, ion_number, level_number. Now I want to make a new DataFrame where atomic_number in (6, 7, 8, 9) - but it is an index. how do I do that? >>>>> >>>>> Cheers >>>>> W >>>>> On 2012-12-14, at 8:15 PM, Th?ger Rivera-Thorsen wrote: >>>>> >>>>>> Like Tyler said, can you be a bit more specific about what you want to >>>>>> obtain? >>>>>> What is your starting point, and where do you want to go from there? >>>>>> >>>>>> I've been looking quite a bit into multiindexing lately, and it is very >>>>>> handy but it does have some caveats. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 12/14/2012 09:04 PM, Wolfgang Kerzendorf wrote: >>>>>>> Hey guys, >>>>>>> >>>>>>> I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? >>>>>>> >>>>>>> Cheers >>>>>>> Wolfgang >>>>>>> _______________________________________________ >>>>>>> AstroPy mailing list >>>>>>> AstroPy at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>> _______________________________________________ >>>>>> AstroPy mailing list >>>>>> AstroPy at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>> _______________________________________________ >>>>> AstroPy mailing list >>>>> AstroPy at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/astropy From trive at astro.su.se Mon Dec 17 19:34:21 2012 From: trive at astro.su.se (=?windows-1252?Q?Th=F8ger_Rivera-Thorsen?=) Date: Tue, 18 Dec 2012 01:34:21 +0100 Subject: [AstroPy] anyone using Pandas? In-Reply-To: References: <50CBCF4A.2020201@astro.su.se> <999E60D0-296D-4EED-8220-C4C73863E1F8@gmail.com> <50CC6292.1030707@astro.su.se> <5DFFE289-582A-4C31-9837-8CA8DC61CB0D@gmail.com> <50CF43D2.9050202@astro.su.se> <50CFA180.7030203@astro.su.se> Message-ID: <50CFBA0D.1060100@astro.su.se> On 12/17/2012 11:59 PM, Wolfgang Kerzendorf wrote: > So this doesn't work for me: > The following should have many more entries > > In [26]: myplasma.levels_data.ix[[14, 26]] > Out[26]: > level_id energy g metastable > atomic_number ion_number level_number > 14 0 14 1400000014 9.392722e-12 3 False > 26 1400000026 9.943114e-12 9 False I think the problem could be that your index is by integers, not strings. Pandas does, IIRC, not support non-unique integer indices. That is, if you use an integer index, it is interpreted as a NumPy-like position based slicing index rather than a label-based slicing. I guess if you convert the numbers in your MultiIndex to strings, it should work as you expect. PS: Let's try and stick to Reply-To-List so others can benefit from this, too :-) > > In [27]: myplasma.levels_data.ix[14] > Out[27]: > > MultiIndex: 752 entries, (0, 0) to (3, 51) > Data columns: > level_id 752 non-null values > energy 752 non-null values > g 752 non-null values > metastable 752 non-null values > dtypes: bool(1), float64(1), int64(2 > > In [28]: myplasma.levels_data.ix[26] > Out[28]: > > MultiIndex: 2486 entries, (0, 0) to (3, 275) > Data columns: > level_id 2486 non-null values > energy 2486 non-null values > g 2486 non-null values > metastable 2486 non-null values > dtypes: bool(1), float64(1), int64(2) > ------ > What I expect is 752+2486 entries which is not the case. > I'm on pandas 0.9.1 maybe that's the problem. > > My approach seems to work as well so it's not crucial. > > Cheers > Wolfgang > > On 2012-12-17, at 5:49 PM, Th?ger Rivera-Thorsen wrote: > >> On 12/17/2012 08:48 PM, Wolfgang Kerzendorf wrote: >>> Hi Thoger, >>> >>> Thanks, I maybe didn't phrase my question properly. I already did know how to do a single slice, what I wanted to know is to slice with multiple atom numbers >>> >>> I tried df.xs([1,3], level=0) which doesn't do it (same with xs). >> For me, with the dataframe from earlier, MyDF.ix[['1', '3']]['Nrg_lvl'] works just fine. >> Note, that there's a subtle difference: MyDF.ix[['1', '3']] will slice the two row'wise indices '1' and '3', while MyDF.ix[('1', '3')] will slice the row-wise index '1', column-wise index '3'. >> >> >> >>> In numpy I can slice with multiple indices by doing array[[1,3,4,5]] - I would like to do the same with Pandas however just using the first index. Here's how I do it now - but maybe there's a better way: >>> >>> levels_atom_filter = atom_data.levels_data['atomic_number'].isin([1,3,4,5]) >>> levels_data = atom_data.levels_data[levels_atom_filter] >>> >>> >>> As for the second question, I now figured that group by can do that. >>> >>> Cheers >>> Wolfgang >>> On 2012-12-17, at 11:09 AM, Th?ger Rivera-Thorsen wrote: >>> >>>> As for the first question: You simply use >>>> >>>> MyDF.xs('1') >>>> >>>> (level=0 is the default). That, or >>>> >>>> MyDF.ix['1'] >>>> >>>> if you want to also be able to change/set values (the .xs method only lets you read values). >>>> >>>> As for the second question, the .ix[] and .xs() methods slices index-wise, while column selection can be done with a simple [] like if you were slicing a Python list or 1D NumPy array: >>>> >>>> In [46]: MyDF.ix['1']['Nrg_lvl'] >>>> >>>> Out[46]: >>>> >>>> levels >>>> >>>> a 0.922009 >>>> >>>> b 0.575255 >>>> >>>> c 0.341259 >>>> >>>> Name: Nrg_lvl >>>> >>>> In [47]: MyDF.xs('1')['Nrg_lvl'] >>>> >>>> Out[47]: >>>> >>>> levels >>>> >>>> a 0.922009 >>>> >>>> b 0.575255 >>>> >>>> c 0.341259 >>>> >>>> Name: Nrg_lvl >>>> >>>> You can access the index[ by the .index parameter: >>>> >>>> MyDF.index >>>> >>>> Out[73]: >>>> >>>> MultiIndex >>>> >>>> [(1, a), (1, b), (1, c), (2, a), (2, b), (2, c), (3, a), (3, b), (3, c), (3, d), (3, e)] >>>> >>>> As you can see, the MultiIndex is essentially a list of tuples, so you can cycle through the list and >>>> for each tuple select the relevant entry and use for whatever you want. >>>> >>>> In general, I'll recommend that you take some time doing what I have done: Read the Pandas documentation - it's quite long, but it's full of examples and very comprehensive, and you can read only the chapters that seem to be relevant for what you are doing right now. Have an iPython interface of your choice open while you do it, and run the examples and maybe try out different variations of them - it gives a feeling of how it works far beyond what any e-mail explanations can provide. Also, I have learned a lot from simply using IPython's Tab-autocompletion - it tells what methods are available for an object and gives an idea about which ones could be interesting. >>>> Of course, there'll still be questions to ask, but you'll get many questions answered quickly that way. >>>> >>>> But before doing that, I think this video (if you haven't already watched it) is a quite thorough introduction without being too long and tehnical: >>>> >>>> http://www.youtube.com/watch?v=MxRMXhjXZos >>>> >>>> >>>> Cheers >>>> Th?ger >>>> >>>> >>>> On 12/16/2012 11:06 PM, Wolfgang Kerzendorf wrote: >>>>> Hi Thoger, >>>>> >>>>> Thanks for your suggestions. I'm using multi indexed levels now - there's still a couple of questions I have: >>>>> >>>>> How can I filter by atoms - in your example only have a data frame with 1s in them? >>>>> >>>>> How can I loop through one of the index - so get 1 and the a list of (0.052846 0.533835 0.185949, ?) >>>>> >>>>> Thanks for helping, >>>>> Wolfgang >>>>> >>>>> >>>>> How can I get back all the >>>>> On 2012-12-15, at 6:44 AM, Th?ger Rivera-Thorsen wrote: >>>>> >>>>>> Hi; >>>>>> >>>>>> It still depends. Your MultiIndex can be made from lists, tuples or >>>>>> arrays; I usually use arrays since I'm familiar with them from using >>>>>> NumPy. For a 3-level index, you need to have 3 1D-arrays of equal >>>>>> length, that civer alle the atomic nr. - Ion - Level combinations you >>>>>> want covered. Toy example; 3 atoms with three levels each, atoms denoted >>>>>> by number and levels by letters. Let's say the two first atoms have 3, >>>>>> and your third atom five levels. Then your arrays should be: >>>>>> atoms = sp.array(['1', '1', '1', '2', '2', '2', '3', '3', '3', '3', '3']) >>>>>> levels = sp.array(['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'd', 'e']) >>>>>> >>>>>> idx = pd.MultiIndex.from_arrays([atoms, levels], names=['atoms', 'levels']) >>>>>> >>>>>> Your data should then be a 11x2 array with energy and g as the columns. >>>>>> You then build the array as: >>>>>> >>>>>> MyDF = pd.DataFrame(dataarray, index=idx, columns=['Nrg_lvl', 'g.']) >>>>>> >>>>>> >>>>>> That gives: >>>>>> >>>>>> In [22]: print MyDF >>>>>> >>>>>> Nrg_lvl g. >>>>>> >>>>>> atoms levels >>>>>> >>>>>> 1 a 0.052846 0.533835 >>>>>> >>>>>> b 0.185949 0.064069 >>>>>> >>>>>> c 0.384630 0.646803 >>>>>> >>>>>> 2 a 0.835958 0.392594 >>>>>> >>>>>> b 0.016399 0.165862 >>>>>> >>>>>> c 0.300874 0.975590 >>>>>> >>>>>> 3 a 0.124640 0.815488 >>>>>> >>>>>> b 0.590613 0.749555 >>>>>> >>>>>> c 0.284481 0.299149 >>>>>> >>>>>> d 0.104408 0.723406 >>>>>> >>>>>> e 0.733087 0.730055 >>>>>> >>>>>> >>>>>> (Here I have just used random numbers for the data). >>>>>> >>>>>> From here, you can do all the Pandas goodness you want: >>>>>> >>>>>> In [23]: print MyDF.xs('b', level=1) >>>>>> >>>>>> Nrg_lvl g. >>>>>> >>>>>> atoms >>>>>> >>>>>> 1 0.185949 0.064069 >>>>>> >>>>>> 2 0.016399 0.165862 >>>>>> >>>>>> 3 0.590613 0.749555 >>>>>> >>>>>> >>>>>> Once you have the basic structure built, it is easy to extend either by >>>>>> concatenating or by using the set_value function. However, most of these >>>>>> operations are returning a modified copy rather than in-place >>>>>> operations, so better build as much of the DataFrame as possible in one >>>>>> go, if you're concerned about memory overhead. ?andas doesn't seem to be >>>>>> designed to gradually build its data structures. >>>>>> >>>>>> Cheers; >>>>>> >>>>>> Th?ger >>>>>> >>>>>> >>>>>> On 12/15/2012 05:52 AM, Wolfgang Kerzendorf wrote: >>>>>>> Sorry before this didn't go to the list: >>>>>>> >>>>>>> Hey guys, >>>>>>> >>>>>>> I'm slowly understanding better what pandas is about. The object I'm representing in a pandas data frame is an atomic database. >>>>>>> Each line in there has atomic_number, ion_number, level_number, energy, g. So I have created a Pandas dataFrame and then set the index to atomic_number, ion_number, level_number. Now I want to make a new DataFrame where atomic_number in (6, 7, 8, 9) - but it is an index. how do I do that? >>>>>>> >>>>>>> Cheers >>>>>>> W >>>>>>> On 2012-12-14, at 8:15 PM, Th?ger Rivera-Thorsen wrote: >>>>>>> >>>>>>>> Like Tyler said, can you be a bit more specific about what you want to >>>>>>>> obtain? >>>>>>>> What is your starting point, and where do you want to go from there? >>>>>>>> >>>>>>>> I've been looking quite a bit into multiindexing lately, and it is very >>>>>>>> handy but it does have some caveats. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 12/14/2012 09:04 PM, Wolfgang Kerzendorf wrote: >>>>>>>>> Hey guys, >>>>>>>>> >>>>>>>>> I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> Wolfgang >>>>>>>>> _______________________________________________ >>>>>>>>> AstroPy mailing list >>>>>>>>> AstroPy at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>>>> _______________________________________________ >>>>>>>> AstroPy mailing list >>>>>>>> AstroPy at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>>> _______________________________________________ >>>>>>> AstroPy mailing list >>>>>>> AstroPy at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>> _______________________________________________ >>>>>> AstroPy mailing list >>>>>> AstroPy at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/astropy From trive at astro.su.se Mon Dec 17 19:41:04 2012 From: trive at astro.su.se (=?windows-1252?Q?Th=F8ger_Rivera-Thorsen?=) Date: Tue, 18 Dec 2012 01:41:04 +0100 Subject: [AstroPy] anyone using Pandas? In-Reply-To: References: <50CBCF4A.2020201@astro.su.se> <999E60D0-296D-4EED-8220-C4C73863E1F8@gmail.com> <50CC6292.1030707@astro.su.se> <5DFFE289-582A-4C31-9837-8CA8DC61CB0D@gmail.com> <50CF43D2.9050202@astro.su.se> <50CFA180.7030203@astro.su.se> Message-ID: <50CFBBA0.5070505@astro.su.se> OK that didn't turn out very clearly; I'll try again: On 12/17/2012 11:59 PM, Wolfgang Kerzendorf wrote: > So this doesn't work for me: > The following should have many more entries > > In [26]: myplasma.levels_data.ix[[14, 26]] > Out[26]: > level_id energy g metastable > atomic_number ion_number level_number > 14 0 14 1400000014 9.392722e-12 3 False > 26 1400000026 9.943114e-12 9 False Using integers for index is interpreted as a Numpy-like position slicing, not a pandas label-based slicing. That is; myplasma.levels_data.ix[[14, 26]] will return the 13th and 25th row in your DataFrame, while myplasma.levels_data.ix[['14', '26']] should return the sub-frame with the atomic numbers 14 and 26, as you expected. > In [27]: myplasma.levels_data.ix[14] > Out[27]: > > MultiIndex: 752 entries, (0, 0) to (3, 51) > Data columns: > level_id 752 non-null values > energy 752 non-null values > g 752 non-null values > metastable 752 non-null values > dtypes: bool(1), float64(1), int64(2 > > In [28]: myplasma.levels_data.ix[26] > Out[28]: > > MultiIndex: 2486 entries, (0, 0) to (3, 275) > Data columns: > level_id 2486 non-null values > energy 2486 non-null values > g 2486 non-null values > metastable 2486 non-null values > dtypes: bool(1), float64(1), int64(2) > ------ > What I expect is 752+2486 entries which is not the case. > I'm on pandas 0.9.1 maybe that's the problem. > > My approach seems to work as well so it's not crucial. > > Cheers > Wolfgang > > On 2012-12-17, at 5:49 PM, Th?ger Rivera-Thorsen wrote: > >> On 12/17/2012 08:48 PM, Wolfgang Kerzendorf wrote: >>> Hi Thoger, >>> >>> Thanks, I maybe didn't phrase my question properly. I already did know how to do a single slice, what I wanted to know is to slice with multiple atom numbers >>> >>> I tried df.xs([1,3], level=0) which doesn't do it (same with xs). >> For me, with the dataframe from earlier, MyDF.ix[['1', '3']]['Nrg_lvl'] works just fine. >> Note, that there's a subtle difference: MyDF.ix[['1', '3']] will slice the two row'wise indices '1' and '3', while MyDF.ix[('1', '3')] will slice the row-wise index '1', column-wise index '3'. >> >> >> >>> In numpy I can slice with multiple indices by doing array[[1,3,4,5]] - I would like to do the same with Pandas however just using the first index. Here's how I do it now - but maybe there's a better way: >>> >>> levels_atom_filter = atom_data.levels_data['atomic_number'].isin([1,3,4,5]) >>> levels_data = atom_data.levels_data[levels_atom_filter] >>> >>> >>> As for the second question, I now figured that group by can do that. >>> >>> Cheers >>> Wolfgang >>> On 2012-12-17, at 11:09 AM, Th?ger Rivera-Thorsen wrote: >>> >>>> As for the first question: You simply use >>>> >>>> MyDF.xs('1') >>>> >>>> (level=0 is the default). That, or >>>> >>>> MyDF.ix['1'] >>>> >>>> if you want to also be able to change/set values (the .xs method only lets you read values). >>>> >>>> As for the second question, the .ix[] and .xs() methods slices index-wise, while column selection can be done with a simple [] like if you were slicing a Python list or 1D NumPy array: >>>> >>>> In [46]: MyDF.ix['1']['Nrg_lvl'] >>>> >>>> Out[46]: >>>> >>>> levels >>>> >>>> a 0.922009 >>>> >>>> b 0.575255 >>>> >>>> c 0.341259 >>>> >>>> Name: Nrg_lvl >>>> >>>> In [47]: MyDF.xs('1')['Nrg_lvl'] >>>> >>>> Out[47]: >>>> >>>> levels >>>> >>>> a 0.922009 >>>> >>>> b 0.575255 >>>> >>>> c 0.341259 >>>> >>>> Name: Nrg_lvl >>>> >>>> You can access the index[ by the .index parameter: >>>> >>>> MyDF.index >>>> >>>> Out[73]: >>>> >>>> MultiIndex >>>> >>>> [(1, a), (1, b), (1, c), (2, a), (2, b), (2, c), (3, a), (3, b), (3, c), (3, d), (3, e)] >>>> >>>> As you can see, the MultiIndex is essentially a list of tuples, so you can cycle through the list and >>>> for each tuple select the relevant entry and use for whatever you want. >>>> >>>> In general, I'll recommend that you take some time doing what I have done: Read the Pandas documentation - it's quite long, but it's full of examples and very comprehensive, and you can read only the chapters that seem to be relevant for what you are doing right now. Have an iPython interface of your choice open while you do it, and run the examples and maybe try out different variations of them - it gives a feeling of how it works far beyond what any e-mail explanations can provide. Also, I have learned a lot from simply using IPython's Tab-autocompletion - it tells what methods are available for an object and gives an idea about which ones could be interesting. >>>> Of course, there'll still be questions to ask, but you'll get many questions answered quickly that way. >>>> >>>> But before doing that, I think this video (if you haven't already watched it) is a quite thorough introduction without being too long and tehnical: >>>> >>>> http://www.youtube.com/watch?v=MxRMXhjXZos >>>> >>>> >>>> Cheers >>>> Th?ger >>>> >>>> >>>> On 12/16/2012 11:06 PM, Wolfgang Kerzendorf wrote: >>>>> Hi Thoger, >>>>> >>>>> Thanks for your suggestions. I'm using multi indexed levels now - there's still a couple of questions I have: >>>>> >>>>> How can I filter by atoms - in your example only have a data frame with 1s in them? >>>>> >>>>> How can I loop through one of the index - so get 1 and the a list of (0.052846 0.533835 0.185949, ?) >>>>> >>>>> Thanks for helping, >>>>> Wolfgang >>>>> >>>>> >>>>> How can I get back all the >>>>> On 2012-12-15, at 6:44 AM, Th?ger Rivera-Thorsen wrote: >>>>> >>>>>> Hi; >>>>>> >>>>>> It still depends. Your MultiIndex can be made from lists, tuples or >>>>>> arrays; I usually use arrays since I'm familiar with them from using >>>>>> NumPy. For a 3-level index, you need to have 3 1D-arrays of equal >>>>>> length, that civer alle the atomic nr. - Ion - Level combinations you >>>>>> want covered. Toy example; 3 atoms with three levels each, atoms denoted >>>>>> by number and levels by letters. Let's say the two first atoms have 3, >>>>>> and your third atom five levels. Then your arrays should be: >>>>>> atoms = sp.array(['1', '1', '1', '2', '2', '2', '3', '3', '3', '3', '3']) >>>>>> levels = sp.array(['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'd', 'e']) >>>>>> >>>>>> idx = pd.MultiIndex.from_arrays([atoms, levels], names=['atoms', 'levels']) >>>>>> >>>>>> Your data should then be a 11x2 array with energy and g as the columns. >>>>>> You then build the array as: >>>>>> >>>>>> MyDF = pd.DataFrame(dataarray, index=idx, columns=['Nrg_lvl', 'g.']) >>>>>> >>>>>> >>>>>> That gives: >>>>>> >>>>>> In [22]: print MyDF >>>>>> >>>>>> Nrg_lvl g. >>>>>> >>>>>> atoms levels >>>>>> >>>>>> 1 a 0.052846 0.533835 >>>>>> >>>>>> b 0.185949 0.064069 >>>>>> >>>>>> c 0.384630 0.646803 >>>>>> >>>>>> 2 a 0.835958 0.392594 >>>>>> >>>>>> b 0.016399 0.165862 >>>>>> >>>>>> c 0.300874 0.975590 >>>>>> >>>>>> 3 a 0.124640 0.815488 >>>>>> >>>>>> b 0.590613 0.749555 >>>>>> >>>>>> c 0.284481 0.299149 >>>>>> >>>>>> d 0.104408 0.723406 >>>>>> >>>>>> e 0.733087 0.730055 >>>>>> >>>>>> >>>>>> (Here I have just used random numbers for the data). >>>>>> >>>>>> From here, you can do all the Pandas goodness you want: >>>>>> >>>>>> In [23]: print MyDF.xs('b', level=1) >>>>>> >>>>>> Nrg_lvl g. >>>>>> >>>>>> atoms >>>>>> >>>>>> 1 0.185949 0.064069 >>>>>> >>>>>> 2 0.016399 0.165862 >>>>>> >>>>>> 3 0.590613 0.749555 >>>>>> >>>>>> >>>>>> Once you have the basic structure built, it is easy to extend either by >>>>>> concatenating or by using the set_value function. However, most of these >>>>>> operations are returning a modified copy rather than in-place >>>>>> operations, so better build as much of the DataFrame as possible in one >>>>>> go, if you're concerned about memory overhead. ?andas doesn't seem to be >>>>>> designed to gradually build its data structures. >>>>>> >>>>>> Cheers; >>>>>> >>>>>> Th?ger >>>>>> >>>>>> >>>>>> On 12/15/2012 05:52 AM, Wolfgang Kerzendorf wrote: >>>>>>> Sorry before this didn't go to the list: >>>>>>> >>>>>>> Hey guys, >>>>>>> >>>>>>> I'm slowly understanding better what pandas is about. The object I'm representing in a pandas data frame is an atomic database. >>>>>>> Each line in there has atomic_number, ion_number, level_number, energy, g. So I have created a Pandas dataFrame and then set the index to atomic_number, ion_number, level_number. Now I want to make a new DataFrame where atomic_number in (6, 7, 8, 9) - but it is an index. how do I do that? >>>>>>> >>>>>>> Cheers >>>>>>> W >>>>>>> On 2012-12-14, at 8:15 PM, Th?ger Rivera-Thorsen wrote: >>>>>>> >>>>>>>> Like Tyler said, can you be a bit more specific about what you want to >>>>>>>> obtain? >>>>>>>> What is your starting point, and where do you want to go from there? >>>>>>>> >>>>>>>> I've been looking quite a bit into multiindexing lately, and it is very >>>>>>>> handy but it does have some caveats. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 12/14/2012 09:04 PM, Wolfgang Kerzendorf wrote: >>>>>>>>> Hey guys, >>>>>>>>> >>>>>>>>> I'm trying to play around with pandas. Currently I have a look at join and it always seems to copy the data. I believe I want to use advanced indexing, but am not quite sure how to do that: any Pandas experts here? >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> Wolfgang >>>>>>>>> _______________________________________________ >>>>>>>>> AstroPy mailing list >>>>>>>>> AstroPy at scipy.org >>>>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>>>> _______________________________________________ >>>>>>>> AstroPy mailing list >>>>>>>> AstroPy at scipy.org >>>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>>> _______________________________________________ >>>>>>> AstroPy mailing list >>>>>>> AstroPy at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>> _______________________________________________ >>>>>> AstroPy mailing list >>>>>> AstroPy at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/astropy From eduardo.balbinot at gmail.com Sat Dec 22 13:22:20 2012 From: eduardo.balbinot at gmail.com (Eduardo Balbinot) Date: Sat, 22 Dec 2012 16:22:20 -0200 Subject: [AstroPy] Maximum likelihood fit to unbinned data Message-ID: A bit off-topic, but.... Anyone could point to a good guide to fitting unbinned data using a maximum likelihood estimator? It would be a nice idea to have it on astropy with most used density profiles (Hubble, Plummer, King, etc...). If I manage to do it we could discuss including it on astropy. Thanks! -- Eduardo Balbinot -------------- next part -------------- An HTML attachment was scrubbed... URL: From pebarrett at gmail.com Sat Dec 22 18:14:05 2012 From: pebarrett at gmail.com (Paul Barrett) Date: Sat, 22 Dec 2012 18:14:05 -0500 Subject: [AstroPy] Maximum likelihood fit to unbinned data In-Reply-To: References: Message-ID: Eduardo, What type of data are you trying to fit; time series, spectral or image? You might want to take a look at what the gamma-ray community has done, since almost all of their analysis involves photon lists. You could also try searching the astrostatistics website. Personally, I would use a Bayesian approach. I have successfully used the Nested Sampling approach to fit FUSE and Chandra spectra. This algorithm succeeded where the traditional Levenberg-Marquardt algorithm did not. The algorithm is described in "Data Analysis: A Bayesian Approach" by D.S. Sivia. One of the best introductions to the subject in my opinion. -- Paul On Sat, Dec 22, 2012 at 1:22 PM, Eduardo Balbinot wrote: > A bit off-topic, but.... > > Anyone could point to a good guide to fitting unbinned data using a maximum > likelihood estimator? > > It would be a nice idea to have it on astropy with most used density > profiles (Hubble, Plummer, King, etc...). If I manage to do it we could > discuss including it on astropy. > > Thanks! > -- Eduardo Balbinot > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From jim.vickroy at noaa.gov Wed Dec 26 17:26:58 2012 From: jim.vickroy at noaa.gov (Jim Vickroy) Date: Wed, 26 Dec 2012 15:26:58 -0700 Subject: [AstroPy] Maximum likelihood fit to unbinned data In-Reply-To: References: Message-ID: <50DB79B2.2070606@noaa.gov> On 12/22/2012 11:22 AM, Eduardo Balbinot wrote: > A bit off-topic, but.... > > Anyone could point to a good guide to fitting unbinned data using a > maximum likelihood estimator? > > It would be a nice idea to have it on astropy with most used density > profiles (Hubble, Plummer, King, etc...). If I manage to do it we > could discuss including it on astropy. > > Thanks! > -- Eduardo Balbinot > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy Hi Eduardo, Have you looked at: http://scikit-learn.org/stable/? -------------- next part -------------- An HTML attachment was scrubbed... URL: From eduardo.balbinot at gmail.com Wed Dec 26 18:15:32 2012 From: eduardo.balbinot at gmail.com (Eduardo Balbinot) Date: Wed, 26 Dec 2012 21:15:32 -0200 Subject: [AstroPy] Maximum likelihood fit to unbinned data In-Reply-To: References: Message-ID: Hi there, thanks for the references. I managed to solve the issue using the reference Paul sent. I was trying to fit a density profile to a distribution of star. The usual way of doing it is to build a histogram of dN/dR (number of points/radial bins). But that introduces biases due to the binning choice. A way to avoid binning is to use a likelihood approach. So I followed this paper 2008ApJ...684.1075M. If anyone is interested in the code I wrote for that please ask! Cheers and thanks again. -- Eduardo Balbinot On Sat, Dec 22, 2012 at 9:14 PM, Paul Barrett wrote: > Eduardo, > > What type of data are you trying to fit; time series, spectral or image? > > You might want to take a look at what the gamma-ray community has > done, since almost all of their analysis involves photon lists. You > could also try searching the astrostatistics website. > > Personally, I would use a Bayesian approach. I have successfully used > the Nested Sampling approach to fit FUSE and Chandra spectra. This > algorithm succeeded where the traditional Levenberg-Marquardt > algorithm did not. The algorithm is described in "Data Analysis: A > Bayesian Approach" by D.S. Sivia. One of the best introductions to > the subject in my opinion. > > -- Paul > > On Sat, Dec 22, 2012 at 1:22 PM, Eduardo Balbinot > wrote: > > A bit off-topic, but.... > > > > Anyone could point to a good guide to fitting unbinned data using a > maximum > > likelihood estimator? > > > > It would be a nice idea to have it on astropy with most used density > > profiles (Hubble, Plummer, King, etc...). If I manage to do it we could > > discuss including it on astropy. > > > > Thanks! > > -- Eduardo Balbinot > > > > _______________________________________________ > > AstroPy mailing list > > AstroPy at scipy.org > > http://mail.scipy.org/mailman/listinfo/astropy > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Thu Dec 27 13:13:32 2012 From: vanderplas at astro.washington.edu (Jake Vanderplas) Date: Thu, 27 Dec 2012 10:13:32 -0800 Subject: [AstroPy] Maximum likelihood fit to unbinned data Message-ID: <50DC8FCC.3070400@astro.washington.edu> Hi, We discuss this sort of thing a bit in our upcoming book. The text not yet published, but the figures and associated code are online. We show one example of unbinned regression of Poisson counts within a Bayesian context, which you can see here: http://astroml.github.com/book_figures/chapter5/fig_poisson_continuous.html The example is perhaps a bit opaque without the accompanying discussion in the text, but I thought you might find it helpful! Jake On 12/22/2012 11:22 AM, Eduardo Balbinot wrote: > A bit off-topic, but.... > > Anyone could point to a good guide to fitting unbinned data using a > maximum likelihood estimator? > > It would be a nice idea to have it on astropy with most used density > profiles (Hubble, Plummer, King, etc...). If I manage to do it we > could discuss including it on astropy. > > Thanks! > -- Eduardo Balbinot