From chanley at stsci.edu Mon May 1 13:14:26 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon, 01 May 2006 13:14:26 -0400 Subject: [AstroPy] PyFITS 1.1 "alpha 2" release Message-ID: <445641F2.8040509@stsci.edu> ------------------ | PYFITS Release | ------------------ Space Telescope Science Institute is pleased to announce the "alpha 2" release of PyFITS 1.1. This release includes support for both the NUMPY and NUMARRAY array packages. This software can be downloaded at: http://www.stsci.edu/resources/software_hardware/pyfits/Download The NUMPY support in PyFITS is not nearly as well tested as the NUMARRAY support. We expect that you will encounter bugs. Please send bug reports to "help at stsci.edu". We intend to support NUMARRAY and NUMPY simultaneously for a transition period of no less than 1 year. Eventually, however, support for NUMARRAY will disappear. During this period, it is likely that new features will appear only for NUMPY. The support for NUMARRAY will primarily be to fix serious bugs and handle platform updates. ----------- | Version | ----------- Version 1.1a2; May 1, 2006 ------------------------------- | Major Changes since v1.1a1 | ------------------------------- * Corrected a bug in which the format for a Column object was not being parsed properly because of format repeats. * Due to a change in NUMPY, recarray field access via attributes will no longer clobber pre-existing class methods or attributes. This behavior was causing PYFITS to fail when attempting to read tables that contained column names that were equivalent to existing class attribute or method names such as "field". This change in numpy was made in revision 2288. ------------------------- | Software Requirements | ------------------------- PyFITS Version 1.1a2 REQUIRES: * Python 2.3 or later * NUMPY or NUMARRAY --------------------- | Installing PyFITS | --------------------- PyFITS 1.1a2 is distributed as a Python distutils module. Installation simply involves unpacking the package and executing % python setup.py install to install it in Python's site-packages directory. Alternatively the command %python setup.py install --local="/destination/directory/" will install PyFITS in an arbitrary directory which should be placed on PYTHONPATH. Once numarray or numpy has been installed, then PyFITS should be available for use under Python. ----------------- | Download Site | ----------------- http://www.stsci.edu/resources/software_hardware/pyfits/Download ---------- | Usage | ---------- Users will issue an "import pyfits" command as in the past. However, the use of the NUMPY or NUMARRAY version of PyFITS will be controlled by an environment variable called NUMERIX. Set NUMERIX to 'numarray' for the NUMARRAY version of PyFITS. Set NUMERIX to 'numpy' for the NUMPY version of pyfits. If only one array package is installed, that package's version of PyFITS will be imported. If both packages are installed the NUMERIX value is used to decide which version to import. If no NUMERIX value is set then the NUMARRAY version of PyFITS will be imported. Anything else will raise an exception upon import. --------------- | Bug Reports | --------------- Please send all PyFITS bug reports to help at stsci.edu ------------------ | Advanced Users | ------------------ Users who would like the "bleeding" edge of PyFITS can retrieve the software from our SUBVERSION repository hosted at: http://astropy.scipy.org/svn/pyfits/trunk We also provide a Trac site at: http://projects.scipy.org/astropy/pyfits/wiki -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From js at abyss.uoregon.edu Mon May 1 17:13:53 2006 From: js at abyss.uoregon.edu (James Schombert) Date: Mon, 01 May 2006 14:13:53 -0700 Subject: [AstroPy] AstroPy and XML Message-ID: <44567A11.2080405@abyss.uoregon.edu> I'm making the plunge to convert my datafiles (and .py output) into XML format/style. Are there any resources/websites with examples of working with astro data and XML format. I'm particularly interested in any accepted format for handling a set of XY datasets. There are many references on goggle to xml.gsfc.nasa.gov, but that site appears to have died. J From tara at physics.usyd.edu.au Mon May 1 17:16:59 2006 From: tara at physics.usyd.edu.au (Tara Murphy) Date: Tue, 2 May 2006 07:16:59 +1000 (EST) Subject: [AstroPy] AstroPy and XML In-Reply-To: <44567A11.2080405@abyss.uoregon.edu> References: <44567A11.2080405@abyss.uoregon.edu> Message-ID: Hi, > I'm making the plunge to convert my datafiles (and .py output) into XML > format/style. Are > there any resources/websites with examples of working with astro data > and XML format. I'm particularly interested in any accepted format for have you come across VOTable? It is a proposed XML standard for astronomy tables being developed by the Virtual Observatory. http://www.us-vo.org/VOTable/ ----- Dr. Tara Murphy ARC Postdoctoral Fellow School of Information | School of Physics Technologies Room G89B | Room 565 University of Sydney | University of Sydney P: +61 2 9351 4723 | P: +61 2 9351 3041 E: tm at it.usyd.edu.au | E: tara at physics.usyd.edu.au http://www.it.usyd.edu.au/~info1903 From emsellem at obs.univ-lyon1.fr Thu May 4 08:33:30 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Thu, 04 May 2006 14:33:30 +0200 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: References: Message-ID: <4459F49A.8000009@obs.univ-lyon1.fr> Hi, I am trying since some time to develop a simple module to wrap up some functionalities of pyfits and matplotlib. The goal is to be able to manipulate FITS files (spectra, images, 3D cubes, including maybe Euro3D format for those who know what it is) and display them (plot slices, display images, etc) in a very simple way. It could be inspired from (although not a cloning of) simple functionalities found in Midas. So you can do things like Image1.fits = Image2.fits / (Image3.fits + 3.4) Table.fits.col1 = Table.fits.col2 * Table.fits.col4 / 2. , or easily create a new Fits image, or table, display them by doing something like: load("Image1.fits", scale=3, cuts=(0.,100.), center=(-1.,6.)) or alternatively: ima = fitstab("Image1.fits") load(ima, scale=3, cuts=(0.,100.), center=-1.,6.) (where Image1.fits is displayed with a scaling factor of 3, with cuts as indicated, and the center of the figure corresponding to coordinates -1,6) or plot("Image1.fits", 0) (plot a cut at vertical coordinate 0) You could say then: use Midas. But Midas is clearly very limited (does not have the python built-in potential of course) and its display/plotting functions are really scarce. The fact that there is now a pyMidas available does not fill the gap since I would like to be able to run my favourite ide (ipython for the moment), and really profit from all python modularity/functions. So the choice for an I/O with fits seems easy: "pyfits" which is a really fantastic module for dealing with FITS files. And the choice for displaying is also not so difficult: "matplotlib" which is also an amazing module and provides paper quality figures. My questions are then: 0/ is this a dum idea? Maybe I am missing something here, so just let me know... 1/ is there already something existing which does what I mention above? (so a wrapper around pyfits+matplotlib). Maybe at StSci...? 2/ If I were to try to develop such a tool, it will result in a rather badly coded module: I will try my best but frankly I should be considered as a newby in python and I can sincerely say that I don't fully understand all the forum matplotlib/numpy/scipy discussions, by far... So the question is: would anybody be interested/willing to, once in a while, help me with this development? Some could answer that using directly pyfits+matplotlib does the job. True. But if you wish to directly deal with fits and display them with some option (zooming, cuts, centering, etc) you still need to go through a number of steps which are not necessarily easy to remember (syntax, etc). Some wrapping could help quite a lot here. Thanks for any input here. cheers Eric P.S.: for the sake of completeness I could send anybody on request the very ugly looking - and unfinished modules I wrote last year in this context, when I seriously engaged in python in replacement of Midas/Iraf/Idl... P.P.S.: this is a duplicate of a post sent to matplotlib, but as suggested by P. Barrett, this post is much more relevant to astropy... Sorry for the duplicate. From emsellem at obs.univ-lyon1.fr Thu May 4 09:11:13 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Thu, 04 May 2006 15:11:13 +0200 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <200605041502.11986.jaime@iaa.es> References: <4459F49A.8000009@obs.univ-lyon1.fr> <200605041502.11986.jaime@iaa.es> Message-ID: <4459FD71.8070108@obs.univ-lyon1.fr> Hi again, Jaime Perea wrote: > I would love such an idea but is there any work already done > by pymidas or pyiraf? well no. As mentioned pymidas is just an emulated version of Midas under python so you can use midas commands and also load things in some python-like arrays/lists, etc. Not necessarily what you wish to have here. Same goes for pyraf. And neither of the two have the functionalities which could be delivered by e.g. matplotlib then. (of course these are great assets anyway as it provides the good sides of both worlds, but I think we would be better off with a true pyfits+matplotlib wrapper for simple-everyday usage). I already wrote some "classes" a year ago: one for the tables and one for "images" (meaning nD images, as spectra - 1D - images - 2D - and cubes -3D) but this was done in a just too simple minded way (I was just beginning to learn python then). I would need an expert eye here, also as you mention to include all the methods pertaining to python. I could try to send a draft version of what I have done so far (just for you to see how bad a python-coder I am) and provide some example of what I wish to realize? as I said: any input is welcome here. Eric -- ==================================================================== Eric Emsellem emsellem at obs.univ-lyon1.fr Centre de Recherche Astrophysique de Lyon 9 av. Charles-Andre tel: +33 (0)4 78 86 83 84 69561 Saint-Genis Laval Cedex fax: +33 (0)4 78 86 83 86 France http://www-obs.univ-lyon1.fr/eric.emsellem ==================================================================== From humufr at yahoo.fr Thu May 4 09:52:33 2006 From: humufr at yahoo.fr (Humufr) Date: Thu, 04 May 2006 09:52:33 -0400 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <4459F49A.8000009@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> Message-ID: <445A0721.2000900@yahoo.fr> Hi, I would like something like this too. I tried to play a little bit to implement it but unfortunatly I find an intrinsic limitation in matplotlib. It's using agg and agg can't have image bigger than 3kx3k pixels or something like this and even before this limit matplotlib take age to plot an image on screen. Regars, N. Eric Emsellem wrote: > Hi, > > I am trying since some time to develop a simple module to wrap up some > functionalities of pyfits and matplotlib. > The goal is to be able to manipulate FITS files (spectra, images, 3D > cubes, including maybe Euro3D format for those who know what it is) and > display them (plot slices, display images, etc) in a very simple way. It > could be inspired from (although not a cloning of) simple > functionalities found in Midas. So you can do things like > > Image1.fits = Image2.fits / (Image3.fits + 3.4) > > Table.fits.col1 = Table.fits.col2 * Table.fits.col4 / 2. > > , or easily create a new Fits image, or table, display them by doing > something like: > > load("Image1.fits", scale=3, cuts=(0.,100.), center=(-1.,6.)) > > or alternatively: > > ima = fitstab("Image1.fits") > load(ima, scale=3, cuts=(0.,100.), center=-1.,6.) > > (where Image1.fits is displayed with a scaling factor of 3, with cuts as > indicated, and the center of the figure corresponding to coordinates -1,6) > > or > > plot("Image1.fits", 0) > > (plot a cut at vertical coordinate 0) > > You could say then: use Midas. But Midas is clearly very limited (does > not have the python built-in potential of course) and its > display/plotting functions are really scarce. The fact that there is now > a pyMidas available does not fill the gap since I would like to be able > to run my favourite ide (ipython for the moment), and really profit from > all python modularity/functions. > > So the choice for an I/O with fits seems easy: "pyfits" which is a > really fantastic module for dealing with FITS files. > And the choice for displaying is also not so difficult: "matplotlib" > which is also an amazing module and provides paper quality figures. > > My questions are then: > > 0/ is this a dum idea? Maybe I am missing something here, so just let me > know... > > 1/ is there already something existing which does what I mention above? > (so a wrapper around pyfits+matplotlib). Maybe at StSci...? > > 2/ If I were to try to develop such a tool, it will result in a rather > badly coded module: I will try my best but frankly I should be > considered as a newby in python and I can sincerely say that I don't > fully understand all the forum matplotlib/numpy/scipy discussions, by > far... So the question is: would anybody be interested/willing to, once > in a while, help me with this development? > > Some could answer that using directly pyfits+matplotlib does the job. > True. But if you wish to directly deal with fits and display them with > some option (zooming, cuts, centering, etc) you still need to go through > a number of steps which are not necessarily easy to remember (syntax, > etc). Some wrapping could help quite a lot here. > > Thanks for any input here. > > cheers > > Eric > P.S.: for the sake of completeness I could send anybody on request the > very ugly looking - and unfinished modules I wrote last year in this > context, when I seriously engaged in python in replacement of > Midas/Iraf/Idl... > P.P.S.: this is a duplicate of a post sent to matplotlib, but as > suggested by P. Barrett, this post is much more relevant to astropy... > Sorry for the duplicate. > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.net > http://www.scipy.net/mailman/listinfo/astropy > > From maurizio at iac.es Thu May 4 10:33:43 2006 From: maurizio at iac.es (Maurizio Panniello) Date: Thu, 04 May 2006 15:33:43 +0100 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <4459F49A.8000009@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> Message-ID: <445A10C7.2090409@iac.es> Very nice idea... It could be a great proyect if it is well done. Especially now that Iraf is no longer sopported and many people are moving to python... Good luck! >Hi, > >I am trying since some time to develop a simple module to wrap up some >functionalities of pyfits and matplotlib. >The goal is to be able to manipulate FITS files (spectra, images, 3D >cubes, including maybe Euro3D format for those who know what it is) and >display them (plot slices, display images, etc) in a very simple way. It >could be inspired from (although not a cloning of) simple >functionalities found in Midas. So you can do things like > >Image1.fits = Image2.fits / (Image3.fits + 3.4) > >Table.fits.col1 = Table.fits.col2 * Table.fits.col4 / 2. > >, or easily create a new Fits image, or table, display them by doing >something like: > >load("Image1.fits", scale=3, cuts=(0.,100.), center=(-1.,6.)) > >or alternatively: > >ima = fitstab("Image1.fits") >load(ima, scale=3, cuts=(0.,100.), center=-1.,6.) > >(where Image1.fits is displayed with a scaling factor of 3, with cuts as >indicated, and the center of the figure corresponding to coordinates -1,6) > >or > >plot("Image1.fits", 0) > >(plot a cut at vertical coordinate 0) > >You could say then: use Midas. But Midas is clearly very limited (does >not have the python built-in potential of course) and its >display/plotting functions are really scarce. The fact that there is now >a pyMidas available does not fill the gap since I would like to be able >to run my favourite ide (ipython for the moment), and really profit from >all python modularity/functions. > >So the choice for an I/O with fits seems easy: "pyfits" which is a >really fantastic module for dealing with FITS files. >And the choice for displaying is also not so difficult: "matplotlib" >which is also an amazing module and provides paper quality figures. > >My questions are then: > >0/ is this a dum idea? Maybe I am missing something here, so just let me >know... > >1/ is there already something existing which does what I mention above? >(so a wrapper around pyfits+matplotlib). Maybe at StSci...? > >2/ If I were to try to develop such a tool, it will result in a rather >badly coded module: I will try my best but frankly I should be >considered as a newby in python and I can sincerely say that I don't >fully understand all the forum matplotlib/numpy/scipy discussions, by >far... So the question is: would anybody be interested/willing to, once >in a while, help me with this development? > >Some could answer that using directly pyfits+matplotlib does the job. >True. But if you wish to directly deal with fits and display them with >some option (zooming, cuts, centering, etc) you still need to go through >a number of steps which are not necessarily easy to remember (syntax, >etc). Some wrapping could help quite a lot here. > >Thanks for any input here. > >cheers > >Eric >P.S.: for the sake of completeness I could send anybody on request the >very ugly looking - and unfinished modules I wrote last year in this >context, when I seriously engaged in python in replacement of >Midas/Iraf/Idl... >P.P.S.: this is a duplicate of a post sent to matplotlib, but as >suggested by P. Barrett, this post is much more relevant to astropy... >Sorry for the duplicate. >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.net >http://www.scipy.net/mailman/listinfo/astropy > > From erwin at mpe.mpg.de Thu May 4 11:18:56 2006 From: erwin at mpe.mpg.de (Peter Erwin) Date: Thu, 4 May 2006 17:18:56 +0200 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment Message-ID: Hi Eric, At 3:11 PM +0200 5/4/06, Eric Emsellem wrote: >Hi again, > >Jaime Perea wrote: >> I would love such an idea but is there any work already done >> by pymidas or pyiraf? > >well no. As mentioned pymidas is just an emulated version of Midas under >python so you can use midas commands and also load things in some >python-like arrays/lists, etc. Not necessarily what you wish to have >here. Same goes for pyraf. And neither of the two have the >functionalities which could be delivered by e.g. matplotlib then. >(of course these are great assets anyway as it provides the good sides >of both worlds, but I think we would be better off with a true >pyfits+matplotlib wrapper for simple-everyday usage). This may not be what you would really like, but I think it should be possible to use pyfits and matplotlib from *within* pyraf -- which is, after all, a sort of front-end to a regular Python interpreter. I have to admit that I don't currently use pyraf in command-line mode (I use its modules in some Python scripts), so I have no advice on how easy or feasible this would be in general. But the following works on my iBook: $ pyraf --> import pyfits --> import pylab --> ff="n3627irac1ss.fit" --> im = pyfits.getdata(ff) --> pylab.plot(im[1:,1000]) [] --> pylab.show() (I'm glossing over problems I have getting the prompt back after the first use of pylab.show(), but that's a matplotlib-related problem that afflicts plain Python sessions as well, at least using the TkAgg interface on my MacOS X system.) This ought to let one use the image-display capabilities of iraf + ds9 (for example), combined with whatever bits of pyfits and matplotlib you wish to use, and whatever wrappers you wish to write to make things simpler (the wrappers could, in principle, mix pyraft *and* pyfits *and* matplotlib...) cheers, Peter -- ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)89 30000 3695 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From jturner at gemini.edu Thu May 4 18:56:17 2006 From: jturner at gemini.edu (James Turner) Date: Thu, 04 May 2006 18:56:17 -0400 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <4459F49A.8000009@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> Message-ID: <445A8691.8070902@gemini.edu> Hi Eric, > Image1.fits = Image2.fits / (Image3.fits + 3.4) > > Table.fits.col1 = Table.fits.col2 * Table.fits.col4 / 2. > > , or easily create a new Fits image, or table, display them by doing > something like: > > load("Image1.fits", scale=3, cuts=(0.,100.), center=(-1.,6.)) Apologies if I have overlooked something basic, being fairly new to Python myself - but could you do what you want by loading the original data with PyFITS and then working in memory with numarray + numdisplay /matplotlib instead of FITS files on disk? Have you looked at the STScI tutorial at http://stsdas.stsci.edu/perry/pydatatut.pdf ? Also, I think ipython and Pyraf are somewhat interchangable (see tutorial), so maybe you can use the IRAF tasks (imarith?) to do what you want without losing your nice Python environment? If you would rather work with FITS files, I'm guessing it's for one of three (good) reasons: 1) you want to save all your intermediate results to disk, 2) you need to propagate headers, variance extensions etc., 3) you need to do some intermediate processing with MIDAS/IRAF or 4) it is just more convenient not to have to load and save the files explicitly. FOUR reasons! (sorry if you don't get the silly Monty Python joke, but it seemed appropriate) If you are concerned about #2, this is something I have been thinking about too, also in the context of IFU data. I'll send a separate email with some ideas on this, since it's perhaps a bit off topic. Cheers, James. -- James E.H. Turner Gemini Observatory Southern Operations Centre, Casilla 603, Tel. (+56) 51 205609 La Serena, Chile. Fax. (+56) 51 205650 From jturner at gemini.edu Thu May 4 19:14:17 2006 From: jturner at gemini.edu (James Turner) Date: Thu, 04 May 2006 19:14:17 -0400 Subject: [AstroPy] Representing astronomical (esp. IFU) datasets as Python objects In-Reply-To: <4459F49A.8000009@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> Message-ID: <445A8AC9.6060008@gemini.edu> Hi Eric et al., I was motivated to write this in reply to Eric's post on wrapping pyfits+matplotlib, but I think the subject is sufficiently different that I'll try to avoid mixing the 2 threads... For reducing astronomical data, I'd like to operate directly on objects in memory that have associated header-type information and methods for 'presenting' the data appropriately to the main program or user. This would be closer to manipulating FITS files in IRAF than working with 'bare' arrays. More importantly, it seems like a natural way to write reusable high-level functionality for use in Pyraf or Python scripts. PyFITS already does a good job of handling FITS files, but it is very generic and tied to the FITS structure. I would like to work with data objects that 'know more' about their own properties and contain methods for doing WCS transformations etc. Rather than tie these directly to FITS files, I started writing a DataSet class that will correspond to a single science data array plus associated WCS information, variance, data quality etc. This is just a layer on top of PyFITS & numarray, of course, although the DataSets don't have to exist on disk. The idea is that DataSets can be copied from or to PyFITS HDULists at the start or end of the script (it could get slightly more complicated with memory mapping etc.). The base class should be able to do generic operations on itself to present the appropriate data to the main program. Something like a Euro3D dataset could be a derived class with its own special methods to interpret that format, eg. allowing the main program to interrogate the DataSet for values by RA and Dec instead of IFU element. I believe it is easy to redefine/overload operators in Python, so it should be possible to do operations like '*', '+' etc. directly on the DataSets once the class is finished, just like in Eric's examples. All the WCS & variance propagation etc. would then get carried through automatically. I think one could use matplotlib or numdisplay to show the results without too much extra work or reinvention. In case anyone gets excited by this idea, I should state that I'm just another busy astronomer and Python newbie and am not about to pull the finished product out of a hat - but anyway I started writing this (a much less sophisticated version) as part of a datacube mosaicing program and I think in principle it could be quite powerful, so feel free to treat this as a "Request For Comments". In future, I hope the work will tie into our IRAF/Pyraf project at Gemini, but at the moment it's just a small personal experiment. I'd be willing to collaborate with others, time permitting, in order to make this better, although it's really a higher priority for me to spend more time on my astronomy research during the coming months, so this may be overambitious... Also, if anyone can point out why this is a stupid idea by an ignorant newbie, or if it has already been done, please tell me before I waste any more time on it! Cheers, James. -- James E.H. Turner Gemini Observatory Southern Operations Centre, Casilla 603, Tel. (+56) 51 205609 La Serena, Chile. Fax. (+56) 51 205650 From erwin at mpe.mpg.de Thu May 4 19:48:49 2006 From: erwin at mpe.mpg.de (Peter Erwin) Date: Fri, 5 May 2006 01:48:49 +0200 Subject: [AstroPy] Python tool for quick and dirty telescopy archive searches Message-ID: Hi, I've been working on a Python-based tool which people on this list might find useful. It's basically a command-line program which simplifies searching various public telescope archives to see if they might have data on a particular astronomical object or part of the sky. This includes the HST and general MAST archive, as well as the ING, ESO, UKIRT, CFHT, and AAT ground-based archives, and Data Release 4 (DR4) from the Sloan Digital Sky Survey. The NOAO Science Archive is also included, though it has very little publically available data at this point. (I hope to add searches of the Gemini and Subaru archives in the future.) The program won't *get* the data for you, of course, or even tell you very much about it -- for that, you still need to visit the individual archive web pages. But it will save you lots of clicking and typing in web-page forms if you just want to find out if there is *any* data available. An example (searching within a 2-arcminute box centered on the planetary nebula NGC 7027; searching on coordinates directly is also possible): $ telarchive "ngc 7027" 2.0 SIMBAD (France): Found object coordinates: RA = 21 07 01.59, Dec = +42 14 10.2 Searching archives for NGC 7027 (RA = 21 07 01.59, dec = +42 14 10.2), with search box = 2.0 arcmin... ESO Archive: No data found. HST Archive (at ESO): Data exists! ("A total of 85 were found") -- 6 WFPC2, 24 WF, 1 PC, 13 NICMOS, 3 FOC, 25 STIS, 13 FOS UKIRT Archive: Data exists! ("A total of 517 were retrieved") AAT Archive: No data found. Sloan Digital Sky Survey (DR4) SQL Search: No data found. NOAO Science Archive: No data found. ING Archive: Data exists! ("A total of 577 were retrieved") 143 images, 391 spectra, 43 unclassified JKT -- RBS (1), FWHL (2), AGBX (102); INT -- WFC (9), PFCU (30), IDS (66), MES (17), FOS_1 (10); WHT -- TAURUS_2 (5), unknown (43), ISIS_BLUE_ARM (159), UES (58), ISIS_RED_ARM (75) CFHT Archive: Data exists! ("A total of 213 records were retrieved") Multimission Archive at STScI (MAST): Data exists! FUSE (2); IUE (34); COPERNICUS (1) It's available as a gzipped tar file here: http://www.mpe.mpg.de/~erwin/code/ There are installation instructions/suggestions in the README file. (And feel free to email with questions or suggestions about it!) cheers, Peter -- ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)89 30000 3695 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From emsellem at obs.univ-lyon1.fr Fri May 5 03:20:36 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Fri, 05 May 2006 09:20:36 +0200 Subject: [AstroPy] Representing astronomical (esp. IFU) datasets as Python objects In-Reply-To: <445A8AC9.6060008@gemini.edu> References: <4459F49A.8000009@obs.univ-lyon1.fr> <445A8AC9.6060008@gemini.edu> Message-ID: <445AFCC4.1030209@obs.univ-lyon1.fr> Hi, just to make things clear there (trying at least), I don't want to get tied to Iraf or Midas here although these are very nice tools indeed. Iraf is full of very useful and complex routines but is not (I think) , easy to use even in a python environment. The fact that pyraf exists is great, but to me it is not achieving what it should I think, partly just because it was not meant as a python module originally. And the graphics/display are not convincing (although again, things like ds9 are powerful and are very useful tools). As for Midas, it has the advantage to be much simpler (and also powerful) but clearly lacks the speed, is kind of clumsy in terms of data addressing and lack a good display/graphics environment. > PyFITS already does a good job of handling FITS files, but it is very > generic and tied to the FITS structure. I would like to work with data > objects that 'know more' about their own properties and contain methods > for doing WCS transformations etc. This is exactly what I am talking about; I would like to use the power of python + numpy + .. to design a direct and easy way to address e.g. Fits files, work with the data (with all the power of python), be able to display/plot things then, and be able to (easily) write back the result. This means indeed to create new classes which would provide direct access to methods, ufuncs, but also embedded structures and info. One example is to be able to open a Euro3D cube, plot a slice, perform some operation on part of the cube, or merge it with another cube (the function being then able to check the consistent nature of the cubes) taking into account e.g. flagged pixels and variance, and maybe write back the result on disk either as another Euro3D cube, or some other formatted file on disK. The classes I am developing so far are indeed another layer on top of pyfits (in fact it inherits from HDUlist), which is very probably the best thing available for dealing with fits files. The fact that I am for the moment tied to fits files is just that this format is widely used in astronomy. We may switch to something else at some point, but we will always have to address binary data, associated flags, associate coordinates, variance, etc. (whatever you can think of which is useful to understand your data). I agree with you Peter that things can be worked out by just using pyfits/pyraf/matplotlib+.... These are fantastic assets! But even using these things everyday I am still everytime looking at some simple example of how to deal with the data before being able to write a new script or module. And I find the everyday use not efficient enough. So I may consider spending time to develop something which will help me to be more efficient later on (not sure about the total gain there, but at least I would have learn a little more of python hopefully!) Eric -- ==================================================================== Eric Emsellem emsellem at obs.univ-lyon1.fr Centre de Recherche Astrophysique de Lyon 9 av. Charles-Andre tel: +33 (0)4 78 86 83 84 69561 Saint-Genis Laval Cedex fax: +33 (0)4 78 86 83 86 France http://www-obs.univ-lyon1.fr/eric.emsellem ==================================================================== From jaime at iaa.es Thu May 4 09:02:11 2006 From: jaime at iaa.es (Jaime Perea) Date: Thu, 4 May 2006 15:02:11 +0200 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <4459F49A.8000009@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> Message-ID: <200605041502.11986.jaime@iaa.es> I would love such an idea but is there any work already done by pymidas or pyiraf? From the point of view of python perhaps the idea will be something as defining a 'fits' class using pyfits and include there all the __add__ __div__ __sub__ and whatever needed methods, really here the language can help a lot. It really does not seem to be so crazy :-) The problem I think is mainly all the conventions that are needed to handle the headers. Regards -- Jaime D. Perea Duarte. Linux registered user #10472 Dep. Astrofisica Extragalactica. Instituto de Astrofisica de Andalucia (CSIC) Apdo. 3004, 18080 Granada, Spain. El Jueves, 4 de Mayo de 2006 14:33, Eric Emsellem escribi?: > Hi, > > I am trying since some time to develop a simple module to wrap up some > functionalities of pyfits and matplotlib. > The goal is to be able to manipulate FITS files (spectra, images, 3D > cubes, including maybe Euro3D format for those who know what it is) and > display them (plot slices, display images, etc) in a very simple way. It > could be inspired from (although not a cloning of) simple > functionalities found in Midas. So you can do things like > > Image1.fits = Image2.fits / (Image3.fits + 3.4) > > Table.fits.col1 = Table.fits.col2 * Table.fits.col4 / 2. > > , or easily create a new Fits image, or table, display them by doing > something like: > > load("Image1.fits", scale=3, cuts=(0.,100.), center=(-1.,6.)) > > or alternatively: > > ima = fitstab("Image1.fits") > load(ima, scale=3, cuts=(0.,100.), center=-1.,6.) > > (where Image1.fits is displayed with a scaling factor of 3, with cuts as > indicated, and the center of the figure corresponding to coordinates > -1,6) > > or > > plot("Image1.fits", 0) > > (plot a cut at vertical coordinate 0) > > You could say then: use Midas. But Midas is clearly very limited (does > not have the python built-in potential of course) and its > display/plotting functions are really scarce. The fact that there is now > a pyMidas available does not fill the gap since I would like to be able > to run my favourite ide (ipython for the moment), and really profit from > all python modularity/functions. > > So the choice for an I/O with fits seems easy: "pyfits" which is a > really fantastic module for dealing with FITS files. > And the choice for displaying is also not so difficult: "matplotlib" > which is also an amazing module and provides paper quality figures. > > My questions are then: > > 0/ is this a dum idea? Maybe I am missing something here, so just let me > know... > > 1/ is there already something existing which does what I mention above? > (so a wrapper around pyfits+matplotlib). Maybe at StSci...? > > 2/ If I were to try to develop such a tool, it will result in a rather > badly coded module: I will try my best but frankly I should be > considered as a newby in python and I can sincerely say that I don't > fully understand all the forum matplotlib/numpy/scipy discussions, by > far... So the question is: would anybody be interested/willing to, once > in a while, help me with this development? > > Some could answer that using directly pyfits+matplotlib does the job. > True. But if you wish to directly deal with fits and display them with > some option (zooming, cuts, centering, etc) you still need to go through > a number of steps which are not necessarily easy to remember (syntax, > etc). Some wrapping could help quite a lot here. > > Thanks for any input here. > > cheers > > Eric > P.S.: for the sake of completeness I could send anybody on request the > very ugly looking - and unfinished modules I wrote last year in this > context, when I seriously engaged in python in replacement of > Midas/Iraf/Idl... > P.P.S.: this is a duplicate of a post sent to matplotlib, but as > suggested by P. Barrett, this post is much more relevant to astropy... > Sorry for the duplicate. > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.net > http://www.scipy.net/mailman/listinfo/astropy From jturner at gemini.edu Fri May 5 11:19:09 2006 From: jturner at gemini.edu (James Turner) Date: Fri, 05 May 2006 11:19:09 -0400 Subject: [AstroPy] Representing astronomical (esp. IFU) datasets as Python objects In-Reply-To: <445AFCC4.1030209@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> <445A8AC9.6060008@gemini.edu> <445AFCC4.1030209@obs.univ-lyon1.fr> Message-ID: <445B6CED.1080801@gemini.edu> Hi Eric, OK - sorry - I wasn't sure that we were talking about exactly the same thing. It sounds like you're ahead of me on a very similar path. I'd be happy to send code if it is useful, although I haven't really done a lot yet, just the bits needed for the script I was writing (mainly simple WCS stuff). I still have to figure out how all the special methods etc. work. Anyway, with input from a few people I think a project like this could be very useful. Do you have other people involved at the moment? I had avoided inheriting from HDUList, partly because it means having to understand all the PyFITS internals (which I wasn't really ready for), as well as keeping up with PyFITS maintenance - but it could make life easier in some ways. Several times I found myself contorting to avoid using the private attributes etc. When I mentioned being 'tied to FITS', I wasn't suggesting using a different file format - just noting that the way data extensions are grouped in FITS files may not be exactly what the main program or user wants to deal with. For example, you might want to open the first SCI extension of several FITS files in a flat list, grouped with their VAR & DQ, rather than manipulating whole FITS files (nested lists), each of which corresponds to some subgroup of extensions. It seemed like the way to deal with this was to keep an arbitrary list of objects that each correspond to a SCI HDU, bundled together with its VAR, DQ, WCS & maybe all the other header info. Those could then be kept either in a normal list or a special kind of list that manages the mapping to FITS files (I was wavering back and forth on this, but never did it). Cheers, James. -- James E.H. Turner Gemini Observatory Southern Operations Centre, Casilla 603, Tel. (+56) 51 205609 La Serena, Chile. Fax. (+56) 51 205650 From perry at stsci.edu Fri May 5 11:42:20 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri, 5 May 2006 11:42:20 -0400 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <4459F49A.8000009@obs.univ-lyon1.fr> References: <4459F49A.8000009@obs.univ-lyon1.fr> Message-ID: <8243824b7b80212421934aef9387cc1d@stsci.edu> Sorry for not responding more quickly, I was away for a day... On May 4, 2006, at 8:33 AM, Eric Emsellem wrote: > Hi, > > I am trying since some time to develop a simple module to wrap up some > functionalities of pyfits and matplotlib. > The goal is to be able to manipulate FITS files (spectra, images, 3D > cubes, including maybe Euro3D format for those who know what it is) and > display them (plot slices, display images, etc) in a very simple way. > It > could be inspired from (although not a cloning of) simple > functionalities found in Midas. So you can do things like > > Image1.fits = Image2.fits / (Image3.fits + 3.4) > This kind of syntax would not work in Python without some sort of special interpretation since Python would not find ImageX.fits in the current namespace. That means you have to think about what context you want this kind of capability to work in. As a special, non-Python command language that gets translated into python (pyraf does that for IRAF-like commands)? > Table.fits.col1 = Table.fits.col2 * Table.fits.col4 / 2. > ditto > , or easily create a new F > its image, or table, display them by doing > something like: > > load("Image1.fits", scale=3, cuts=(0.,100.), center=(-1.,6.)) > This is much easier. > or alternatively: > > ima = fitstab("Image1.fits") > load(ima, scale=3, cuts=(0.,100.), center=-1.,6.) > > (where Image1.fits is displayed with a scaling factor of 3, with cuts > as > indicated, and the center of the figure corresponding to coordinates > -1,6) > > or > > plot("Image1.fits", 0) > > (plot a cut at vertical coordinate 0) > As is this. [...] > My questions are then: > > 0/ is this a dum idea? Maybe I am missing something here, so just let > me > know... > I suppose I would say the idea needs a bit more definition as to whether it is enhancement to python libraries or a new kind of shell. > 1/ is there already something existing which does what I mention above? > (so a wrapper around pyfits+matplotlib). Maybe at StSci...? > Not to the level that you describe. Doing the latter two examples would be fairly easy I think. > 2/ If I were to try to develop such a tool, it will result in a rather > badly coded module: I will try my best but frankly I should be > considered as a newby in python and I can sincerely say that I don't > fully understand all the forum matplotlib/numpy/scipy discussions, by > far... So the question is: would anybody be interested/willing to, once > in a while, help me with this development? > Yes, but this is a bad time for us at STScI for the next month or two (we are really spread thinly over several projects; hopefully it won't be so bad in a couple months) So I'd say that it would be hard for us to provide much help or even advice in the very short term. > Some could answer that using directly pyfits+matplotlib does the job. > True. But if you wish to directly deal with fits and display them with > some option (zooming, cuts, centering, etc) you still need to go > through > a number of steps which are not necessarily easy to remember (syntax, > etc). Some wrapping could help quite a lot here. > Certainly. Perry Greenfield From perry at stsci.edu Fri May 5 11:46:52 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri, 5 May 2006 11:46:52 -0400 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <445A8691.8070902@gemini.edu> References: <4459F49A.8000009@obs.univ-lyon1.fr> <445A8691.8070902@gemini.edu> Message-ID: <62f79c97647351b8af73317c5cd031d7@stsci.edu> On May 4, 2006, at 6:56 PM, James Turner wrote: > If you would rather work with FITS files, I'm guessing it's for one of > three (good) reasons: 1) you want to save all your intermediate results > to disk, 2) you need to propagate headers, variance extensions etc., Note that 2) is a common desire, but think about how easy it is in practice. When presented with two headers with conflicting header info, which does one use, the first, or some combination when adding two fits files? This is always much harder to do well in practice. Given the complex arrangements FITS files can have, I have my doubts about this being a really useful reason outside of doing very simple things (in essence, saying that one doesn't care that much about what is in the headers or how different entities are related in fits files) Perry Greenfield From perry at stsci.edu Fri May 5 11:56:00 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri, 5 May 2006 11:56:00 -0400 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <445A8691.8070902@gemini.edu> References: <4459F49A.8000009@obs.univ-lyon1.fr> <445A8691.8070902@gemini.edu> Message-ID: <068638d6cb3f13e44248a7710cb1a185@stsci.edu> > Apologies if I have overlooked something basic, being fairly new to > Python myself - but could you do what you want by loading the original > data with PyFITS and then working in memory with numarray + numdisplay > /matplotlib instead of FITS files on disk? Have you looked at the STScI > tutorial at http://stsdas.stsci.edu/perry/pydatatut.pdf ? Also, I think > ipython and Pyraf are somewhat interchangable (see tutorial), so maybe > you can use the IRAF tasks (imarith?) to do what you want without > losing your nice Python environment? > By the way, Todd Miller has recently added ipython integration to pyraf (such that if one does: pyraf -ipython, it uses ipython as the shell). It isn't in our last stsci_python release but is in our subversion repository for those who can't wait. Perry Greenfield From emsellem at obs.univ-lyon1.fr Fri May 5 12:39:45 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Fri, 05 May 2006 18:39:45 +0200 Subject: [AstroPy] Representing astronomical (esp. IFU) datasets as Python objects In-Reply-To: <445B6CED.1080801@gemini.edu> References: <4459F49A.8000009@obs.univ-lyon1.fr> <445A8AC9.6060008@gemini.edu> <445AFCC4.1030209@obs.univ-lyon1.fr> <445B6CED.1080801@gemini.edu> Message-ID: <445B7FD1.7020404@obs.univ-lyon1.fr> Hi James, well, any input is welcome here. And at the moment I am alone working on this although I have some colleagues here who are willing to maybe help me in the task. So there is hope.... cheers Eric James Turner wrote: > Hi Eric, > > OK - sorry - I wasn't sure that we were talking about exactly the same > thing. It sounds like you're ahead of me on a very similar path. I'd be > happy to send code if it is useful, although I haven't really done a lot > yet, just the bits needed for the script I was writing (mainly simple > WCS stuff). I still have to figure out how all the special methods etc. > work. Anyway, with input from a few people I think a project like this > could be very useful. Do you have other people involved at the moment? > > I had avoided inheriting from HDUList, partly because it means having > to understand all the PyFITS internals (which I wasn't really ready for), > as well as keeping up with PyFITS maintenance - but it could make life > easier in some ways. Several times I found myself contorting to avoid > using the private attributes etc. > > When I mentioned being 'tied to FITS', I wasn't suggesting using a > different file format - just noting that the way data extensions are > grouped in FITS files may not be exactly what the main program or user > wants to deal with. For example, you might want to open the first SCI > extension of several FITS files in a flat list, grouped with their VAR > & DQ, rather than manipulating whole FITS files (nested lists), each of > which corresponds to some subgroup of extensions. It seemed like the > way to deal with this was to keep an arbitrary list of objects that > each correspond to a SCI HDU, bundled together with its VAR, DQ, WCS > & maybe all the other header info. Those could then be kept either in > a normal list or a special kind of list that manages the mapping to > FITS files (I was wavering back and forth on this, but never did it). > > Cheers, > > James. > -- ==================================================================== Eric Emsellem emsellem at obs.univ-lyon1.fr Centre de Recherche Astrophysique de Lyon 9 av. Charles-Andre tel: +33 (0)4 78 86 83 84 69561 Saint-Genis Laval Cedex fax: +33 (0)4 78 86 83 86 France http://www-obs.univ-lyon1.fr/eric.emsellem ==================================================================== From emsellem at obs.univ-lyon1.fr Fri May 5 12:49:27 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Fri, 05 May 2006 18:49:27 +0200 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <8243824b7b80212421934aef9387cc1d@stsci.edu> References: <4459F49A.8000009@obs.univ-lyon1.fr> <8243824b7b80212421934aef9387cc1d@stsci.edu> Message-ID: <445B8217.3040904@obs.univ-lyon1.fr> >> >> Image1.fits = Image2.fits / (Image3.fits + 3.4) >> > This kind of syntax would not work in Python without some sort of > special interpretation since Python would not find ImageX.fits in the > current namespace. That means you have to think about what context you > want this kind of capability to work in. As a special, non-Python > command language that gets translated into python (pyraf does that for > IRAF-like commands)? yes indeed this was a puzzle I wanted to solve (include the potential of a syntax interpretation) but did not know how to address. I will first see if this is really necessary at this stage since it may a major complication (for me). >> > I suppose I would say the idea needs a bit more definition as to whether > it is enhancement to python libraries or a new kind of shell. agreed. I need to first work a little more on this, see what I really need to develop and what I can just inherit from python+pyfits etc . I guess I will have a try at it, send the whole thing with some examples and wait for reactions... > Yes, but this is a bad time for us at STScI for the next month or two > (we are really spread thinly over several projects; hopefully it won't > be so bad in a couple months) > So I'd say that it would be hard for us to provide much help or even > advice in the very short term. well, it is always good to know there are experts around. I have waited 1 year so far to really start things. Waiting another number of months is not a problem (the sooner the better, but this is only a "best effort", so..). On May 4, 2006, at 6:56 PM, James Turner wrote: >> If you would rather work with FITS files, I'm guessing it's for one of >> three (good) reasons: 1) you want to save all your intermediate results >> to disk, 2) you need to propagate headers, variance extensions etc., >Note that 2) is a common desire, but think about how easy it is in practice. >When presented with two headers with conflicting header info, which does one use, >the first, or some combination when adding two fits files? This is always much harder >to do well in practice. Given the complex arrangements FITS files can have, I have my >doubts about this being a really useful reason outside of doing very simple things >(in essence, saying that one doesn't care that much about what is in the headers or >how different entities are related in fits files) agreed, If this is the way to go, we need CLEAR but also USEFUL! rules. I already thought a little about this and there are simple cases we can treat (as when we wish to do spec1 + spec2, where we check the step and start - to see if these are concomitant - and then do the work on the intersection only for example). Again, this has to be worth it. and thanks for the warning about pyraf and ipython. Useful indeed. (but I am not an Iraf fan here, as a practical tool I mean) thanks a lot for the input. Cheers Eric -- ==================================================================== Eric Emsellem emsellem at obs.univ-lyon1.fr Centre de Recherche Astrophysique de Lyon 9 av. Charles-Andre tel: +33 (0)4 78 86 83 84 69561 Saint-Genis Laval Cedex fax: +33 (0)4 78 86 83 86 France http://www-obs.univ-lyon1.fr/eric.emsellem ==================================================================== From jturner at gemini.edu Fri May 5 12:59:56 2006 From: jturner at gemini.edu (James Turner) Date: Fri, 05 May 2006 12:59:56 -0400 Subject: [AstroPy] A simple python module, wrapping pytfits+matplotlib: fits I/O + graphic/display environment In-Reply-To: <62f79c97647351b8af73317c5cd031d7@stsci.edu> References: <4459F49A.8000009@obs.univ-lyon1.fr> <445A8691.8070902@gemini.edu> <62f79c97647351b8af73317c5cd031d7@stsci.edu> Message-ID: <445B848C.2050109@gemini.edu> Hi Perry, >> to disk, 2) you need to propagate headers, variance extensions etc., > > Note that 2) is a common desire, but think about how easy it is in > practice. When presented with two headers with conflicting header info, > which does one use, the first, or some combination when adding two fits > files? This is always much harder to do well in practice. For me, I suppose the key thing is to have enough of the header info available to support methods that do things like WCS transformations or whatever. For that particular case, I think it should be possible to ensure that the values are always correct for the output. More generally, with the Gemini data reduction software, we usually just pass through all the useful miscellaneous keywords from the first header when combining several input files (we also name the combined output file after the first input file number, but with a suffix like "_add"). I suppose that does mean that things like time stamps will not be as meaningful as they should be, but it seems to work well enough in practice. Of course the Gemini reduction process is a relatively special, well-defined case - I'm sure you have to deal with much more general problems (I'm just a humble astronomer, if there is such a thing) - but I still think that even simplistic functionality in this area would be useful in practice. Cheers, James. -- James E.H. Turner Gemini Observatory Southern Operations Centre, Casilla 603, Tel. (+56) 51 205609 La Serena, Chile. Fax. (+56) 51 205650 From a.jaffe at imperial.ac.uk Thu May 11 07:11:46 2006 From: a.jaffe at imperial.ac.uk (Andrew Jaffe) Date: Thu, 11 May 2006 12:11:46 +0100 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: References: Message-ID: <17507.7154.923159.12653@god.ph.ic.ac.uk> Hi All, Thanks very much for providing and supporting the various astropy packages. Two questions: - are there plans to upgrade to numpy from numarray/numeric? - I've got to convert a large number -- potentially millions -- of points between coordinate systems. Is there a recommended and/or easy way to do this? (I think this is two separate questions, really: how should I store millions of positions, and how should I then do the conversion?) Thanks, Andrew -- ______________________________________________________________________ Andrew Jaffe a.jaffe at imperial.ac.uk Astrophysics Group +44 207 594-7526 Blackett Laboratory, Room 1013 FAX 7541 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From rowen at u.washington.edu Thu May 11 12:35:53 2006 From: rowen at u.washington.edu (Russell E Owen) Date: Thu, 11 May 2006 09:35:53 -0700 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: <17507.7154.923159.12653@god.ph.ic.ac.uk> References: <17507.7154.923159.12653@god.ph.ic.ac.uk> Message-ID: At 12:11 PM +0100 5/11/06, Andrew Jaffe wrote: >Hi All, > >Thanks very much for providing and supporting the various astropy >packages. Two >questions: > >- are there plans to upgrade to numpy from numarray/numeric? > >- I've got to convert a large number -- potentially millions -- of > points between coordinate systems. Is there a recommended and/or easy > way to do this? (I think this is two separate questions, really: how > should I store millions of positions, and how should I then do the > conversion?) One way to do coordinate conversions is to use the RO.Astro package RO is pure python (and uses Numeric), so I'm not sure it'll be fast enough, but it is worth trying. slalib (C code) would be another option, but you'll have to either do the whole job in c or write the c/python interface (though at one time there was a swig wrapper for slalib, I have not been able to find it recently). As to storing a million positions...you could use a text file or a database. The latter makes more sense if you want to search the results. -- Russell From laidler at stsci.edu Thu May 11 12:52:25 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Thu, 11 May 2006 12:52:25 -0400 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: <17507.7154.923159.12653@god.ph.ic.ac.uk> References: <17507.7154.923159.12653@god.ph.ic.ac.uk> Message-ID: <44636BC9.3070406@stsci.edu> Hi Andrew, We are just about to release version 0.2 of the AstroLib coords package. http://www.scipy.org/AstroLibCoordsHome It uses Jeff Percival's TPM (Telescope Pointing Machine) software to do the actual coordinate transformations. I've coded up a Python interface for the most common sets (J2000, B1950, galactic, ecliptic), and there's a hook for direct access to TPM if you need something more complex. TPM is in C; the coords package is pure Python, and uses numarray as its array package. It's not optimized for handling large numbers of coordinates, but that was indeed on our list to think about next. So it would be useful to have a real-life use case to help direct that development. cheers, Vicki Laidler Andrew Jaffe wrote: >Hi All, > >Thanks very much for providing and supporting the various astropy packages. Two >questions: > >- are there plans to upgrade to numpy from numarray/numeric? > >- I've got to convert a large number -- potentially millions -- of > points between coordinate systems. Is there a recommended and/or easy > way to do this? (I think this is two separate questions, really: how > should I store millions of positions, and how should I then do the > conversion?) > >Thanks, > >Andrew > > > From perry at stsci.edu Thu May 11 13:05:55 2006 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 11 May 2006 13:05:55 -0400 Subject: [AstroPy] numpy; bulk coordinate conversions References: Message-ID: <66F4524B-D3A8-4EC9-B59B-96013F7CC8B3@stsci.edu> On May 11, 2006, at 7:11 AM, Andrew Jaffe wrote: > Hi All, > > Thanks very much for providing and supporting the various astropy > packages. Two > questions: > > - are there plans to upgrade to numpy from numarray/numeric? > Yes. We've already done that for PyFITS, and now we are working on doing our internal applications (e.g., multidrizzle). In that process we are generating utilities and instructions on how best to transition numarray code. Perry Greenfield From laidler at stsci.edu Thu May 11 13:05:17 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Thu, 11 May 2006 13:05:17 -0400 Subject: [AstroPy] AstroLib coords v0.2 release Message-ID: <44636ECD.8020801@stsci.edu> Hello folks, We'd like to announce the release of v0.2 of the coords package for AstroLib. This package is intended to provide much of the IDL "astron" functionality that pertains to coordinate manipulations in an OO framework. Our target user is a typical astronomer who needs to analyze data, work with catalogs, prepare observing proposals, and prepare for observing runs. With this version, it is possible to * define a Position object - using hmsdms, decimal degrees, or radians - in celestial J2000, celestial B1950, galactic, or ecliptic coordinates * access the coordinates of the Position in any of these coordinate systems * measure the angular separation between two Positions, no matter how defined * determine whether two Positions are within a specified angular separation All coordinate transformations are handled by the TPM (Telescope Pointing Machine) library, which was graciously contributed by Jeff Percival. This package does not yet handle arrays of coordinates, but we plan to add that in a future version. The AstroDate module is also part of this package. It can be used to define a date by specifiying it as a Julian year, Besselian year, Julian date, or Modified Julian Date, and transform between them. More information is available at the project wiki: http://www.scipy.org/AstroLibCoordsHome The software and documentation are available here: http://stsdas.stsci.edu/astrolib/ Please download it and give it a try. Comments and suggestions are welcome on the project wiki. Cheers, Vicki Laidler From mkuemmel at eso.org Thu May 11 16:06:08 2006 From: mkuemmel at eso.org (mkuemmel at eso.org) Date: Thu, 11 May 2006 22:06:08 +0200 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: References: <17507.7154.923159.12653@god.ph.ic.ac.uk> Message-ID: <1147377968.44639930b9e6f@webmail.eso.org> > > > >- I've got to convert a large number -- potentially millions -- of > > points between coordinate systems. Is there a recommended and/or easy > > way to do this? (I think this is two separate questions, really: how > > should I store millions of positions, and how should I then do the > > conversion?) > > As to storing a million positions...you could use a text file or a > database. The latter makes more sense if you want to search the > results. For large scale project involving lots of people dealing with the data, yes, take a data base. If it's just few people using the data, take text files and the AstroAsciiData package. Then you have just everything in python, and it is fast enough once the files are read in. Cheers, Martin From a.jaffe at imperial.ac.uk Thu May 11 16:44:59 2006 From: a.jaffe at imperial.ac.uk (Andrew Jaffe) Date: Thu, 11 May 2006 21:44:59 +0100 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: <1147377968.44639930b9e6f@webmail.eso.org> References: <17507.7154.923159.12653@god.ph.ic.ac.uk> <1147377968.44639930b9e6f@webmail.eso.org> Message-ID: Hi- Actually, I wasn't talking about offline storage, really -- I was really talking nted to know what sort of array/list/etc to use within python. I was already planning on using the routines from coord.position, but it seems that there's likely a lot of overhead in storing many individual position objects, especially when I'll want to do the same conversions on each of them. Andrew On 11 May 2006, at 21:06, mkuemmel at eso.org wrote: >>> >>> - I've got to convert a large number -- potentially millions -- of >>> points between coordinate systems. Is there a recommended and/ >>> or easy >>> way to do this? (I think this is two separate questions, >>> really: how >>> should I store millions of positions, and how should I then do the >>> conversion?) >> >> As to storing a million positions...you could use a text file or a >> database. The latter makes more sense if you want to search the >> results. > > For large scale project involving lots of people dealing with the > data, > yes, take a data base. > If it's just few people using the data, take text files and the > AstroAsciiData > package. Then you have just everything in python, and it is fast > enough > once the files are read in. > > Cheers, > Martin ______________________________________________________________________ Andrew Jaffe a.jaffe at imperial.ac.uk Astrophysics Group +44 207 594-7526 Blackett Laboratory, Room 1013 FAX 7541 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From laidler at stsci.edu Thu May 11 17:01:53 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Thu, 11 May 2006 17:01:53 -0400 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: References: <17507.7154.923159.12653@god.ph.ic.ac.uk> <1147377968.44639930b9e6f@webmail.eso.org> Message-ID: <4463A641.3010002@stsci.edu> Andrew Jaffe wrote: >Hi- > >Actually, I wasn't talking about offline storage, really -- I was >really talking >nted to know what sort of array/list/etc to use within python. I was >already planning on using the routines from coord.position, but it >seems that there's likely a lot of overhead in storing many >individual position objects, especially when I'll want to do the same >conversions on each of them. > > > Hi Andrew, I would give it a try, anyway - I've worked with a catalog of 10**4 objects without difficulty, although I was doing angular separation calculations rather than coordinate transformations. I used a simple list of Positions, if I recall correctly, since I was just looping through them. Or maybe a dictionary, with object id as key and Position as value. Eventually we do want to either: - enhance Position so it can internally hold an array of positions, all with the same equinox, coordinate system, etc; or, - define a new Catalog object, which would be an array of positions, all with the same equinox, coordinate system, etc; and had a similar set of methods as Position that would operate on the whole catalog -- as well as some methods that only make sense on Catalogs. The latter makes more sense to me, but there may be some software maintenance reasons to opt for the former. What else are you going to do with these positions, besides transform them? Maybe a real use case will help settle this question. Vicki From a.jaffe at imperial.ac.uk Fri May 12 05:48:36 2006 From: a.jaffe at imperial.ac.uk (Andrew Jaffe) Date: Fri, 12 May 2006 10:48:36 +0100 Subject: [AstroPy] numpy; bulk coordinate conversions In-Reply-To: <4463A641.3010002@stsci.edu> References: <17507.7154.923159.12653@god.ph.ic.ac.uk> <1147377968.44639930b9e6f@webmail.eso.org> <4463A641.3010002@stsci.edu> Message-ID: <17508.23028.571182.994031@god.ph.ic.ac.uk> Hi Victoria, Victoria G. Laidler writes: > Andrew Jaffe wrote: > > >Hi- > > > >Actually, I wasn't talking about offline storage, really -- I was > >really talking nted to know what sort of array/list/etc to use > >within python. I was already planning on using the routines from > >coord.position, but it seems that there's likely a lot of overhead > >in storing many individual position objects, especially when I'll > >want to do the same conversions on each of them. > > > Hi Andrew, > > I would give it a try, anyway - I've worked with a catalog of 10**4 > objects without difficulty, although I was doing angular separation > calculations rather than coordinate transformations. > > I used a simple list of Positions, if I recall correctly, since I was > just looping through them. Or maybe a dictionary, with object id as > key and Position as value. Well, I think it would be useful to be well-integrated into numpy as it becomes more and more standard. I'm not sure if that argues for an object array or some sort of specific array subclass to avoid duplication of information. > Eventually we do want to either: > > - enhance Position so it can internally hold an array of positions, > all with the same equinox, coordinate system, etc; or, > > - define a new Catalog object, which would be an array of positions, > all with the same equinox, coordinate system, etc; and had a similar > set of methods as Position that would operate on the whole catalog -- > as well as some methods that only make sense on Catalogs. > > The latter makes more sense to me, but there may be some software > maintenance reasons to opt for the former. Both of these sound very much like an internal representation of at least some aspects of the VOTable object. Are there thoughts about doing any of this in a VO context? [I guess one interesting question not quite on the same topic is whether there are plans to make something similar to pyfits for VOTable files...] > What else are you going to do with these positions, besides transform > them? Maybe a real use case will help settle this question. And for my personal work -- CMB -- the 'catalog' isn't really the fundamental object; rather it's a 'map' with some pixelization (usually healpix for large-ish maps). Indeed for me the ideal object would carry along all of healpix's capabilities, including transformations to spherical-harmonic components. Additionally, I would need to be able to get angular separations between points (potentially O(N^2) operations like correlation functions), compare to other maps (and catalogs!), etc. Yours, Andrew -- ______________________________________________________________________ Andrew Jaffe a.jaffe at imperial.ac.uk Astrophysics Group +44 207 594-7526 Blackett Laboratory, Room 1013 FAX 7541 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From William.T.Bridgman.1 at gsfc.nasa.gov Tue May 16 09:49:51 2006 From: William.T.Bridgman.1 at gsfc.nasa.gov (W.T. Bridgman) Date: Tue, 16 May 2006 09:49:51 -0400 Subject: [AstroPy] Handling significant figures in tables? Message-ID: <8A237F88-81E5-4809-A3D7-45EC67A8C036@gsfc.nasa.gov> Is anyone on the list trying to generate tables in LaTeX or HTML using python? I'm interested in generating some tables but need a way to handle significant figures and/or error bounds. Google reveals a SigFig package on ASPN which does not support scientific notation, but I do need that support. Anyone made use of other clever solutions for this issue? Thanks, Tom -- Dr. William T."Tom" Bridgman Scientific Visualization Studio Global Science & Technology, Inc. NASA/Goddard Space Flight Center Email: William.T.Bridgman.1 at gsfc.nasa.gov Code 610.3 Phone: 301-286-1346 Greenbelt, MD 20771 FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ From chanley at stsci.edu Mon May 22 13:01:42 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon, 22 May 2006 13:01:42 -0400 Subject: [AstroPy] PYFITS 1.1 "beta" release Message-ID: <4471EE76.9030906@stsci.edu> ------------------ | PYFITS Release | ------------------ Space Telescope Science Institute is pleased to announce the "beta" release of PyFITS 1.1. This release includes support for both the NUMPY and NUMARRAY array packages. This software can be downloaded at: http://www.stsci.edu/resources/software_hardware/pyfits/Download The NUMPY support in PyFITS is not nearly as well tested as the NUMARRAY support. We expect that you will encounter bugs. Please send bug reports to "help at stsci.edu". We intend to support NUMARRAY and NUMPY simultaneously for a transition period of no less than 1 year. Eventually, however, support for NUMARRAY will disappear. During this period, it is likely that new features will appear only for NUMPY. The support for NUMARRAY will primarily be to fix serious bugs and handle platform updates. ----------- | Version | ----------- Version 1.1b; May 22, 2006 ------------------------------- | Major Changes since v1.1a2 | ------------------------------- * Modified verify method of _Verify class to eliminate printing of warning messages when an exception is raised. * The following problems have been corrected in the NUMPY version of PYFITS as of NUMPY version 0.9.8: - It is now possible to create single field recarrays using string format input. - NUMPY has corrected a problem in which field access in recarrays was broken when the field name matched a class method or attribute. - NUMPY now properly handles byte-ordering when using index arrays. ------------------------- | Software Requirements | ------------------------- PyFITS Version 1.1a2 REQUIRES: * Python 2.3 or later * NUMPY 0.9.8 or NUMARRAY --------------------- | Installing PyFITS | --------------------- PyFITS 1.1b is distributed as a Python distutils module. Installation simply involves unpacking the package and executing % python setup.py install to install it in Python's site-packages directory. Alternatively the command %python setup.py install --local="/destination/directory/" will install PyFITS in an arbitrary directory which should be placed on PYTHONPATH. Once numarray or numpy has been installed, then PyFITS should be available for use under Python. ----------------- | Download Site | ----------------- http://www.stsci.edu/resources/software_hardware/pyfits/Download ---------- | Usage | ---------- Users will issue an "import pyfits" command as in the past. However, the use of the NUMPY or NUMARRAY version of PyFITS will be controlled by an environment variable called NUMERIX. Set NUMERIX to 'numarray' for the NUMARRAY version of PyFITS. Set NUMERIX to 'numpy' for the NUMPY version of pyfits. If only one array package is installed, that package's version of PyFITS will be imported. If both packages are installed the NUMERIX value is used to decide which version to import. If no NUMERIX value is set then the NUMARRAY version of PyFITS will be imported. Anything else will raise an exception upon import. --------------- | Bug Reports | --------------- Please send all PyFITS bug reports to help at stsci.edu ------------------ | Advanced Users | ------------------ Users who would like the "bleeding" edge of PyFITS can retrieve the software from our SUBVERSION repository hosted at: http://astropy.scipy.org/svn/pyfits/trunk We also provide a Trac site at: http://projects.scipy.org/astropy/pyfits/wiki -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338