From perry at stsci.edu Wed Feb 27 15:56:03 2002 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 27 Feb 2002 15:56:03 -0500 Subject: [AstroPy] PyFITS 0.6.2 available Message-ID: We are announcing the availability of PyFITS, a Python module to provide a means of reading and writing FITS format files. The FITS (Flexible Image Transport System) format is standardized data format widely used for astronomical data (http://fits.gsfc.nasa.gov/). This module is based on the PyFITS module initially developed by Paul Barrett while he was at NASA/Goddard but has since been modified and adapted for use by the Space Telescope Science Institute. It has been developed primarily by Paul Barrett, Jin-Chung Hsu, and Todd Miller, with assistance from Warren Hack, Phil Hodge, and Michele De La Pena. It is being released under an Open Source License. A web page for PyFITS is available: http://stsdas.stsci.edu/pyfits There is a preliminary draft for a PyFITS manual available from that page. It is still in the early stages and is incomplete (particularly with regard to the details of how to manipulate table objects). However, there should be enough information to indicate how to perform basic operations with FITS files. PyFITS requires that numarray v0.2 be installed (http://stsdas.stsci.edu/numarray). Numarray is a relatively new replacement for the Numeric module (and largely backward- compatible at the Python level). It also currently lacks all of the 3rd-party libraries that Numeric has (this will begin to change within a couple of months). However, it is not compatible at the C-API level nor are numarray and Numeric arrays interchangeable. It is possible to have both modules loaded simultaneously and to convert between numarray and Numeric arrays using the tostring/fromstring mechanism each provides (at the expense of extra memory usage, of course) The PyFITS module is simply a single Python file (pyfits.py). Installation consists of placing that file in a directory in the Python search path. Besides the use of numarray, there are significant differences with the interface provided by the original PyFITS. We do not expect many future backward-incompatible changes to the interface of PyFITS. (Though more methods and functions will almost certainly be added.) This is an early version and undoubtably bugs will be discovered when used with a greater variety of FITS files. Currently PyFITS take a fairly strict interpretation of FITS files. There are likely to be problems with FITS data that do not strictly conform to the standard. We intend to accommodate such variances, particularly if they involve widely used or available data (so please let us know when such problems occur). Things not yet supported but are part of future development: 1) Verification and/or correction of FITS objects being written to disk so that they are legal FITS. This is being added now and should be available in about a month. Currently, one may construct FITS headers that are inconsistent with the data and write such FITS objects to disk. Future versions will provide options to either a) correct discrepancies and warn, b) correct discrepancies silently, c) throw a Python exception, or d) write illegal FITS (for test purposes!). 2) Support for ascii tables or random groups format. Support for ASCII tables will be done soon (~1 month). When random group support is added is uncertain. 3) Support for memory mapping FITS data (to reduce memory demands). We expect to provide this capability in about 3 months. 4) Support for columns in binary tables having scaled values (e.g. BSCALE or BZERO) or boolean values. Currently booleans are stored as Int8 arrays and users must explicitly convert them into a boolean array. Likewise, scaled columns must be copied with scaling and offset by testing for those attributes explicitly. Future versions will produce such copies automatically. 5) Support for tables with TNULL values. This awaits an enhancement to numarray to support mask arrays (planned). (At least a couple months off) Please contact help at stsci.edu for with questions about it usage, bug reports, or requests for enhancements. Perry Greenfield Science Software Group _____________________________________________________ AstroPy mailing list - astropy at stsci.edu http://lheawww.gsfc.nasa.gov/~bridgman/AstroPy/ From cgp at star.le.ac.uk Thu Feb 28 04:04:15 2002 From: cgp at star.le.ac.uk (Clive Page) Date: Thu, 28 Feb 2002 09:04:15 +0000 (GMT) Subject: [AstroPy] PyFITS 0.6.2 available In-Reply-To: Message-ID: On Wed, 27 Feb 2002, Perry Greenfield wrote: > We are announcing the availability of PyFITS, a Python module to > provide a means of reading and writing FITS format files. Perry Would you (or someone) like to post a comparison between PyFITS and pcFITSIO (see http://www.stecf.org/~npirzkal/)? It's nice to see that Python and FITS are so successful that there are two competing packages of bindings between them, but eventually the astronomical community may find that's one too many. One basic question to which I could not find an answer in your web pages: does PyFITS use Bill Pence's cFITSIO library, or have you implemented all the required functionality independently? -- Clive Page, Dept of Physics & Astronomy, University of Leicester, Tel +44 116 252 3551 Leicester, LE1 7RH, U.K. Fax +44 116 252 3311 _____________________________________________________ AstroPy mailing list - astropy at stsci.edu http://lheawww.gsfc.nasa.gov/~bridgman/AstroPy/ From andrew at physics.uwa.edu.au Thu Feb 28 07:32:06 2002 From: andrew at physics.uwa.edu.au (Andrew Williams) Date: Thu, 28 Feb 2002 20:32:06 +0800 Subject: [AstroPy] PyFITS 0.6.2 available In-Reply-To: Message-ID: <200202281237.g1SCbJe05855@linseed.longtable.org> On Thu, 28 Feb 2002 09:04:15 +0000 (GMT), Clive Page wrote: >pcFITSIO (see http://www.stecf.org/~npirzkal/)? It's nice to see that >Python and FITS are so successful that there are two competing packages of >bindings between them, but eventually the astronomical community may find >that's one too many. Three, actually, mine's not really 'competing' but it fills a small niche... If you want a pure Python interface to read and write simple FITS headers, have a look at for FITS.py - if you have Numeric installed, you can read and write simple (2 dimensional image) data in assorted data formats, otherwise it works fine just as a way of reading and parsing headers (making them available as a pair of dictionary attributes, 'headers' and 'comments'). As far as I know, it's the only FITS interface that can deal with (admittedly simple) FITS headers in a single pure python file, with no extension packages needed. I wrote it because the code I was writing was going to several remote sites, where the people who would be using it had very little python knowledge, and no root access to the machines to install non-standard packages. I added the optional use of Numeric so that I could use it on my own system as well, as part of our automated telescope/camera control system - 'improc.py' on the same page demonstrates the sort of preprocessing we do with it. I'm sure both of the other packages are more flexible about data types and FITS format issues. >One basic question to which I could not find an answer in your web pages: >does PyFITS use Bill Pence's cFITSIO library, or have you implemented all >the required functionality independently? >From the PDF docs, it seems to need only 'numarray'. Andrew -- Dr Andrew Williams andrew at physics.uwa.edu.au Perth Observatory, andrew at ucc.gu.uwa.edu.au Walnut rd, Bickley, Western Australia. andrew at longtable.org _____________________________________________________ AstroPy mailing list - astropy at stsci.edu http://lheawww.gsfc.nasa.gov/~bridgman/AstroPy/ From perry at stsci.edu Thu Feb 28 13:23:34 2002 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 28 Feb 2002 13:23:34 -0500 Subject: [AstroPy] PyFITS 0.6.2 available Message-ID: Clive Page writes: > Would you (or someone) like to post a comparison between PyFITS and > pcFITSIO (see http://www.stecf.org/~npirzkal/)? It's nice to see that > Python and FITS are so successful that there are two competing packages of > bindings between them, but eventually the astronomical community may find > that's one too many. > I'm not sure I can go into a detailed comparison but I will outline many of the reasons we decided to go a different way (albeit more costly in development costs). I haven't used pCFITSIO so I may inadvertently misrepresent some things; I hope Nor will correct me in such cases. Basically CFITSIO is a fairly low-level interface while we desired a more object-oriented interface to FITS files. It would probably be possible to construct a low-level interface to CFITSIO (much like pCFITSIO) with a more object-oriented wrapper library. But to the extent the low-level library doesn't map well into the expected behavior, that can become difficult. It depends on lots of details. We wanted an interface that made the components of a FITS file have corresponding objects in Python, and that accessing such objects would not entail a huge drain on resources (either in time, I/O, or memory usage). For example, it would be very nice to give headers much of the capabilities that Python dictionaries have. Yet would we want every dictionary access to result in I/O to the file containing the header? CFITIO is intrinsically file-oriented. Changes to the elements of a FITS file may require an immediate rewriting of the entire file, something we wanted to avoid. We wanted the interface to FITS objects to have more memory-like (and hence more dynamic) behavior. To give some specific examples: 1) If I want to add keywords to the header that cause it to need more blocks that will cause (as I understand it) CFITIO to rewrite the file to make space for it. If the header is in memory, that step can be deferred until all such changes are made (vs. many rewrites of the file). If we're forced to keep a copy of the header in memory and provide means for updating it, we've already begun a significant replacement of CFITSIO functionality. Besides, mapping all header changes to file I/O can be quite inefficient. 2) If I want to insert HDU's into a FITS file, the wrapper library must manage such HDU's as file objects rather than memory objects. This requires that I/O be done for all such manipulations, even when plenty of memory may be available for the inserted HDU's. It often would be better to have in-memory HDU's and defer the I/O overhead to the time when all the insertions and changes are completed. 3) It is not possible to memory map FITS files in CFITSIO. This is a capability that we feel is important for minimizing memory usage within Python. There really is no way of adding this to a wrapper library without replacing CFITSIO. 4) CFITSIO does limited verification checking of the FITS files written. We wish to see better mechanisms for preventing one from writing inconsistent or illegal FITS files. 5) CFITSIO table access is relatively slow because of how table data are buffered. We desired a faster means of creating arrays from table columns. It is the case that this version of PyFITS does not currently supply all the functionality of CFITIO (e.g., ascii tables, random groups, compression options, and extended filename syntax.) It is also true that the current version provides no means of reading subsets of large data sets (if you want the data for an HDU, it must all be read into memory). This will change when memory mapping support is added (and if there are platforms that do not support memory mapping sufficiently well, or not at all, we will add means for reading subsets of the data). But all in all, I think people will find the PyFITS approach a much easier-to-use and more flexible approach to dealing with FITS data when working within Python. We are using at STScI for developing new applications and calibration pipelines so we have a strong commitment to long-term support. We have also developed a new array module (numarray) to provide sufficient capabilities for efficient and transparent access to FITS data that was not present in the existing array module (Numeric). In the short run this is a drawback since numarray does not yet have many of the associated libraries to make it as useful as Numeric currently is, but that will begin to change within a couple of months. (In fact, much of the delay in making PyFITS public has been driven by how best to provide data access. We concluded that we needed to get a workable version of numarray to build on rather work with the restrictions that Numeric would have required.) To summarize in more user-oriented terms the difference between the two approaches: with CFITSIO you must manage many of the details yourself. Many things are cumbersome. Try inserting a keyword just just after another keyword in the header. Try inserting an HDU into the middle of an existing FITS file. Generally you will have to make many calls to do either of these. Both are relatively easy in PyFITS. In short, CFITSIO (or pCFITSIO) will require you to do a lot more bookkeeping. Mind you, I'm not trying to put down CFITSIO. It is very solid, well supported, has many features, and is well documented. If you are writing programs in C, it's clearly the library to use. We just didn't think it mapped well to the Python/array-oriented paradigm > One basic question to which I could not find an answer in your web pages: > does PyFITS use Bill Pence's cFITSIO library, or have you implemented all > the required functionality independently? > No, it is not layered on CFITSIO, and yes all the functionality present (not all "required" functionality is there as mentioned in the announcement, e.g., ascii tables or random groups) is implemented independently. By the way, if many feel it is useful to provide access to FITS headers without having numarray available (as mentioned in the subsequent email by Andrew Williams), I don't believe that would be very hard for us to implement (but access to data would certainly require numarray). PyFITS itself is pure Python. Perry Greenfield _____________________________________________________ AstroPy mailing list - astropy at stsci.edu http://lheawww.gsfc.nasa.gov/~bridgman/AstroPy/ From jmiller at stsci.edu Thu Feb 28 19:07:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Thu, 28 Feb 2002 19:07:04 -0500 Subject: [AstroPy] Numarray problems with GCC and SPARC Message-ID: <3C7EC628.8080406@stsci.edu> Hi, Two people have reported problems with compiling Numarray-0.11 or 0.2 with GCC on SPARC. There are two problems: 1. Compiling the _ufuncmodule.c using gcc-2.95 on a SPARC (with the default switches) uses tons of virtual memory and typically fails. a. This can be avoided by adding the compilation flags: EXTRA_COMPILE_ARGS=["-O0", "-Wno-return-type"] to your setup.py when compiling * *numarray**. b. Alternately, you can wait for numarray-0.3 which will partition the ufuncmodule into smaller compilation units. We suspect these will avoid the problem naturally and permit the use of optimization. c. Lastly, if you have Sun cc, you might want to try using it instead of gcc. This is what we do at STSCI. You need to recompile Python itself if you want to do this and your python was already compiled with gcc. 2. Python compiled with gcc generates misaligned storage within buffer objects. Numarray-0.2 is dependent on the problematic variant of the buffer object so if you want to use Float64 or Complex128 on a SPARC you may experience core dumps. a. I have a non-portable patch which worked for me with gcc-2.95 on SPARC. I can e-mail this to anyone interested. Apply this patch and recompile **python **. b. You might be able to fix this with gcc compilation switches for Python: try -munaligned-doubles and recompile * *python**. c. Numarray-0.3 will address this issue by providing its own minimal memory object which features correctly aligned storage. This solution will not require recompiling python, but won't be available until numarray-0.3. d. Python compiled with Sun cc using the default switches doesn't manifest this bug. If you have Sun cc, you may want to recompile * *python** using that. In general, I think the "better part of valor" is probably to wait 3 weeks for numarray-0.3 when both issues should be addressed. If you want to try numarray-0.2 now with GCC on a SPARC, I hope some of these ideas work for you. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSG (410) 338 4576 _____________________________________________________ AstroPy mailing list - astropy at stsci.edu http://lheawww.gsfc.nasa.gov/~bridgman/AstroPy/