From thomas.robitaille at gmail.com Mon Oct 2 11:55:11 2017 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 2 Oct 2017 16:55:11 +0100 Subject: [AstroPy] ANN: astropy-healpix v0.1, a new BSD-licensed HEALPix Python package Message-ID: Hi everyone, We are happy to announce the first release of a new package: astropy-healpix - A BSD-licensed Astropy-compatible HEALPix implementation * Docs: http://astropy-healpix.readthedocs.io * Code: https://github.com/astropy/astropy-healpix * PyPI: https://pypi.python.org/pypi/astropy-healpix This new package uses C HEALPix code originally developed by Dustin Lang as part of astrometry.net (developed based on the HEALPix papers), and provides a high-level Pythonic API using Astropy objects (including e.g. astropy.coordinates objects). At this time the package should still be considered experimental and we may still change the API in future depending on feedback on this first release. The main motivations for developing this package (which is independent from the GPL-licensed HEALPix and healpy packages) are to provide a BSD-licensed HEALPix implementation that can be used in packages in the Astropy ecosystem such as the BSD-licensed reproject, hips, and regions packages without worrying about potential licensing issues, and to have a lightweight and portable implementation that works on all major platforms (Linux, MacOS X, and Windows). To try it out, you can easily install it with pip: pip install astropy-healpix Or with conda: conda install -c conda-forge astropy-healpix and take a look at the documentation to get started on using it: http://astropy-healpix.readthedocs.io Please do report any issues, feature requests, or suggestions for improvements in the issue tracker for the repository: https://github.com/astropy/astropy-healpix/issues We would also be really interested in hearing from you if you would like to help out with the development/maintenance of the package - the Python part of the package is very approachable and there are a number of low-hanging issues to solve. Cheers, Tom and Christoph From shbhuk at gmail.com Fri Oct 6 15:31:25 2017 From: shbhuk at gmail.com (Shubham Kanodia) Date: Fri, 6 Oct 2017 15:31:25 -0400 Subject: [AstroPy] Earth Orientation Parameters Message-ID: Hi, I am looking for a way to query the IERS bulletin and obtain the Earth Orientation Parameters in Python. The analogue of the IDL hprstatn.pro procedure. Just checking if this is implemented in some form in Astropy. I know that I can query the IERS bulletin and obtain the table, but is the final product already done? My end goal is to obtain the x,y,z,vx,vy,vz of an observatory wrt the barycenter taking into consideration these movements. Has anyone worked on this before, or has any tips? *Shubham Kanodia* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcbeth at broggs.org Fri Oct 6 15:51:23 2017 From: mcbeth at broggs.org (Jeffrey Brent McBeth) Date: Fri, 6 Oct 2017 15:51:23 -0400 Subject: [AstroPy] Earth Orientation Parameters In-Reply-To: References: Message-ID: <20171006195123.frcunvh7bmjg5six@broggs.org> On Fri, Oct 06, 2017 at 03:31:25PM -0400, Shubham Kanodia wrote: > Hi, > I am looking for a way to query the IERS bulletin and obtain the Earth > Orientation Parameters in Python. The analogue of the > IDL?[1]hprstatn.pro?procedure. > Just checking if this is implemented in some form in Astropy. I know that > I can query the IERS bulletin and obtain the table, but is the final > product already done? > My end goal is to obtain the x,y,z,vx,vy,vz of an observatory wrt the > barycenter taking into consideration these movements. > Has anyone worked on this before, or has any tips? > Shubham Kanodia I have a pending pull request (since Feb) with Astropy to extract the last bits of the table necessary to do the calculation (dX,dY). I am then computing the rest using the astropy/erfa bindings rather than through the official interface :( Jeffrey McBeth -- "The man who does not read good books has no advantage over the man who cannot read them." -- not Mark Twain, maybe a southen librarian in 1910 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: not available URL: From Stefano.DellaTorre at mib.infn.it Mon Oct 9 12:09:28 2017 From: Stefano.DellaTorre at mib.infn.it (Stefano Della Torre) Date: Mon, 9 Oct 2017 18:09:28 +0200 Subject: [AstroPy] conversion from ECI o HCRS Message-ID: <21edd73b-44a5-3de2-fae3-62e9f34417fa@mib.infn.it> Hello, I'm looking a way to transform a spacecraft position expressed in ECI in a heliocentric reference system. I tried the following code that should convert Earth center in heliocentri coordinates: c = SkyCoord(x=0,y=0,z=0,unit='km', representation='cartesian') d=c.transform_to('hcrs') In [105]: d.distance.astronomical_unit Out[105]: 0.0076673051912451405 The so obtained distance in Astronomical unit is far from the expected value 1 AU. I'm doing something wrong in the conversion? Best regards Stefano From d.f.evans at keele.ac.uk Mon Oct 9 13:36:02 2017 From: d.f.evans at keele.ac.uk (Daniel Evans) Date: Mon, 9 Oct 2017 18:36:02 +0100 Subject: [AstroPy] conversion from ECI o HCRS In-Reply-To: <21edd73b-44a5-3de2-fae3-62e9f34417fa@mib.infn.it> References: <21edd73b-44a5-3de2-fae3-62e9f34417fa@mib.infn.it> Message-ID: > I tried the following code that should convert Earth center in heliocentric coordinates: Without defining a frame of reference, SkyCoord() defaults to ICRS, which is centred on the barycentre. The distance of 1,147,012km is about the difference I'd expect from Barycentric to Heliocentric coordinates. Assuming the the Geocentric Celestial Reference System (GCRS) is the frame your satellite positions are specified in, you would specify it as follows: >>> c = SkyCoord(x=0,y=0,z=0,unit='km', representation='cartesian', frame='gcrs') >>> c.transform_to("hcrs").distance.astronomical_unit 0.9833276664706555 Regards, Daniel On 9 October 2017 at 17:09, Stefano Della Torre < Stefano.DellaTorre at mib.infn.it> wrote: > Hello, I'm looking a way to transform a spacecraft position expressed in > ECI in a heliocentric reference system. > > I tried the following code that should convert Earth center in heliocentri > coordinates: > > > c = SkyCoord(x=0,y=0,z=0,unit='km', representation='cartesian') > ( 0., 0., 0.)> > > d=c.transform_to('hcrs') > ( 20.35057636, 6.91379769, 1147012.53061733)> > > In [105]: d.distance.astronomical_unit > Out[105]: 0.0076673051912451405 > > The so obtained distance in Astronomical unit is far from the expected > value 1 AU. > I'm doing something wrong in the conversion? > > Best regards > Stefano > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcraig at mnstate.edu Mon Oct 16 15:46:16 2017 From: mcraig at mnstate.edu (Matthew Craig) Date: Mon, 16 Oct 2017 19:46:16 +0000 Subject: [AstroPy] Python in Astronomy 2018 Applications open Message-ID: <922781D8-43C6-44BC-BC3B-43DCFB6B714C@mnstate.edu> Hi, It is my pleasure to announce, on behalf of the Scientific Organizing Committee, that applications are now open for Python in Astronomy 2018, to be held 30 Apr - 4 May 2018 at the Center for Computational Astrophysics at the Flatiron Institute in New York, NY. Though the application form will be open until 23:59UTC on Dec 9, 2017, I encourage you to complete the form soon to make sure you don?t miss the deadline. The application form is at: https://goo.gl/forms/AGuoU8mUFv3WJ8Qn2 More information about the conference, and links to past years, is available at: Finally, a brief excerpt from the description of the conference: In addition to sharing information about state-of-the art Python Astronomy packages, the workshop will focus on improving interoperability between astronomical Python packages, mentoring open-source contributors, and developing educational materials for Python in Astronomy. The meeting is therefore not only aimed at current developers and users, but also educators or others who are interested in being involved in these efforts. Participant selection will be made with the goal of enhancing the Python in Astronomy community and we particularly encourage requests to attend from junior astronomers and people who are new to contributing to open source software. Effort will also be made to select participants who have a demonstrated potential to contribute meaningfully to the Python in Astronomy infrastructure via providing educational materials, documentation, and/or coding. This conference is neither intended to be an introduction to Python nor only for expert-level Python developers. Thanks, Matt Craig, on behalf of the SOC: Azalee Bostroem Daniela Huppenkothen Drew Leonard Duncan Macleod Brigitta Sipocz Erik Tollerud ?? Professor Department of Physics and Astronomy Minnesota State University Moorhead 1104 7th Ave S, Moorhead MN 56563 office: Hagen 307F phone: (218) 477-2439 fax: (218) 477-2290 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcraig at mnstate.edu Mon Oct 16 16:02:09 2017 From: mcraig at mnstate.edu (Matthew Craig) Date: Mon, 16 Oct 2017 20:02:09 +0000 Subject: [AstroPy] Python in Astronomy 2018 Applications open In-Reply-To: <922781D8-43C6-44BC-BC3B-43DCFB6B714C@mnstate.edu> References: <922781D8-43C6-44BC-BC3B-43DCFB6B714C@mnstate.edu> Message-ID: <0B763EE0-6EFD-4CBA-9AE7-67542A24FB64@mnstate.edu> Hi, Sorry, in my excitement to send out the announcement I missed the link to the pyastro web site: http://openastronomy.org/pyastro/2018/ Matt Craig ?? Professor Department of Physics and Astronomy Minnesota State University Moorhead 1104 7th Ave S, Moorhead MN 56563 office: Hagen 307F phone: (218) 477-2439 fax: (218) 477-2290 On Oct 16, 2017, at 2:46 PM, Matthew Craig > wrote: Hi, It is my pleasure to announce, on behalf of the Scientific Organizing Committee, that applications are now open for Python in Astronomy 2018, to be held 30 Apr - 4 May 2018 at the Center for Computational Astrophysics at the Flatiron Institute in New York, NY. Though the application form will be open until 23:59UTC on Dec 9, 2017, I encourage you to complete the form soon to make sure you don?t miss the deadline. The application form is at: https://goo.gl/forms/AGuoU8mUFv3WJ8Qn2 More information about the conference, and links to past years, is available at: Finally, a brief excerpt from the description of the conference: In addition to sharing information about state-of-the art Python Astronomy packages, the workshop will focus on improving interoperability between astronomical Python packages, mentoring open-source contributors, and developing educational materials for Python in Astronomy. The meeting is therefore not only aimed at current developers and users, but also educators or others who are interested in being involved in these efforts. Participant selection will be made with the goal of enhancing the Python in Astronomy community and we particularly encourage requests to attend from junior astronomers and people who are new to contributing to open source software. Effort will also be made to select participants who have a demonstrated potential to contribute meaningfully to the Python in Astronomy infrastructure via providing educational materials, documentation, and/or coding. This conference is neither intended to be an introduction to Python nor only for expert-level Python developers. Thanks, Matt Craig, on behalf of the SOC: Azalee Bostroem Daniela Huppenkothen Drew Leonard Duncan Macleod Brigitta Sipocz Erik Tollerud ?? Professor Department of Physics and Astronomy Minnesota State University Moorhead 1104 7th Ave S, Moorhead MN 56563 office: Hagen 307F phone: (218) 477-2439 fax: (218) 477-2290 _______________________________________________ AstroPy mailing list AstroPy at python.org https://mail.python.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From deil.christoph at googlemail.com Tue Oct 17 02:16:29 2017 From: deil.christoph at googlemail.com (Christoph Deil) Date: Tue, 17 Oct 2017 08:16:29 +0200 Subject: [AstroPy] ANN: pyregion 2.0 released Message-ID: Hi everyone, We are happy to announce the release of pyregion version 2.0. pyregion is a Python package to work with DS9 region files. * Docs: http://pyregion.readthedocs.io * Code: https://github.com/astropy/pyregion * PyPI: https://pypi.python.org/pypi/pyregion Detailed information about changes in this release is here: http://pyregion.readthedocs.io/en/latest/changelog.html#oct-14-2017 To try it out, you can easily install it with pip: pip install pyregion --upgrade Or with conda: conda install -c conda-forge pyregion Please do report any issues you find with pyregion at: https://github.com/astropy/pyregion/issues Cheers, Jae-Joon Lee, Joseph Booker, Adam Ginsburg, Brigitta Sipocz, Tom Robitaille and Christoph Deil From matteo at matteobachetti.it Wed Oct 18 06:26:29 2017 From: matteo at matteobachetti.it (Matteo) Date: Wed, 18 Oct 2017 10:26:29 +0000 Subject: [AstroPy] ANN: HENDRICS 3.0 beta 1 released In-Reply-To: References: Message-ID: Dear all, I would like to announce the first beta release of HENDRICS, High ENergy Data Reduction Interface from the Command Shell. This provisionally-accepted Astropy affiliated package is a set of scripts, based on the spectral timing library Stingray, to analyze the fast variability of X-ray data from multiple missions. It is an evolution of MaLTPyNT, an in-development affiliated software package originally designed for the analysis of NuSTAR data. Currently supported missions include also RXTE and XMM (but most of the analysis can be done on any compatible event files from other missions). This beta release is the first with the new API based on Stingray. The code is currently in feature freeze, and a stable version will be made available after a testing period of approximately one month (unless major bugs appear). Code: https://github.com/StingraySoftware/HENDRICS Docs: http://hendrics.readthedocs.io Pypi: https://pypi.python.org/pypi/hendrics Cheers, Matteo Bachetti on behalf of the StingraySoftware collaboration ( github.com/stingraysoftware/) -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.f.evans at keele.ac.uk Wed Oct 18 11:55:57 2017 From: d.f.evans at keele.ac.uk (Daniel Evans) Date: Wed, 18 Oct 2017 16:55:57 +0100 Subject: [AstroPy] Fitting in 3 dimensions Message-ID: Dear all, I have a cube of images derived from an integral field spectrograph, with axes (x, y, wavelen), with a size of something like (32, 32, 39). In these images is a star which I would like to use astropy.modelling to fit an AiryDisk2D to, doing so simultaneously in my N wavelength slices. I'd like to do it simultaneously in order to better constrain the PSF position and size using data from all wavelength slices simultaneously - I don't have any obvious "correct" answer to fix the inputs to. I envision doing this by fitting N Airy disk models, with x_0 and y_0 fixed between them, the radius dependent on the wavelength in slice N, and the amplitude being free to vary. A semi-psudocode version is of what I imagine the custom model's evaluate() function to be is: #### def evaluate(x, y, wavelength, x_cen, y_cen, radius_0, amplitude_0...): z = numpy.empty_like(x) global wavelengthValues for N in range(len(wavelengthValues)): radius = radius_0 * wavelengthValues[N] / wavelengthValues[0] amplitude = amplitude_N wavelengthMask = wavelength == N z[wavelengthMask] = models.AiryDisk2D.evaluate( x[wavelengthMask], y[wavelengthMask], amplitude, x_cen, y_cen, radius) return z #### 1. This requires fitting in 3D, which I can't see any simple way to do in astropy - everything seems to support x,y as input and z as output. Have I missed something, or would I have to really go back to basics? 2. My suggested model is also relatively clunky due to having to define N amplitude parameters. I glossed over this in the code above, but in reality the only way I've thought of to do it is to define N input parameters, make a list from those N input parameters, and then select the correct one from the list. That's a lot of typing! Is there a more elegant way to do this? 3. Alternatively, can I attack this problem in 2D? Could I easily tile my N wavelength slices together into a single image, and then called a series of AiryDisk2D() models that generate data only for the relevant (x, y) of each tile? Regards, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From bardeau at iram.fr Mon Oct 23 09:05:35 2017 From: bardeau at iram.fr (=?UTF-8?Q?S=c3=a9bastien_Bardeau?=) Date: Mon, 23 Oct 2017 15:05:35 +0200 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords Message-ID: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Dear astropy community, I am facing a FITS image which contains the following keywords in its primary header: NAXIS?? =??????????????????? 4???????? / NAXIS1? =????????????????? 256???????? / NAXIS2? =????????????????? 256???????? / NAXIS3? =??????????????????? 1???????? / NAXIS4? =??????????????????? 1???????? / As you can see it provides a 2D image, but there are 2 trailing degenerate dimensions. Tools like ds9 have no problem understanding this is a 2D image, but another user reports that matplotlib returns an error. As an intermediate solution, I wanted to patch the header by setting NAXIS to 2 and removing NAXIS3 and NAXIS4. You will find the script below. Unfortunately, none of the modifications I will do on the NAXIS* keywords will be saved in the result file, even if I keep these keywords perfectly consistent with the actual binary image. On the other hand, modifying, deleting, or adding other keywords works correctly. I remember having the same problem with pyfits in the past. I can also add that I can perfectly imaging doing the opposite transformation, i.e. adding extra degenerate dimensions to transform an image into a cube or more. Is this a bug or a feature? I believe this is a kind of protection against inconsistent modifications of the header, but can't we imagine to relax a bit this protection, by e.g. checking "only" that the total number of elements are preserved? I can provide a test file if needed. Thanks, Sebastien #!/bin/env python import astropy.io.fits import sys a = astropy.io.fits.open(sys.argv[1],mode='update') print "Before: ",a[0].header['NAXIS'] a[0].header['NAXIS'] = 2? # Won't be saved print "After:? ",a[0].header['NAXIS'] del a[0].header['NAXIS3']? # Won't be deleted del a[0].header['NAXIS4']? # Won't be deleted a[0].header['OBJECT'] = 'ANOTHERONE'? # Will be changed del a[0].header['ORIGIN']? # Will be deleted a[0].header['MYKEYWOR'] = 'Some string'? # Will be added a.close()? # Save changes del a # Check a = astropy.io.fits.open(sys.argv[1],mode='readonly') print "Current:",a[0].header['NAXIS']? # Shows unmodified value a.close() del a -- Sebastien BARDEAU IRAM 300 rue de la Piscine F - 38406 Saint Martin d'Heres Tel: (+33) 4 76 82 49 86 www.iram.fr/IRAMFR/GILDAS/ From jzuhone at gmail.com Mon Oct 23 09:47:41 2017 From: jzuhone at gmail.com (John ZuHone) Date: Mon, 23 Oct 2017 09:47:41 -0400 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: <6E50868A-0193-4CF1-B803-36FF4F93F3EC@gmail.com> I?m a bit confused about the statement that ?matplotlib returns an error?. Unless you are using APLpy or wcsaxes, Matplotlib doesn?t know anything about FITS file headers. Could you elaborate? Sent from my iPhone > On Oct 23, 2017, at 9:05 AM, S?bastien Bardeau wrote: > > Dear astropy community, > > I am facing a FITS image which contains the following keywords in its > primary header: > > NAXIS = 4 / > NAXIS1 = 256 / > NAXIS2 = 256 / > NAXIS3 = 1 / > NAXIS4 = 1 / > > As you can see it provides a 2D image, but there are 2 trailing > degenerate dimensions. Tools like ds9 have no problem understanding this > is a 2D image, but another user reports that matplotlib returns an > error. As an intermediate solution, I wanted to patch the header by > setting NAXIS to 2 and removing NAXIS3 and NAXIS4. You will find the > script below. Unfortunately, none of the modifications I will do on the > NAXIS* keywords will be saved in the result file, even if I keep these > keywords perfectly consistent with the actual binary image. On the other > hand, modifying, deleting, or adding other keywords works correctly. > > I remember having the same problem with pyfits in the past. I can also > add that I can perfectly imaging doing the opposite transformation, i.e. > adding extra degenerate dimensions to transform an image into a cube or > more. > > Is this a bug or a feature? I believe this is a kind of protection > against inconsistent modifications of the header, but can't we imagine > to relax a bit this protection, by e.g. checking "only" that the total > number of elements are preserved? I can provide a test file if needed. > > Thanks, > > Sebastien > > > #!/bin/env python > > import astropy.io.fits > import sys > > a = astropy.io.fits.open(sys.argv[1],mode='update') > > print "Before: ",a[0].header['NAXIS'] > a[0].header['NAXIS'] = 2 # Won't be saved > print "After: ",a[0].header['NAXIS'] > del a[0].header['NAXIS3'] # Won't be deleted > del a[0].header['NAXIS4'] # Won't be deleted > > a[0].header['OBJECT'] = 'ANOTHERONE' # Will be changed > del a[0].header['ORIGIN'] # Will be deleted > a[0].header['MYKEYWOR'] = 'Some string' # Will be added > > a.close() # Save changes > del a > > # Check > a = astropy.io.fits.open(sys.argv[1],mode='readonly') > print "Current:",a[0].header['NAXIS'] # Shows unmodified value > a.close() > del a > > > > -- > Sebastien BARDEAU > IRAM > 300 rue de la Piscine > F - 38406 Saint Martin d'Heres > Tel: (+33) 4 76 82 49 86 > www.iram.fr/IRAMFR/GILDAS/ > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy From d.f.evans at keele.ac.uk Mon Oct 23 09:49:14 2017 From: d.f.evans at keele.ac.uk (Daniel Evans) Date: Mon, 23 Oct 2017 14:49:14 +0100 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: Dear Sebastien, I suspect you're right that astropy is undoing your modifications as it thinks the header no longer matches the data. The solutions I can think of for your problem are: 1. Simply modify the array once read into Python to turn it into a 2D array, suitable for matplotlib.pyplot.imshow (or similar). This is relatively standard for viewing images in datacubes anyway, and requires very little effort in Python: a = astropy.io.fits.open(sys.argv[1]) image4D = a[0].data a.close() image2D = image4D[0, 0] matplotlib.pyplot.imshow(image2D) matplotlib.pyplot.show() 2. If you do want to modify the fits file, you can convert the array from 4D to 2D, and reinsert this into the file. astropy will update the header to match: a = astropy.io.fits.open(sys.argv[1], mode = "update") a[0].data = a[0].data[0, 0] a.close() John ZuHone - the issue is presumably that matplotlib is being asked to display a 2D image, but is being passed a 4D array - I've done this many times with 3D datacubes! Regards, Daniel On 23 October 2017 at 14:05, S?bastien Bardeau wrote: > Dear astropy community, > > I am facing a FITS image which contains the following keywords in its > primary header: > > NAXIS = 4 / > NAXIS1 = 256 / > NAXIS2 = 256 / > NAXIS3 = 1 / > NAXIS4 = 1 / > > As you can see it provides a 2D image, but there are 2 trailing > degenerate dimensions. Tools like ds9 have no problem understanding this > is a 2D image, but another user reports that matplotlib returns an > error. As an intermediate solution, I wanted to patch the header by > setting NAXIS to 2 and removing NAXIS3 and NAXIS4. You will find the > script below. Unfortunately, none of the modifications I will do on the > NAXIS* keywords will be saved in the result file, even if I keep these > keywords perfectly consistent with the actual binary image. On the other > hand, modifying, deleting, or adding other keywords works correctly. > > I remember having the same problem with pyfits in the past. I can also > add that I can perfectly imaging doing the opposite transformation, i.e. > adding extra degenerate dimensions to transform an image into a cube or > more. > > Is this a bug or a feature? I believe this is a kind of protection > against inconsistent modifications of the header, but can't we imagine > to relax a bit this protection, by e.g. checking "only" that the total > number of elements are preserved? I can provide a test file if needed. > > Thanks, > > Sebastien > > > #!/bin/env python > > import astropy.io.fits > import sys > > a = astropy.io.fits.open(sys.argv[1],mode='update') > > print "Before: ",a[0].header['NAXIS'] > a[0].header['NAXIS'] = 2 # Won't be saved > print "After: ",a[0].header['NAXIS'] > del a[0].header['NAXIS3'] # Won't be deleted > del a[0].header['NAXIS4'] # Won't be deleted > > a[0].header['OBJECT'] = 'ANOTHERONE' # Will be changed > del a[0].header['ORIGIN'] # Will be deleted > a[0].header['MYKEYWOR'] = 'Some string' # Will be added > > a.close() # Save changes > del a > > # Check > a = astropy.io.fits.open(sys.argv[1],mode='readonly') > print "Current:",a[0].header['NAXIS'] # Shows unmodified value > a.close() > del a > > > > -- > Sebastien BARDEAU > IRAM > 300 rue de la Piscine > F - 38406 Saint Martin d'Heres > Tel: (+33) 4 76 82 49 86 > www.iram.fr/IRAMFR/GILDAS/ > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bardeau at iram.fr Mon Oct 23 10:02:13 2017 From: bardeau at iram.fr (=?UTF-8?Q?S=c3=a9bastien_Bardeau?=) Date: Mon, 23 Oct 2017 16:02:13 +0200 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: Dear Daniel, On 10/23/2017 03:49 PM, Daniel Evans wrote: > Dear Sebastien, > > I suspect you're right that astropy is undoing your modifications as > it thinks the header no longer matches the data. The solutions I can > think of for your problem are: > > 1. Simply modify the array once read into Python to turn it into a 2D > array, suitable for matplotlib.pyplot.imshow (or similar). This is > relatively standard for viewing images in datacubes anyway, and > requires very little effort in Python: > > ??? a = astropy.io.fits.open(sys.argv[1]) > ??? image4D = a[0].data > ??? a.close() > > ??? image2D = image4D[0, 0] > ??? matplotlib.pyplot.imshow(image2D) > ??? matplotlib.pyplot.show() > > 2. If you do want to modify the fits file, you can convert the array > from 4D to 2D, and reinsert this into the file. astropy will update > the header to match: > > ??? a = astropy.io.fits.open(sys.argv[1], mode = "update") > ??? a[0].data = a[0].data[0, 0] > ??? a.close() I was focusing on editing the header, but actually "editing" the data works great! Thanks. Do you know what would be the best syntax to add a trailing dimension from a 2D image? I mean, in a clever way without doubling the memory consumption. > > John ZuHone - the issue is presumably that matplotlib is being asked > to display a 2D image, but is being passed a 4D array - I've done this > many times with 3D datacubes! @ John ZuHone: I am not a Matplotlib user (I am in charge of reading/writing the FITS files), and I had just a comment about a Matplotlib error without more details. But Daniel is probably right here. Thanks, Sebastien -------------- next part -------------- An HTML attachment was scrubbed... URL: From bushouse at stsci.edu Mon Oct 23 09:52:29 2017 From: bushouse at stsci.edu (Howard Bushouse) Date: Mon, 23 Oct 2017 13:52:29 +0000 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: <6E50868A-0193-4CF1-B803-36FF4F93F3EC@gmail.com> References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> <6E50868A-0193-4CF1-B803-36FF4F93F3EC@gmail.com> Message-ID: <5CC82CCB-DA1D-448F-8629-0B7569FE6849@stsci.edu> Regarding the updates to the NAXISn keywords, it's highly likely that NAXIS and NAXISn are determined automatically within the fits library by the shape and size of the data array and hence you can't change the keyword values without actually changing the shape of the data array itself. If the data array is currently 4-D, slice off the first 2 axes to convert it to 2-D and then resave the file. The NAXISn keywords should update automatically. -Howard > On Oct 23, 2017, at 9:47 AM, John ZuHone wrote: > > I?m a bit confused about the statement that ?matplotlib returns an error?. Unless you are using APLpy or wcsaxes, Matplotlib doesn?t know anything about FITS file headers. Could you elaborate? > > Sent from my iPhone > >> On Oct 23, 2017, at 9:05 AM, S?bastien Bardeau wrote: >> >> Dear astropy community, >> >> I am facing a FITS image which contains the following keywords in its >> primary header: >> >> NAXIS = 4 / >> NAXIS1 = 256 / >> NAXIS2 = 256 / >> NAXIS3 = 1 / >> NAXIS4 = 1 / >> >> As you can see it provides a 2D image, but there are 2 trailing >> degenerate dimensions. Tools like ds9 have no problem understanding this >> is a 2D image, but another user reports that matplotlib returns an >> error. As an intermediate solution, I wanted to patch the header by >> setting NAXIS to 2 and removing NAXIS3 and NAXIS4. You will find the >> script below. Unfortunately, none of the modifications I will do on the >> NAXIS* keywords will be saved in the result file, even if I keep these >> keywords perfectly consistent with the actual binary image. On the other >> hand, modifying, deleting, or adding other keywords works correctly. >> >> I remember having the same problem with pyfits in the past. I can also >> add that I can perfectly imaging doing the opposite transformation, i.e. >> adding extra degenerate dimensions to transform an image into a cube or >> more. >> >> Is this a bug or a feature? I believe this is a kind of protection >> against inconsistent modifications of the header, but can't we imagine >> to relax a bit this protection, by e.g. checking "only" that the total >> number of elements are preserved? I can provide a test file if needed. >> >> Thanks, >> >> Sebastien >> >> >> #!/bin/env python >> >> import astropy.io.fits >> import sys >> >> a = astropy.io.fits.open(sys.argv[1],mode='update') >> >> print "Before: ",a[0].header['NAXIS'] >> a[0].header['NAXIS'] = 2 # Won't be saved >> print "After: ",a[0].header['NAXIS'] >> del a[0].header['NAXIS3'] # Won't be deleted >> del a[0].header['NAXIS4'] # Won't be deleted >> >> a[0].header['OBJECT'] = 'ANOTHERONE' # Will be changed >> del a[0].header['ORIGIN'] # Will be deleted >> a[0].header['MYKEYWOR'] = 'Some string' # Will be added >> >> a.close() # Save changes >> del a >> >> # Check >> a = astropy.io.fits.open(sys.argv[1],mode='readonly') >> print "Current:",a[0].header['NAXIS'] # Shows unmodified value >> a.close() >> del a >> >> >> >> -- >> Sebastien BARDEAU >> IRAM >> 300 rue de la Piscine >> F - 38406 Saint Martin d'Heres >> Tel: (+33) 4 76 82 49 86 >> www.iram.fr/IRAMFR/GILDAS/ >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1642 bytes Desc: not available URL: From adam.g.ginsburg at gmail.com Mon Oct 23 11:10:27 2017 From: adam.g.ginsburg at gmail.com (Adam Ginsburg) Date: Mon, 23 Oct 2017 17:10:27 +0200 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: > Do you know what would be the best syntax to add a trailing dimension from > a 2D image? I mean, in a clever way without doubling the memory consumption. > > I'm not sure this is what you meant to ask, but I'll answer: The simplest way to expand an array's dimensions without expanding its memory consumption is with an indexing trick. This will turn a 2D array (assuming the original file was 2D) into a 4D array with dimensions 3 and 4 having size 1: data = fits.getdata(filename) data[:, :, None, None] -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Mon Oct 23 11:14:21 2017 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Mon, 23 Oct 2017 12:14:21 -0300 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: On Mon, Oct 23, 2017 at 12:10 PM, Adam Ginsburg wrote: > > >> Do you know what would be the best syntax to add a trailing dimension >> from a 2D image? I mean, in a clever way without doubling the memory >> consumption. >> >> > I'm not sure this is what you meant to ask, but I'll answer: > > The simplest way to expand an array's dimensions without expanding its > memory consumption is with an indexing trick. This will turn a 2D array > (assuming the original file was 2D) into a 4D array with dimensions 3 and 4 > having size 1: > > data = fits.getdata(filename) > data[:, :, None, None] > One can also use np.newaxis, which is just an alias to None. It's a tiny bit more readable for people who aren't aware of how None works in numpy's indexing system. > > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.f.evans at keele.ac.uk Mon Oct 23 10:56:13 2017 From: d.f.evans at keele.ac.uk (Daniel Evans) Date: Mon, 23 Oct 2017 15:56:13 +0100 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: Dear Sebastien, > Do you know what would be the best syntax to add a trailing dimension from a 2D image? I mean, in a clever way without doubling the memory consumption. The docs for numpy.reshape() state that the returned object "[...] will be a new view object if possible; otherwise, it will be a copy.", suggesting that a change will be done in place if possible. The following test returns True for me, indicating that the data is not duplicated: import numpy foo = numpy.zeros((256, 256)) # "Reshape" foo into a 3D array bar = foo.reshape((-1, 256, 256)) # If bar is a new view of foo, then they are both referencing the same data # Therefore, a modification of foo will propogate to bar foo[0, 0] = 1 print("foo and bar are the same data: ", foo[0, 0] == bar[0, 0, 0]) Regards, Daniel On 23 October 2017 at 15:02, S?bastien Bardeau wrote: > Dear Daniel, > > > On 10/23/2017 03:49 PM, Daniel Evans wrote: > > Dear Sebastien, > > I suspect you're right that astropy is undoing your modifications as it > thinks the header no longer matches the data. The solutions I can think of > for your problem are: > > 1. Simply modify the array once read into Python to turn it into a 2D > array, suitable for matplotlib.pyplot.imshow (or similar). This is > relatively standard for viewing images in datacubes anyway, and requires > very little effort in Python: > > a = astropy.io.fits.open(sys.argv[1]) > image4D = a[0].data > a.close() > > image2D = image4D[0, 0] > matplotlib.pyplot.imshow(image2D) > matplotlib.pyplot.show() > > 2. If you do want to modify the fits file, you can convert the array from > 4D to 2D, and reinsert this into the file. astropy will update the header > to match: > > a = astropy.io.fits.open(sys.argv[1], mode = "update") > a[0].data = a[0].data[0, 0] > a.close() > > > I was focusing on editing the header, but actually "editing" the data > works great! Thanks. > > Do you know what would be the best syntax to add a trailing dimension from > a 2D image? I mean, in a clever way without doubling the memory consumption. > > > > John ZuHone - the issue is presumably that matplotlib is being asked to > display a 2D image, but is being passed a 4D array - I've done this many > times with 3D datacubes! > > > @ John ZuHone: I am not a Matplotlib user (I am in charge of > reading/writing the FITS files), and I had just a comment about a > Matplotlib error without more details. But Daniel is probably right here. > > Thanks, > > Sebastien > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bardeau at iram.fr Mon Oct 23 12:03:13 2017 From: bardeau at iram.fr (=?UTF-8?Q?S=c3=a9bastien_Bardeau?=) Date: Mon, 23 Oct 2017 18:03:13 +0200 Subject: [AstroPy] astropy.io.fits won't allow me to modify the NAXIS keywords In-Reply-To: References: <7b967307-dd7b-c2dd-3c40-a3116b866a98@iram.fr> Message-ID: Dear Daniel and Adams, both of your solutions are satisfying (using data.reshape(-1,256,256) or data[None,None,:,:] ). I slightly prefer the 2nd one as it is more compact, it is quite symmetric to the opposite operation, and I do not need to know the actual dimensions. Thanks for your help everyone. On 10/23/2017 05:10 PM, Adam Ginsburg wrote: > ? > > Do you know what would be the best syntax to add a trailing > dimension from a 2D image? I mean, in a clever way without > doubling the memory consumption. > > > I'm not sure this is what you meant to ask, but I'll answer: > > The simplest way to expand an array's dimensions without expanding its > memory consumption is with an indexing trick.? This will turn a 2D > array (assuming the original file was 2D) into a 4D array with > dimensions 3 and 4 having size 1: > > data = fits.getdata(filename) > data[:, :, None, None] > ? > > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy -- Sebastien BARDEAU IRAM 300 rue de la Piscine F - 38406 Saint Martin d'Heres Tel: (+33) 4 76 82 49 86 www.iram.fr/IRAMFR/GILDAS/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pweilbacher at aip.de Wed Oct 25 09:28:03 2017 From: pweilbacher at aip.de (Peter Weilbacher) Date: Wed, 25 Oct 2017 15:28:03 +0200 (CEST) Subject: [AstroPy] (Table) unit formatting Message-ID: Dear all, I'm trying to add units to table columns with AstroPy (v2.0.1) and save the resulting table as FITS (for now). So I enter unit strings like this table['colname'].unit = 'erg/s/cm**2' and write the table to disk. Unfortunately, I not only get the annoying units warning about the multiple slashes (any way to turn that off?) but also discover that the strong got converted to 'cm-2 erg s-1' which while correct is really ugly -- nobody would write it like that in a paper! And actually, when I use print to look at the table before saving, I see that units as 'erg / (cm2 s)' which is not nice, either, but that I can live with. Can I adjust this behavior of do I have to edit the TUNITn manually after saving the file? By the way, I was also looking for a way to influence the display of the columns, i.e. the TDISPn keywords. If I set a format using t['colname'].format = '%4.2f' this is used when printing the table, but does not end up in the corresponding TDISPn after writing to FITS. Am I missing something? Finally, I'd like to give potential users of my file a longer description, and hoped there was a way to write something to the FITS comment part (after the slash of the TTYPEn keywords). Setting t['colname'].description = 'a descriptive message' works, but influences neither printing of the table nor writing to FITS. Any hints? Peter. -- Dr. Peter M. Weilbacher http://www.aip.de/People/PWeilbacher Phone +49 331 74 99-667 encryption key ID 7D6B4AA0 ------------------------------------------------------------------------ Leibniz-Institut f?r Astrophysik Potsdam (AIP) An der Sternwarte 16, D-14482 Potsdam Vorstand: Prof. Dr. Matthias Steinmetz, Matthias Winker Stiftung b?rgerlichen Rechts, Stiftungsverz. Brandenburg: 26 742-00/7026 From stephenbailey at lbl.gov Thu Oct 26 23:30:08 2017 From: stephenbailey at lbl.gov (Stephen Bailey) Date: Thu, 26 Oct 2017 20:30:08 -0700 Subject: [AstroPy] importing astropy without reading the config file Message-ID: We're using astropy in a massively parallel mpi4py program where 4800 ranks wake up simultaneously and try to import astropy. The code itself is installed to a filesystem optimized for the millions (literally!) of tiny file accesses that this causes. But we're having trouble with all 4800 ranks trying to read the astropy config file under our home directory, which is not optimized for such a load. This intermittently produces the traceback below when one of them gets a corrupted read. Although one could blame the filesystem for not being able to handle an arbitrarily large load, is there any way to mitigate this, e.g. * some environment variable to tell astropy not to load the config file upon import, plus some way of setting the config by hand * some environment variable to tell astropy to load it from a different location than $HOME * other ideas? For those less familiar with MPI, this is a "feature" of MPI that all ranks need to independently load the modules that they need, and they all start nearly simultaneously. Unlike other parallel programming paradigms, it isn't an option with MPI itself to have one process wake up and import stuff, and then fork other processes that inherit the same environment. (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that isn't widely implemented/deployed yet.) Our full scale option is to use docker with astropy installed into the image, but even there we have to create a fake $HOME in the docker image so that astropy will get its config files from there instead of reaching out of the container to get them from the real $HOME directory. It works, but yuck, because it means that nothing else in the program can access $HOME because we've hidden that from the container just to keep astropy imports from abusing it. Thanks for the help, Stephen File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py", line 21, in import astropy.io.fits File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line 287, in log = _init_log() File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, in _init_log log._set_defaults() File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, in _set_defaults self.setLevel(conf.log_level) File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", line 278, in __get__ return self() File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", line 401, in __call__ sec = get_config(self.module) File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", line 535, in get_config cobj = configobj.ConfigObj(cfgfn, interpolation=False) File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", line 1231, in __init__ self._load(infile, configspec) File "/global/common/software/desi/edison/desiconda/20170920-1.2. 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", line 1320, in _load raise error astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00 -------------- next part -------------- An HTML attachment was scrubbed... URL: From baweaver at lbl.gov Fri Oct 27 02:45:07 2017 From: baweaver at lbl.gov (Benjamin Alan Weaver) Date: Thu, 26 Oct 2017 23:45:07 -0700 Subject: [AstroPy] importing astropy without reading the config file In-Reply-To: References: Message-ID: Hello Stephen, It looks like XDG_CONFIG_HOME is a possibility, as discussed here: http://docs.astropy.org/en/stable/config/index.html, but I'd be concerned that other services & applications outside of Python might also see that environment variable and look for config files in the directory it specifies. Then again, maybe that's not a bad thing, since the fewer things that touch the real $HOME the better. Kia ora koe, Benjamin Alan Weaver On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey wrote: > We're using astropy in a massively parallel mpi4py program where 4800 > ranks wake up simultaneously and try to import astropy. The code itself is > installed to a filesystem optimized for the millions (literally!) of tiny > file accesses that this causes. But we're having trouble with all 4800 > ranks trying to read the astropy config file under our home directory, > which is not optimized for such a load. This intermittently produces the > traceback below when one of them gets a corrupted read. > > Although one could blame the filesystem for not being able to handle an > arbitrarily large load, is there any way to mitigate this, e.g. > > * some environment variable to tell astropy not to load the config file > upon import, plus some way of setting the config by hand > * some environment variable to tell astropy to load it from a different > location than $HOME > * other ideas? > > For those less familiar with MPI, this is a "feature" of MPI that all > ranks need to independently load the modules that they need, and they all > start nearly simultaneously. Unlike other parallel programming paradigms, > it isn't an option with MPI itself to have one process wake up and import > stuff, and then fork other processes that inherit the same environment. > (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that > isn't widely implemented/deployed yet.) > > Our full scale option is to use docker with astropy installed into the > image, but even there we have to create a fake $HOME in the docker image so > that astropy will get its config files from there instead of reaching out > of the container to get them from the real $HOME directory. It works, but > yuck, because it means that nothing else in the program can access $HOME > because we've hidden that from the container just to keep astropy imports > from abusing it. > > Thanks for the help, > > Stephen > > > > File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py", > line 21, in > import astropy.io.fits > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line 287, > in > log = _init_log() > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, in > _init_log > log._set_defaults() > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, in > _set_defaults > self.setLevel(conf.log_level) > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", > line 278, in __get__ > return self() > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", > line 401, in __call__ > sec = get_config(self.module) > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", > line 535, in get_config > cobj = configobj.ConfigObj(cfgfn, interpolation=False) > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", > line 1231, in __init__ > self._load(infile, configspec) > File "/global/common/software/desi/edison/desiconda/20170920-1.2. > 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", > line 1320, in _load > raise error > astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00 > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > > -- With great responsibility seldom comes great power. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnfarias at uc.cl Fri Oct 27 21:43:29 2017 From: dnfarias at uc.cl (Diego Farias) Date: Fri, 27 Oct 2017 22:43:29 -0300 Subject: [AstroPy] BSCALE/BZERO Message-ID: Hi everyone, I have posted this message wrongly in the 'issue' of astropy github, but it is better doing it by the mailing list. I've been struggling with a problem that I quite don't understand and a possible solution by scale function of astropy.io (that's why I'm asking advice). I have images with BITPIX -32,(max pixel values ~ 70000, min ~0) and I was instructed to 'change' them to - *BITPIX=16, BSCALE=1, and BZERO=32767 (or whatever value is needed to make the dynamic range from 0 to 65535)* That's why I thought I could use scale('uint16') in order to do this. But I'm pretty lost on whether I'm doing it good or wrong with this method. It was told me that a pixel value of e.g. 66030.2 in -32 image will be int(66030.2 - 65535) because an integer image (BITPIX=16) can only have 65536 distinct values. I get that, but what I don't understand is this: *BITPIX etc does not change the value, it only changes how that value is saved* What do you think about this problem? Thanks, Diego -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcraig at mnstate.edu Sat Oct 28 12:18:00 2017 From: mcraig at mnstate.edu (Matthew Craig) Date: Sat, 28 Oct 2017 16:18:00 +0000 Subject: [AstroPy] BSCALE/BZERO In-Reply-To: References: Message-ID: <503F38EB-E763-4B3A-ADE3-1DF1E41A817D@mnstate.edu> Hi Diego, Great question! You are right that to convert to unsigned 16-bit integers you would do something like scale(?uint16') but first some comments on whether you should convert: You are also right to be concerned about doing this with your data?uint16 goes only from 0 to 65535, but values large or smaller than those limits are *not* necessary just truncated. When convert a value of -4000 to a uint16 I get 61536, for example; if convert 95000 to uint16 I get 29464.0. As a result, converting your data with values outside the range 0 to 65535 will not just ?squash? or compress it to that range, it will give you incorrect data values. With those caveats, some more explanation about what the conversion options are and what they do. Assume for all of these that you have read in an HDU or HDUList called hdu. 1. Convert on output using scale; doing this: hdu.scale(?uint16?) does a couple of things: it sets BITPIX/BSCALE/BZERO in the header and converts the data to unsigned int16. Consider this example: import numpy as np from astropy.io import fits my_data = np.array([[90000, 2.0], [-1.0, -4000]]) my_hdu = fits.PrimaryHDU(my_data) This HDU has the header keywords appropriate for a float image: print(my_hdu.header) # Output: SIMPLE = T / conforms to FITS standard BITPIX = -64 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 2 NAXIS2 = 2 EXTEND = T END and the data looks like what was fed into PrimaryHDU print(my_hdu.data) # Output: [[ 9.00000000e+04 2.00000000e+00] [ -1.00000000e+00 -4.00000000e+03]] Now let?s scale and then print again: my_hdu.scale('uint16?) print(my_hdu.header) # Output: SIMPLE = T / conforms to FITS standard BITPIX = 16 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 2 NAXIS2 = 2 EXTEND = T BSCALE = 1 BZERO = 32768 END Notice BITPIX has changed to 16, and BZERO and BSCALE are now present. The data has been changed too: print(my_hdu.data) # Output: [[24464 2] [65535 61536]] Note that almost none of these entries are what is desired: 90000 has changed to 24464, -1 has changed to 65535, and -4000 has changed to 61536. 2. Alternatively, you can change that data?s type, which updates both the data values and the header: my_hdu.data = np.uint16(my_hdu.data) To wrap up I want to emphasize that the original data values are permanently gone. If I write out this HDU and read it back (and convert it to floating point) then you do not get back the original data values: my_hdu.writeto('oof.fits?) new_hdu = fits.open('oof.fits') new_hdu = new_hdu[0] # just get the first HDU in the HDUList new_hdu.scale('float32') # This conversion would automatically happen the first time # you used the data in a floating point operation print(new_hdu.data) # Output: [[ 57232., 32770.], [ 32767., 28768.]] Matt Craig schedule: http://physics.mnstate.edu/craig ?? Professor Department of Physics and Astronomy Minnesota State University Moorhead 1104 7th Ave S, Moorhead MN 56563 office: Hagen 307F phone: (218) 477-2439 fax: (218) 477-2290 On Oct 27, 2017, at 8:43 PM, Diego Farias > wrote: Hi everyone, I have posted this message wrongly in the 'issue' of astropy github, but it is better doing it by the mailing list. I've been struggling with a problem that I quite don't understand and a possible solution by scale function of astropy.io (that's why I'm asking advice). I have images with BITPIX -32,(max pixel values ~ 70000, min ~0) and I was instructed to 'change' them to * BITPIX=16, BSCALE=1, and BZERO=32767 (or whatever value is needed to make the dynamic range from 0 to 65535) That's why I thought I could use scale('uint16') in order to do this. But I'm pretty lost on whether I'm doing it good or wrong with this method. It was told me that a pixel value of e.g. 66030.2 in -32 image will be int(66030.2 - 65535) because an integer image (BITPIX=16) can only have 65536 distinct values. I get that, but what I don't understand is this: BITPIX etc does not change the value, it only changes how that value is saved What do you think about this problem? Thanks, Diego _______________________________________________ AstroPy mailing list AstroPy at python.org https://mail.python.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnfarias at uc.cl Sat Oct 28 14:10:25 2017 From: dnfarias at uc.cl (Diego Farias) Date: Sat, 28 Oct 2017 15:10:25 -0300 Subject: [AstroPy] BSCALE/BZERO Message-ID: Hi Matt and everyone: I did the same 'experiments'! and I realized that I 'loose' information if I scale to 'uint16' . That's why I don't understand the problem because I have been told that the *PIXEL VALUES *should remain the same it's just that the dynamic range with the BITPIX=16,BZERO=32767 should be from 0 to 65535. As I'm not clever enough I'm stuck in this step.. I'm guessing that later we need to use another astronomical tool that requires the images with BITPIX = 16. In my despair, I found this: https://github.com/varenius/salsa/blob/master/Control_program/spectrum.py where the important step is #Since using int16 as datatype we use bscale and bzero to keep dynamic range. #SalsaJ cannot read bitpix correctly except 16 bit. If SalsaJ could read bitpix #we could just have BITPIX -64 and skip Bscale, Bzero, i.e. just remove #astype above. datamin = np.min(self.data) datamax = np.max(self.data) bscale = (datamax-datamin)/65534.0 #shouldn't be 65535.0? bzero = datamin+bscale*32767.0 scaledata = (self.data - bzero)/bscale hdu.data = scaledata.reshape(1, 1, self.nchans).*astype(np.int16)* Shouldn't this *also changes *the values? (in this case, spectrum values). Also, bscale ~ 1., bzero ~ 326767. Thanks a lot, Diego -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcraig at mnstate.edu Sat Oct 28 16:52:58 2017 From: mcraig at mnstate.edu (Matthew Craig) Date: Sat, 28 Oct 2017 20:52:58 +0000 Subject: [AstroPy] BSCALE/BZERO In-Reply-To: References: Message-ID: Hi, That's why I don't understand the problem because I have been told that the PIXEL VALUES should remain the same it's just that the dynamic range with the BITPIX=16,BZERO=32767 should be from 0 to 65535. I think this is true as long as the data is already in the range 0 to 65535; in that case the data is actually stored in the FITS files as signed 16 bit integers which are transparently changed to unsigned int when you read it with astropy.io.fits or other FITS tools. In your case you will need to transform the pixel values because you need to change the range of your data. If I understand your use case, you need to squeeze the data into an int16 range because another tool handles that. Two options are to manually calculate a BZERO/BSCALE and scale the data with that or to convert the data int16 or uint16 and let astropy.io.fits set BZERO/BSCALE for you. I would recommend the latter?it will not affect the dynamic range of the data and saves you the hassle of calculating those things yourself. The other reason is that numpy has a way of representing uint16 or int16. I?ll address how to do want you want by converting the data to uint16. In your case you need to scale the values so that they fit in the range 0 to 65535. I would do something like (using the same example data as before)(have not tested this :)): from __future__ import division # if you are running Python 2 my_data = np.array([[90000, 2.0], [-1.0, -4000]]) my_hdu = fits.PrimaryHDU(my_data) data_min = my_hdu.data.min() data_max = my_hdu.data.max() # Yes, the maximum value should be 65535 not 65534 new_data = 65535 * (my_hdu.data - data_min) / (data_max - data_min) # The line below will automatically revise BITPIX/BZERO/BSCALE my_hdu.data = new_data.astype(np.uint16) my_hdu.writeto('ui16_image.fits?) mh_hdu.data array([[65535, 2790], [ 2788, 0]], dtype=uint16) # Note that the largest and smallest pixel values in the original array are still the largest and smallest # pixels in the result. Matt Craig schedule: http://physics.mnstate.edu/craig ?? Professor Department of Physics and Astronomy Minnesota State University Moorhead 1104 7th Ave S, Moorhead MN 56563 office: Hagen 307F phone: (218) 477-2439 fax: (218) 477-2290 On Oct 28, 2017, at 1:10 PM, Diego Farias > wrote: Hi Matt and everyone: I did the same 'experiments'! and I realized that I 'loose' information if I scale to 'uint16' . That's why I don't understand the problem because I have been told that the PIXEL VALUES should remain the same it's just that the dynamic range with the BITPIX=16,BZERO=32767 should be from 0 to 65535. As I'm not clever enough I'm stuck in this step.. I'm guessing that later we need to use another astronomical tool that requires the images with BITPIX = 16. In my despair, I found this: https://github.com/varenius/salsa/blob/master/Control_program/spectrum.py where the important step is #Since using int16 as datatype we use bscale and bzero to keep dynamic range. #SalsaJ cannot read bitpix correctly except 16 bit. If SalsaJ could read bitpix #we could just have BITPIX -64 and skip Bscale, Bzero, i.e. just remove #astype above. datamin = np.min(self.data) datamax = np.max(self.data) bscale = (datamax-datamin)/65534.0 #shouldn't be 65535.0? bzero = datamin+bscale*32767.0 scaledata = (self.data - bzero)/bscale hdu.data = scaledata.reshape(1, 1, self.nchans).astype(np.int16) Shouldn't this also changes the values? (in this case, spectrum values). Also, bscale ~ 1., bzero ~ 326767. Thanks a lot, Diego _______________________________________________ AstroPy mailing list AstroPy at python.org https://mail.python.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From vittorio.braga at roma2.infn.it Tue Oct 31 14:22:35 2017 From: vittorio.braga at roma2.infn.it (Vittorio_Roma2) Date: Tue, 31 Oct 2017 19:22:35 +0100 Subject: [AstroPy] Aplpy FITSFigure does not open any window Message-ID: Dear all I am having a problem with aplpy. I am using Python 3.5.2 under ubuntu 16.04 and installed aplpy version 1.1.1 through pip3 install aplpy The first problem arises when importing aplpy: import/home/vittorio/.local/lib/python3.5/site-packages/matplotlib/cbook/deprecation.py:106: MatplotlibDeprecationWarning: The mpl_toolkits.axes_grid module was deprecated in version 2.1. Use mpl_toolkits.axes_grid1 and mpl_toolkits.axisartist provies the same functionality instead. ? warnings.warn(message, mplDeprecation, stacklevel=1) I have tried to substitute mpl_toolkits.axes_grid1 with mpl_toolkits.axes_grid in core.py but then I got an error (and not a warning)? saying that axes_grid1 does not have some method. This will just return a failure and stop everything. Anyways, when I do >>> fitsfile='somefile.fits' >>> F=aplpy.FITSFigure(fitsfile) WARNING: Cannot determine equinox. Assuming J2000. [aplpy.wcs_util] WARNING: Cannot determine equinox. Assuming J2000. [aplpy.wcs_util] >>> F.show_grayscale() INFO: Auto-setting vmin to? 3.416e+03 [aplpy.core] INFO: Auto-setting vmax to? 2.212e+04 [aplpy.core] No window at all is opening. I have also tried giving as input of FITSFIgure astropy.io.fits.open objects but again, no graphical window shows up. I have found some workaround, that is, after the above instructions, to give >>>hdu_list=fits.open(fitsfile) >>>image_data=hdu_list[0].data >>>plt.imshow(image_data,'gray',clim=[4500,9000]) >>>plt.show(block=False) This does open an astrometrized image, with the labels of the X and Y axis in the right place. However, this does not allow me to use any of the FITSFigure methods. Neither I can add subplots to put annotations. Finally, to add markers, I am forced to do another workaround: to give pixel coordinates like this >>>plt.scatter(pixcoords[:,0],pixcoords[:,1]) where pixcoords comes from WCS transformations. I have also tried a workaround, giving,e.g. >>>F.add_label(0.5,0.5,'ssss',relative=True) and do the plt.imshow and plt.show later, but this still does not work. To sum up, while it is true that? I can live with the workarounds for some very basic figures, all of this is slowing down the script and I can't call any of the FITSFigure methods Any help would really be appreciated Best regards, Vittorio From stephenbailey at lbl.gov Tue Oct 31 15:01:34 2017 From: stephenbailey at lbl.gov (Stephen Bailey) Date: Tue, 31 Oct 2017 12:01:34 -0700 Subject: [AstroPy] importing astropy without reading the config file In-Reply-To: References: Message-ID: For the record: setting $XDG_CONFIG_HOME did not work but I don't know why. It didn't create any .astropy/ config files in the new location, and when I moved my $HOME/.astropy out of the way, it still created a new $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME was set to a different location. I didn't try debugging beyond that. In the meantime we've moved on with a pragmatic hack that performs a random sleep between 0 and 60 seconds per rank before doing any astropy imports, thus spreading out the load. That buys us a bit more time for development work; we still plan to use docker/shifter for production work with a redefined $HOME. Stephen On Thu, Oct 26, 2017 at 11:45 PM, Benjamin Alan Weaver wrote: > Hello Stephen, > > It looks like XDG_CONFIG_HOME is a possibility, as discussed here: > http://docs.astropy.org/en/stable/config/index.html, but I'd be concerned > that other services & applications outside of Python might also see that > environment variable and look for config files in the directory it > specifies. Then again, maybe that's not a bad thing, since the fewer > things that touch the real $HOME the better. > > Kia ora koe, > Benjamin Alan Weaver > > On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey > wrote: > >> We're using astropy in a massively parallel mpi4py program where 4800 >> ranks wake up simultaneously and try to import astropy. The code itself is >> installed to a filesystem optimized for the millions (literally!) of tiny >> file accesses that this causes. But we're having trouble with all 4800 >> ranks trying to read the astropy config file under our home directory, >> which is not optimized for such a load. This intermittently produces the >> traceback below when one of them gets a corrupted read. >> >> Although one could blame the filesystem for not being able to handle an >> arbitrarily large load, is there any way to mitigate this, e.g. >> >> * some environment variable to tell astropy not to load the config file >> upon import, plus some way of setting the config by hand >> * some environment variable to tell astropy to load it from a different >> location than $HOME >> * other ideas? >> >> For those less familiar with MPI, this is a "feature" of MPI that all >> ranks need to independently load the modules that they need, and they all >> start nearly simultaneously. Unlike other parallel programming paradigms, >> it isn't an option with MPI itself to have one process wake up and import >> stuff, and then fork other processes that inherit the same environment. >> (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that >> isn't widely implemented/deployed yet.) >> >> Our full scale option is to use docker with astropy installed into the >> image, but even there we have to create a fake $HOME in the docker image so >> that astropy will get its config files from there instead of reaching out >> of the container to get them from the real $HOME directory. It works, but >> yuck, because it means that nothing else in the program can access $HOME >> because we've hidden that from the container just to keep astropy imports >> from abusing it. >> >> Thanks for the help, >> >> Stephen >> >> >> >> File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py", >> line 21, in >> import astropy.io.fits >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line 287, >> in >> log = _init_log() >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, >> in _init_log >> log._set_defaults() >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, >> in _set_defaults >> self.setLevel(conf.log_level) >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >> line 278, in __get__ >> return self() >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >> line 401, in __call__ >> sec = get_config(self.module) >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >> line 535, in get_config >> cobj = configobj.ConfigObj(cfgfn, interpolation=False) >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", >> line 1231, in __init__ >> self._load(infile, configspec) >> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", >> line 1320, in _load >> raise error >> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00 >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy >> >> > > > -- > With great responsibility seldom comes great power. > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From baweaver at lbl.gov Tue Oct 31 15:59:51 2017 From: baweaver at lbl.gov (Benjamin Alan Weaver) Date: Tue, 31 Oct 2017 12:59:51 -0700 Subject: [AstroPy] importing astropy without reading the config file In-Reply-To: References: Message-ID: Hello Stephen, According to the documentation, http://docs.astropy.org/en/stable/config/index.html, $XDG_CONFIG_HOME/astropy must *already* exist. You can take a look at https://github.com/astropy/astropy/blob/master/astropy/config/paths.py to see the logic, but to summarize: 1. $XDG_CONFIG_HOME must be set. 2. $XDG_CONFIG_HOME/astropy/ must *exist*. Note that's 'astropy' not '.astropy'. 3. The configuration files go in $XDG_CONFIG_HOME/astropy/, NOT $XDG_CONFIG_HOME/astropy/config/. This one confused me! You might also want to consider setting XDG_CONFIG_CACHE, so that cache files are written to some other filesystem than $HOME. Kia ora koe, Benjamin Alan Weaver On Tue, Oct 31, 2017 at 12:01 PM, Stephen Bailey wrote: > For the record: setting $XDG_CONFIG_HOME did not work but I don't know > why. It didn't create any .astropy/ config files in the new location, and > when I moved my $HOME/.astropy out of the way, it still created a new > $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME > was set to a different location. I didn't try debugging beyond that. > > In the meantime we've moved on with a pragmatic hack that performs a > random sleep between 0 and 60 seconds per rank before doing any astropy > imports, thus spreading out the load. That buys us a bit more time for > development work; we still plan to use docker/shifter for production work > with a redefined $HOME. > > Stephen > > > On Thu, Oct 26, 2017 at 11:45 PM, Benjamin Alan Weaver > wrote: > >> Hello Stephen, >> >> It looks like XDG_CONFIG_HOME is a possibility, as discussed here: >> http://docs.astropy.org/en/stable/config/index.html, but I'd be >> concerned that other services & applications outside of Python might also >> see that environment variable and look for config files in the directory it >> specifies. Then again, maybe that's not a bad thing, since the fewer >> things that touch the real $HOME the better. >> >> Kia ora koe, >> Benjamin Alan Weaver >> >> On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey >> wrote: >> >>> We're using astropy in a massively parallel mpi4py program where 4800 >>> ranks wake up simultaneously and try to import astropy. The code itself is >>> installed to a filesystem optimized for the millions (literally!) of tiny >>> file accesses that this causes. But we're having trouble with all 4800 >>> ranks trying to read the astropy config file under our home directory, >>> which is not optimized for such a load. This intermittently produces the >>> traceback below when one of them gets a corrupted read. >>> >>> Although one could blame the filesystem for not being able to handle an >>> arbitrarily large load, is there any way to mitigate this, e.g. >>> >>> * some environment variable to tell astropy not to load the config file >>> upon import, plus some way of setting the config by hand >>> * some environment variable to tell astropy to load it from a different >>> location than $HOME >>> * other ideas? >>> >>> For those less familiar with MPI, this is a "feature" of MPI that all >>> ranks need to independently load the modules that they need, and they all >>> start nearly simultaneously. Unlike other parallel programming paradigms, >>> it isn't an option with MPI itself to have one process wake up and import >>> stuff, and then fork other processes that inherit the same environment. >>> (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that >>> isn't widely implemented/deployed yet.) >>> >>> Our full scale option is to use docker with astropy installed into the >>> image, but even there we have to create a fake $HOME in the docker image so >>> that astropy will get its config files from there instead of reaching out >>> of the container to get them from the real $HOME directory. It works, but >>> yuck, because it means that nothing else in the program can access $HOME >>> because we've hidden that from the container just to keep astropy imports >>> from abusing it. >>> >>> Thanks for the help, >>> >>> Stephen >>> >>> >>> >>> File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py", >>> line 21, in >>> import astropy.io.fits >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line >>> 287, in >>> log = _init_log() >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, >>> in _init_log >>> log._set_defaults() >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, >>> in _set_defaults >>> self.setLevel(conf.log_level) >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >>> line 278, in __get__ >>> return self() >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >>> line 401, in __call__ >>> sec = get_config(self.module) >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >>> line 535, in get_config >>> cobj = configobj.ConfigObj(cfgfn, interpolation=False) >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", >>> line 1231, in __init__ >>> self._load(infile, configspec) >>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", >>> line 1320, in _load >>> raise error >>> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00 >>> >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at python.org >>> https://mail.python.org/mailman/listinfo/astropy >>> >>> >> >> >> -- >> With great responsibility seldom comes great power. >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy >> >> > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > > -- With great responsibility seldom comes great power. -------------- next part -------------- An HTML attachment was scrubbed... URL: From baweaver at lbl.gov Tue Oct 31 16:00:57 2017 From: baweaver at lbl.gov (Benjamin Alan Weaver) Date: Tue, 31 Oct 2017 13:00:57 -0700 Subject: [AstroPy] importing astropy without reading the config file In-Reply-To: References: Message-ID: Hello Stephen, PS, just to be clear, when I followed all three steps, it did work for me. Kia ora koe, Benjamin Alan Weaver On Tue, Oct 31, 2017 at 12:59 PM, Benjamin Alan Weaver wrote: > Hello Stephen, > > According to the documentation, http://docs.astropy.org/en/stable/config/ > index.html, $XDG_CONFIG_HOME/astropy must *already* exist. You can take a > look at https://github.com/astropy/astropy/blob/master/astropy/ > config/paths.py to see the logic, but to summarize: > > 1. $XDG_CONFIG_HOME must be set. > > 2. $XDG_CONFIG_HOME/astropy/ must *exist*. Note that's 'astropy' not > '.astropy'. > > 3. The configuration files go in $XDG_CONFIG_HOME/astropy/, NOT > $XDG_CONFIG_HOME/astropy/config/. This one confused me! > > You might also want to consider setting XDG_CONFIG_CACHE, so that cache > files are written to some other filesystem than $HOME. > > Kia ora koe, > Benjamin Alan Weaver > > > > On Tue, Oct 31, 2017 at 12:01 PM, Stephen Bailey > wrote: > >> For the record: setting $XDG_CONFIG_HOME did not work but I don't know >> why. It didn't create any .astropy/ config files in the new location, and >> when I moved my $HOME/.astropy out of the way, it still created a new >> $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME >> was set to a different location. I didn't try debugging beyond that. >> >> In the meantime we've moved on with a pragmatic hack that performs a >> random sleep between 0 and 60 seconds per rank before doing any astropy >> imports, thus spreading out the load. That buys us a bit more time for >> development work; we still plan to use docker/shifter for production work >> with a redefined $HOME. >> >> Stephen >> >> >> On Thu, Oct 26, 2017 at 11:45 PM, Benjamin Alan Weaver >> wrote: >> >>> Hello Stephen, >>> >>> It looks like XDG_CONFIG_HOME is a possibility, as discussed here: >>> http://docs.astropy.org/en/stable/config/index.html, but I'd be >>> concerned that other services & applications outside of Python might also >>> see that environment variable and look for config files in the directory it >>> specifies. Then again, maybe that's not a bad thing, since the fewer >>> things that touch the real $HOME the better. >>> >>> Kia ora koe, >>> Benjamin Alan Weaver >>> >>> On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey >>> wrote: >>> >>>> We're using astropy in a massively parallel mpi4py program where 4800 >>>> ranks wake up simultaneously and try to import astropy. The code itself is >>>> installed to a filesystem optimized for the millions (literally!) of tiny >>>> file accesses that this causes. But we're having trouble with all 4800 >>>> ranks trying to read the astropy config file under our home directory, >>>> which is not optimized for such a load. This intermittently produces the >>>> traceback below when one of them gets a corrupted read. >>>> >>>> Although one could blame the filesystem for not being able to handle an >>>> arbitrarily large load, is there any way to mitigate this, e.g. >>>> >>>> * some environment variable to tell astropy not to load the config file >>>> upon import, plus some way of setting the config by hand >>>> * some environment variable to tell astropy to load it from a different >>>> location than $HOME >>>> * other ideas? >>>> >>>> For those less familiar with MPI, this is a "feature" of MPI that all >>>> ranks need to independently load the modules that they need, and they all >>>> start nearly simultaneously. Unlike other parallel programming paradigms, >>>> it isn't an option with MPI itself to have one process wake up and import >>>> stuff, and then fork other processes that inherit the same environment. >>>> (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that >>>> isn't widely implemented/deployed yet.) >>>> >>>> Our full scale option is to use docker with astropy installed into the >>>> image, but even there we have to create a fake $HOME in the docker image so >>>> that astropy will get its config files from there instead of reaching out >>>> of the container to get them from the real $HOME directory. It works, but >>>> yuck, because it means that nothing else in the program can access $HOME >>>> because we've hidden that from the container just to keep astropy imports >>>> from abusing it. >>>> >>>> Thanks for the help, >>>> >>>> Stephen >>>> >>>> >>>> >>>> File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py", >>>> line 21, in >>>> import astropy.io.fits >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line >>>> 287, in >>>> log = _init_log() >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, >>>> in _init_log >>>> log._set_defaults() >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, >>>> in _set_defaults >>>> self.setLevel(conf.log_level) >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >>>> line 278, in __get__ >>>> return self() >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >>>> line 401, in __call__ >>>> sec = get_config(self.module) >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py", >>>> line 535, in get_config >>>> cobj = configobj.ConfigObj(cfgfn, interpolation=False) >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", >>>> line 1231, in __init__ >>>> self._load(infile, configspec) >>>> File "/global/common/software/desi/edison/desiconda/20170920-1.2. >>>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py", >>>> line 1320, in _load >>>> raise error >>>> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00 >>>> >>>> >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at python.org >>>> https://mail.python.org/mailman/listinfo/astropy >>>> >>>> >>> >>> >>> -- >>> With great responsibility seldom comes great power. >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at python.org >>> https://mail.python.org/mailman/listinfo/astropy >>> >>> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy >> >> > > > -- > With great responsibility seldom comes great power. > -- With great responsibility seldom comes great power. -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Tue Oct 31 15:44:05 2017 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 Oct 2017 20:44:05 +0100 Subject: [AstroPy] importing astropy without reading the config file In-Reply-To: References: Message-ID: <96A90894-F5B7-4F0F-A71E-55D97E53AB98@astro.physik.uni-goettingen.de> On 31 Oct 2017, at 8:01 pm, Stephen Bailey wrote: > > For the record: setting $XDG_CONFIG_HOME did not work but I don't know why. It didn't create any .astropy/ config files in the new location, and when I moved my $HOME/.astropy out of the way, it still created a new $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME was set to a different location. I didn't try debugging beyond that. Looking into astropy/config/paths.py, this should be $XDG_CONFIG_HOME/astropy , and that directory has to already exist to be used as the config subdirectory - the docstring of get_config_dir() can be a little bit misleading in that respect. If setting $XDG_CONFIG_HOME should still be causing troubles and modifying your astropy installation is an option, you might also try to hack paths.py to have get_config_dir additionally search a third location. HTH, Derek From jh at physics.ucf.edu Tue Oct 31 16:38:45 2017 From: jh at physics.ucf.edu (Joe Harrington) Date: Tue, 31 Oct 2017 16:38:45 -0400 Subject: [AstroPy] verify behavior Message-ID: I have some non-compliant FITS images from an old instrument that I use in my class (why? because the data are perfect for an example of median combination, and the strange format and non-compliance help me teach writing wrapper functions). I had thought that doing hdulist.verify('silentfix') would silently fix the broken cards. However, it doesn't. It only seems to set the option for how to do it later, should the need arise. For example, the cards get fixed if printed. Otherwise, when trying to write the file, it errors out. It's not a solution to write with 'silentfix', because I want to fix the data on read so the user doesn't know it was ever bad. So, my goal is to hide the weirdness of this instrument's data inside a wrapper script, which means I need to print the header for each image as it's read, which is not desired. So, my hack is to open /dev/null (portably) and print it to that. Is this the best option, or is it a bug and hdulist.verify() should actually DO the fix, or is there something else I can run to make it do the fix on or right after the read that is cleaner? The example script is below, set up to comment in and out various lines to try stuff. An example FITS(ish) file is here: https://physics.ucf.edu/~jh/dark_13s_5.fits --jh-- #! /usr/bin/env python3 # Astropy verify bug test case. # Joseph Harrington 31 October 2017 # Note that floats header entries have lowercase 'e' in file, but they # have uppercase 'E' in astropy FITS header object. It gets fixed # automatically. However, it isn't really fixed. # Note: CDELT1, CDELT2, RA_OFFS, DEC_OFFS, RA_RATE, DEC_RATE. # Comment in and out the lines below. # BAD cases: # Simple read and write: prints errors, nothing written. # read, silentfix, write: prints errors, nothing written. # read, fix, write: prints warnings, prints errors, nothing written. # NOT GREAT cases (can't hide fixes in reader function): # read, print, write: prints warnings, writes good data. # read and write with 'silentfix': writes good data # GOOD cases: # use hdulist.verify('silentfix') to set the kind of fixing option # print the header to /dev/null or equivalent to fix the header # write import numpy as np import astropy.io.fits as fits import os infile = 'dark_13s_5.fits' outfile = 'writefix.fits' hdulist = fits.open(infile) hdulist.verify('silentfix') header = hdulist[0].header data = hdulist[0].data #print("Printing header to /dev/null or OS equivalent") with open(os.devnull, 'w') as null: print(header, file=null) print("Writing file "+outfile) fits.writeto(outfile, data, header, overwrite=True) # This is NOT a solution, as user shouldn't know about problem. #fits.writeto(outfile, data, header, overwrite=True, output_verify='silentfix') From stephenbailey at lbl.gov Tue Oct 31 16:49:00 2017 From: stephenbailey at lbl.gov (Stephen Bailey) Date: Tue, 31 Oct 2017 13:49:00 -0700 Subject: [AstroPy] importing astropy without reading the config file In-Reply-To: <96A90894-F5B7-4F0F-A71E-55D97E53AB98@astro.physik.uni-goettingen.de> References: <96A90894-F5B7-4F0F-A71E-55D97E53AB98@astro.physik.uni-goettingen.de> Message-ID: Thanks Derek and Ben. Ben's step (2) was the bit I was missing: $XDG_CONFIG_HOME/astropy/ instead of $XDG_CONFIG_HOME/.astropy/ I had been including the extra "." and thus not pre-creating the directory that it was looking for. I confirm that it also works for me with that change. Stephen On Tue, Oct 31, 2017 at 12:44 PM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > On 31 Oct 2017, at 8:01 pm, Stephen Bailey wrote: > > > > For the record: setting $XDG_CONFIG_HOME did not work but I don't know > why. It didn't create any .astropy/ config files in the new location, and > when I moved my $HOME/.astropy out of the way, it still created a new > $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME > was set to a different location. I didn't try debugging beyond that. > > Looking into astropy/config/paths.py, this should be > $XDG_CONFIG_HOME/astropy , > and that directory has to already exist to be used as the config > subdirectory - the docstring > of get_config_dir() can be a little bit misleading in that respect. > If setting $XDG_CONFIG_HOME should still be causing troubles and modifying > your > astropy installation is an option, you might also try to hack paths.py to > have get_config_dir > additionally search a third location. > > HTH, > Derek > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: