From oneaufs at gmail.com Fri Jul 1 03:38:43 2011 From: oneaufs at gmail.com (Prasanth) Date: Fri, 1 Jul 2011 13:08:43 +0530 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E0D20A4.7090507@du.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> Message-ID: Hello, > I am, however, a little concerned that the current vision statement > seems to exclude the use of existing officially sanctioned (i.e. by the > IAU) routines that are written in other languages. I am not sure if anyone is against such libraries, it is just that we haven't reached a stage where we debate on how to include C/Fortan extensions in the common package. > For instance, the Standards of Fundamental Astronomy > (http://www.iausofa.org/) package contains IAU-sanctioned routines for > very common tasks (Calendards, Time Scales, Sidereal times, Ephemerids, > star motion, star catalog conversion, etc). This package is written in > C and is quite easily wrapped using SWIG (see the pysofa package > http://pypi.python.org/pypi/pysofa). If the interface layer is > implemented correctly, the function calls have minimal overhead and, in > some circumstances, can benefit from being implemented in lower-level > languages. I have found put that Cython is much better than SWIG when it comes to implementing interfaces to C packages. I was able to come up with a much more pythonic and easier to use interface for PyTPM and COORDS, using Cython. The speed gain was also noticeable. The pysofa package, mentioned above, is written using ctypes and I haven't played around with it. Writing the interface itself is very easy; I have code that interfaces with NOVAS and SOFA, to varying degrees of completeness. The challenge is in reducing the time of execution while handling multiple input values and providing a pythonic interface. I have a suggestion that while the common plan is being developed, sub-groups can be formed and these groups can have their own discussions on how to choose libraries and so on. Of-course the exact nature of the products can wait till the common dependencies are sorted out, but considerable progress can be achieved in the mean time. Also these discussions can provide feedback on what should go into the common package. Prasanth From erwin at mpe.mpg.de Fri Jul 1 07:05:28 2011 From: erwin at mpe.mpg.de (Peter Erwin) Date: Fri, 1 Jul 2011 13:05:28 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> Message-ID: <4A6A8E45-5F93-4D56-8318-220255B915A1@mpe.mpg.de> On Jul 1, 2011, at 2:02 AM, Erik Tollerud wrote: > Just to quickly address the process here: The idea was that the > vision document in particular needed to be drafted with a somewhat > coherent vision, so we wanted to offer it for a vote quickly so as to > get a sense as to whether everyone was on the same page here (so far > at least, in seems most people are, given the votes and clarifications > here). > > For future documents such as coding style and other planning > documents, we will likely be issuing drafts and solicit > comments/suggestions before putting a modified version to a vote. In > this particular case we felt that speed and consistency of vision > would help the process proceed more quickly and stay organized, > especially given that most of the details in the document are still > open for change if it's desired. > > > On the topic of GUIs, the thought was that it would be difficult to > come to a consensus given how many different toolkits are in use and > the fact that the learning curve is a fair bit steeper... so it would > be great to settle on one, but it might alienate a fair number of > otherwise interested developers, or reduce the knowledge base so much > that we couldn't get off the ground for GUIs. That said, I personally > favor TraitsUI and the related Enthought tools, but I know there are > other opinions... > I think avoiding trying to specify a particular GUI is an excellent idea, particularly at this stage. Ironically, I think part of the reasons for matplotlib's success is that *it* doesn't depend on a particular GUI -- instead, they went for the difficult path of supporting enough GUIs that it becomes likely you can install and use matplotlib without having to install large, complex GUI packages (because you probably already have one of the supported GUIs on your system). Not that I'd recommend a "support every GUI in detail" approach for astropy. cheers, Peter ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)89 30000 3695 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From erik.tollerud at gmail.com Fri Jul 1 13:59:10 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Fri, 1 Jul 2011 10:59:10 -0700 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> Message-ID: Technically, we're waiting until the poll closes (9pm EST tonight), but as you say, there's not much doubt as to the result at this point. We (the Coordination Committee) are currently working on the important starting points (coding style, documentation style, and similar) mentioned in the vision. Once we have drafts, we will send them out for review by the community - this shouldn't take too long, as most of this is fairly straightforward (although very crucial to agree on at the outset). Once that's done, we will implement the necessary infrastructure code that the vision calls for (e.g. documentation outlines, the template package, configuration/data code, and the installer script), ASAP. This shouldn't take too long (I speculate a few weeks), as it's mostly stuff that's already been implemented in other forms and just needs tweaking. There's been no decision yet as to whether or not this will be a "release," although once the initial work is done, although at that point it makes sense for people to start using at least the development version. At the same time, we will start identifying the initial areas of work for modules, and community members can begin work either writing new packages to satisfy those needs, or start adapting existing packages to the astropy framework. Additionally, we will start accepting affiliated packages once we have the index of packages up and running. On Thu, Jun 30, 2011 at 5:44 PM, Paul Barrett wrote: > Since, it appears that this vision for an astropy suite will be > adopted, given the overwhelming number of yeses at the current time, > do you have a timeline in mind for how to proceed, i.e., what modules > are being considered for the initial release and when will it occur? > > ?-- Paul > -- Erik Tollerud From thomas.robitaille at gmail.com Sat Jul 2 04:05:26 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sat, 2 Jul 2011 10:05:26 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: Message-ID: Hi everyone, For the record, here are the final results of the poll: The initial vision for the common Astronomy package has been approved, with 79 'yes' votes (91.9%) against 7 'no' votes (8.1%) The name 'astropy' for the common Astronomy package has been approved, with 89 'yes' votes (98.9%) against 1 'no' vote (1.1%) As mentioned several times in this thread, the vision is an initial version, and people should feel free to bring up suggestions for changes or additions any time in the future. As Erik already mentioned, we're preparing a draft of the initial areas of work and coding guidelines which we'll put up for discussion very soon! I would just like to add a note that it's great to see so many people involved at this stage - thanks for voting! Cheers, Tom On 27 June 2011 22:16, Thomas Robitaille wrote: > Hi everyone, > > In the last week, Erik, Perry, and myself have been discussing how > best to coordinate the development of the common Python Astronomy > package. We have now converged on a common vision, and would now like > to know whether you would be happy with it too. The vision and a > poll are available at the following pages: > > vision: http://astropy.wikispaces.com/vision > > poll: http://astropy.wikispaces.com/vision-polls > > In addition, 'astropy' has been suggested by several people as a name > for this common package, so rather than creating a multi-option poll, > we've created a simple yes/no poll to find out whether you would agree > with this name. The idea is to have a name that does not endorse any > specific existing project, is in line with numpy/scipy, and reflects > its initial development via this mailing list. The poll is located at > the same URL as before. > > The polls will be open unti Friday 1st July at 9pm EST. > > Thanks! > Thomas > From aldcroft at head.cfa.harvard.edu Sun Jul 3 16:02:52 2011 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Sun, 3 Jul 2011 16:02:52 -0400 Subject: [AstroPy] "double free or corruption" crash: PyFITS + NumPy 1.6.0 Message-ID: I'm seeing some sort of memory or allocation error (glibc detected) when trying to read a particular FITS file with the following setup: OS: CentOS-5.6 (64-bit) Python: 2.7.1.4 (ActiveState distribution) NumPy: 1.6.0 PyFits: 2.4.0 If I downgrade to NumPy 1.5.0 and re-build everything then I do not see this problem, while PyFITS 2.2.2 + NumPy 1.6.0 did show this crash. I've put the troublesome file on: http://hea-www.harvard.edu/~aldcroft/tmp/glibc_detected.fits.gz This particular file is the output of the Chandra X-ray Center automated processing which converts raw Chandra telemetry to "Level-0 engineering data". I've read hundreds of thousands of such files with Python 2.6, PyFits 2.2.2 and NumPy 1.3.0 without issue. I recently upgraded Python, PyFits and NumPy and ran into this problem. Other similar Level-0 files were read OK with the new versions, but not the one in question. I did a significant amount of regression testing before doing this version upgrade and didn't see any differences, but my testing didn't hit this particular file type. For now I'm back at NumPy 1.5.0. Regards, Tom Aldcroft ----------------- ccosmos% python ActivePython 2.7.1.4 (ActiveState Software Inc.) based on Python 2.7.1 (r271:86832, Feb 7 2011, 11:30:54) [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pyfits >>> hdus = pyfits.open('glibc_detected.fits.gz') >>> dat = hdus[1].data >>> print dat *** glibc detected *** /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python: double free or corruption (!prev): 0x0000000017aea970 *** ======= Backtrace: ========= /lib64/libc.so.6[0x390d07245f] /lib64/libc.so.6(cfree+0x4b)[0x390d0728bb] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362a94f2] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736462061] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736475b27] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b973647739b] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4fd4)[0x494954] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x421f07] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x46e99f] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x68bf)[0x49623f] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5c52)[0x4955d2] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_CallObjectWithKeywords+0x55)[0x48ed15] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362873b4] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(_PyObject_Str+0x69)[0x449cc9] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Str+0x13)[0x449da3] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x449f7d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyFile_WriteObject+0x7f)[0x42e04f] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4b8d)[0x49450d] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCode+0x32)[0x4970e2] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveOneFlags+0x190)[0x4b9e10] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveLoopFlags+0x4e)[0x4ba01e] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_AnyFileExFlags+0x4c)[0x4ba12c] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(Py_Main+0xa0e)[0x415bee] /lib64/libc.so.6(__libc_start_main+0xf4)[0x390d01d994] /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x414df9] ======= Memory map: ======== 00400000-0054b000 r-xp 00000000 08:04 106168389 /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python 0064b000-0068f000 rw-p 0014b000 08:04 106168389 /data/cosmos2/ska/arcAbort From sienkiew at stsci.edu Tue Jul 5 13:45:27 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 05 Jul 2011 13:45:27 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E0D20A4.7090507@du.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> Message-ID: <4E134DB7.8030404@stsci.edu> Brian Kloppenborg wrote: > I am, however, a little concerned that the current vision statement > seems to exclude the use of existing officially sanctioned (i.e. by the > IAU) routines that are written in other languages. > I don't see that the vision statement excludes this case. I only see where it says that these other routines will not be REQUIRED in order for Astropy to work at all. Are you proposing that I must install the SOFA software before I can install Astropy? Are you proposing that I must agree to the SOFA terms of use (which are more restrictive than GPL or BSD licensing) before I can use Astropy? If this is the case, a user might be reluctant even to try Astropy because of the complications involved. I can tell you from first hand experience that I have decide not to even evaluate A LOT of free software projects because of excessive number/difficulty of prerequisites. If not, are you proposing that Astropy contain a package that can make use of pysofa for optional features? That is, Astropy can work without pysofa, but may be able to do more things if pysofa is present? I think this case is explicitly called out in the vision statement. If you look in the current astrolib.pywcs, you'll see that it makes use of a wcs library by incorporating the entire source code of the third party library. This is particularly convenient for users because they can just install the python package and expect it to work. This is a way that you can make third party routines available without creating additional dependencies, though it does involve some work for the developer. (But then, that is what developers do, right? They do some extra work to make things easier for the user.) So, I don't think that the kind of thing you are looking for is explicitly excluded, but there are some complications involved. Mark p.s. Specifically for SOFA, you might want some way for a python program to tell you whether it used any part of SOFA during this run, and is therefore affected by the SOFA license requirements. From rowen at uw.edu Tue Jul 5 19:15:02 2011 From: rowen at uw.edu (Russell Owen) Date: Tue, 5 Jul 2011 16:15:02 -0700 Subject: [AstroPy] "double free or corruption" crash: PyFITS + NumPy 1.6.0 In-Reply-To: References: Message-ID: I don't see this on my MacOS X 10.6 box. I can load the file just fine. However, I've been getting memory problems with numpy and Python 2.7 as well. I've reluctantly decided to go back to Python 2.6 to release applications that others can reliably use. I spent some time trying to reproduce the problem today and found one case that reliably causes numpy.subtract to segfault on MacOS X 10.5 (pyfits is not involved), but so far no luck on 10.6. I submitted a ticket. Regards, -- Russell On Jul 3, 2011, at 1:02 PM, Tom Aldcroft wrote: > I'm seeing some sort of memory or allocation error (glibc detected) > when trying to read a particular FITS file with the following setup: > > OS: CentOS-5.6 (64-bit) > Python: 2.7.1.4 (ActiveState distribution) > NumPy: 1.6.0 > PyFits: 2.4.0 > > If I downgrade to NumPy 1.5.0 and re-build everything then I do not > see this problem, while PyFITS 2.2.2 + NumPy 1.6.0 did show this > crash. I've put the troublesome file on: > > http://hea-www.harvard.edu/~aldcroft/tmp/glibc_detected.fits.gz > > This particular file is the output of the Chandra X-ray Center > automated processing which converts raw Chandra telemetry to "Level-0 > engineering data". I've read hundreds of thousands of such files with > Python 2.6, PyFits 2.2.2 and NumPy 1.3.0 without issue. I recently > upgraded Python, PyFits and NumPy and ran into this problem. Other > similar Level-0 files were read OK with the new versions, but not the > one in question. I did a significant amount of regression testing > before doing this version upgrade and didn't see any differences, but > my testing didn't hit this particular file type. For now I'm back at > NumPy 1.5.0. > > Regards, > Tom Aldcroft > > ----------------- > > ccosmos% python > ActivePython 2.7.1.4 (ActiveState Software Inc.) based on > Python 2.7.1 (r271:86832, Feb 7 2011, 11:30:54) > [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import pyfits >>>> hdus = pyfits.open('glibc_detected.fits.gz') >>>> dat = hdus[1].data >>>> print dat > *** glibc detected *** > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python: double free > or corruption (!prev): 0x0000000017aea970 *** > ======= Backtrace: ========= > /lib64/libc.so.6[0x390d07245f] > /lib64/libc.so.6(cfree+0x4b)[0x390d0728bb] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362a94f2] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736462061] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736475b27] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b973647739b] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4fd4)[0x494954] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x421f07] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x46e99f] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x68bf)[0x49623f] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5c52)[0x4955d2] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_CallObjectWithKeywords+0x55)[0x48ed15] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362873b4] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(_PyObject_Str+0x69)[0x449cc9] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Str+0x13)[0x449da3] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x449f7d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyFile_WriteObject+0x7f)[0x42e04f] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4b8d)[0x49450d] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCode+0x32)[0x4970e2] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveOneFlags+0x190)[0x4b9e10] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveLoopFlags+0x4e)[0x4ba01e] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_AnyFileExFlags+0x4c)[0x4ba12c] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(Py_Main+0xa0e)[0x415bee] > /lib64/libc.so.6(__libc_start_main+0xf4)[0x390d01d994] > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x414df9] > ======= Memory map: ======== > 00400000-0054b000 r-xp 00000000 08:04 106168389 > /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python > 0064b000-0068f000 rw-p 0014b000 08:04 106168389 > /data/cosmos2/ska/arcAbort > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From rowen at uw.edu Tue Jul 5 20:03:04 2011 From: rowen at uw.edu (Russell Owen) Date: Tue, 5 Jul 2011 17:03:04 -0700 Subject: [AstroPy] "double free or corruption" crash: PyFITS + NumPy 1.6.0 In-Reply-To: References: Message-ID: <0DDF036F-7B64-41C7-8E6C-F7346DF5F5DD@uw.edu> Just for completeness: the error I see on MacOS X 10.6 is: Python(5086,0xa08a0540) malloc: *** error for object 0x22c66c04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug but I've not yet tracked it down far enough to submit a ticket. -- Russell On Jul 5, 2011, at 4:15 PM, Russell Owen wrote: > I don't see this on my MacOS X 10.6 box. I can load the file just fine. > > However, I've been getting memory problems with numpy and Python 2.7 as well. I've reluctantly decided to go back to Python 2.6 to release applications that others can reliably use. I spent some time trying to reproduce the problem today and found one case that reliably causes numpy.subtract to segfault on MacOS X 10.5 (pyfits is not involved), but so far no luck on 10.6. I submitted a ticket. > > Regards, > > -- Russell > > On Jul 3, 2011, at 1:02 PM, Tom Aldcroft wrote: > >> I'm seeing some sort of memory or allocation error (glibc detected) >> when trying to read a particular FITS file with the following setup: >> >> OS: CentOS-5.6 (64-bit) >> Python: 2.7.1.4 (ActiveState distribution) >> NumPy: 1.6.0 >> PyFits: 2.4.0 >> >> If I downgrade to NumPy 1.5.0 and re-build everything then I do not >> see this problem, while PyFITS 2.2.2 + NumPy 1.6.0 did show this >> crash. I've put the troublesome file on: >> >> http://hea-www.harvard.edu/~aldcroft/tmp/glibc_detected.fits.gz >> >> This particular file is the output of the Chandra X-ray Center >> automated processing which converts raw Chandra telemetry to "Level-0 >> engineering data". I've read hundreds of thousands of such files with >> Python 2.6, PyFits 2.2.2 and NumPy 1.3.0 without issue. I recently >> upgraded Python, PyFits and NumPy and ran into this problem. Other >> similar Level-0 files were read OK with the new versions, but not the >> one in question. I did a significant amount of regression testing >> before doing this version upgrade and didn't see any differences, but >> my testing didn't hit this particular file type. For now I'm back at >> NumPy 1.5.0. >> >> Regards, >> Tom Aldcroft >> >> ----------------- >> >> ccosmos% python >> ActivePython 2.7.1.4 (ActiveState Software Inc.) based on >> Python 2.7.1 (r271:86832, Feb 7 2011, 11:30:54) >> [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import pyfits >>>>> hdus = pyfits.open('glibc_detected.fits.gz') >>>>> dat = hdus[1].data >>>>> print dat >> *** glibc detected *** >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python: double free >> or corruption (!prev): 0x0000000017aea970 *** >> ======= Backtrace: ========= >> /lib64/libc.so.6[0x390d07245f] >> /lib64/libc.so.6(cfree+0x4b)[0x390d0728bb] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362a94f2] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736462061] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b9736475b27] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/umath.so[0x2b973647739b] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4fd4)[0x494954] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x421f07] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x46e99f] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x68bf)[0x49623f] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5c52)[0x4955d2] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x5bbd)[0x49553d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x4f7717] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Call+0x62)[0x41a032] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_CallObjectWithKeywords+0x55)[0x48ed15] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b97362873b4] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(_PyObject_Str+0x69)[0x449cc9] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyObject_Str+0x13)[0x449da3] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x449f7d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyFile_WriteObject+0x7f)[0x42e04f] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalFrameEx+0x4b8d)[0x49450d] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCodeEx+0x675)[0x496de5] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyEval_EvalCode+0x32)[0x4970e2] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveOneFlags+0x190)[0x4b9e10] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_InteractiveLoopFlags+0x4e)[0x4ba01e] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(PyRun_AnyFileExFlags+0x4c)[0x4ba12c] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python(Py_Main+0xa0e)[0x415bee] >> /lib64/libc.so.6(__libc_start_main+0xf4)[0x390d01d994] >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python[0x414df9] >> ======= Memory map: ======== >> 00400000-0054b000 r-xp 00000000 08:04 106168389 >> /data/cosmos2/ska/arch/x86_64-linux_CentOS-5/bin/python >> 0064b000-0068f000 rw-p 0014b000 08:04 106168389 >> /data/cosmos2/ska/arcAbort >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From oneaufs at gmail.com Wed Jul 6 02:19:21 2011 From: oneaufs at gmail.com (Prasanth) Date: Wed, 6 Jul 2011 11:49:21 +0530 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E134DB7.8030404@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: Hello, > Are you proposing that I must agree to the SOFA terms > of use (which are more restrictive than GPL or BSD licensing) before I > can use Astropy? > I think that the SOFA license and terms of use are much more liberal than GPL. The only requirement is that the developer retains the copyright notice, and none of the derived components carry the prefix "iau". It can be used for commercial purposes. Also, I couldn't find anything in the SOFA license that prevents a developer from even making a closed source application. Prasanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From astropy at liska.ath.cx Wed Jul 6 04:06:20 2011 From: astropy at liska.ath.cx (Ole Streicher) Date: Wed, 06 Jul 2011 10:06:20 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E134DB7.8030404@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: <4E14177C.5090707@liska.ath.cx> Am 05.07.2011 19:45, schrieb Mark Sienkiewicz: > If you look in the current astrolib.pywcs, you'll see that it makes use > of a wcs library by incorporating the entire source code of the third > party library. This is particularly convenient for users because they > can just install the python package and expect it to work. This is a > way that you can make third party routines available without creating > additional dependencies, though it does involve some work for the > developer. (But then, that is what developers do, right? They do some > extra work to make things easier for the user.) I am not completely happy with this: On some systems, the third-party-package may be already installed, or may be installed with the package management system. For pywcs, there are already wcslib packages available for Fedora and Ubuntu, so there would be no need to install them again -- resolving the dependency is done by the package management system. Another point is that one may run into difficulties since bug fixes in the original package take some time to find their way into the python package. For this reason, it would be good if there was a choice of using the original package in case it is already installed instead of the one that was packed with the python package. This would make the life for Linux (Ubuntu, Fedora, OpenSUSE) package creators much easier. Best regards Ole From erik.tollerud at gmail.com Wed Jul 6 04:56:06 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Wed, 6 Jul 2011 01:56:06 -0700 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: I suspect the most challenging part of the SOFA license is this clause: "The source code of your derived work must contain descriptions of how the derived work is based upon, contains and/or differs from the original SOFA software." This implies that all work derived from it needs to describe explicitly how SOFA is used/altered. I suspect that's incompatible with most other open source licenses (although I'm certainly no lawyer, and for that matter I don't know if the SOFA board or IAU pay attention to this too closely). Regardless, though, Mark's interpretation is the one we had in mind for the vision - external C libraries could be used in Astropy, but as an optional feature, not as a requirement for the main functionality. The particular instance of SOFA, though, is perhaps an example of the problem we facing in wrapping existing libraries. IMHO, they are usually quite un-pythonic in both implementation and interface, and wrappings typically follow similar forms (e.g. pysofa follows the sofa interface very closely). If the goal here is a python library and framework, we want to leverage the advantages of the language itself in terms of style and programming idioms. And especially given the compilation/packaging issues noted by Mark, Ole, and many others, it has to be worth the trouble to write wrappers that end up with such in interface... SWIG is great at making direct wrappings, but I haven't really seen it go beyond that. More recent schemes like ctypes or Cython seem to be easier to compile and are easier to make more pythonic, but at some point it becomes more efficient to just implement the algorithm using numpy (possibly with the help of a small C extension if speed is a problem). This doesn't mean there isn't a place for such official libraries, though - they certainly should be involved in the debugging and test suite of the Astropy implementations. After all, they are meant to be reference implementations. And if an externally sanctioned/supported library really can be wrapped in a way that matches the community's needs and is simple and portable to compile/package, it certainly could have a place in the core. I personally am just a bit skeptical that this can or often will be done. On Tue, Jul 5, 2011 at 11:19 PM, Prasanth wrote: > Hello, > >> >> Are you proposing that I must agree to the SOFA terms >> of use (which are more restrictive than GPL or BSD licensing) before I >> can use Astropy? > > > I think that the SOFA license and terms of use are much more liberal than > GPL. The only requirement is that the developer retains the copyright > notice, and none of the derived components carry the prefix "iau". It can be > used for commercial purposes. Also, I couldn't find anything in the SOFA > license that prevents a developer from even making a closed source > application. > Prasanth > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- Erik Tollerud From oneaufs at gmail.com Wed Jul 6 07:02:08 2011 From: oneaufs at gmail.com (Prasanth) Date: Wed, 6 Jul 2011 16:32:08 +0530 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: Hello, The particular instance of SOFA, though, is perhaps an example of the > problem we facing in wrapping existing libraries. IMHO, they are > usually quite un-pythonic in both implementation and interface, and > wrappings typically follow similar forms (e.g. pysofa follows the sofa > interface very closely). If the goal here is a python library and > framework, we want to leverage the advantages of the language itself > in terms of style and programming idioms. And especially given the > compilation/packaging issues noted by Mark, Ole, and many others, it > has to be worth the trouble to write wrappers that end up with such in > interface... SWIG is great at making direct wrappings, but I haven't > really seen it go beyond that. More recent schemes like ctypes or > Cython seem to be easier to compile and are easier to make more > pythonic, but at some point it becomes more efficient to just > implement the algorithm using numpy (possibly with the help of a small > C extension if speed is a problem). > > In my experience, the pythonic part of the implementation essentially comes down to having proper Python objects that map the data structures used by the C library. It is quite easy to come up with such objects in Cython. I switched to Cython after finding this easier than SWIG. Need for other objects will depend on the application that uses the library. For example, an application may need a Position object, where as this may not be necessary for just wrapping the library. The Position object methods can call wrapped functions, for example to convert angle to HMS format or calculate separation between two points. I have an example of using objects, that wrap C structures, from the PyTPM library, which is an interface to the TPM library. The equivalent C code is at this url. This is just a wrapper to the library and not a full astrometry application, and hence don't have objects that are not needed for calling C functions. This is also pretty fast. I use this script to read in the Hipparcos data set into IPython (the data i/o takes a while since I am doing simple list.append operations), and then ran the %timeit magic command, to perform FK5 J2000 to Galactic conversion. This was done on a laptop running Ubuntu 11.04 and IPython .10 with Python 2.6. >>> %timeit convert.convert(ra=raj2000, dec=dej2000,s2=tpm.TPM_S04) 1 loops, best of 3: 4.86 s per loop >>> %timeit convert.convert(ra=raj2000, dec=dej2000,s2=tpm.TPM_S04) 1 loops, best of 3: 4.97 s per loop >>> %timeit convert.convert(ra=raj2000, dec=dej2000,s2=tpm.TPM_S04) 1 loops, best of 3: 5.21 s per loop >>> %timeit convert.convert(ra=raj2000, dec=dej2000,s2=tpm.TPM_S04) 1 loops, best of 3: 4.95 s per loop >>> %timeit convert.convert(ra=raj2000, dec=dej2000,s2=tpm.TPM_S04) 1 loops, best of 3: 5.1 s per loop The PyTPM library does not use Numpy, but since it simply iterates over input values passing Numpy 1D arrays will cause no problem. I do this in the unofficial /unsanctioned fork of the astrolib.coords package, which includes PyTPM, available here . The code there is fast as well. This package would be an example of an astrometry application, something that can be added to astropy, and hence needs to have objects etc., that make it easier to use. As of now it just has some functions. Prasanth On Tue, Jul 5, 2011 at 11:19 PM, Prasanth wrote: > > Hello, > > > >> > >> Are you proposing that I must agree to the SOFA terms > >> of use (which are more restrictive than GPL or BSD licensing) before I > >> can use Astropy? > > > > > > I think that the SOFA license and terms of use are much more liberal than > > GPL. The only requirement is that the developer retains the copyright > > notice, and none of the derived components carry the prefix "iau". It can > be > > used for commercial purposes. Also, I couldn't find anything in the SOFA > > license that prevents a developer from even making a closed source > > application. > > Prasanth > > _______________________________________________ > > AstroPy mailing list > > AstroPy at scipy.org > > http://mail.scipy.org/mailman/listinfo/astropy > > > > > > > > -- > Erik Tollerud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Wed Jul 6 07:06:08 2011 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Wed, 6 Jul 2011 07:06:08 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: On Wed, Jul 6, 2011 at 4:56 AM, Erik Tollerud wrote: > I suspect the most challenging part of the SOFA license is this > clause: "The source code of your derived work must contain > descriptions of how the derived work is based upon, contains and/or > differs from the original SOFA software." This implies that all work > derived from it needs to describe explicitly how SOFA is used/altered. > I suspect that's incompatible with most other open source licenses > (although I'm certainly no lawyer, and for that matter I don't know if > the SOFA board or IAU pay attention to this too closely). This publication clause seems a little worrying here because (as I believe Mark alluded) there needs to be way for users of astropy to know they they used the SOFA library: "In any published work or commercial products which includes results achieved by using the SOFA software, you shall acknowledge that the SOFA software was used in obtaining those results." Of course this is a somewhat theoretical matter, in reality astronomers rarely acknowledge the software they used and nobody is going to sue them for this. I don't believe the intent of the SOFA license is that every code that calls unaltered SOFA becomes tainted with the SOFA license, but rather to make sure that no user distributes C/Fortran routines that *look* like official SOFA-endorsed algorithms but are actually altered. > Regardless, though, Mark's interpretation is the one we had in mind > for the vision - external C libraries could be used in Astropy, but as > an optional feature, not as a requirement for the main functionality. > > The particular instance of SOFA, though, is perhaps an example of the > problem we facing in wrapping existing libraries. IMHO, they are > usually quite un-pythonic in both implementation and interface, and > wrappings typically follow similar forms (e.g. pysofa follows the sofa > interface very closely). If the goal here is a python library and > framework, we want to leverage the advantages of the language itself > in terms of style and programming idioms. > Assuming that astropy will include a full-featured date and time capability in the core, there is a lot of useful functionality in SOFA. Taking advantage of legacy, well-tested software is a good thing (libwcs, LAPACK, etc) and can save a lot of developer time. There is certainly an opportunity to put a nicer Pythonic interface around the library, and this is the kind of project that could attract developers (as opposed to the tedious mechanics of doing time conversions and getting it right). I should make it clear that I'm not advocating the SOFA library per se (maybe there is another library that has similar functionality and more agreeable license terms), but just being practical about using legacy code where it makes sense. There will always be a trade-off between time spent writing something from scratch and the pain of repackaging existing code. Maybe someone will step up in a big way and volunteer to re-implement the SOFA functionality in Python?? - Tom From fred.grollier at gmail.com Wed Jul 6 08:07:27 2011 From: fred.grollier at gmail.com (=?iso-8859-1?Q?Fr=E9d=E9ric?= Grollier) Date: Wed, 6 Jul 2011 14:07:27 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: <20110706120726.GA6523@free.fr> In my opinion, development/packaging/deployment problems when using third-party libraries are quite easily addressed, and the real point on this is just about license. That said, I can't find anything in the "vision" document about licensing of the proposed astropy software, and I guess this discussion can continue endlessly until something has been decided. I'm pretty sure almost everyone has "something free" in mind, but the devil is in the details... On the other hand, there's the possibility to make the reverse choice: we decide that we *want* to use SOFA/NOVAS/TMP/whatever, so we pick the type of license which is appropriate (note that I'm not very fond of this approach). Regarding this particular SOFA example, I agree with Tom and Mark that the viral "acknowledgment statement" is something that may be a bit challenging to handle. On Wed, Jul 06, 2011 at 07:06:08AM -0400, Tom Aldcroft wrote: > Assuming that astropy will include a full-featured date and time > capability in the core, Arguably, having an astro-specialized datetime-like object which is able to handle switching between calendar representations, timescales, extended range of years, leap seconds, enhanced strftime(), etc. is clearly something that should be included in the core. (Once upon a time I wrote a C extension implementing such an object, but the design choices I made proved to be inefficient and difficult to code with, so was never published) Fred. From jturner at gemini.edu Wed Jul 6 08:27:46 2011 From: jturner at gemini.edu (James Turner) Date: Wed, 6 Jul 2011 08:27:46 -0400 Subject: [AstroPy] Proliferating py-astro-libs In-Reply-To: References: <4DF0FAC3.6050505@hs.uni-hamburg.de> <75493BE8-D47D-45E0-8293-4E0603349DE7@stsci.edu> <4DF647F9.2060409@gemini.edu> <4DF64C59.3010501@gemini.edu> <4DF678BF.3070505@gemini.edu> <2A288F6CE58C40CE9E0824285A9C82B6@gmail.com> <3296B214-71D7-4BEC-9EE3-F45690D15F7B@stsci.edu> <4DF8B2AE.9070505@stsci.edu> <4DF8C7FC.7010407@gemini.edu> Message-ID: <4E1454C2.5060502@gemini.edu> Hi everyone, I have added a poll for SciPy attendees to figure out the best time to meet up next week. Please select which day(s) you could make an astronomy/AstroPy discussion. The SciPy organizers are nominally expecting BoFs to be on the Wed and Thurs evenings, but I've included all the days in the poll, as well as Wed/Thurs lunchtimes (only select these if you can't make the evening), just in case another time is better. We can update the wiki and the conference page with a time and place once it's decided. Thanks! James. On 15/06/11 10:59, Thomas Robitaille wrote: >> Can I suggest adding a section to the wiki for SciPy atendees to >> add their name? I was going to add one last night, but noticed that >> the wiki says "email us to suggest a new section" or something >> like that. > > I created a page: > > http://astropy.wikispaces.com/SciPy+2011 > > In future, feel free to create this kind of page directly if you can > (not sure whether you need special permissions) > > Cheers, > Tom From jturner at gemini.edu Wed Jul 6 08:28:13 2011 From: jturner at gemini.edu (James Turner) Date: Wed, 6 Jul 2011 08:28:13 -0400 Subject: [AstroPy] Proliferating py-astro-libs In-Reply-To: References: <4DF0FAC3.6050505@hs.uni-hamburg.de> <75493BE8-D47D-45E0-8293-4E0603349DE7@stsci.edu> <4DF647F9.2060409@gemini.edu> <4DF64C59.3010501@gemini.edu> <4DF678BF.3070505@gemini.edu> <2A288F6CE58C40CE9E0824285A9C82B6@gmail.com> <3296B214-71D7-4BEC-9EE3-F45690D15F7B@stsci.edu> <4DF8B2AE.9070505@stsci.edu> <4DF8C7FC.7010407@gemini.edu> Message-ID: <4E1454DD.4050801@gemini.edu> Hi everyone, I have added a poll for SciPy attendees to figure out the best time to meet up next week. Please select which day(s) you could make an astronomy/AstroPy discussion. The SciPy organizers are nominally expecting BoFs to be on the Wed and Thurs evenings, but I've included all the days in the poll, as well as Wed/Thurs lunchtimes (only select these if you can't make the evening), just in case another time is better. http://astropy.wikispaces.com/SciPy+2011 We can update the wiki and the conference page with a time and place once it's decided. Thanks! James. On 15/06/11 10:59, Thomas Robitaille wrote: >> Can I suggest adding a section to the wiki for SciPy atendees to >> add their name? I was going to add one last night, but noticed that >> the wiki says "email us to suggest a new section" or something >> like that. > > I created a page: > > http://astropy.wikispaces.com/SciPy+2011 > > In future, feel free to create this kind of page directly if you can > (not sure whether you need special permissions) > > Cheers, > Tom From sienkiew at stsci.edu Wed Jul 6 09:58:21 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Wed, 06 Jul 2011 09:58:21 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: <4E1469FD.4020109@stsci.edu> Prasanth wrote: > Hello, > > > Are you proposing that I must agree to the SOFA terms > of use (which are more restrictive than GPL or BSD licensing) before I > can use Astropy? > > > I think that the SOFA license and terms of use are much more liberal > than GPL. The only requirement is that the developer retains the > copyright notice, and none of the derived components carry the prefix > "iau". It can be used for commercial purposes. Also, I couldn't find > anything in the SOFA license that prevents a developer from even > making a closed source application. I draw your attention to paragraph 4 of http://www.iausofa.org/tandc.html : 4. In any published work or commercial products which includes results achieved by using the SOFA software, you shall acknowledge that the SOFA software was used in obtaining those results. GPL wants to attach conditions to your program; SOFA wants to attach conditions to the data that your program processed. I consider that more restrictive. This is why I suggested that you would want the program to report on whether any SOFA routines were used or not -- if SOFA routines are used, then you have additional accounting/reporting requirements. Mark From sergiopr at fis.ucm.es Wed Jul 6 10:18:58 2011 From: sergiopr at fis.ucm.es (Sergio Pascual) Date: Wed, 6 Jul 2011 16:18:58 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E1469FD.4020109@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E1469FD.4020109@stsci.edu> Message-ID: Hi all Regarding the current SOFA discussion, I wouldn't agree in having a package that is not free and thus cannot be included precompiled in most linux distributions. I have sent a mail to the Fedora legal mailing list asking if software with this license is suitable to be included in Fedora. Perhaps we can ask other institutions, such as the Free Software Foundation or the Open Source Initiative (http://www.opensource.org/) in order to end our doubts. Regards, Sergio Sergio Pascual ? ? ? ? ? ? ? ? ? ? ? ?http://guaix.fis.ucm.es/~spr gpg fingerprint: 5203 B42D 86A0 5649 410A F4AC A35F D465 F263 BCCC Departamento de Astrof?sica -- Universidad Complutense de Madrid (Spain) From sienkiew at stsci.edu Wed Jul 6 10:21:44 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Wed, 06 Jul 2011 10:21:44 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E14177C.5090707@liska.ath.cx> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> Message-ID: <4E146F78.3090404@stsci.edu> Ole Streicher wrote: > Am 05.07.2011 19:45, schrieb Mark Sienkiewicz: > >> If you look in the current astrolib.pywcs, you'll see that it makes use >> of a wcs library by incorporating the entire source code of the third >> party library. This is particularly convenient for users because they >> can just install the python package and expect it to work. This is a >> way that you can make third party routines available without creating >> additional dependencies, though it does involve some work for the >> developer. (But then, that is what developers do, right? They do some >> extra work to make things easier for the user.) >> > > I am not completely happy with this: On some systems, the > third-party-package may be already installed, or may be installed with > the package management system. For pywcs, there are already wcslib > packages available for Fedora and Ubuntu, so there would be no need to > install them again -- resolving the dependency is done by the package > management system. Another point is that one may run into difficulties > since bug fixes in the original package take some time to find their way > into the python package. > I often have the opposite problem -- the package that the system provides is the one that is older and broken. For example, every one of my Solaris machines already has BLAS installed, and every one of them is missing a function that scipy needs. At times, I've had to make difficult hacks to autoconfigs to make it use the copy of some library that I provide. > For this reason, it would be good if there was a choice of using the > original package in case it is already installed instead of the one that > was packed with the python package. This would make the life for Linux > (Ubuntu, Fedora, OpenSUSE) package creators much easier. > If there is an ACTIVE choice made by the user, I agree. The user could choose to use the already-installed package if they need to do that. They can also take responsibility for using a library that our python code was never tested with. I would object if there is an automatic detector that prefers the system package, primarily because that is a scenario that causes problems for me all the time both as a user and as a support engineer. Mark From astropy at liska.ath.cx Wed Jul 6 11:03:59 2011 From: astropy at liska.ath.cx (Ole Streicher) Date: Wed, 06 Jul 2011 17:03:59 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E146F78.3090404@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> Message-ID: <4E14795F.9020204@liska.ath.cx> Am 06.07.2011 16:21, schrieb Mark Sienkiewicz: > I often have the opposite problem -- the package that the system > provides is the one that is older and broken. For example, every one of > my Solaris machines already has BLAS installed, and every one of them is > missing a function that scipy needs. At times, I've had to make > difficult hacks to autoconfigs to make it use the copy of some library > that I provide. Sorry, but I would call "Solaris" a really special case -- there is probably only a very small percentage of users in that use this system. IMO, the easy installations should run on common systems, which are the major Linux flavors and MacOSX. >> For this reason, it would be good if there was a choice of using the >> original package in case it is already installed instead of the one that >> was packed with the python package. This would make the life for Linux >> (Ubuntu, Fedora, OpenSUSE) package creators much easier. > If there is an ACTIVE choice made by the user, I agree. The user could > choose to use the already-installed package if they need to do that. I do not speak so much about "users", but about "package maintainers", who compile the software and create packages that can then be installed by the users, taking care of about all the dependencies one would need. For example, when I packaged pywcs for Ubuntu, I created in two packages: one with the wcslib (since wcslib was not available for Ubuntu), and another for the python wrapper itself. The latter package has the dependency for the lib; if needed one could even specify that a specific version should be installed. On systems that allow a smart package management, we should allow (and encourage) to use it as much as possible. Wouldn't his help for Solaris? > They can also take responsibility for using a library that our python > code was never tested with. I see the problem of incompatible libraries today much less than 10 years ago -- libraries tend to get better documented now, better tested and have more stable APIs. From this point of view, I dont see a big difference in updating a required external library, or the system's libm. BTW, are there ideas for regression tests of astropy? Especially in the case you mentioned, this would be *very* helpful to check that everything works in the target system. > I would object if there is an automatic detector that prefers the system > package, primarily because that is a scenario that causes problems for > me all the time both as a user and as a support engineer. My goal is that the use any external package that is used in astropy is well-documented (which versions are compatible, where to get it etc.) and if the external package is included in the python package, there should be a switch to use the external library instead. Best regards Ole From laidler at stsci.edu Wed Jul 6 11:35:41 2011 From: laidler at stsci.edu (Victoria G. Laidler) Date: Wed, 6 Jul 2011 11:35:41 -0400 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package) In-Reply-To: <4E14795F.9020204@liska.ath.cx> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> Message-ID: <4E1480CD.2060404@stsci.edu> Ole Streicher wrote: > BTW, are there ideas for regression tests of astropy? Especially in the > case you mentioned, this would be *very* helpful to check that > everything works in the target system. I was thinking about this as well. I suggest that there be minimum testing standards for astropy packages. Packages should come with tests that: - can be run using a standard syntax (what?) - verify that all significant elements of the package can successfully execute - verify that all significant elements of the package, for some minimum set of common use cases, give the right answer when run - come with documentation that in some way describe test coverage (how much of the system do the tests cover) and the source of the test answers (ie, how did you determine the "right answer" used in the tests) This last point is especially important if we want to get people to adopt astropy libraries in place of their current favorite tool. In this case, testing serves as a confidence builder: astropy package X is at least as good as OtherPackage Y. I'm using two phrases in the above that are meant to have a basic definition but also allow some wiggle room for package developers: - all significant elements of the package: the basic parts that most people who use the package at all will use, similar to the dependency discussion) - minimum set of common use cases: the typical use cases that most people will encounter This allows a package to be included in astropy without requiring the developer to provide an exhaustive set of tests that verify correct behavior in every possible corner of the domain space. This is especially important for software whose behavior is data-dependent. Of course, more tests are better, and would produce better coverage. Vicki Laidler From bkloppenborg at gmail.com Wed Jul 6 11:52:27 2011 From: bkloppenborg at gmail.com (Brian Kloppenborg) Date: Wed, 06 Jul 2011 09:52:27 -0600 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package) In-Reply-To: <4E1480CD.2060404@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1480CD.2060404@stsci.edu> Message-ID: <4E1484BB.1010403@du.edu> On 07/06/2011 09:35 AM, Victoria G. Laidler wrote: > Ole Streicher wrote: >> BTW, are there ideas for regression tests of astropy? Especially in the >> case you mentioned, this would be *very* helpful to check that >> everything works in the target system. > I was thinking about this as well. I suggest that there be minimum > testing standards for astropy packages. Packages should come with tests > that: > - can be run using a standard syntax (what?) > - verify that all significant elements of the package can successfully > execute > - verify that all significant elements of the package, for some minimum > set of common use cases, give the right answer when run > - come with documentation that in some way describe test coverage (how > much of the system do the tests cover) and the source of the test > answers (ie, how did you determine the "right answer" used in the tests) There is a nice unit testing package in python that could be used for this: http://docs.python.org/library/unittest.html I use it quite frequently on larger projects. Implementing is as simple as a series of Assert statements whose outcome is known (i.e. hard-coded). Brian From laidler at stsci.edu Wed Jul 6 12:01:43 2011 From: laidler at stsci.edu (Victoria G. Laidler) Date: Wed, 6 Jul 2011 12:01:43 -0400 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package) In-Reply-To: <4E1484BB.1010403@du.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1480CD.2060404@stsci.edu> <4E1484BB.1010403@du.edu> Message-ID: <4E1486E7.4080609@stsci.edu> Brian Kloppenborg wrote: > > > On 07/06/2011 09:35 AM, Victoria G. Laidler wrote: >> Ole Streicher wrote: >>> BTW, are there ideas for regression tests of astropy? Especially in the >>> case you mentioned, this would be *very* helpful to check that >>> everything works in the target system. >> I was thinking about this as well. I suggest that there be minimum >> testing standards for astropy packages. Packages should come with tests >> that: >> - can be run using a standard syntax (what?) >> - verify that all significant elements of the package can successfully >> execute >> - verify that all significant elements of the package, for some minimum >> set of common use cases, give the right answer when run >> - come with documentation that in some way describe test coverage (how >> much of the system do the tests cover) and the source of the test >> answers (ie, how did you determine the "right answer" used in the tests) > There is a nice unit testing package in python that could be used for > this: > http://docs.python.org/library/unittest.html > > I use it quite frequently on larger projects. Implementing is as > simple as a series of Assert statements whose outcome is known (i.e. > hard-coded). Unittest has recently undergone a major rewrite for Python 2.7. There are some backwards incompatibilities, so we'd need to be somewhat careful here. Unittest is also a fairly primitive testing framework (and deliberately written to match the Java JUnit package, which isn't always a very pythonic way of doing things). There are a variety of test-related packages out there, many of them summarized here: http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy although I suppose once we go outside the standard library, the issue of "which of these tools will astropy ship with to support its internal testing" becomes important. :/ Although, another possible approach is to segment the tests, so you can run the minimum tests (described above) without installing anything extra, and additional tests if you feel like also installing whatever the package developer likes for testing. There's also a very active Testing in Python mailing list, to which you can subscribe from the taxonomy page. Vicki From oneaufs at gmail.com Wed Jul 6 12:35:28 2011 From: oneaufs at gmail.com (Prasanth) Date: Wed, 6 Jul 2011 22:05:28 +0530 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E1469FD.4020109@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E1469FD.4020109@stsci.edu> Message-ID: Hello, > I draw your attention to paragraph 4 of http://www.iausofa.org/tandc.** > html : > > 4. In any published work or commercial products which includes results > achieved by using the SOFA software, you shall acknowledge that the SOFA > software was used in obtaining those results. > > > Isn't this similar to the conditions for using software such as IRAF and data sources such as 2MASS? I am not saying that SOFA should become part of astropy and this is not an assertion but is a question. Perhaps, in the former cases users are not required to do this? Prasanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From embray at stsci.edu Wed Jul 6 12:55:06 2011 From: embray at stsci.edu (Erik Bray) Date: Wed, 6 Jul 2011 12:55:06 -0400 Subject: [AstroPy] Proliferating py-astro-libs In-Reply-To: <4E1454DD.4050801@gemini.edu> References: <4DF0FAC3.6050505@hs.uni-hamburg.de> <75493BE8-D47D-45E0-8293-4E0603349DE7@stsci.edu> <4DF647F9.2060409@gemini.edu> <4DF64C59.3010501@gemini.edu> <4DF678BF.3070505@gemini.edu> <2A288F6CE58C40CE9E0824285A9C82B6@gmail.com> <3296B214-71D7-4BEC-9EE3-F45690D15F7B@stsci.edu> <4DF8B2AE.9070505@stsci.edu> <4DF8C7FC.7010407@gemini.edu> <4E1454DD.4050801@gemini.edu> Message-ID: <4E14936A.5030700@stsci.edu> On 07/06/2011 08:28 AM, James Turner wrote: > Hi everyone, > > I have added a poll for SciPy attendees to figure out the best time > to meet up next week. Please select which day(s) you could make an > astronomy/AstroPy discussion. The SciPy organizers are nominally > expecting BoFs to be on the Wed and Thurs evenings, but I've included > all the days in the poll, as well as Wed/Thurs lunchtimes (only > select these if you can't make the evening), just in case another > time is better. > > http://astropy.wikispaces.com/SciPy+2011 > > We can update the wiki and the conference page with a time and place > once it's decided. > > Thanks! > > James. From looking at the poll currently it seems like Tuesday or Wednesday evening are probably the best bet. Thanks for doing this--I know you had e-mailed me a while back about this, but I got buried in other things and dropped the ball on it. Erik From embray at stsci.edu Wed Jul 6 13:06:40 2011 From: embray at stsci.edu (Erik Bray) Date: Wed, 6 Jul 2011 13:06:40 -0400 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package) In-Reply-To: <4E1486E7.4080609@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1480CD.2060404@stsci.edu> <4E1484BB.1010403@du.edu> <4E1486E7.4080609@stsci.edu> Message-ID: <4E149620.6000903@stsci.edu> On 07/06/2011 12:01 PM, Victoria G. Laidler wrote: > although I suppose once we go outside the standard library, the issue of > "which of these tools will astropy ship with to support its internal > testing" becomes important. :/ Although, another possible approach is to > segment the tests, so you can run the minimum tests (described above) > without installing anything extra, and additional tests if you feel like > also installing whatever the package developer likes for testing. > > There's also a very active Testing in Python mailing list, to which you > can subscribe from the taxonomy page. > > Vicki As a side note on testing: setuptools/distribute supports on option in setup.py files called 'tests_require' (or something like that) that lists all required packages (nose, unittest2, etc) for running that particular project's tests. Those requirements won't be installed unless the user wants to run the tests. There are also discussions for adding a similar feature in distutils2/packaging, though the details haven't been finalized yet. Point being, there are ways to solve that particular issue... Erik From perry at stsci.edu Wed Jul 6 13:55:44 2011 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 6 Jul 2011 13:55:44 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E1469FD.4020109@stsci.edu> Message-ID: <4BFB25EF-EA6E-45F6-9652-0D5E7F428D0E@stsci.edu> Here is the current IRAF statement with regard to that: (http://iraf.noao.edu/faq/FAQsec01.html#1008) Is IRAF public domain software? Although IRAF is available free of charge over the network to anyone who may find it useful it is not considered public domain software since it is copyrighted. Here is part of the copyright notice that appears in the distributed system. ----------------- Copyright(c) 1986 Association of Universities for Research in Astronomy Inc. The IRAF software is publicly available, but is NOT in the public domain. The difference is that copyrights granting rights for unrestricted use and redistribution have been placed on all of the software to identify its authors. You are allowed and encouraged to take this software and use it as you wish, subject to the restrictions outlined below. Permission to use, copy, modify, and distribute this software and its documentation is hereby granted without fee, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that references to the Association of Universities for Research in Astronomy Inc. (AURA), the National Optical Astronomy Observatories (NOAO), or the Image Reduction and Analysis Facility (IRAF) not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission from NOAO. NOAO makes no representations about the suitability of this software for any purpose. It is provided "as is" without express or implied warranty. NOAO DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL NOAO BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ****** So no, there is no similar requirement for IRAF that you must acknowledge its use. Perry On Jul 6, 2011, at 12:35 PM, Prasanth wrote: > Hello, > > > I draw your attention to paragraph 4 of http://www.iausofa.org/tandc.html > : > > 4. In any published work or commercial products which includes > results achieved by using the SOFA software, you shall acknowledge > that the SOFA software was used in obtaining those results. > > > Isn't this similar to the conditions for using software such as IRAF > and data sources such as 2MASS? I am not saying that SOFA should > become part of astropy and this is not an assertion but is a > question. Perhaps, in the former cases users are not required to do > this? > > Prasanth > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From perry at stsci.edu Wed Jul 6 14:06:41 2011 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 6 Jul 2011 14:06:41 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E14795F.9020204@liska.ath.cx> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> Message-ID: <555B4939-6E0D-4E82-A0A1-946F37CDE528@stsci.edu> On Jul 6, 2011, at 11:03 AM, Ole Streicher wrote: > Am 06.07.2011 16:21, schrieb Mark Sienkiewicz: >> I often have the opposite problem -- the package that the system >> provides is the one that is older and broken. For example, every >> one of >> my Solaris machines already has BLAS installed, and every one of >> them is >> missing a function that scipy needs. At times, I've had to make >> difficult hacks to autoconfigs to make it use the copy of some >> library >> that I provide. > > Sorry, but I would call "Solaris" a really special case -- there is > probably only a very small percentage of users in that use this > system. > IMO, the easy installations should run on common systems, which are > the > major Linux flavors and MacOSX. > Sure, Solaris is the extreme example, but not the only case. We do see similar issues on other platforms at non-negligible rates. MacOSX has this issue due partly to the number of different ways people may install unix-derived software. I'll also note that existing package dependency systems do not solve all dependency conflicts either. There is simply no substitute for testing known combinations of versions of libraries as opposed to hoping that the one you pick up on a particular system actually works with what you have. Things may be better now than in the past, but they sure are far from foolproof yet. Thus, bundling as much together (particularly the troublesome libraries) is about the only sure way of ensuring consistency. This is the approach that Guido has long advocated for ensuring reliable distributions (well he used to, I don't know he's since changed his mind). And if it doesn't collide with the other versions of the library on the system (e.g., you don't use ld_library_path or its equivalent), is the wasted disk space really an issue these days given terabyte drives are a dime a dozen (or will be soon :-) Perry From perry at stsci.edu Wed Jul 6 14:22:06 2011 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 6 Jul 2011 14:22:06 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: On Jul 6, 2011, at 7:06 AM, Tom Aldcroft wrote: > On Wed, Jul 6, 2011 at 4:56 AM, Erik Tollerud > wrote: >> I suspect the most challenging part of the SOFA license is this >> clause: "The source code of your derived work must contain >> descriptions of how the derived work is based upon, contains and/or >> differs from the original SOFA software." This implies that all work >> derived from it needs to describe explicitly how SOFA is used/ >> altered. >> I suspect that's incompatible with most other open source licenses >> (although I'm certainly no lawyer, and for that matter I don't know >> if >> the SOFA board or IAU pay attention to this too closely). > > This publication clause seems a little worrying here because (as I > believe Mark alluded) there needs to be way for users of astropy to > know they they used the SOFA library: "In any published work or > commercial products which includes results achieved by using the SOFA > software, you shall acknowledge that the SOFA software was used in > obtaining those results." Of course this is a somewhat theoretical > matter, in reality astronomers rarely acknowledge the software they > used and nobody is going to sue them for this. > > I don't believe the intent of the SOFA license is that every code that > calls unaltered SOFA becomes tainted with the SOFA license, but rather > to make sure that no user distributes C/Fortran routines that *look* > like official SOFA-endorsed algorithms but are actually altered. IANAL, but when it says: 3. You (the user) may copy and distribute SOFA source code to others, and use and adapt its code and algorithms in your own software, on a world-wide, royalty-free basis. That portion of your distribution that does not consist of intact and unchanged copies of SOFA source code files is a "derived work" that must comply with the following requirements: ? Your work shall be marked or carry a statement that it (i) uses routines and computations derived by you from software provided by SOFA under license to you; and (ii) does not itself constitute software provided by and/or endorsed by SOFA. ? The source code of your derived work must contain descriptions of how the derived work is based upon, contains and/or differs from the original SOFA software. ? The name(s) of all routine(s) in your derived work shall not include the prefix "iau". ? The origin of the SOFA components of your derived work must not be misrepresented; you must not claim that you wrote the original software, nor file a patent application for SOFA software or algorithms embedded in the SOFA software. ? These requirements must be reproduced intact in any source distribution and shall apply to anyone to whom you have granted a further right to modify the source code of your derived work. ******* The phrase '"derived work" that must comply with the following requirements' sure seems to applying new terms to the original license of the derived work. It's not clear that the SOFA license is GPL- compatible for example (the requirement on usage--i.e., publishing data using it requiring acknowledgement, or the iau prefix restriction for example). Perry From embray at stsci.edu Wed Jul 6 14:33:00 2011 From: embray at stsci.edu (Erik Bray) Date: Wed, 6 Jul 2011 14:33:00 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <555B4939-6E0D-4E82-A0A1-946F37CDE528@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <555B4939-6E0D-4E82-A0A1-946F37CDE528@stsci.edu> Message-ID: <4E14AA5C.8060104@stsci.edu> On 07/06/2011 02:06 PM, Perry Greenfield wrote: > > On Jul 6, 2011, at 11:03 AM, Ole Streicher wrote: > >> Am 06.07.2011 16:21, schrieb Mark Sienkiewicz: >>> I often have the opposite problem -- the package that the system >>> provides is the one that is older and broken. For example, every >>> one of >>> my Solaris machines already has BLAS installed, and every one of >>> them is >>> missing a function that scipy needs. At times, I've had to make >>> difficult hacks to autoconfigs to make it use the copy of some >>> library >>> that I provide. >> >> Sorry, but I would call "Solaris" a really special case -- there is >> probably only a very small percentage of users in that use this >> system. >> IMO, the easy installations should run on common systems, which are >> the >> major Linux flavors and MacOSX. >> > Sure, Solaris is the extreme example, but not the only case. We do see > similar issues on other platforms at non-negligible rates. MacOSX has > this issue due partly to the number of different ways people may > install unix-derived software. > > I'll also note that existing package dependency systems do not solve > all dependency conflicts either. There is simply no substitute for > testing known combinations of versions of libraries as opposed to > hoping that the one you pick up on a particular system actually works > with what you have. Things may be better now than in the past, but > they sure are far from foolproof yet. Thus, bundling as much together > (particularly the troublesome libraries) is about the only sure way of > ensuring consistency. This is the approach that Guido has long > advocated for ensuring reliable distributions (well he used to, I > don't know he's since changed his mind). And if it doesn't collide > with the other versions of the library on the system (e.g., you don't > use ld_library_path or its equivalent), is the wasted disk space > really an issue these days given terabyte drives are a dime a dozen > (or will be soon :-) > > Perry I'll add that in the case of pywcs, although it contains its own copy of wcslib, it only compiles the parts of it that it needs and bundles them in _pywcs.so. It does not rely on linking with a separately installed copy of wcslib. So in cases were users already have a wcslib installed by the system, this at least will not interfere with it in any way. I definitely agree that it's ideal to use the platform's packaging system to manage all dependencies, but it doesn't hurt to include the source code for troublesome dependencies as well. So long as it's easy for system packages to not use the bundled version if desired. In the case of pywcs it's mostly a matter of removing the wcslib portions from the _pywcs C extension and instead have it link _pywcs.so with the existing wcslib. Erik From aldcroft at head.cfa.harvard.edu Wed Jul 6 14:45:16 2011 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Wed, 6 Jul 2011 14:45:16 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> Message-ID: On Wed, Jul 6, 2011 at 2:22 PM, Perry Greenfield wrote: > > On Jul 6, 2011, at 7:06 AM, Tom Aldcroft wrote: > >> On Wed, Jul 6, 2011 at 4:56 AM, Erik Tollerud >> wrote: >>> >>> I suspect the most challenging part of the SOFA license is this >>> clause: "The source code of your derived work must contain >>> descriptions of how the derived work is based upon, contains and/or >>> differs from the original SOFA software." ?This implies that all work >>> derived from it needs to describe explicitly how SOFA is used/altered. >>> I suspect that's incompatible with most other open source licenses >>> (although I'm certainly no lawyer, and for that matter I don't know if >>> the SOFA board or IAU pay attention to this too closely). >> >> This publication clause seems a little worrying here because (as I >> believe Mark alluded) there needs to be way for users of astropy to >> know they they used the SOFA library: ? "In any published work or >> commercial products which includes results achieved by using the SOFA >> software, you shall acknowledge that the SOFA software was used in >> obtaining those results." ?Of course this is a somewhat theoretical >> matter, in reality astronomers rarely acknowledge the software they >> used and nobody is going to sue them for this. >> >> I don't believe the intent of the SOFA license is that every code that >> calls unaltered SOFA becomes tainted with the SOFA license, but rather >> to make sure that no user distributes C/Fortran routines that *look* >> like official SOFA-endorsed algorithms but are actually altered. > > > IANAL, but when it says: > > 3. You (the user) may copy and distribute SOFA source code to others, and > use and adapt its code and algorithms in your own software, on a world-wide, > royalty-free basis. That portion of your distribution that does not consist > of intact and unchanged copies of SOFA source code files is a "derived work" > that must comply with the following requirements: > ? ? ? ?? Your work shall be marked or carry a statement that it (i) uses > routines and computations derived by you from software provided by SOFA > under license to you; and (ii) does not itself constitute software provided > by and/or endorsed by SOFA. > ? ? ? ?? The source code of your derived work must contain descriptions of > how the derived work is based upon, contains and/or differs from the > original SOFA software. > ? ? ? ?? The name(s) of all routine(s) in your derived work shall not > include the prefix "iau". > ? ? ? ?? The origin of the SOFA components of your derived work must not be > misrepresented; you must not claim that you wrote the original software, nor > file a patent application for SOFA software or algorithms embedded in the > SOFA software. > ? ? ? ?? These requirements must be reproduced intact in any source > distribution and shall apply to anyone to whom you have granted a further > right to modify the source code of your derived work. > > ******* > > The phrase '"derived work" that must comply with the following requirements' > sure seems to applying new terms to the original license of the derived > work. It's not clear that the SOFA license is GPL-compatible for example > (the requirement on usage--i.e., publishing data using it requiring > acknowledgement, ?or the iau prefix restriction for example). > > Perry > Earlier today I sent a message to the contact email for the IAU SOFA tools asking them to look over this discussion thread and weigh in with their interpretation of their license agreement. Can't hurt to ask, and if it turns out they really intend this restrictive and viral license then it's everyone's loss. - Tom From erik.tollerud at gmail.com Wed Jul 6 15:29:56 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Wed, 6 Jul 2011 12:29:56 -0700 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package) In-Reply-To: <4E1480CD.2060404@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1480CD.2060404@stsci.edu> Message-ID: We (the Coordinating Committee) are finalizing a draft coding standards document that we will soon send out to the list. This does indeed include unit tests/regression tests, although the way we've phrased it is that tests are "encouraged for all public classes/methods/functions". The idea is that we don't want a strict requirement, but we (the committee) will reserve the right to not accept packages if they are not tested enough (and of course "enough" depends on how critical a given component is). As for the testing framework, we were thinking nose (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). Nose is very nice for a project like this because it's super easy to run and also easy to write tests with, but is also compatible with the stdlib unittests. I also think that the dependencies requirements for the tests need not be as strict as for the package itself (the vision does *not* restrict the requirements for running tests). Requiring an additional dependency or two is not a huge burden (especially with the 'test_requires' option Erik points out). And if we end up using external libraries for comparison (e.g. SOFA as the other thread mentions) which is probably a very good idea, that will be a much bigger pain anyway. That's also another reason to go with nose - it's easy to switch tests or groups of tests on and off with nose (i.e. if the user doesn't have SOFA installed or something). On Wed, Jul 6, 2011 at 8:35 AM, Victoria G. Laidler wrote: > Ole Streicher wrote: >> BTW, are there ideas for regression tests of astropy? Especially in the >> case you mentioned, this would be *very* helpful to check that >> everything works in the target system. > I was thinking about this as well. I suggest that there be minimum > testing standards for astropy packages. Packages should come with tests > that: > - can be run using a standard syntax (what?) > - verify that all significant elements of the package can successfully > execute > - verify that all significant elements of the package, for some minimum > set of common use cases, give the right answer when run > - come with documentation that in some way describe test coverage (how > much of the system do the tests cover) and the source of the test > answers (ie, how did you determine the "right answer" used in the tests) > > This last point is especially important if we want to get people to > adopt astropy libraries in place of their current favorite tool. In this > case, testing serves as a confidence builder: astropy package X is at > least as good as OtherPackage Y. > > I'm using two phrases in the above that are meant to have a basic > definition but also allow some wiggle room for package developers: > - all significant elements of the package: the basic parts that most > people who use the package at all will use, similar to the dependency > discussion) > - minimum set of common use cases: the typical use cases that most > people will encounter > > This allows a package to be included in astropy without requiring the > developer to provide an exhaustive set of tests that verify correct > behavior in every possible corner of the domain space. This is > especially important for software whose behavior is data-dependent. Of > course, more tests are better, and would produce better coverage. > > Vicki Laidler > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud From laidler at stsci.edu Wed Jul 6 22:17:12 2011 From: laidler at stsci.edu (Vicki Laidler) Date: Thu, 7 Jul 2011 02:17:12 +0000 Subject: [AstroPy] Thoughts on growing a productive open-source community Message-ID: <173C0F068F6E49449797E90DF0B59EF8B84CBA@EXCHMAIL1.stsci.edu> This essay by the Volunteer Development Coordinater for Wikimedia makes some interesting points that I thought might be useful as we try to organize around astropy, in particular as regards including people who are relative newbs -- in the case of astropy, that might be astronomers who have written their own one-off scripts or evolving programs for their own research before, but not done anything more formal in the way of software development. http://geekfeminism.org/2011/07/02/put-up-or-shut-up/ Anybody have links to other resources about growing a FLOSS community, or inspiring anecdotes or cautionary tales on the subject? From laidler at stsci.edu Thu Jul 7 12:44:10 2011 From: laidler at stsci.edu (Victoria G. Laidler) Date: Thu, 7 Jul 2011 12:44:10 -0400 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package)] Message-ID: <4E15E25A.80209@stsci.edu> Erik Tollerud wrote: > We (the Coordinating Committee) are finalizing a draft coding > standards document that we will soon send out to the list. This does > indeed include unit tests/regression tests, although the way we've > phrased it is that tests are "encouraged for all public > classes/methods/functions". The idea is that we don't want a strict > requirement, but we (the committee) will reserve the right to not > accept packages if they are not tested enough (and of course "enough" > depends on how critical a given component is). > I suggest adding a specific requirement for end to end testing of common use cases. Testing one method at a time doesn't guarantee that they all play together nicely as they should. > As for the testing framework, we were thinking nose > (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). Nose is > very nice for a project like this because it's super easy to run and > also easy to write tests with, but is also compatible with the stdlib > unittests. > Last I heard, nose had been more or less orphaned, because Kumar doesn't have time to continue maintenance so didn't want to commit to it going forward. There was some discussion about a TIP-community-driven successor that would be more cleanly compatible with Python 3, and I know something started in that direction, but I don't know the status. I can inquire. I've become less fond of nose the more I've used it for heavy testing, because its support for reasonably complex tests (eg, perform exactly the same tests for 25 different sets of input) is not very robust or easy to use. I know there are other frameworks out there that claim to do better at this, but haven't had time to investigate them. I should be able to get to that this summer. I've also added a section on "Testing" on the wiki so people can sign up to contribute efforts in this area. Vicki > > On Wed, Jul 6, 2011 at 8:35 AM, Victoria G. Laidler wrote: > >> Ole Streicher wrote: >> >>> BTW, are there ideas for regression tests of astropy? Especially in the >>> case you mentioned, this would be *very* helpful to check that >>> everything works in the target system. >>> >> I was thinking about this as well. I suggest that there be minimum >> testing standards for astropy packages. Packages should come with tests >> that: >> - can be run using a standard syntax (what?) >> - verify that all significant elements of the package can successfully >> execute >> - verify that all significant elements of the package, for some minimum >> set of common use cases, give the right answer when run >> - come with documentation that in some way describe test coverage (how >> much of the system do the tests cover) and the source of the test >> answers (ie, how did you determine the "right answer" used in the tests) >> >> This last point is especially important if we want to get people to >> adopt astropy libraries in place of their current favorite tool. In this >> case, testing serves as a confidence builder: astropy package X is at >> least as good as OtherPackage Y. >> >> I'm using two phrases in the above that are meant to have a basic >> definition but also allow some wiggle room for package developers: >> - all significant elements of the package: the basic parts that most >> people who use the package at all will use, similar to the dependency >> discussion) >> - minimum set of common use cases: the typical use cases that most >> people will encounter >> >> This allows a package to be included in astropy without requiring the >> developer to provide an exhaustive set of tests that verify correct >> behavior in every possible corner of the domain space. This is >> especially important for software whose behavior is data-dependent. Of >> course, more tests are better, and would produce better coverage. >> >> Vicki Laidler >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> > > > > From sienkiew at stsci.edu Thu Jul 7 15:07:16 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Thu, 07 Jul 2011 15:07:16 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E14795F.9020204@liska.ath.cx> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> Message-ID: <4E1603E4.6020208@stsci.edu> Ole Streicher wrote: > Am 06.07.2011 16:21, schrieb Mark Sienkiewicz: > >> I often have the opposite problem -- the package that the system >> provides is the one that is older and broken. For example, every one of >> my Solaris machines already has BLAS installed, and every one of them is >> missing a function that scipy needs. At times, I've had to make >> difficult hacks to autoconfigs to make it use the copy of some library >> that I provide. >> > > Sorry, but I would call "Solaris" a really special case -- I don't think Solaris has any kind of monopoly on crappy software. You can pretend I said "Mac OSX had a bad BLAS/LAPACK", or "autoconf found something that I didn't know that Fink installed", or "Linux had a too-old copy of freetype", or "my linux machine comes with subversion 1.1". All of these are issues that I've had to deal with. I'm talking about a large class of problems. That one particular library on one particular operating system was just an example. >> If there is an ACTIVE choice made by the user, I agree. The user could >> choose to use the already-installed package if they need to do that. >> > > I do not speak so much about "users", but about "package maintainers", > who compile the software and create packages that can then be installed > by the users, taking care of about all the dependencies one would need. > I absolutely need to talk about users. It is a mistake to think that users only ever install software from the packages that package maintainers are creating. Users typically know just enough to make their computer work, and (judging from the support tickets I handle) some of them don't even know that. You should expect that because they are concentrating on their application (e.g. astronomy) rather than the computer. For them, all the default choices should work as smoothly and easily as possible. > For example, when I packaged pywcs for Ubuntu, I created in two > packages: one with the wcslib (since wcslib was not available for > Ubuntu), and another for the python wrapper itself. The latter package > has the dependency for the lib; if needed one could even specify that a > specific version should be installed. On systems that allow a smart > package management, we should allow (and encourage) to use it as much as > possible. Wouldn't his help for Solaris? > It is not clear. I am quite reluctant to use things that automatically select software for me, because I have had them choose badly on so many occasions. >> I would object if there is an automatic detector that prefers the system >> package, primarily because that is a scenario that causes problems for >> me all the time both as a user and as a support engineer. >> > > My goal is that the use any external package that is used in astropy is > well-documented (which versions are compatible, where to get it etc.) > and if the external package is included in the python package, there > should be a switch to use the external library instead. > At the end, this is the solution that we can both agree on. You can tell it to use the external library when you want to, and I can tell it not to. As long as it responds correctly to our respective directives, it works well for everybody. Mark From pebarrett at gmail.com Thu Jul 7 21:03:42 2011 From: pebarrett at gmail.com (Paul Barrett) Date: Thu, 7 Jul 2011 21:03:42 -0400 Subject: [AstroPy] Tests for AstroPy (was POLL: vision for a common Astronomy package) In-Reply-To: References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1480CD.2060404@stsci.edu> Message-ID: I would suggest dividing tests into at least four categories: 1. unit tests: that are simple and quick, and test the class interface, 2. functional tests: that test the classes functionality, 3. regression tests: that test common use cases for the class and may require more time and processing power, and 4. integration tests: that test compatibility with other modules. The unit and functional tests should require little or no dependencies and therefore can be run at any time. The other two test categories may require dependencies and are therefore optional. I would recommend that the unit and functional tests be required for package acceptance. These are the categories that we are using for our project. In addition we need to report the percentage of statement and function coverage, which is another aspect of testing that we may want to consider. This sounds like a lot of testing, but if we want this code to become the standard library used by all other developers of astronomical software, then it should not be taken lightly. We want developers and users to have confidence in our libraries, so they will be readily adopted by the community. -- Paul On Wed, Jul 6, 2011 at 3:29 PM, Erik Tollerud wrote: > We (the Coordinating Committee) are finalizing a draft coding > standards document that we will soon send out to the list. ?This does > indeed include unit tests/regression tests, although the way we've > phrased it is that tests are "encouraged for all public > classes/methods/functions". ?The idea is that we don't want a strict > requirement, but we (the committee) will reserve the right to not > accept packages if they are not tested enough (and of course "enough" > depends on how critical a given component is). > > As for the testing framework, we were thinking nose > (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). ? Nose is > very nice for a project like this because it's super easy to run and > also easy to write tests with, but is also compatible with the stdlib > unittests. > > I also think that the dependencies requirements for the tests need not > be as strict as for the package itself (the vision does *not* restrict > the requirements for running tests). Requiring an additional > dependency or two is not a huge burden ?(especially with the > 'test_requires' option Erik points out). ?And if we end up using > external libraries for comparison (e.g. SOFA as the other thread > mentions) which is probably a very good idea, that will be a much > bigger pain anyway. ?That's also another reason to go with nose - it's > easy to switch tests or groups of tests on and off with nose (i.e. if > the user doesn't have SOFA installed or something). > > > On Wed, Jul 6, 2011 at 8:35 AM, Victoria G. Laidler wrote: >> Ole Streicher wrote: >>> BTW, are there ideas for regression tests of astropy? Especially in the >>> case you mentioned, this would be *very* helpful to check that >>> everything works in the target system. >> I was thinking about this as well. I suggest that there be minimum >> testing standards for astropy packages. Packages should come with tests >> that: >> - can be run using a standard syntax (what?) >> - verify that all significant elements of the package can successfully >> execute >> - verify that all significant elements of the package, for some minimum >> set of common use cases, give the right answer when run >> - come with documentation that in some way describe test coverage (how >> much of the system do the tests cover) and the source of the test >> answers (ie, how did you determine the "right answer" used in the tests) >> >> This last point is especially important if we want to get people to >> adopt astropy libraries in place of their current favorite tool. In this >> case, testing serves as a confidence builder: astropy package X is at >> least as good as OtherPackage Y. >> >> I'm using two phrases in the above that are meant to have a basic >> definition but also allow some wiggle room for package developers: >> - all significant elements of the package: the basic parts that most >> people who use the package at all will use, similar to the dependency >> discussion) >> - minimum set of common use cases: the typical use cases that most >> people will encounter >> >> This allows a package to be included in astropy without requiring the >> developer to provide an exhaustive set of tests that verify correct >> behavior in every possible corner of the domain space. This is >> especially important for software whose behavior is data-dependent. Of >> course, more tests are better, and would produce better coverage. >> >> Vicki Laidler >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From astropy at liska.ath.cx Fri Jul 8 04:04:25 2011 From: astropy at liska.ath.cx (Ole Streicher) Date: Fri, 08 Jul 2011 10:04:25 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E1603E4.6020208@stsci.edu> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1603E4.6020208@stsci.edu> Message-ID: <4E16BA09.1080102@liska.ath.cx> Am 07.07.2011 21:07, schrieb Mark Sienkiewicz: > I absolutely need to talk about users. It is a mistake to think that > users only ever install software from the packages that package > maintainers are creating. As long as the packages fullfill their needs, it is (from the user's point of view) the easiest solution. I think nothing can beat a "select astropy in the Ubuntu software center" or a one-click install on openSUSE; so the preferred way (from the user's view) should be even that. Today's Linux versions have a defined support cycle, and I would just stick to their philosophy for the default installation. > Users typically know just enough to make their computer work, and > (judging from the support tickets I handle) some of them don't even know > that. You should expect that because they are concentrating on their > application (e.g. astronomy) rather than the computer. For them, all > the default choices should work as smoothly and easily as possible. That's why I am propagating to make the installation as seamless as possible: use the standard package management system that is provided by the system as default. > I am quite reluctant to use things that automatically select software > for me, because I have had them choose badly on so many occasions. This may happen if the packages are broken. But since we are speaking about *creating* packages, we take care of this ourself, right? I think, "standard" for a user meas always "use the package management provided with the system": this is what the user is familar with, especially on Linux. > At the end, this is the solution that we can both agree on. You can > tell it to use the external library when you want to, and I can tell it > not to. As long as it responds correctly to our respective directives, > it works well for everybody. Agreed. Best regards Ole From mrdavis at stsci.edu Fri Jul 8 09:02:31 2011 From: mrdavis at stsci.edu (Matt Davis) Date: Fri, 8 Jul 2011 13:02:31 +0000 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E16BA09.1080102@liska.ath.cx> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1603E4.6020208@stsci.edu> <4E16BA09.1080102@liska.ath.cx> Message-ID: <351B1682-EF2C-4378-ACBD-091DA8DF8C22@stsci.edu> > That's why I am propagating to make the installation as seamless as > possible: use the standard package management system that is provided by > the system as default. As food for thought I'll just point out that Macs do not come with a package management system and the third party managers that exist tend to be under populated and/or not work very well. As a Mac user, package managers are not my first choice for installing software. In another thread on this list it has come up that package mangers often require admin privileges, preventing institutional users from installing from repos. We should support package managers, of course, but let's not make things overly difficult for people who must install manually. Best, Matt Davis From mrdavis at stsci.edu Fri Jul 8 09:02:32 2011 From: mrdavis at stsci.edu (Matt Davis) Date: Fri, 8 Jul 2011 13:02:32 +0000 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E16BA09.1080102@liska.ath.cx> References: <4E099F94.3060200@hs.uni-hamburg.de> <4E0D20A4.7090507@du.edu> <4E134DB7.8030404@stsci.edu> <4E14177C.5090707@liska.ath.cx> <4E146F78.3090404@stsci.edu> <4E14795F.9020204@liska.ath.cx> <4E1603E4.6020208@stsci.edu> <4E16BA09.1080102@liska.ath.cx> Message-ID: <6C34C8D7-CD88-40CF-A02B-EF9D507A4A92@stsci.edu> > That's why I am propagating to make the installation as seamless as > possible: use the standard package management system that is provided by > the system as default. I'll just point out as food for thought that Macs do not have a package management system provided by the system and the third party package managers that exist tend to be under-populated and/or not work very well. As a Mac user package managers are not my first choice for installing software. Something else that has come up on this mailing list recently is that package managers often require admin privileges which a lot of institutional users do not have. Best, Matt Davis From jslavin at cfa.harvard.edu Fri Jul 8 13:18:24 2011 From: jslavin at cfa.harvard.edu (Jonathan Slavin) Date: Fri, 08 Jul 2011 13:18:24 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package Message-ID: <1310145504.6158.15.camel@shevek> > I think, "standard" for a user meas always "use the package management > provided with the system": this is what the user is familar with, > especially on Linux. That's only true if you have root privileges -- which not all of us have. At home I use the package management system (rpm/yum) but at work that's not an option. I have been using "python setup.py install --user" method, which installs under ~/.local/lib/python/site-packages/, though I have heard about different (perhaps more robust) approaches. In any case, I think we should have an install option available for non-root users. Jon From nicholas.nell at colorado.edu Fri Jul 8 13:38:06 2011 From: nicholas.nell at colorado.edu (Nico Nell) Date: Fri, 8 Jul 2011 11:38:06 -0600 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <1310145504.6158.15.camel@shevek> References: <1310145504.6158.15.camel@shevek> Message-ID: I'd just like to make a small clarification for people that don't understand what the linux package managers are doing. The package manager is just executing a build script put together by a maintainer. It generally executes something along the lines of python setup.py . This script then handles dependencies, conflicts, paths, upgrades, etc. So the idea that package managers are supported in no way excludes a manual install. If anything it pretty much implies that a manual install method is available in the source tarball. The idea of supporting package managers for various operating systems is a step above the manual method. It's just the idea of trying to make it easy for a wide variety of users to get the software. So don't worry, there pretty much always has to be a manual install method. ~Nico On Fri, Jul 8, 2011 at 11:18 AM, Jonathan Slavin wrote: > >> I think, "standard" for a user meas always "use the package management >> provided with the system": this is what the user is familar with, >> especially on Linux. > > That's only true if you have root privileges -- which not all of us > have. ?At home I use the package management system (rpm/yum) but at work > that's not an option. ?I have been using "python setup.py install > --user" method, which installs under ~/.local/lib/python/site-packages/, > though I have heard about different (perhaps more robust) approaches. > In any case, I think we should have an install option available for > non-root users. > > Jon > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From astropy at liska.ath.cx Fri Jul 8 13:42:19 2011 From: astropy at liska.ath.cx (astropy at liska.ath.cx) Date: Fri, 08 Jul 2011 19:42:19 +0200 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <1310145504.6158.15.camel@shevek> References: <1310145504.6158.15.camel@shevek> Message-ID: <4E17417B.3030702@liska.ath.cx> Am 08.07.2011 19:18, schrieb Jonathan Slavin: >> I think, "standard" for a user meas always "use the package management >> provided with the system": this is what the user is familar with, >> especially on Linux. > That's only true if you have root privileges -- which not all of us > have. That's why I said "standard": Of course there are many situations where the standard does not fit -- it is the same situation if you need to use the latest version of LibreOffice on your terminal PC and the sysadmin does not want to install it as root. However, in case that astropy will be successfully meet its goals -- especially to provide a uniform astrophysical library -- I don't see a reason why it should not be installed on default (just like, for example, TeX) in an astrophysical environment. As a astropy developer, you may of course want to have the latest development version available, but this is already a special requirement. > In any case, I think we should have an install option available for > non-root users. Definitely. I think just that one should go the standard way as long as it takes us and only leave it in the cases when it is really necessary. I think that the solution to provide an option for the use of external packages is a solution everyone agreed about? Best regards Ole From sienkiew at stsci.edu Fri Jul 8 14:36:01 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Fri, 8 Jul 2011 14:36:01 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E17417B.3030702@liska.ath.cx> References: <1310145504.6158.15.camel@shevek> <4E17417B.3030702@liska.ath.cx> Message-ID: <4E174E11.5030909@stsci.edu> On 7/8/11 1:42 PM, astropy at liska.ath.cx wrote: > I think that the solution to provide an option for the use of external > packages is a solution everyone agreed about? Yes. To summarize for anybody who has not been following this discussion closely: -- Some portion of Astropy may want to use some C code that comes from an external library. We can provide a copy of that library code along with Astropy. For a user installing from source code, the python install just does the right thing; there is no need to download/compile/install the external package separately. If we do provide the library, we also provide an option to the setup.py to explicitly say "use the library that is already installed instead of the C sources distributed with Astropy". A packager can use this option to integrate with other packages provided by the linux distribution. -- So, we have a solution. All of the rest of the discussion has centered around which group of users you think you need to worry about: can use a package, or cannot use a package. There are substantial use cases for both categories. We have a solution that works well for both and at very little cost. Otherwise, I think all we need to do is acknowledge that both use cases are important in different situations, and that we are happy to make them both work smoothly. Mark S. From jturner at gemini.edu Fri Jul 8 15:43:09 2011 From: jturner at gemini.edu (James Turner) Date: Fri, 8 Jul 2011 15:43:09 -0400 Subject: [AstroPy] POLL: vision for a common Astronomy package In-Reply-To: <4E174E11.5030909@stsci.edu> References: <1310145504.6158.15.camel@shevek> <4E17417B.3030702@liska.ath.cx> <4E174E11.5030909@stsci.edu> Message-ID: <4E175DCD.1040304@gemini.edu> > To summarize for anybody who has not been following this discussion closely: Thanks. That's very helpful whilst on vacation... :-) > -- > > Some portion of Astropy may want to use some C code that comes from an > external library. > > We can provide a copy of that library code along with Astropy. For a > user installing from source code, the python install just does the right > thing; there is no need to download/compile/install the external package > separately. > > If we do provide the library, we also provide an option to the setup.py > to explicitly say "use the library that is already installed > instead of the C sources distributed with Astropy". A packager can use > this option to integrate with other packages provided by the linux > distribution. > > -- > > So, we have a solution. All of the rest of the discussion has centered > around which group of users you think you need to worry about: can use > a package, or cannot use a package. > > There are substantial use cases for both categories. We have a solution > that works well for both and at very little cost. > > Otherwise, I think all we need to do is acknowledge that both use cases > are important in different situations, and that we are happy to make > them both work smoothly. > > Mark S. > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- James E.H. Turner Gemini Observatory Southern Operations Centre, Casilla 603, Tel. (+56) 51 205609 La Serena, Chile. Fax. (+56) 51 205650 From sienkiew at stsci.edu Fri Jul 8 15:57:44 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Fri, 8 Jul 2011 15:57:44 -0400 Subject: [AstroPy] Testing Frameworks ( was Tests for AstroPy) In-Reply-To: <4E15E25A.80209@stsci.edu> References: <4E15E25A.80209@stsci.edu> Message-ID: <4E176138.10500@stsci.edu> >> As for the testing framework, we were thinking nose >> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). Nose is >> very nice for a project like this because it's super easy to run and >> also easy to write tests with, but is also compatible with the stdlib >> unittests. "use nose" is a little ambiguous. numpy "uses nose" but if you look inside numpy.test() it looks like of like they implemented their own testing system using nose as a library. I've never succeeded in using nosetests to run the numpy tests, for example. If you choose nose, my preference is to say: Tests come in a separate directory from the source. You can run all the tests with the command "nosetests tests/". This is not to exclude the possibility of installing some of the tests or of providing a test() function, but I also want to run the tests directly with nosetests. I actually have an ulterior motive here: I use a nose plugin that feeds a report generator. If you only provide a foo.test() function, I can't necessarily get it to use the plugin even if you base the whole thing on nose. You might not think that was a big deal, but it gets important when you have the test volume that I regularly deal with: > 100 000 tests per night, sometimes > 10 000 fail, depending what the developers are up to. b.t.w. This doesn't mean I have any attachment to nose; it just happens to be one of a small number of test runners that I use right now. > Last I heard, nose had been more or less orphaned, because Kuma doesn't > have time to > continue maintenance so didn't want to commit to it going forward. > There was some discussion about a TIP-community-driven > successor that would be more cleanly compatible with Python 3, and I know > something started in that direction, but I don't know the status. I can > inquire. I think this concern is somewhat less important now (than earlier) because nose is effectively a mature product. That is, it does what it needs to do and further development is not really necessary. That said, I've looked inside nose and not liked what I saw -- I really wouldn't want to have to fix a bug if one turned up. nose 1.0.0 claims to work in python 3. I only know of one serious alternative to nose: py.test seems to do pretty much the same thing, and relatively cleanly. The other test frameworks for python that I've looked at have been either narrowly defined for specific purposes or less flexible than nose/py.test. > I've become less fond of nose the more I've used it for heavy testing, > because its support > for reasonably complex tests (eg, perform exactly the same tests for 25 > different sets of input) > is not very robust or easy to use. I know there are other frameworks out > there that claim to do > better at this, but haven't had time to investigate them. I should be > able to get to that this summer. py.test does it a different way than nose, but it looks pretty reasonable. http://doc.pytest.org/en/latest/funcargs.html#parametrizing-tests From jturner at gemini.edu Fri Jul 8 16:50:34 2011 From: jturner at gemini.edu (James Turner) Date: Fri, 8 Jul 2011 16:50:34 -0400 Subject: [AstroPy] AstroPy BoF meeting at SciPy Message-ID: <4E176D9A.8060008@gemini.edu> Hi everyone, The results of the SciPy BoF poll were evenly split between Tues, Wed & Thurs evenings as the preferred time. Given the organizers' plans and Perry's availability, I have asked for a slot on **Thursday evening**. It looks like one person who voted can't make that time, so please do feel free to grab the rest of us for a chat before then. http://astropy.wikispaces.com/SciPy+2011 I don't have any rigid agenda in mind for this; it was clear from previous discussion that people wanted to meet at SciPy, so I have just set up a time that everyone knows about. Following on from the AstroPy vision noted down by the co-ordinators, we might have some more detailed discussion on how to organize collaboration, who is willing to contribute what, ideas for the proposed meeting later this year, consensus views on some of the questions that have come up in the last week etc. I think it would be good if people could also very briefly (in a couple of minutes) summarize what they're working on. I'll think about priorities some more on the way. Whether or not you will be at SciPy, if you have specific topics you think it would be useful to discuss and provide input to the AstroPy co-ordinators on, feel free to reply to this and propose something. Tom and Erik, I believe you won't be there, so let us know if there's something you'd like input on. Of course we'll probably only have an hour or perhaps two so we can't solve all the problems of the Universe or do anything very concrete just yet (unless anyone wants to use sprint time on Fri-Sat for that). Cheers, James. From mrdavis at stsci.edu Fri Jul 8 18:13:05 2011 From: mrdavis at stsci.edu (Matt Davis) Date: Fri, 8 Jul 2011 22:13:05 +0000 Subject: [AstroPy] AstroPy BoF meeting at SciPy In-Reply-To: <4E176D9A.8060008@gemini.edu> References: <4E176D9A.8060008@gemini.edu> Message-ID: <63AE9AA4-5C6D-4F0D-99C9-952C5D3887AA@stsci.edu> James, thanks for organizing! I'll just warn you that of the 9 people who have signed on at the wiki, six of us are developers at STSCI. Though I am planning to drag in another outside voice in the form of an astronomy grad student. As someone who does not do research I'm very curious to learn more about what astronomers would actually find most useful in astropy. Maybe one thing we could do at SciPy is brainstorm some features we think are likely to be the most popular in astropy (and provide the best foundation), and then we can poll the community for input before the fall meeting. Looking forward to SciPy! Best, Matt Davis P.S. I will be around during the sprints. On Jul 8, 2011, at 4:50 PM, James Turner wrote: > Hi everyone, > > The results of the SciPy BoF poll were evenly split between Tues, > Wed & Thurs evenings as the preferred time. Given the organizers' > plans and Perry's availability, I have asked for a slot on > **Thursday evening**. It looks like one person who voted can't make > that time, so please do feel free to grab the rest of us for a chat > before then. > > http://astropy.wikispaces.com/SciPy+2011 > > I don't have any rigid agenda in mind for this; it was clear from > previous discussion that people wanted to meet at SciPy, so I have > just set up a time that everyone knows about. Following on from the > AstroPy vision noted down by the co-ordinators, we might have some > more detailed discussion on how to organize collaboration, who is > willing to contribute what, ideas for the proposed meeting later > this year, consensus views on some of the questions that have come > up in the last week etc. I think it would be good if people could > also very briefly (in a couple of minutes) summarize what they're > working on. I'll think about priorities some more on the way. > > Whether or not you will be at SciPy, if you have specific topics you > think it would be useful to discuss and provide input to the AstroPy > co-ordinators on, feel free to reply to this and propose something. > Tom and Erik, I believe you won't be there, so let us know if > there's something you'd like input on. Of course we'll probably only > have an hour or perhaps two so we can't solve all the problems of > the Universe or do anything very concrete just yet (unless anyone > wants to use sprint time on Fri-Sat for that). > > Cheers, > > James. > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From mrdavis at stsci.edu Fri Jul 8 23:05:42 2011 From: mrdavis at stsci.edu (Matt Davis) Date: Sat, 9 Jul 2011 03:05:42 +0000 Subject: [AstroPy] EPD Free Message-ID: A new option for non-academic users to get the analytic Python basics. This will probably be useful (http://enthought.com/products/epd_free.php): OUR NEW LIGHTWEIGHT DISTRIBUTION OF SCIENTIFIC PYTHON ESSENTIALS: SCIPY, NUMPY, IPYTHON, MATPLOTLIB, TRAITS, & CHACO EPD Free provides a free, cross-platform installer of six of the libraries we consider fundamental for scientists, engineers, and analysts. It is perfect for: * Beginners who seek a simple but powerful Python stack * Developers who want to distribute a small, reliable environment for their applications * Anyone can (using the enpkg command) quickly install any of th 10,000+ libraries from our Python Package Index build mirror without the pain of manually building them $ enpkg Support for EPD Free is provided by the user and developer community on the EPD-Users mailing list. For details on usage, see the EPD Free license. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Sun Jul 10 01:39:46 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 9 Jul 2011 22:39:46 -0700 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) Message-ID: Hello everyone, Now that the vision and name for astropy have been ratified, the Coordinating Committee (Thomas, Perry, and I) have been working on a draft of coding guidelines/requirements for the astropy package. The intent is to use these items to determine if code in affiliated packages can be merged with the core package (as outlined in the vision). I've posted our draft of these guidelines to the wikispaces page: http://astropy.wikispaces.com/Astropy+Coding+Guidelines Feel free to ask questions about or comment on any of these items. We want to make sure the community is on board with a common set of guidelines, so we have tried to take into account discussions that have already occurred on this list in this draft. Many of these items have not yet been discussed, however, so we will certainly adjust these guidelines (or add new ones) based on your feedback once consensus is reached on the list. We do ask, though, that the comments/discussion be on the mailing list instead of the wiki page, so that the archive of the mailing list preserves the full discussion. Also, note that the guidelines reference both a documentation and testing guidelines document. I have created pages for these documents as well: Documentation: http://astropy.wikispaces.com/Astropy+Documentation+Guidelines Testing: http://astropy.wikispaces.com/Astropy+Testing+Guidelines These documents are not fleshed-out, as we wish to solicit ideas from the community here. I will send additional messages to this mailing list shortly to begin the discussion these topics, and ask again that the discussion for each take place in their associated threads. In the near future we (the committee) will also be sending out a message on determining project hosting and choice of VCS, as well as a set of priorities to guide development. Once we (the community) have reached consensus on this items, we can begin the actual work in earnest! Thanks in advance for your ideas and comments! -- Erik Tollerud From erik.tollerud at gmail.com Sun Jul 10 02:08:50 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 9 Jul 2011 23:08:50 -0700 Subject: [AstroPy] Testing Guidelines Message-ID: As I mentioned in my message a moment ago, I have added a page on the astropy wikispaces page dealing with testing for the astropy package: http://astropy.wikispaces.com/Astropy+Testing+Guidelines The page itself has basically no content - we will fill it in once we've had a good discussion here on the list as to what the community is comfortable with. There has already been some discussion on this topic on the list, so this message is to serve as a place to house the "official" discussion of what the actual package guidelines document should say. Once item I would like to put out for discussion: unless the community strongly disagrees, we (the coordinating committee) would prefer not to *require* complete coverage of unit or compliance testing, as we are concerned that this might be a major burden on some packages/submodules that would overly slow development. Regardless, probably the most important starting point is to determine the framework (Mark already began this discussion, so I've pasted his message below - sorry to hijack the thread, but I'm hoping to use the guideline documents to organization the discussion). Once that's decided, we can discuss the best arrangement and policies for how to organize tests. On that topic, prompted by Mark's comments below, I'll clarify that the "using nose" that I had in mind in my earlier comments was to use nose as it is, using only built-in plugins that seem to be fairly stable as it stands right now (although I've never personally used it with python 3.x - has anyone?). My experiences with nose have been pretty painless, and it's nice that it allows us to combine most of the different testing options into one framework (in particular, the combination of doctests and tests in a separate "tests" directory is attractive). It might be reasonable to use nose as a unit test framework, and then have a separate "compliance" test suite that might depend on external tools (as suggested by both Paul and Vicki). I think having 4 different test suites as (at least, I think) suggested by Paul might be a bit burdensome, though. Has anyone actually used py.test for a project that can chime in and give a sense of how it compares to nose? The last time I looked it wasn't nearly as well-documented as it seems to be now, so it may be a better alternative... -- Erik Tollerud On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz wrote: > >>> As for the testing framework, we were thinking nose >>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). ? Nose is >>> very nice for a project like this because it's super easy to run and >>> also easy to write tests with, but is also compatible with the stdlib >>> unittests. > > > "use nose" is a little ambiguous. ?numpy "uses nose" but if you look > inside numpy.test() it looks like of like they implemented their own > testing system using nose as a library. ?I've never succeeded in using > nosetests to run the numpy tests, for example. > > If you choose nose, my preference is to say: ?Tests come in a separate > directory from the source. ?You can run all the tests with the command > "nosetests tests/". > > This is not to exclude the possibility of installing some of the tests > or of providing a test() function, but I also want to run the tests > directly with nosetests. > > I actually have an ulterior motive here: ?I use a nose plugin that feeds > a report generator. ?If you only provide a foo.test() function, I can't > necessarily get it to use the plugin even if you base the whole thing on > nose. ?You might not think that was a big deal, but it gets important > when you have the test volume that I regularly deal with: > 100 000 > tests per night, sometimes > 10 000 fail, depending what the developers > are up to. > > b.t.w. ?This doesn't mean I have any attachment to nose; it just happens > to be one of a small number of test runners that I use right now. > > >> Last I heard, nose had been more or less orphaned, because Kuma doesn't >> have time to >> continue maintenance so didn't want to commit to it going forward. >> There was some discussion about a TIP-community-driven >> successor that would be more cleanly compatible with Python 3, and I know >> something started in that direction, but I don't know the status. I can >> inquire. > > > I think this concern is somewhat less important now (than earlier) > because nose is effectively a mature product. ?That is, it does what it > needs to do and further development is not really necessary. > > That said, I've looked inside nose and not liked what I saw -- I really > wouldn't want to have to fix a bug if one turned up. > > nose 1.0.0 claims to work in python 3. > > > I only know of one serious alternative to nose: ?py.test seems to do > pretty much the same thing, and relatively cleanly. ?The other test > frameworks for python that I've looked at have been either narrowly > defined for specific purposes or less flexible than nose/py.test. > > >> I've become less fond of nose the more I've used it for heavy testing, >> because its support >> for reasonably complex tests (eg, perform exactly the same tests for 25 >> different sets of input) >> is not very robust or easy to use. I know there are other frameworks out >> there that claim to do >> better at this, but haven't had time to investigate them. I should be >> able to get to that this summer. > > py.test does it a different way than nose, but it looks pretty reasonable. > > http://doc.pytest.org/en/latest/funcargs.html#parametrizing-tests From erik.tollerud at gmail.com Sun Jul 10 02:21:28 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 9 Jul 2011 23:21:28 -0700 Subject: [AstroPy] Documentation Guidelines Message-ID: As mentioned in the previous messages, I have added a wikispaces page on the topic of documentation guidelines: http://astropy.wikispaces.com/Astropy+Documentation+Guidelines that page is obviously light on details, primarily because we (the coordinating committee) want to solicit suggestions and opinions from the list as to the best way to format docstrings. This is the main item that must be determined ASAP, because we expect people will be writing docstrings as they start working on their code, and it's much easier to agree on the docstring syntax beforehand than it is to go back and fix it later. The exact layout/style of the documentation is probably less important right now, although I'm happy to entertain discussion on that topic, as well. As we see it, the main goals for the function and class docstrings are 1) readability in plain-text 2) clean-looking APIs when added to sphinx documentation. Any suggestions for preferred formats? -- Erik Tollerud From erik.tollerud at gmail.com Sun Jul 10 02:26:32 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 9 Jul 2011 23:26:32 -0700 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: I (speaking just as myself rather than as a committee member) have generally used the following syntax, because it's easy to use with sphinx while remaining rather readable in plain-text. def func(arg1,arg2,**kwargs): """ A one-sentence description of the function. Possibly much longer discussion of the function. This may be multiple paragraphs. It can include sphinx markup, like referencing the class :class:`AClass`. :param int arg1: Describe the first parameter, an integer. :param arg2: Include a longer description here because this maybe can be different types. :param kwargs: The function accepts additional arbitrary keywords, which are passed into some other function. :returns: Describe the return value here. :except TypeError: Describe any exceptions here. On Sat, Jul 9, 2011 at 11:21 PM, Erik Tollerud wrote: > As mentioned in the previous messages, I have added a wikispaces page > on the topic of documentation guidelines: > > http://astropy.wikispaces.com/Astropy+Documentation+Guidelines > > that page is obviously light on details, primarily because we (the > coordinating committee) want to solicit suggestions and opinions from > the list as to the best way to format docstrings. ?This is the main > item that must be determined ASAP, because we expect people will be > writing docstrings as they start working on their code, and it's much > easier to agree on the docstring syntax beforehand than it is to go > back and fix it later. ?The exact layout/style of the documentation is > probably less important right now, although I'm happy to entertain > discussion on that topic, as well. > > As we see it, the main goals for the function and class docstrings are > 1) readability in plain-text 2) clean-looking APIs when added to > sphinx documentation. ?Any suggestions for preferred formats? > > -- > Erik Tollerud > -- Erik Tollerud From oneaufs at gmail.com Sun Jul 10 04:32:32 2011 From: oneaufs at gmail.com (Prasanth) Date: Sun, 10 Jul 2011 14:02:32 +0530 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: Hello, Is it possible to develop/discuss the core package layout/template along with the coding guidelines? The coding guidelines is very general, as it will be until a consensus is reached. But I find it is easier to think about some of the guidelines, if I can think in terms of something concrete. For example, on the question of Python 3.x compatibility and meeting the core layout/template at the same time, I would like to think about how I should manage the practical aspects of my coding such as the style, directory layout, setuptools/distutils usage etc., in a way that it make it easier to meet these. Similarly, if I am to use a global configuration setting, how would I go about coding the package so that I can test my package, before submitting it for inclusion? I am likely to understand and make suggestions on the guidelines, if I can think in terms of what I already know, as it allows me to weigh the pros and cons. Thanks, Prasanth On Sun, Jul 10, 2011 at 11:09 AM, Erik Tollerud wrote: > Hello everyone, > > Now that the vision and name for astropy have been ratified, the > Coordinating Committee (Thomas, Perry, and I) have been working on a > draft of coding guidelines/requirements for the astropy package. The > intent is to use these items to determine if code in affiliated > packages can be merged with the core package (as outlined in the > vision). I've posted our draft of these guidelines to the wikispaces > page: > > http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > Feel free to ask questions about or comment on any of these items. We > want to make sure the community is on board with a common set of > guidelines, so we have tried to take into account discussions that > have already occurred on this list in this draft. Many of these items > have not yet been discussed, however, so we will certainly adjust > these guidelines (or add new ones) based on your feedback once > consensus is reached on the list. We do ask, though, that the > comments/discussion be on the mailing list instead of the wiki page, > so that the archive of the mailing list preserves the full discussion. > > > Also, note that the guidelines reference both a documentation and > testing guidelines document. I have created pages for these documents > as well: > > Documentation: http://astropy.wikispaces.com/Astropy+Documentation+Guidelines > Testing: http://astropy.wikispaces.com/Astropy+Testing+Guidelines > > These documents are not fleshed-out, as we wish to solicit ideas from > the community here. I will send additional messages to this mailing > list shortly to begin the discussion these topics, and ask again that > the discussion for each take place in their associated threads. > > > In the near future we (the committee) will also be sending out a > message on determining project hosting and choice of VCS, as well as a > set of priorities to guide development. Once we (the community) have > reached consensus on this items, we can begin the actual work in > earnest! > > Thanks in advance for your ideas and comments! > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oneaufs at gmail.com Sun Jul 10 04:49:04 2011 From: oneaufs at gmail.com (Prasanth) Date: Sun, 10 Jul 2011 14:19:04 +0530 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: Hello, The following format: :param v: Some desc. :type v: float will create an output that looks something like v(float) - Some desc. in the Sphinx output. This also make it suitable for epydoc. Similarly if we align the parameter names in all "param" and "type" fileds, then it will be easier to read it as docstring in the terminal: :param name1: :type name1: :param n2: :type n2: Prasanth On Sun, Jul 10, 2011 at 11:56 AM, Erik Tollerud wrote: > I (speaking just as myself rather than as a committee member) have > generally used the following syntax, because it's easy to use with > sphinx while remaining rather readable in plain-text. > > def func(arg1,arg2,**kwargs): > """ > A one-sentence description of the function. > > Possibly much longer discussion of the function. This may > be multiple paragraphs. It can include sphinx markup, like > referencing the class :class:`AClass`. > > > :param int arg1: Describe the first parameter, an integer. > :param arg2: > Include a longer description here because this maybe > can be different types. > :param kwargs: > The function accepts additional arbitrary keywords, > which are passed into some other function. > > :returns: Describe the return value here. > :except TypeError: Describe any exceptions here. > > > > On Sat, Jul 9, 2011 at 11:21 PM, Erik Tollerud > wrote: > > As mentioned in the previous messages, I have added a wikispaces page > > on the topic of documentation guidelines: > > > > http://astropy.wikispaces.com/Astropy+Documentation+Guidelines > > > > that page is obviously light on details, primarily because we (the > > coordinating committee) want to solicit suggestions and opinions from > > the list as to the best way to format docstrings. This is the main > > item that must be determined ASAP, because we expect people will be > > writing docstrings as they start working on their code, and it's much > > easier to agree on the docstring syntax beforehand than it is to go > > back and fix it later. The exact layout/style of the documentation is > > probably less important right now, although I'm happy to entertain > > discussion on that topic, as well. > > > > As we see it, the main goals for the function and class docstrings are > > 1) readability in plain-text 2) clean-looking APIs when added to > > sphinx documentation. Any suggestions for preferred formats? > > > > -- > > Erik Tollerud > > > > > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Sun Jul 10 18:29:55 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sun, 10 Jul 2011 15:29:55 -0700 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: That format actually produces the exact same sphinx output as :param float v: Some desc. The advantage of the single line format is that in plain-text it's easier to read because it's only one entry for each parameter. The disadvantage is that only a single-word plain text type is allowed - e.g. you can do neither ":param float or None v:", nor ":param :class:`AClass` v:". Separate ":param v:" and ":type v:" allow any arbitrary sphinx markup to be used as a type. (those who are interested can see more details at http://sphinx.pocoo.org/markup/desc.html#info-field-lists) I have personally found that anytime I need more than a single-word type description, it ends up difficult to read (e.g. "v(float or int or array or None) - some desc." is harder to understand even in sphinx than "v - A thing that does this if it's a float, this if it's an int, this if it's an array, or this if None." So it seems to me like the extra type line is *usually* unnecessary. On Sun, Jul 10, 2011 at 1:49 AM, Prasanth wrote: > Hello, > The following format: > > :param v: Some desc. > :type ? ?v: float > will create an output that looks something like > v(float) - Some desc. > in the Sphinx output. This also make it suitable for epydoc. > Similarly?if we?align?the parameter names in all "param" and "type" fileds, > then it will be easier to read it as docstring in the terminal: > :param name1: > :type ? ?name1: > :param n2: > :type ? ?n2: > Prasanth > On Sun, Jul 10, 2011 at 11:56 AM, Erik Tollerud > wrote: >> >> I (speaking just as myself rather than as a committee member) have >> generally used the following syntax, because it's easy to use with >> sphinx while remaining rather readable in plain-text. >> >> def func(arg1,arg2,**kwargs): >> ? ?""" >> ? ?A one-sentence description of the function. >> >> ? ?Possibly much longer discussion of the function. ?This may >> ? ?be multiple paragraphs. It can include sphinx markup, like >> ? ?referencing the class :class:`AClass`. >> >> >> ? ?:param int arg1: Describe the first parameter, an integer. >> ? ?:param arg2: >> ? ? ? ?Include a longer description here because this maybe >> ? ? ? ?can be different types. >> ? ?:param kwargs: >> ? ? ? ?The function accepts additional arbitrary keywords, >> ? ? ? ?which are passed into some other function. >> >> ? ?:returns: Describe the return value here. >> ? ?:except TypeError: Describe any exceptions here. >> >> >> >> On Sat, Jul 9, 2011 at 11:21 PM, Erik Tollerud >> wrote: >> > As mentioned in the previous messages, I have added a wikispaces page >> > on the topic of documentation guidelines: >> > >> > http://astropy.wikispaces.com/Astropy+Documentation+Guidelines >> > >> > that page is obviously light on details, primarily because we (the >> > coordinating committee) want to solicit suggestions and opinions from >> > the list as to the best way to format docstrings. ?This is the main >> > item that must be determined ASAP, because we expect people will be >> > writing docstrings as they start working on their code, and it's much >> > easier to agree on the docstring syntax beforehand than it is to go >> > back and fix it later. ?The exact layout/style of the documentation is >> > probably less important right now, although I'm happy to entertain >> > discussion on that topic, as well. >> > >> > As we see it, the main goals for the function and class docstrings are >> > 1) readability in plain-text 2) clean-looking APIs when added to >> > sphinx documentation. ?Any suggestions for preferred formats? >> > >> > -- >> > Erik Tollerud >> > >> >> >> >> -- >> Erik Tollerud >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > -- Erik Tollerud From mdroe at stsci.edu Sun Jul 10 21:17:42 2011 From: mdroe at stsci.edu (Michael Droettboom) Date: Sun, 10 Jul 2011 21:17:42 -0400 Subject: [AstroPy] Fwd: Error in vo on Mac OS 10.6.8, Python 2.7 In-Reply-To: References: Message-ID: <4E1A4F36.4060405@stsci.edu> Sorry I'm just getting to this message now... (This strangely ended up being classified as spam...) I'd be curious to see what the arraysize declaration looks like. It's possible that TOPCAT and KXML aren't really parsing the PARAM elements, but just storing them to spit them out verbatim again, so they aren't really "deeply" parsing the value. Cheers, Mike On 07/05/2011 03:47 AM, Jean-Baptiste Marquette wrote: > Hi Micheal, > > I got the point: a wrong arraysize declaration of a PARAM in > SExtractor while creating the VOTables. I just informed Emmanuel > Bertin, and have to sed these 201,000 catalogues to correct them. > But it is strange that both TOPCAT and KXML Editor load the tables > without complaining... > > D?but du message r?exp?di? : > >> *De : *Jean-Baptiste Marquette > >> *Date : *4 juillet 2011 10:46:01 HAEC >> *? : *Michael Droettboom > >> *Objet : **R?exp : [AstroPy] Error in vo on Mac OS 10.6.8, Python 2.7* >> >> Hi Michael, >> >> I investigated a bit further on the issue below. I installed KXML >> Editor on my Mac Pro, which reads my table seamlessly, as TOPCAT did. >> So I think this is an actual bug in vo. Should I open a ticket ? >> >> Cheers, >> Jean-Baptiste >> >> D?but du message r?exp?di? : >> >>> *De : *Jean-Baptiste Marquette >> > >>> *Date : *30 juin 2011 19:46:10 HAEC >>> *? : *astropy at scipy.org >>> *Objet : **[AstroPy] Error in vo on Mac OS 10.6.8, Python 2.7* >>> >>> Dear vo gurus, >>> >>> I use vo through ATpy on VOTables which are well read by TOPCAT. >>> Thus I didn't find any violation of VO specifications. >>> >>> The traceback is: >>> >>> bs300.date processing ... >>> /Volumes/pepperland/erosdata/sexcatcalib/bs/bs300/bs3001/bs3001_6f2079.cat:3:0: >>> W20: No version number specified in file. Assuming 1.0 >>> Traceback (most recent call last): >>> File "/Users/marquett/workspace/EpochsCat/src/PutEpochs.py", line >>> 28, in >>> Table = atpy.Table(Cat, type='vo', pedantic=False, tid=2) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/atpy/basetable.py", >>> line 167, in __init__ >>> self.read(*args, **kwargs) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/atpy/basetable.py", >>> line 213, in read >>> atpy._readers[table_type](self, *args, **kwargs) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/atpy/votable.py", >>> line 86, in read >>> votable = parse(filename, pedantic=pedantic) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/table.py", >>> line 103, in parse >>> return tree.VOTableFile(config=config, pos=(1, >>> 1)).parse(iterator, config) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2963, in parse >>> iterator, tag, data, config, pos) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2897, in _add_resource >>> resource.parse(self, iterator, config) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2732, in parse >>> tag_mapping.get(tag, self._add_unknown_tag)(iterator, tag, data, >>> config, pos) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2710, in _add_resource >>> resource.parse(self._votable, iterator, config) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2732, in parse >>> tag_mapping.get(tag, self._add_unknown_tag)(iterator, tag, data, >>> config, pos) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2710, in _add_resource >>> resource.parse(self._votable, iterator, config) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2732, in parse >>> tag_mapping.get(tag, self._add_unknown_tag)(iterator, tag, data, >>> config, pos) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 2698, in _add_param >>> param = Param(self._votable, config=config, pos=pos, **data) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 1232, in __init__ >>> id=id, config=config, pos=pos) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 915, in __init__ >>> self._setup(config) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 1251, in _setup >>> self.value = self._value >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>> line 1239, in _set_value >>> self._value = self.converter.parse(value, self._config, self._pos)[0] >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/converters.py", >>> line 433, in parse >>> config, pos) >>> File >>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/voexceptions.py", >>> line 109, in vo_raise >>> raise exc(message) >>> vo.voexceptions.VOTableSpecError: >>> /Volumes/pepperland/erosdata/sexcatcalib/bs/bs300/bs3001/bs3001_6f2079.cat:87041:3: >>> E02: Incorrect number of elements in array, expected 0, got 1 >>> >>> As I said this table is well loaded by TOPCAT, and was created by >>> Emmanuel Bertin's SExtractor. It is 20 Mb large. >>> 87041 (if it is a rank) is far beyond the index bound. >>> If necessary, I can provide the table in my ftp, together with my >>> python script. >>> I checked that the issue remains the same on other tables of the >>> same type. >>> >>> Thanks for your help, >>> >>> Jean-Baptiste Marquette >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >> >> > > ==================================================================== > Bien cordialement / Very truly yours / Mit freundlichen Gruessen, > Jean-Baptiste Marquette > Institut d'Astrophysique de Paris > CNRS - UMR 7095 > Universit? Pierre & Marie Curie > 98bis Bd Arago > 75014 Paris - France > Tel +33 (0)1 4432 8196 > Fax +33 (0)1 4432 8001 > Web : http://www2.iap.fr/users/marquett > ==================================================================== > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oneaufs at gmail.com Mon Jul 11 00:06:33 2011 From: oneaufs at gmail.com (Prasanth) Date: Mon, 11 Jul 2011 09:36:33 +0530 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: Hello, One place the extra line can be useful is while writing docstrings that are meant to be parsed by epydoc, in addition to Sphinx. So the guidelines can be made flexible enough for incorporating both use cases. Prasanth On Mon, Jul 11, 2011 at 3:59 AM, Erik Tollerud wrote: > That format actually produces the exact same sphinx output as > > :param float v: Some desc. > > The advantage of the single line format is that in plain-text it's > easier to read because it's only one entry for each parameter. The > disadvantage is that only a single-word plain text type is allowed - > e.g. you can do neither ":param float or None v:", nor ":param > :class:`AClass` v:". Separate ":param v:" and ":type v:" allow any > arbitrary sphinx markup to be used as a type. (those who are > interested can see more details at > http://sphinx.pocoo.org/markup/desc.html#info-field-lists) > > I have personally found that anytime I need more than a single-word > type description, it ends up difficult to read (e.g. "v(float or int > or array or None) - some desc." is harder to understand even in sphinx > than "v - A thing that does this if it's a float, this if it's an int, > this if it's an array, or this if None." So it seems to me like the > extra type line is *usually* unnecessary. > > On Sun, Jul 10, 2011 at 1:49 AM, Prasanth wrote: > > Hello, > > The following format: > > > > :param v: Some desc. > > :type v: float > > will create an output that looks something like > > v(float) - Some desc. > > in the Sphinx output. This also make it suitable for epydoc. > > Similarly if we align the parameter names in all "param" and "type" > fileds, > > then it will be easier to read it as docstring in the terminal: > > :param name1: > > :type name1: > > :param n2: > > :type n2: > > Prasanth > > On Sun, Jul 10, 2011 at 11:56 AM, Erik Tollerud > > > wrote: > >> > >> I (speaking just as myself rather than as a committee member) have > >> generally used the following syntax, because it's easy to use with > >> sphinx while remaining rather readable in plain-text. > >> > >> def func(arg1,arg2,**kwargs): > >> """ > >> A one-sentence description of the function. > >> > >> Possibly much longer discussion of the function. This may > >> be multiple paragraphs. It can include sphinx markup, like > >> referencing the class :class:`AClass`. > >> > >> > >> :param int arg1: Describe the first parameter, an integer. > >> :param arg2: > >> Include a longer description here because this maybe > >> can be different types. > >> :param kwargs: > >> The function accepts additional arbitrary keywords, > >> which are passed into some other function. > >> > >> :returns: Describe the return value here. > >> :except TypeError: Describe any exceptions here. > >> > >> > >> > >> On Sat, Jul 9, 2011 at 11:21 PM, Erik Tollerud > > >> wrote: > >> > As mentioned in the previous messages, I have added a wikispaces page > >> > on the topic of documentation guidelines: > >> > > >> > http://astropy.wikispaces.com/Astropy+Documentation+Guidelines > >> > > >> > that page is obviously light on details, primarily because we (the > >> > coordinating committee) want to solicit suggestions and opinions from > >> > the list as to the best way to format docstrings. This is the main > >> > item that must be determined ASAP, because we expect people will be > >> > writing docstrings as they start working on their code, and it's much > >> > easier to agree on the docstring syntax beforehand than it is to go > >> > back and fix it later. The exact layout/style of the documentation is > >> > probably less important right now, although I'm happy to entertain > >> > discussion on that topic, as well. > >> > > >> > As we see it, the main goals for the function and class docstrings are > >> > 1) readability in plain-text 2) clean-looking APIs when added to > >> > sphinx documentation. Any suggestions for preferred formats? > >> > > >> > -- > >> > Erik Tollerud > >> > > >> > >> > >> > >> -- > >> Erik Tollerud > >> _______________________________________________ > >> AstroPy mailing list > >> AstroPy at scipy.org > >> http://mail.scipy.org/mailman/listinfo/astropy > > > > > > > > -- > Erik Tollerud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jh at physics.ucf.edu Mon Jul 11 01:10:39 2011 From: jh at physics.ucf.edu (Joe Harrington) Date: Mon, 11 Jul 2011 01:10:39 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: (astropy-request@scipy.org) References: Message-ID: > http://astropy.wikispaces.com/Astropy+Coding+Guidelines I would strongly encourage: 1. Use the numpy docstring format. We spent a *lot* of time designing this format and getting community feedback to ensure that projects like this one can use it, in addition to the core numpy doc project. It's been vetted in the writing of over 1000 pages of docs. It can handle images, formulae, and cross references and there are guidelines for how to do that and how to make sure that the ASCII version is also sensible. It's what numpy and scipy users are now used to. It's based on Sphinx, so it makes a PDF, HTML, and of course ASCII, has doctests, etc. There's a place for references, and a way to separate the basic info everyone needs to know from the details ("notes") only specialists care about. There is one mod that the Matplotlib folks have, for the special case of families of routines that have dozens or hundreds of identical, optional parameters (common for graphics routines). Important: Keep the one-line summary truly to less than about 60 characters. It can be used to make lists of routines, one line per routine, including the name, for easy indexing. This breaks if the "one line" goes on and on. Here is the standard: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt If you want to see what the results look like, just type help(np.whatever), or look at the PDF or HTML pages at docs.scipy.org. 2. Use the docs.scipy.org wiki system, pydocweb. Make a project either there or on the astropy system; the code is OSS. This will allow normal users who would not otherwise have access to the code repo to submit doc fixes, examples, and tests; perform peer review; and discuss specific pages. This is extremely helpful if the programmer is lazy (or even functionally illiterate ;-), or when importing a package that's already written and has poor docs (e.g., numpy 3 years ago). It can sync in both directions with the VCS, gives metrics of who contributed what, what needs editing now, what got worked on this week, etc. You can see who wrote what, and revert if there's a problem. Check out these features at http://docs.scipy.org/numpy/ A description of the workflow and links to the style guidelines, templates, and examples are here: http://docs.scipy.org/numpy/Front%20Page/ Source code to pydocweb is here: http://code.google.com/p/pydocweb/ The project at numpy is fairly dormant now, the main docs having been written (the community fell flat on its face and didn't step up to review), but the folks are all available on the scipy-dev mailing list if you want to ask questions about it. The main developer is Pauli Virtanen. We wrote status articles in the 2008 and 2009 SciPy Conference proceedings: http://conference.scipy.org/proceedings/SciPy2008/paper_7/ http://conference.scipy.org/proceedings/SciPy2009/paper_14/ I hope the community will accept this format and system. It's field-tested! --jh-- From mdroe at stsci.edu Mon Jul 11 08:35:30 2011 From: mdroe at stsci.edu (Michael Droettboom) Date: Mon, 11 Jul 2011 08:35:30 -0400 Subject: [AstroPy] Error in vo on Mac OS 10.6.8, Python 2.7 In-Reply-To: References: <4E1A4F36.4060405@stsci.edu> Message-ID: <4E1AEE12.6050800@stsci.edu> This is very helpful. It's an invalid VOTable file. kxml accepts it because it's just a generic XML editor and doesn't know about VOTable datatype/arraysize specifications. Looking at the TOPCAT source code, it appears that the "value" attribute isn't parsed until the value is actually requested. So simply loading the file doesn't exercise the value parser for PARAM elements. However, I suspect (this is untested) calling getObject() on that ParamElement would raise an exception. That said, this could probably be loosened in the vo library. If "pedantic" is False, we could just raise a warning (rather than an exception) and fill as many array elements as we can. In this case, the arraysize would still be "0", and no values would be set, but at least it would continue to parse the rest of the file. Cheers, Mike On 07/11/2011 05:25 AM, Jean-Baptiste Marquette wrote: > > Hi Mike, > > The declaration looks like: > > ucd="instr.sensitivity;obs.param" value="0"/> > > Let me know if you wish me to put a complete table on my ftp. > > Cheers > Jean-Baptiste > > > >> Sorry I'm just getting to this message now... (This strangely ended >> up being classified as spam...) >> >> I'd be curious to see what the arraysize declaration looks like. >> It's possible that TOPCAT and KXML aren't really parsing the PARAM >> elements, but just storing them to spit them out verbatim again, so >> they aren't really "deeply" parsing the value. >> >> Cheers, >> Mike >> >> On 07/05/2011 03:47 AM, Jean-Baptiste Marquette wrote: >>> Hi Micheal, >>> >>> I got the point: a wrong arraysize declaration of a PARAM in >>> SExtractor while creating the VOTables. I just informed Emmanuel >>> Bertin, and have to sed these 201,000 catalogues to correct them. >>> But it is strange that both TOPCAT and KXML Editor load the tables >>> without complaining... >>> >>> D?but du message r?exp?di? : >>> >>>> *De : *Jean-Baptiste Marquette >>> > >>>> *Date : *4 juillet 2011 10:46:01 HAEC >>>> *? : *Michael Droettboom > >>>> *Objet : **R?exp : [AstroPy] Error in vo on Mac OS 10.6.8, Python 2.7* >>>> >>>> Hi Michael, >>>> >>>> I investigated a bit further on the issue below. I installed KXML >>>> Editor on my Mac Pro, which reads my table seamlessly, as TOPCAT >>>> did. So I think this is an actual bug in vo. Should I open a ticket ? >>>> >>>> Cheers, >>>> Jean-Baptiste >>>> >>>> D?but du message r?exp?di? : >>>> >>>>> *De : *Jean-Baptiste Marquette >>>> > >>>>> *Date : *30 juin 2011 19:46:10 HAEC >>>>> *? : *astropy at scipy.org >>>>> *Objet : **[AstroPy] Error in vo on Mac OS 10.6.8, Python 2.7* >>>>> >>>>> Dear vo gurus, >>>>> >>>>> I use vo through ATpy on VOTables which are well read by TOPCAT. >>>>> Thus I didn't find any violation of VO specifications. >>>>> >>>>> The traceback is: >>>>> >>>>> bs300.date processing ... >>>>> /Volumes/pepperland/erosdata/sexcatcalib/bs/bs300/bs3001/bs3001_6f2079.cat:3:0: >>>>> W20: No version number specified in file. Assuming 1.0 >>>>> Traceback (most recent call last): >>>>> File "/Users/marquett/workspace/EpochsCat/src/PutEpochs.py", line >>>>> 28, in >>>>> Table = atpy.Table(Cat, type='vo', pedantic=False, tid=2) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/atpy/basetable.py", >>>>> line 167, in __init__ >>>>> self.read(*args, **kwargs) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/atpy/basetable.py", >>>>> line 213, in read >>>>> atpy._readers[table_type](self, *args, **kwargs) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/atpy/votable.py", >>>>> line 86, in read >>>>> votable = parse(filename, pedantic=pedantic) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/table.py", >>>>> line 103, in parse >>>>> return tree.VOTableFile(config=config, pos=(1, >>>>> 1)).parse(iterator, config) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2963, in parse >>>>> iterator, tag, data, config, pos) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2897, in _add_resource >>>>> resource.parse(self, iterator, config) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2732, in parse >>>>> tag_mapping.get(tag, self._add_unknown_tag)(iterator, tag, >>>>> data, config, pos) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2710, in _add_resource >>>>> resource.parse(self._votable, iterator, config) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2732, in parse >>>>> tag_mapping.get(tag, self._add_unknown_tag)(iterator, tag, >>>>> data, config, pos) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2710, in _add_resource >>>>> resource.parse(self._votable, iterator, config) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2732, in parse >>>>> tag_mapping.get(tag, self._add_unknown_tag)(iterator, tag, >>>>> data, config, pos) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 2698, in _add_param >>>>> param = Param(self._votable, config=config, pos=pos, **data) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 1232, in __init__ >>>>> id=id, config=config, pos=pos) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 915, in __init__ >>>>> self._setup(config) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 1251, in _setup >>>>> self.value = self._value >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/tree.py", >>>>> line 1239, in _set_value >>>>> self._value = self.converter.parse(value, self._config, >>>>> self._pos)[0] >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/converters.py", >>>>> line 433, in parse >>>>> config, pos) >>>>> File >>>>> "/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/site-packages/vo/voexceptions.py", >>>>> line 109, in vo_raise >>>>> raise exc(message) >>>>> vo.voexceptions.VOTableSpecError: >>>>> /Volumes/pepperland/erosdata/sexcatcalib/bs/bs300/bs3001/bs3001_6f2079.cat:87041:3: >>>>> E02: Incorrect number of elements in array, expected 0, got 1 >>>>> >>>>> As I said this table is well loaded by TOPCAT, and was created by >>>>> Emmanuel Bertin's SExtractor. It is 20 Mb large. >>>>> 87041 (if it is a rank) is far beyond the index bound. >>>>> If necessary, I can provide the table in my ftp, together with my >>>>> python script. >>>>> I checked that the issue remains the same on other tables of the >>>>> same type. >>>>> >>>>> Thanks for your help, >>>>> >>>>> Jean-Baptiste Marquette >>>>> _______________________________________________ >>>>> AstroPy mailing list >>>>> AstroPy at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> >>>> >>> >>> ==================================================================== >>> Bien cordialement / Very truly yours / Mit freundlichen Gruessen, >>> Jean-Baptiste Marquette >>> Institut d'Astrophysique de Paris >>> CNRS - UMR 7095 >>> Universit? Pierre & Marie Curie >>> 98bis Bd Arago >>> 75014 Paris - France >>> Tel +33 (0)1 4432 8196 >>> Fax +33 (0)1 4432 8001 >>> Web : http://www2.iap.fr/users/marquett >>> ==================================================================== >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sienkiew at stsci.edu Mon Jul 11 11:37:21 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Mon, 11 Jul 2011 11:37:21 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: <4E1B18B1.4060803@stsci.edu> On 7/10/11 1:39 AM, Erik Tollerud wrote: > http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > Feel free to ask questions about or comment on any of these items. It looks pretty mature already. Here are various comments: ---- " # The package should be importable with no dependencies other than the Python Standard Library (v2.6), NumPy, SciPy, matplotlib, and components already in the core Astronomy library." * I was wondering which "Astronomy" library this refers to and what part is "core". I think it means "core Astropy", and I think later it defines "core Astropy" as any package that has already been accepted into astropy as astropy.whatever; If this is correct, I recommend "in the core Astronomy library" -> "already in the Astropy package". ---- I recommend that the template package include some small amount of C code and the Cython example for how to use it. I note that the Cython recommendation implies an additional external dependency for somebody. Is this for anybody who installs Astropy from source, or is it a preprocessor that you only run occasionally like yacc/lex? ---- " * If an external C library is needed, the source code for the library should be bundled with the astropy core. Additionally, the package must be compatible with using a system-installed library instead of the version included in astropy." add "... using the mechanism shown in the template package." We want everybody to provide the same flags, same interface, etc, so we show one way to do it and ask everybody to do it that way. ---- If you want to require PEP 8, I suggest that you make your examples PEP 8 compliant. ---- I think we should explicitly recommend the new relative import mechanism any time you import another module/package that is in the same package you are part of. Say, for example, that you want to debug a package and you would like to run both the original and a modified copy at the same time. With relative imports, you don't need to edit every import line in the whole package to make your modified copy. This is not compliant with PEP 8, which recommends the bad practice of always using absolute imports. It made sense when relative imports were ambiguous, but the Python language no longer has that particular weakness. ---- The property/get/set example is weak: 1) Take out the functions that are not relevant to get/set; you don't need them cluttering the example. 2) Show a user interacting with the object: x = star() # use this x.color = 100 print x.color # not this x.set_color(100) print x.get_color() Realistically, I think people are likely to want to use the get_color()/set_color() approach because it is more natural. It also makes it more clear what is happening. I know that hard core OO guys think properties are cool, but I've never seen the attraction myself. ---- The super() example actually shows more instances of doing it the wrong way than the right way, and it does not clearly mark which one to use. It should show one complete implementation using super() [marked "don't do this"] and another complete implementation using direct calls [marked "do this"]. One of the common cases for wanting to call into the superclass is to invoke the __init__() method. The example should show how to do this, including the case of multiple inheritance. ---- In the "import *" example, "__init__" is shown as "init" (two places). It looks like it needs a wiki escape. From mdroe at stsci.edu Mon Jul 11 12:05:38 2011 From: mdroe at stsci.edu (Michael Droettboom) Date: Mon, 11 Jul 2011 12:05:38 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E1B18B1.4060803@stsci.edu> References: <4E1B18B1.4060803@stsci.edu> Message-ID: <4E1B1F52.8080000@stsci.edu> With regard to: "C extensions will be allowed only when the provide a significant performance enhancement over pure python. When C extensions are used, the python interface must meet interface guidelines, and the use of Cython is strongly recommended." In the case of something like pywcs, a C extension is used because the WCS standard is so difficult to implement that it was easier to use existing and well-tested code. It's not clear whether there was a performance advantage over a Numpy-based implementation (I'm not taking a position either way, I'm saying that such an experiment was never done, and performance was not a motivating factor.) There are a lot of good reasons beyond performance to use non-Python extension code, so I don't know if we should artificially limit ourselves here. With respect to: "Packages can include data in [path tbd] as long as it is less than about 100 kb. These data should be accessed via the astropy.config.[funcname tbd] mechanism. If the data exceeds this size, it should be hosted outside the source code repository and downloaded using the astropy.config.[funcname tbd] mechanism." I worry that there are often version dependencies between the code and the data. (Matplotlib has a dependency on specific versions of the STIX math fonts, for example). Having the data in a separate code repository makes keeping this synchronized trickier. I would instead say: large data required for functioning of the library goes in the repository, but under a special directory that is not distributed as part of the binary distribution. The mechanism for accessing this data when needed should be aware of the source code revision and download the corresponding revision of the data. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From rays at blue-cove.com Mon Jul 11 12:23:51 2011 From: rays at blue-cove.com (RayS) Date: Mon, 11 Jul 2011 09:23:51 -0700 Subject: [AstroPy] EPD Free In-Reply-To: References: Message-ID: <20110711162411.OPWT18463.fed1rmfepo102.cox.net@fed1rmimpo01.cox.net> At 08:05 PM 7/8/2011, Matt Davis wrote: >A new option for non-academic users to get the analytic Python >basics. This will probably be useful >(http://enthought.com/products/epd_free.php): > Additionally: I bought and use the $200 EPD distribution for its MKL linkage; numpy is linked to the Intel MKL and made accessible via an attribute: numpy.use_fastnumpy = True import mkl For instance, the numpy MKl fft is 4x faster with N=2**11 and >6x with N>2**13 on a Core 2. All non-numpy MKL functions are also available via ctypes. The free version does not have the MKL linkage. I do find it odd and a little sad that a number of Python web sites still use PHP... Ray -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubutsu at gmail.com Mon Jul 11 12:24:47 2011 From: ubutsu at gmail.com (Taro Sato) Date: Mon, 11 Jul 2011 13:24:47 -0300 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E1B1F52.8080000@stsci.edu> References: <4E1B18B1.4060803@stsci.edu> <4E1B1F52.8080000@stsci.edu> Message-ID: On Mon, Jul 11, 2011 at 1:05 PM, Michael Droettboom wrote: > With regard to: "C extensions will be allowed only when the provide a > significant performance enhancement over pure python. When C extensions are > used, the python interface must meet interface guidelines, and the use of > Cython is strongly recommended." What about weave? I've fallen in love with using inline C codes for the obvious performance advantage when the stock NumPy/SciPy routines don't cut it... just for quick and tiny hacks and such. It's much less cumbersome than writing full blown C extensions for small tasks. It's probably not particularly elegant or good for maintenance purposes. But the interfaces can of course remain Pythonic. Taro From oneaufs at gmail.com Mon Jul 11 12:42:16 2011 From: oneaufs at gmail.com (Prasanth) Date: Mon, 11 Jul 2011 22:12:16 +0530 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E1B18B1.4060803@stsci.edu> References: <4E1B18B1.4060803@stsci.edu> Message-ID: > > > I recommend that the template package include some small amount of C > code and the Cython example for how to use it. > > I note that the Cython recommendation implies an additional external > dependency for somebody. Is this for anybody who installs Astropy from > source, or is it a preprocessor that you only run occasionally like > yacc/lex? > > I am assuming that the dependency here is about having to install Cython. This is not required. The source distribution that is checked into astropy can just include the Cython generated C file along with the C library. The user then compiles all the C files together, while using setup.py commands. Prasanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From mperrin at stsci.edu Mon Jul 11 12:52:06 2011 From: mperrin at stsci.edu (Marshall Perrin) Date: Mon, 11 Jul 2011 16:52:06 +0000 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: <75AC6B80-9CEE-4181-8A4E-1C0CD80F7E8B@stsci.edu> On Jul 11, 2011, at 1:10 AM, Joe Harrington wrote: >> http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > I would strongly encourage: > > 1. Use the numpy docstring format. We spent a *lot* of time designing > this format and getting community feedback to ensure that projects > like this one can use it, in addition to the core numpy doc > project. Strongly seconded. This is a case where we really don't need to reinvent the wheel. I've used the numpy docstring format in all my own projects for some time now, and I've found it very straightforward to work with and the support for inline math and images and code execution allows for really top-quality web page outputs with minimal effort. Given all the competing demands for everyone's limited brain cells these days, I'd really rather not have to learn/memorize a new and incompatible docstring standard, unless someone can articulate a compelling reasons why the numpy/scipy docstring format is inadequate for our needs. - Marshall From sienkiew at stsci.edu Mon Jul 11 14:12:39 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Mon, 11 Jul 2011 14:12:39 -0400 Subject: [AstroPy] Testing Guidelines In-Reply-To: References: Message-ID: <4E1B3D17.4010604@stsci.edu> On 7/10/11 2:08 AM, Erik Tollerud wrote: > On that topic, prompted by Mark's comments below, I'll clarify that > the "using nose" that I had in mind in my earlier comments was to use > nose as it is, using only built-in plugins that seem to be fairly > stable as it stands right now (although I've never personally used it > with python 3.x - has anyone?). I have successfully run simple examples in nose in python 3. I am not doing much with python 3, but if Astropy becomes a reality, I will be running its tests in my CI system, which implies that I will use my python 3 environment more regularly. > It might be reasonable to use nose as a unit test > framework, and then have a separate "compliance" test suite that might > depend on external tools (as suggested by both Paul and Vicki). That is a possibility. b.t.w. I suggest that you differentiate the _purpose_ of a test from the _mechanism_ of the test. nose is "a unit test framework", but I use nose to run integration tests and regression tests every day, stsci_regtest is a "regression test system", but I use it to run tests that could easily qualify as unit tests. In my regular work, I don't distinguish the categories that Paul suggests because I really don't care. If a test points up an anomaly, somebody has to deal with it no matter what kind of test it is. I think it is more useful to divide tests according to the difficulty of running them. I think this is very similar (or even the same) to what you are talking about with "compliance" tests, but I am thinking "simple to run" and "harder to run" with no particular meaning attached to the name of the category. For example, you might have a set of tests that are very easy to run and require no special setup. You could give these to the user to run on their installed system. I would not deny the user an opportunity to run some of those tests because I think of them as "integration" instead of "unit". Likewise, you might have some tests that need an elaborate installed environment to run. Maybe you make this a different group of tests, not because they are "integration" tests, but because they are hard to run. > Has anyone actually used py.test for a project that can chime in and > give a sense of how it compares to nose? The last time I looked it > wasn't nearly as well-documented as it seems to be now, so it may be a > better alternative... I found py.test to be approximately interchangeable with nose. It works about the same way and is similarly easy to use. The documentation is now in pretty good condition. For simple test functions or unittest objects, nose and py.test can run the same test source files. If you use generators, I think the details are different, but the capabilities are the same. For internal use here, I probably would have recommended switching to py.test already except for one thing: I already have a nose plugin that feeds results to my reporting system, and I do not yet have one for py.test. But that is an accident of history. If somebody writes me a plugin for py.test, I would start using it immediately. Mark S. b.t.w. I experimented with py.test plugins a bit. I found the architecture cleaner/easier than nose, though I never actually finished writing my plugin for it. This may be interesting to me, but maybe not to the rest of you: if you need a plugin to test your package, you should think about whether you might be doing something wrong. From jh at physics.ucf.edu Mon Jul 11 14:23:13 2011 From: jh at physics.ucf.edu (Joe Harrington) Date: Mon, 11 Jul 2011 14:23:13 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: (message from Erik Tollerud on Sat, 9 Jul 2011 23:21:28 -0700) References: Message-ID: I apologize profusely for replying on the wrong thread! I've re-posted here so discussion can follow in the right one. Sorry for the extra traffic. > http://astropy.wikispaces.com/Astropy+Coding+Guidelines I would strongly encourage: 1. Use the numpy docstring format. We spent a *lot* of time designing this format and getting community feedback to ensure that projects like this one can use it, in addition to the core numpy doc project. It's been vetted in the writing of over 1000 pages of docs. It can handle images, formulae, and cross references and there are guidelines for how to do that and how to make sure that the ASCII version is also sensible. It's what numpy and scipy users are now used to. It's based on Sphinx, so it makes a PDF, HTML, and of course ASCII, has doctests, etc. There's a place for references, and a way to separate the basic info everyone needs to know from the details ("notes") only specialists care about. There is one mod that the Matplotlib folks have, for the special case of families of routines that have dozens or hundreds of identical, optional parameters (common for graphics routines). Important: Keep the one-line summary truly to less than about 60 characters. It can be used to make lists of routines, one line per routine, including the name, for easy indexing. This breaks if the "one line" goes on and on. Here is the standard: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt If you want to see what the results look like, just type help(np.whatever), or look at the PDF or HTML pages at docs.scipy.org. 2. Use the docs.scipy.org wiki system, pydocweb. Make a project either there or on the astropy system; the code is OSS. This will allow normal users who would not otherwise have access to the code repo to submit doc fixes, examples, and tests; perform peer review; and discuss specific pages. This is extremely helpful if the programmer is lazy (or even functionally illiterate ;-), or when importing a package that's already written and has poor docs (e.g., numpy 3 years ago). It can sync in both directions with the VCS, gives metrics of who contributed what, what needs editing now, what got worked on this week, etc. You can see who wrote what, and revert if there's a problem. Check out these features at http://docs.scipy.org/numpy/ A description of the workflow and links to the style guidelines, templates, and examples are here: http://docs.scipy.org/numpy/Front%20Page/ Source code to pydocweb is here: http://code.google.com/p/pydocweb/ The project at numpy is fairly dormant now, the main docs having been written (the community fell flat on its face and didn't step up to review), but the folks are all available on the scipy-dev mailing list if you want to ask questions about it. The main developer is Pauli Virtanen. We wrote status articles in the 2008 and 2009 SciPy Conference proceedings: http://conference.scipy.org/proceedings/SciPy2008/paper_7/ http://conference.scipy.org/proceedings/SciPy2009/paper_14/ I hope the community will accept this format and system. It's field-tested! --jh-- From msk at astro.umd.edu Mon Jul 11 14:36:13 2011 From: msk at astro.umd.edu (Michael S. Kelley) Date: Mon, 11 Jul 2011 14:36:13 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: Hi all, I agree with the strong encouragement and arguments for the numpy docstring format. In addition to the fact that the numpydoc format is what numpy and scipy users are used to reading, it will also be frequently encountered by astropy users when they read the numpy/scipy documentation (and, to some extent, matplotlib). I also find the numpydoc format easy to parse and understand in the on-line help (i.e., via python help() or ipython "?"). I think the alternate format discussed by Erik T. and Prasanth is OK, and I've encountered it before (e.g., while reading the AstroLibCoords documentation). However, I think this format is more cumbersome to read because the use of the additional ":param" or ":returns" at the beginning of each variable definition, and the use of ":type". For me, these phrases make the text more difficult to quickly search and find the variable name that I would be looking for when using the on-line help. numpydoc's headings "Parameters", "Returns", etc. is much easier to read. - Mike On Mon, Jul 11, 2011 at 2:23 PM, Joe Harrington wrote: > I apologize profusely for replying on the wrong thread! ?I've > re-posted here so discussion can follow in the right one. ?Sorry for > the extra traffic. > >> http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > I would strongly encourage: > > 1. Use the numpy docstring format. ?We spent a *lot* of time designing > ? this format and getting community feedback to ensure that projects > ? like this one can use it, in addition to the core numpy doc > ? project. ?It's been vetted in the writing of over 1000 pages of > ? docs. ?It can handle images, formulae, and cross references and > ? there are guidelines for how to do that and how to make sure that > ? the ASCII version is also sensible. ?It's what numpy and scipy > ? users are now used to. ?It's based on Sphinx, so it makes a PDF, > ? HTML, and of course ASCII, has doctests, etc. ?There's a place for > ? references, and a way to separate the basic info everyone needs to > ? know from the details ("notes") only specialists care about. ?There > ? is one mod that the Matplotlib folks have, for the special case of > ? families of routines that have dozens or hundreds of identical, > ? optional parameters (common for graphics routines). > > ? Important: Keep the one-line summary truly to less than about 60 > ? characters. ?It can be used to make lists of routines, one line per > ? routine, including the name, for easy indexing. ?This breaks if the > ? "one line" goes on and on. > > ? Here is the standard: > > ? https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt > > ? If you want to see what the results look like, just type > ? help(np.whatever), or look at the PDF or HTML pages at > ? docs.scipy.org. > > 2. Use the docs.scipy.org wiki system, pydocweb. ?Make a project > ? either there or on the astropy system; the code is OSS. ?This will > ? allow normal users who would not otherwise have access to the code > ? repo to submit doc fixes, examples, and tests; perform peer review; > ? and discuss specific pages. ?This is extremely helpful if the > ? programmer is lazy (or even functionally illiterate ;-), or when > ? importing a package that's already written and has poor docs (e.g., > ? numpy 3 years ago). ?It can sync in both directions with the VCS, > ? gives metrics of who contributed what, what needs editing now, what > ? got worked on this week, etc. ?You can see who wrote what, and > ? revert if there's a problem. ?Check out these features at > > ? http://docs.scipy.org/numpy/ > > ? A description of the workflow and links to the style guidelines, > ? templates, and examples are here: > > ? http://docs.scipy.org/numpy/Front%20Page/ > > ? Source code to pydocweb is here: > > ? http://code.google.com/p/pydocweb/ > > ? The project at numpy is fairly dormant now, the main docs having > ? been written (the community fell flat on its face and didn't step > ? up to review), but the folks are all available on the scipy-dev > ? mailing list if you want to ask questions about it. ?The main > ? developer is Pauli Virtanen. ?We wrote status articles in the 2008 > ? and 2009 SciPy Conference proceedings: > > ? http://conference.scipy.org/proceedings/SciPy2008/paper_7/ > ? http://conference.scipy.org/proceedings/SciPy2009/paper_14/ > > > ? I hope the community will accept this format and system. ?It's > ? field-tested! > > --jh-- > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Michael S Kelley Department of Astronomy University of Maryland -- College Park 301-405-3796 From derek at astro.physik.uni-goettingen.de Mon Jul 11 14:54:37 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Mon, 11 Jul 2011 20:54:37 +0200 Subject: [AstroPy] Testing Guidelines In-Reply-To: References: Message-ID: <00945222-7EE6-421B-AB46-A534A4E4A733@astro.physik.uni-goettingen.de> On 10.07.2011, at 8:08AM, Erik Tollerud wrote: > As I mentioned in my message a moment ago, I have added a page on the > astropy wikispaces page dealing with testing for the astropy package: > > http://astropy.wikispaces.com/Astropy+Testing+Guidelines > > The page itself has basically no content - we will fill it in once > we've had a good discussion here on the list as to what the community > is comfortable with. There has already been some discussion on this > topic on the list, so this message is to serve as a place to house the > "official" discussion of what the actual package guidelines document > should say. > > Once item I would like to put out for discussion: unless the community > strongly disagrees, we (the coordinating committee) would prefer not > to *require* complete coverage of unit or compliance testing, as we > are concerned that this might be a major burden on some > packages/submodules that would overly slow development. > What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion. I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly developed or revised code, also to encourage test-driven development practices. > Regardless, probably the most important starting point is to determine > the framework (Mark already began this discussion, so I've pasted his > message below - sorry to hijack the thread, but I'm hoping to use the > guideline documents to organization the discussion). Once that's > decided, we can discuss the best arrangement and policies for how to > organize tests. > > > > On that topic, prompted by Mark's comments below, I'll clarify that > the "using nose" that I had in mind in my earlier comments was to use > nose as it is, using only built-in plugins that seem to be fairly > stable as it stands right now (although I've never personally used it > with python 3.x - has anyone?). My experiences with nose have been > pretty painless, and it's nice that it allows us to combine most of > the different testing options into one framework (in particular, the > combination of doctests and tests in a separate "tests" directory is > attractive). It might be reasonable to use nose as a unit test > framework, and then have a separate "compliance" test suite that might > depend on external tools (as suggested by both Paul and Vicki). I > think having 4 different test suites as (at least, I think) suggested > by Paul might be a bit burdensome, though. > I can confirm nose-1.0.0 supports Python 3 - just about as well as Python 2.x, afaikt (Python 3 support indeed seemed to have stalled until a few months ago, allowing the impression nose development was not very much alive...). I found Mark's comments about numpy and nose possibly a bit confusing - numpy.test() definitely depends on nose, and in fact I cannot find vastly different behaviour in the current numpy master to calling nosetests directly. For example, calling nosetests2.7 [3.2] in site-packages/numpy is running 3615 [3614] tests with 10 [13] errors, while numpy.test('full') runs 3609 [3608] tests with 4 [5] known failures with the respective python versions. So the main difference seems to be the way nosetests deals (or deals not) with tests decorated as known failures or otherwise to be skipped. But otherwise, as you can call nosetests in the "tests" subdir of every subpackage, it allows of course for quote some more flexibility. > Has anyone actually used py.test for a project that can chime in and > give a sense of how it compares to nose? The last time I looked it > wasn't nearly as well-documented as it seems to be now, so it may be a > better alternative... I have no experience with it - so personally, and also because there are plenty of examples in the numpy/scipy code etc. to build on, I am biased towards nose, but of course this does not mean one should exclude a possibly superior alternative (especially not at this point). Cheers, Derek > On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz wrote: >> >>>> As for the testing framework, we were thinking nose >>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). Nose is >>>> very nice for a project like this because it's super easy to run and >>>> also easy to write tests with, but is also compatible with the stdlib >>>> unittests. >> >> >> "use nose" is a little ambiguous. numpy "uses nose" but if you look >> inside numpy.test() it looks like of like they implemented their own >> testing system using nose as a library. I've never succeeded in using >> nosetests to run the numpy tests, for example. >> >> If you choose nose, my preference is to say: Tests come in a separate >> directory from the source. You can run all the tests with the command >> "nosetests tests/". >> >> This is not to exclude the possibility of installing some of the tests >> or of providing a test() function, but I also want to run the tests >> directly with nosetests. >> >> I actually have an ulterior motive here: I use a nose plugin that feeds >> a report generator. If you only provide a foo.test() function, I can't >> necessarily get it to use the plugin even if you base the whole thing on >> nose. You might not think that was a big deal, but it gets important >> when you have the test volume that I regularly deal with: > 100 000 >> tests per night, sometimes > 10 000 fail, depending what the developers >> are up to. >> >> b.t.w. This doesn't mean I have any attachment to nose; it just happens >> to be one of a small number of test runners that I use right now. >> > From perry at stsci.edu Mon Jul 11 15:31:35 2011 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 11 Jul 2011 15:31:35 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E1B1F52.8080000@stsci.edu> References: <4E1B18B1.4060803@stsci.edu> <4E1B1F52.8080000@stsci.edu> Message-ID: <6B8BE994-2670-439E-9A65-0985CBADD1E7@stsci.edu> On Jul 11, 2011, at 12:05 PM, Michael Droettboom wrote: > With regard to: "C extensions will be allowed only when the provide > a significant performance enhancement over pure python. When C > extensions are used, the python interface must meet interface > guidelines, and the use of Cython is strongly recommended." > > In the case of something like pywcs, a C extension is used because > the WCS standard is so difficult to implement that it was easier to > use existing and well-tested code. It's not clear whether there was > a performance advantage over a Numpy-based implementation (I'm not > taking a position either way, I'm saying that such an experiment was > never done, and performance was not a motivating factor.) There are > a lot of good reasons beyond performance to use non-Python extension > code, so I don't know if we should artificially limit ourselves here. > That's a good point and we should be clearer about that. Where a good C library already exists, and it doesn't pose any serious design compromises to use it, it should be fine to do so (that's what much of scipy is all about after all). > With respect to: "Packages can include data in [path tbd] as long as > it is less than about 100 kb. These data should be accessed via the > astropy.config.[funcname tbd] mechanism. If the data exceeds this > size, it should be hosted outside the source code repository and > downloaded using the astropy.config.[funcname tbd] mechanism." > > I worry that there are often version dependencies between the code > and the data. (Matplotlib has a dependency on specific versions of > the STIX math fonts, for example). Having the data in a separate > code repository makes keeping this synchronized trickier. I would > instead say: large data required for functioning of the library goes > in the repository, but under a special directory that is not > distributed as part of the binary distribution. The mechanism for > accessing this data when needed should be aware of the source code > revision and download the corresponding revision of the data. The intent was to prevent the inclination of just dumping all binary data into repositories. This has happened with FITS files for us in the past and it wasn't a good thing to do in general (and there usually aren't significant version couplings there). We've had people continue to do it with some our repositories even when the policy says not to. There are certainly good exceptions to this (and the matplotlib example of one of these). I suppose the exception has to be illustrated by likely version dependencies in the binary data. Does that seem reasonable? But I think the default should be to disallow it and require someone to argue for doing it rather than the other way around. Perry From erik.tollerud at gmail.com Mon Jul 11 18:58:51 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 11 Jul 2011 15:58:51 -0700 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: I can definitely see the value of the numpy/scipy system, given the reasons Joe and Michael have given here. In particular, the actual sectioning/organization of the docstring definitely makes sense for our needs and consistency with numpy/scipy is an added bonus. However, when I thought about it earlier, I saw a number of issues with using the full numpy build doc system + pydocweb for the particular needs of astropy. I will say right now that any of these may simply be due to my lack of knowledge of how the system works, so if there are easily solutions I've missed, I'd love to hear them! * First, and most importantly, it's not clear to me how to adapt the numpy doc builder to our needs. I've actually never gotten it to run completely on numpy, despite trying on three different platforms. I'm willing to concede this could well be user error, but it implies that it might be difficult to use on a bunch of new projects - there seem to be 8 separate extensions in numpy's "sphinxet". Remember that the plan we have set out here involves a large number of packages that operate semi-independently until they are merged into the core, and they must be able to build their documentation independently of the core, at least until they are merged (and many will probably never be merged). Is there a separate project that is documented and can be easily adapted/simplified to suit our needs? * Does the pydocweb system still work since scipy moved to github (or in general is it VCS-agnostic)? I have edited scipy docs using the system before, but it was never clear to me how the web interface and the VCS interplayed with each other. I would say we want to keep the docs themselves in sync with the VCS version of the code they apply to - is that always the case with pydocweb? *The pydocweb workflow diagram requires quite a few steps to completion, and as Joe points out, it seems the review system ran out of steam - and indeed looking at the numpy docs nearly everything is still listed as "need review." Given that our community is probably smaller and has fewer resources to devote to reviewing docs, I'm concerned this workflow will not work for astropy (particularly for the affiliated packages). My understanding is that the review system was designed basically because there was a bunch of poorly-documented functionality, so the idea was to bring other people in to do the documentation. With the vision in place for astropy, nothing even makes it into the core in the first place if it doesn't have fairly complete documentation. That's one of the most important jobs of the coordinating committee. So with that in mind, might the full pydocweb scheme actually be too *complex* for our needs? Or is it easy to simplify it for our needs (perhaps just a single "review" stage that comes immediately after the code is written?) *How does pydocweb handle properties? We definitely want to be able to document properties directly, and have them still end up in the sphinx docs alongside the docs for the class' attributes. *pydocweb requires django to operate, according to it's docs - so that implies that either we have to have a central host to hold all of the affiliated projects' pydocweb projects while they are being written, or we have to expect people to figure that out on their own... By contrast, a simpler sphinx setup only requires "python setup.py build_docs upload_docs" and then the latest docs are at packages.python.org, and fully accessible via PyPI. Or is there a way to bring pydocweb onto packages.python.org ? Given that sphinx now has tools to generate APIs entirely from docstrings (the built-in autodoc and autosummary extensions) without any extra scripts needed, might it make more sense to use the numpy docstring format, but couple it instead to these sphinx tools? Sphinx also has markup specific to all of the sections that the numpy docs call for, also, so it's simple to use the ".. seealso::" and ".. notes" constructs to get the same effect as the numpy docstrings without the additional numpy scripts. This of would mean we'd either have to change the syntax of the numpydoc scheme (although hopefully not the sectioning), or write a sphinx extension to suit our needs. > I think the alternate format discussed by Erik T. and Prasanth is OK, > and I've encountered it before (e.g., while reading the AstroLibCoords > documentation). ?However, I think this format is more cumbersome to > read because the use of the additional ":param" or ":returns" at the > beginning of each variable definition, and the use of ":type". ?For > me, these phrases make the text more difficult to quickly search and > find the variable name that I would be looking for when using the > on-line help. ?numpydoc's headings "Parameters", "Returns", etc. is > much easier to read. I agree this is a bit more cumbersome to read, but it produces (at least, to my eye) much prettier sphinx APIs if you use it with autodoc/autosummary. So I guess the question is: do we want the docstrings as they appear with ipython's "?" or help() to be prioritized over the sphinx output, vice-versa, or what? (Oh, and there's only need for one "returns" clause, and as I was saying before, the "type" can be excluded if it's judged to make it too hard to read. The "param name" of course has to be there, though). On Mon, Jul 11, 2011 at 11:23 AM, Joe Harrington wrote: > I apologize profusely for replying on the wrong thread! ?I've > re-posted here so discussion can follow in the right one. ?Sorry for > the extra traffic. > >> http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > I would strongly encourage: > > 1. Use the numpy docstring format. ?We spent a *lot* of time designing > ? this format and getting community feedback to ensure that projects > ? like this one can use it, in addition to the core numpy doc > ? project. ?It's been vetted in the writing of over 1000 pages of > ? docs. ?It can handle images, formulae, and cross references and > ? there are guidelines for how to do that and how to make sure that > ? the ASCII version is also sensible. ?It's what numpy and scipy > ? users are now used to. ?It's based on Sphinx, so it makes a PDF, > ? HTML, and of course ASCII, has doctests, etc. ?There's a place for > ? references, and a way to separate the basic info everyone needs to > ? know from the details ("notes") only specialists care about. ?There > ? is one mod that the Matplotlib folks have, for the special case of > ? families of routines that have dozens or hundreds of identical, > ? optional parameters (common for graphics routines). > > ? Important: Keep the one-line summary truly to less than about 60 > ? characters. ?It can be used to make lists of routines, one line per > ? routine, including the name, for easy indexing. ?This breaks if the > ? "one line" goes on and on. > > ? Here is the standard: > > ? https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt > > ? If you want to see what the results look like, just type > ? help(np.whatever), or look at the PDF or HTML pages at > ? docs.scipy.org. > > 2. Use the docs.scipy.org wiki system, pydocweb. ?Make a project > ? either there or on the astropy system; the code is OSS. ?This will > ? allow normal users who would not otherwise have access to the code > ? repo to submit doc fixes, examples, and tests; perform peer review; > ? and discuss specific pages. ?This is extremely helpful if the > ? programmer is lazy (or even functionally illiterate ;-), or when > ? importing a package that's already written and has poor docs (e.g., > ? numpy 3 years ago). ?It can sync in both directions with the VCS, > ? gives metrics of who contributed what, what needs editing now, what > ? got worked on this week, etc. ?You can see who wrote what, and > ? revert if there's a problem. ?Check out these features at > > ? http://docs.scipy.org/numpy/ > > ? A description of the workflow and links to the style guidelines, > ? templates, and examples are here: > > ? http://docs.scipy.org/numpy/Front%20Page/ > > ? Source code to pydocweb is here: > > ? http://code.google.com/p/pydocweb/ > > ? The project at numpy is fairly dormant now, the main docs having > ? been written (the community fell flat on its face and didn't step > ? up to review), but the folks are all available on the scipy-dev > ? mailing list if you want to ask questions about it. ?The main > ? developer is Pauli Virtanen. ?We wrote status articles in the 2008 > ? and 2009 SciPy Conference proceedings: > > ? http://conference.scipy.org/proceedings/SciPy2008/paper_7/ > ? http://conference.scipy.org/proceedings/SciPy2009/paper_14/ -- Erik Tollerud From thomas.robitaille at gmail.com Mon Jul 11 19:27:24 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 11 Jul 2011 19:27:24 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: Just a quick note regarding the numpy sphinx ext. It can be installed as a standalone package from http://pypi.python.org/pypi/numpydoc/0.3.1 (though I'm not sure how up to date that is). Maybe it can also be directly imported from numpy? (I haven't delved deep enough into this) I've created a fake package and docs here to demonstrate the various formats we are talking about: https://github.com/astrofrog/astropy_doc_test Note, this is just a personal repo, nothing official. You need to clone the repo and open doc/build/index.html. Note that all I had to do to get the numpy format to work correctly was install numpydoc above and change: extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest'] to extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'numpydoc'] in conf.py, so it doesn't seem too hard. Of course, there's other features in the numpy docstring format that I haven't demonstrated here, this is just for a simple case. I called Erik and Prasanth's format "Sphinx-suggested" since it's what the sphinx docs suggest. If you have other ideas of docstrings, feel free to fork the repo and optionally merge back via pull requests! Cheers, Tom On 11 July 2011 18:58, Erik Tollerud wrote: > I can definitely see the value of the numpy/scipy system, given the > reasons Joe and Michael have given here. ?In particular, the actual > sectioning/organization of the docstring definitely makes sense for > our needs and consistency with numpy/scipy is an added bonus. > > However, when I thought about it earlier, I saw a number of issues > with using the full numpy build doc system + pydocweb for the > particular needs of astropy. ?I will say right now that any of these > may simply be due to my lack of knowledge of how the system works, so > if there are easily solutions I've missed, I'd love to hear them! > > * First, and most importantly, it's not clear to me how to adapt the > numpy doc builder to our needs. ?I've actually never gotten it to run > completely on numpy, despite trying on three different platforms. ?I'm > willing to concede this could well be user error, but it implies that > it might be difficult to use on a bunch of new projects - there seem > to be 8 separate extensions in numpy's "sphinxet". Remember that the > plan we have set out here involves a large number of packages that > operate semi-independently until they are merged into the core, and > they must be able to build their documentation independently of the > core, at least until they are merged (and many will probably never be > merged). ?Is there a separate project that is documented and can be > easily adapted/simplified to suit our needs? > > * Does the pydocweb system still work since scipy moved to github (or > in general is it VCS-agnostic)? I have edited scipy docs using the > system before, but it was never clear to me how the web interface and > the VCS interplayed with each other. ?I would say we want to keep the > docs themselves in sync with the VCS version of the code they apply to > - is that always the case with pydocweb? > > *The pydocweb workflow diagram requires quite a few steps to > completion, and as Joe points out, it seems the review system ran out > of steam - and indeed looking at the numpy docs nearly everything is > still listed as "need review." ?Given that our community is probably > smaller and has fewer resources to devote to reviewing docs, I'm > concerned this workflow will not work for astropy (particularly for > the affiliated packages). ?My understanding is that the review system > was designed basically because there was a bunch of poorly-documented > functionality, so the idea was to bring other people in to do the > documentation. ?With the vision in place for astropy, nothing even > makes it into the core in the first place if it doesn't have fairly > complete documentation. ?That's one of the most important jobs of the > coordinating committee. ?So with that in mind, might the full pydocweb > scheme actually be too *complex* for our needs? Or is it easy to > simplify it for our needs (perhaps just a single "review" stage that > comes immediately after the code is written?) > > *How does pydocweb handle properties? ?We definitely want to be able > to document properties directly, and have them still end up in the > sphinx docs alongside the docs for the class' attributes. > > *pydocweb requires django to operate, according to it's docs - so that > implies that either we have to have a central host to hold all of the > affiliated projects' pydocweb projects while they are being written, > or we have to expect people to figure that out on their own... By > contrast, a simpler sphinx setup only requires "python setup.py > build_docs upload_docs" and then the latest docs are at > packages.python.org, and fully accessible via PyPI. ?Or is there a way > to bring pydocweb onto packages.python.org ? > > Given that sphinx now has tools to generate APIs entirely from > docstrings (the built-in autodoc and autosummary extensions) without > any extra scripts needed, might it make more sense to use the numpy > docstring format, but couple it instead to these sphinx tools? Sphinx > also has markup specific to all of the sections that the numpy docs > call for, also, so it's simple to use the ".. seealso::" and ".. > notes" constructs to get the same effect as the numpy docstrings > without the additional numpy scripts. ?This of would mean we'd either > have to change the syntax of the numpydoc scheme (although hopefully > not the sectioning), or write a sphinx extension to suit our needs. > > > >> I think the alternate format discussed by Erik T. and Prasanth is OK, >> and I've encountered it before (e.g., while reading the AstroLibCoords >> documentation). ?However, I think this format is more cumbersome to >> read because the use of the additional ":param" or ":returns" at the >> beginning of each variable definition, and the use of ":type". ?For >> me, these phrases make the text more difficult to quickly search and >> find the variable name that I would be looking for when using the >> on-line help. ?numpydoc's headings "Parameters", "Returns", etc. is >> much easier to read. > > I agree this is a bit more cumbersome to read, but it produces (at > least, to my eye) much prettier sphinx APIs if you use it with > autodoc/autosummary. ?So I guess the question is: do we want the > docstrings as they appear with ipython's "?" or help() to be > prioritized over the sphinx output, vice-versa, or what? > > (Oh, and there's only need for one "returns" clause, and as I was > saying before, the "type" can be excluded if it's judged to make it > too hard to read. ?The "param name" of course has to be there, > though). > > > > On Mon, Jul 11, 2011 at 11:23 AM, Joe Harrington wrote: >> I apologize profusely for replying on the wrong thread! ?I've >> re-posted here so discussion can follow in the right one. ?Sorry for >> the extra traffic. >> >>> http://astropy.wikispaces.com/Astropy+Coding+Guidelines >> >> I would strongly encourage: >> >> 1. Use the numpy docstring format. ?We spent a *lot* of time designing >> ? this format and getting community feedback to ensure that projects >> ? like this one can use it, in addition to the core numpy doc >> ? project. ?It's been vetted in the writing of over 1000 pages of >> ? docs. ?It can handle images, formulae, and cross references and >> ? there are guidelines for how to do that and how to make sure that >> ? the ASCII version is also sensible. ?It's what numpy and scipy >> ? users are now used to. ?It's based on Sphinx, so it makes a PDF, >> ? HTML, and of course ASCII, has doctests, etc. ?There's a place for >> ? references, and a way to separate the basic info everyone needs to >> ? know from the details ("notes") only specialists care about. ?There >> ? is one mod that the Matplotlib folks have, for the special case of >> ? families of routines that have dozens or hundreds of identical, >> ? optional parameters (common for graphics routines). >> >> ? Important: Keep the one-line summary truly to less than about 60 >> ? characters. ?It can be used to make lists of routines, one line per >> ? routine, including the name, for easy indexing. ?This breaks if the >> ? "one line" goes on and on. >> >> ? Here is the standard: >> >> ? https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt >> >> ? If you want to see what the results look like, just type >> ? help(np.whatever), or look at the PDF or HTML pages at >> ? docs.scipy.org. >> >> 2. Use the docs.scipy.org wiki system, pydocweb. ?Make a project >> ? either there or on the astropy system; the code is OSS. ?This will >> ? allow normal users who would not otherwise have access to the code >> ? repo to submit doc fixes, examples, and tests; perform peer review; >> ? and discuss specific pages. ?This is extremely helpful if the >> ? programmer is lazy (or even functionally illiterate ;-), or when >> ? importing a package that's already written and has poor docs (e.g., >> ? numpy 3 years ago). ?It can sync in both directions with the VCS, >> ? gives metrics of who contributed what, what needs editing now, what >> ? got worked on this week, etc. ?You can see who wrote what, and >> ? revert if there's a problem. ?Check out these features at >> >> ? http://docs.scipy.org/numpy/ >> >> ? A description of the workflow and links to the style guidelines, >> ? templates, and examples are here: >> >> ? http://docs.scipy.org/numpy/Front%20Page/ >> >> ? Source code to pydocweb is here: >> >> ? http://code.google.com/p/pydocweb/ >> >> ? The project at numpy is fairly dormant now, the main docs having >> ? been written (the community fell flat on its face and didn't step >> ? up to review), but the folks are all available on the scipy-dev >> ? mailing list if you want to ask questions about it. ?The main >> ? developer is Pauli Virtanen. ?We wrote status articles in the 2008 >> ? and 2009 SciPy Conference proceedings: >> >> ? http://conference.scipy.org/proceedings/SciPy2008/paper_7/ >> ? http://conference.scipy.org/proceedings/SciPy2009/paper_14/ > > > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From sienkiew at stsci.edu Tue Jul 12 14:00:33 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 12 Jul 2011 14:00:33 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: <4E1C8BC1.7060103@stsci.edu> > I agree this is a bit more cumbersome to read, but it produces (at > least, to my eye) much prettier sphinx APIs if you use it with > autodoc/autosummary. So I guess the question is: do we want the > docstrings as they appear with ipython's "?" or help() to be > prioritized over the sphinx output, vice-versa, or what? I think help() should produce useful output. I also think a programmer who is reading the source code for the function should be able to read the docstring directly in the source code. Fortunately, I don't think we need to make a compromise. Here is an example: https://trac6.assembla.com/astrolib/browser/trunk/pywcs/lib/pywcs/pywcs.py#L126 http://www.astrolib.org/docs/development/html/pywcs/api_wcs.html#wcs If that is good enough, then we don't need to consider more complicated markup in the docstring. From thomas.robitaille at gmail.com Tue Jul 12 14:11:39 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 12 Jul 2011 14:11:39 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: <4E1C8BC1.7060103@stsci.edu> References: <4E1C8BC1.7060103@stsci.edu> Message-ID: I've added this to the options discussed so far shown at https://github.com/astrofrog/astropy_doc_test Cheers, Tom On 12 July 2011 14:00, Mark Sienkiewicz wrote: > >> I agree this is a bit more cumbersome to read, but it produces (at >> least, to my eye) much prettier sphinx APIs if you use it with >> autodoc/autosummary. ?So I guess the question is: do we want the >> docstrings as they appear with ipython's "?" or help() to be >> prioritized over the sphinx output, vice-versa, or what? > > > I think help() should produce useful output. ?I also think a programmer > who is reading the source code for the function should be able to read > the docstring directly in the source code. > > Fortunately, I don't think we need to make a compromise. ?Here is an > example: > > https://trac6.assembla.com/astrolib/browser/trunk/pywcs/lib/pywcs/pywcs.py#L126 > > http://www.astrolib.org/docs/development/html/pywcs/api_wcs.html#wcs > > If that is good enough, then we don't need to consider more complicated > markup in the docstring. > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From sienkiew at stsci.edu Tue Jul 12 15:26:03 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 12 Jul 2011 15:26:03 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: <4E1C8BC1.7060103@stsci.edu> Message-ID: <4E1C9FCB.3090501@stsci.edu> On 7/12/11 2:11 PM, Thomas Robitaille wrote: > I've added this to the options discussed so far shown at > https://github.com/astrofrog/astropy_doc_test I thought that file is supposed to be numpydoc format. Perhaps I am mistaken. The point stands, though, that we don't need a particularly cluttered format to make the sphinx docs look reasonable. Mark > Cheers, > Tom > > On 12 July 2011 14:00, Mark Sienkiewicz wrote: >>> I agree this is a bit more cumbersome to read, but it produces (at >>> least, to my eye) much prettier sphinx APIs if you use it with >>> autodoc/autosummary. So I guess the question is: do we want the >>> docstrings as they appear with ipython's "?" or help() to be >>> prioritized over the sphinx output, vice-versa, or what? >> >> I think help() should produce useful output. I also think a programmer >> who is reading the source code for the function should be able to read >> the docstring directly in the source code. >> >> Fortunately, I don't think we need to make a compromise. Here is an >> example: >> >> https://trac6.assembla.com/astrolib/browser/trunk/pywcs/lib/pywcs/pywcs.py#L126 >> >> http://www.astrolib.org/docs/development/html/pywcs/api_wcs.html#wcs >> >> If that is good enough, then we don't need to consider more complicated >> markup in the docstring. >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> From erik.tollerud at gmail.com Tue Jul 12 15:26:27 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Tue, 12 Jul 2011 12:26:27 -0700 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: <4E1C8BC1.7060103@stsci.edu> Message-ID: To see all the docs actually rendered in sphinx (from Tom's repo), go to http://eteq.github.com/astropy_doc_test/ - that compares the formats suggested so far. > I think help() should produce useful output. ?I also think a programmer > who is reading the source code for the function should be able to read > the docstring directly in the source code. Oh, I think we all agree with that - I was asking if "slightly harder to read but still pretty easy" is an acceptable trade-off if the sphinx docs look significantly better. > If that is good enough, then we don't need to consider more complicated > markup in the docstring. Do you have any sectioning in the pywcs format? Some of the astropy classes will no doubt be rather complex, and hence they will require things like lists and "seealso" sections and references... I'm concerned that if the format doesn't have any syntactic hints (that sphinx will process to organize the doc), it will be difficult to read for more elaborate docstrings. On Tue, Jul 12, 2011 at 11:11 AM, Thomas Robitaille wrote: > I've added this to the options discussed so far shown at > https://github.com/astrofrog/astropy_doc_test > > Cheers, > Tom > > On 12 July 2011 14:00, Mark Sienkiewicz wrote: >> >>> I agree this is a bit more cumbersome to read, but it produces (at >>> least, to my eye) much prettier sphinx APIs if you use it with >>> autodoc/autosummary. ?So I guess the question is: do we want the >>> docstrings as they appear with ipython's "?" or help() to be >>> prioritized over the sphinx output, vice-versa, or what? >> >> >> I think help() should produce useful output. ?I also think a programmer >> who is reading the source code for the function should be able to read >> the docstring directly in the source code. >> >> Fortunately, I don't think we need to make a compromise. ?Here is an >> example: >> >> https://trac6.assembla.com/astrolib/browser/trunk/pywcs/lib/pywcs/pywcs.py#L126 >> >> http://www.astrolib.org/docs/development/html/pywcs/api_wcs.html#wcs >> >> If that is good enough, then we don't need to consider more complicated >> markup in the docstring. >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud From bkloppenborg at gmail.com Tue Jul 12 16:16:12 2011 From: bkloppenborg at gmail.com (Brian Kloppenborg) Date: Tue, 12 Jul 2011 14:16:12 -0600 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: <4E1C8BC1.7060103@stsci.edu> Message-ID: <4E1CAB8C.1040201@du.edu> Tom and others, On 07/12/2011 12:11 PM, Thomas Robitaille wrote: > I've added this to the options discussed so far shown at > https://github.com/astrofrog/astropy_doc_test > > Cheers, > Tom Thank you for making this display example. I vastly prefer the Numpy/Scipy format. The documentation is easy to read both in the source code and in the Sphinx output. It also will be in line with the dominant math/science package in Python so it appears a clear first choice to me. Commenting on the other two formats: The "Sphinx suggested Format" does produce nicely condensed output, but the source code and docstrings are much more difficult to read. Just to find the parameter names you have to visually parse :param str file_name: compared to file_name : str which, in my opinion, destroys the ease-of-lookup factor. The PyWCS format is by far my least favorite as the ouput parameter names from Sphinx are not in bold. Can this be changed? Brian From sienkiew at stsci.edu Tue Jul 12 16:23:29 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Tue, 12 Jul 2011 16:23:29 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: <4E1C8BC1.7060103@stsci.edu> Message-ID: <4E1CAD41.4060407@stsci.edu> >> If that is good enough, then we don't need to consider more complicated >> markup in the docstring. > Do you have any sectioning in the pywcs format? Some of the astropy > classes will no doubt be rather complex, and hence they will require > things like lists and "seealso" sections and references... I'm > concerned that if the format doesn't have any syntactic hints (that > sphinx will process to organize the doc), it will be difficult to read > for more elaborate docstrings. Well, what I know is pywcs is maintained here, and we are supposed to be moving to using numpy docstring format. Obviously, I picked out the wrong example, but it doesn't matter now that you have put examples up on your web page. We should just use the numpy format because it is so much easier to read than the sphinx format. The coding standards it should include a link to what the format is. b.t.w. If a package showed up with docstrings that look like the pywcs example, I wouldn't necessarily turn it away. (Rejecting sub-standard product is what the standards are about, after all.) If a package showed up with docstrings epydoc format, I might ask for the docstrings to be revised before accepting it into astropy. From jlu at astro.caltech.edu Tue Jul 12 19:57:07 2011 From: jlu at astro.caltech.edu (Jessica Lu) Date: Tue, 12 Jul 2011 16:57:07 -0700 Subject: [AstroPy] pysynphot issue with changes in numpy Message-ID: <53E92CA3-CCCE-4437-8027-400EE769FC2D@astro.caltech.edu> Hi, I use pysynphot and I have encountered a bug in the latest scisoft installation of pysynphot v0.8.2 and numpy v1.5.1. I have uploaded some atmosphere models from the literature (AMES dusty) and then I try to access them using the following call import pysynphot sp = pysynphot.Icat('AMESdusty', 3000.5, 0, 3.51) where Teff = 3000.5 K, metallicity = 0 ([M/H]) and log_gravity = 3.51. The issue appears to be that the only log_g values at 3100 K are [4, 4.5, 5, 5.5, 6.0] and the log_g values at 3000 K are [3.5, 4, 4.5, 5, 5.5, 6.0]. In catalog.py there is an uncaught exception at: lowerArray = MA.masked_inside(array, lower, par) from: list6,list7 = self._breakList(list2, 2, log_g) This appears to be because this call returns something different now: lower = MA.maximum(less) than it used to in previous versions of numpy. If less was already entirely masked, then numpy.ma.maximum(less) would return the whole less array over again, but now it appears to be set to a one-element masked array containing a numpy.ma.MaskedContent object. And masked_inside doesn't know what to do with this type of object (in numpy.ma.core.masked_inside, the condition = statement is what actually throws the exception as it expects both lower and par to be floats). Here is a spot test: import numpy log_g_array = numpy.array([4.0, 4.5]) log_g = 3.0 less = numpy.ma.masked_greater(log_g_array, log_g) lower = numpy.ma.maximum(less) # Here comes the exception, didn't used to do this. lowerArray = numpy.ma.masked_inside(log_g_array, lower, log_g) The fix I have is probably not optimal but seems to work. Replace the lowerArray = (and upperArray = ) lines with: # Catch the cases when the upper/lower return # is not valid (returns a list). This happens when # all the array values in greater/less are masked. if hasattr(upper, '__contains__'): upperArray = greater else: upperArray = MA.masked_inside(array, par, upper) if hasattr(lower, '__contains__'): lowerArray = less else: lowerArray = MA.masked_inside(array, lower, par) There is one other use-case which I am not sure how this code would deal with. If the various spectra in the model tables are not pre-sorted by increasing Teff, metallicity, and log_g, then some funny results could result from the lines in Icat.__init__(): sp1 = self._getSpectrum(list6[0], catdir) sp2 = self._getSpectrum(list7[0], catdir) ... because list6[0] is only the correct one to use if the are pre-sorted. Thanks, Jessica Lu From oneaufs at gmail.com Wed Jul 13 02:09:20 2011 From: oneaufs at gmail.com (Prasanth) Date: Wed, 13 Jul 2011 11:39:20 +0530 Subject: [AstroPy] Documentation Guidelines In-Reply-To: <4E1CAD41.4060407@stsci.edu> References: <4E1C8BC1.7060103@stsci.edu> <4E1CAD41.4060407@stsci.edu> Message-ID: Hello, Seeing numpy docstrings in action, I think that would be preferable to the Sphinx-suggested format. numpydoc can be pip installed, so installing it won't be a problem. And, like epydoc, we can generate inheritance diagrams using the sphinx.ext.inheritance_diagram extension. Prasanth On Wed, Jul 13, 2011 at 1:53 AM, Mark Sienkiewicz wrote: > > >> If that is good enough, then we don't need to consider more complicated > >> markup in the docstring. > > Do you have any sectioning in the pywcs format? Some of the astropy > > classes will no doubt be rather complex, and hence they will require > > things like lists and "seealso" sections and references... I'm > > concerned that if the format doesn't have any syntactic hints (that > > sphinx will process to organize the doc), it will be difficult to read > > for more elaborate docstrings. > > Well, what I know is pywcs is maintained here, and we are supposed to be > moving to using numpy docstring format. Obviously, I picked out the > wrong example, but it doesn't matter now that you have put examples up > on your web page. > > We should just use the numpy format because it is so much easier to read > than the sphinx format. The coding standards it should include a link > to what the format is. > > b.t.w. If a package showed up with docstrings that look like the pywcs > example, I wouldn't necessarily turn it away. (Rejecting sub-standard > product is what the standards are about, after all.) If a package > showed up with docstrings epydoc format, I might ask for the docstrings > to be revised before accepting it into astropy. > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Wed Jul 13 18:54:24 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Wed, 13 Jul 2011 15:54:24 -0700 Subject: [AstroPy] Testing Guidelines In-Reply-To: <00945222-7EE6-421B-AB46-A534A4E4A733@astro.physik.uni-goettingen.de> References: <00945222-7EE6-421B-AB46-A534A4E4A733@astro.physik.uni-goettingen.de> Message-ID: > b.t.w. ?I suggest that you differentiate the _purpose_ of a test from > the _mechanism_ of the test. This is a very good point - when I think about it more, it makes sense to have the same test-runner framework. With either nose or py.test it seems reasonable to divide the testing into tests which always depend only on the core modules, and a separate suite that is more general, potentially using e.g. pysofa to test coordinate conversions and the like. As Mark says, there isn't necessarily a need (at least at this stage) to carefully divide tests into categories, as long as testing coverage is good for the actual purpose of the module. > What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion. > I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly ?developed or revised code, also to encourage test-driven development practices. Absolutely agreed - the plan is to make it clear in the example package that everything should be unit-tested if it can be, as that's what new packages will presumably use. Now the nose vs. py.test question... From Mark's summary, it sounds like it's difficult to choose between them because they're too similar! Given Mark and Victoria's comments (and looking over the docs in more detail myself), I'm beginning to think that py.test has more potential. Especially important is that fact that it appears that py.test can directly use tests written for nose, while the converse is not true... this would make integrate already-existing projects with nose much easier and lower the re-learning penalty to pretty much nothing. Anyone else have experience with py.test that may be of relevance? > > Cheers, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek > >> On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz wrote: >>> >>>>> As for the testing framework, we were thinking nose >>>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). ? Nose is >>>>> very nice for a project like this because it's super easy to run and >>>>> also easy to write tests with, but is also compatible with the stdlib >>>>> unittests. >>> >>> >>> "use nose" is a little ambiguous. ?numpy "uses nose" but if you look >>> inside numpy.test() it looks like of like they implemented their own >>> testing system using nose as a library. ?I've never succeeded in using >>> nosetests to run the numpy tests, for example. >>> >>> If you choose nose, my preference is to say: ?Tests come in a separate >>> directory from the source. ?You can run all the tests with the command >>> "nosetests tests/". >>> >>> This is not to exclude the possibility of installing some of the tests >>> or of providing a test() function, but I also want to run the tests >>> directly with nosetests. >>> >>> I actually have an ulterior motive here: ?I use a nose plugin that feeds >>> a report generator. ?If you only provide a foo.test() function, I can't >>> necessarily get it to use the plugin even if you base the whole thing on >>> nose. ?You might not think that was a big deal, but it gets important >>> when you have the test volume that I regularly deal with: > 100 000 >>> tests per night, sometimes > 10 000 fail, depending what the developers >>> are up to. >>> >>> b.t.w. ?This doesn't mean I have any attachment to nose; it just happens >>> to be one of a small number of test runners that I use right now. >>> >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud From geert at barentsen.be Fri Jul 15 12:14:55 2011 From: geert at barentsen.be (Geert Barentsen) Date: Fri, 15 Jul 2011 17:14:55 +0100 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: Hi Erik et al, Many thanks for your efforts on astropy, this seems very worthwile ! My suggestion for the coding guidelines would be to urge people not to use multiple inheritance altogether. Your guideline to avoid "super()" does not remove ambiguity in the case where a subclass does -not- override a method that is implemented by 2 or more superclasses (in this case the super-mechanism still works its magic... unless you demand that every subclass in the whole chain overrides -every- method?) I don't think it is ever possible to use multiple inheritance without understanding the somewhat complex details. Simply having code guidelines which say "do not inherit from more than one class at a time" are far less scary than the complex story about super(). I'm happy to change my opinion if someone comes up with a good example where multiple inheritance makes life better :-) ... when I studied computer science in university the consensus was they are rare! Geert On 10 July 2011 06:39, Erik Tollerud wrote: > Hello everyone, > > Now that the vision and name for astropy have been ratified, the > Coordinating Committee (Thomas, Perry, and I) have been working on a > draft of coding guidelines/requirements for the astropy package. The > intent is to use these items to determine if code in affiliated > packages can be merged with the core package (as outlined in the > vision). I've posted our draft of these guidelines to the wikispaces > page: > > http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > Feel free to ask questions about or comment on any of these items. We > want to make sure the community is on board with a common set of > guidelines, so we have tried to take into account discussions that > have already occurred on this list in this draft. Many of these items > have not yet been discussed, however, so we will certainly adjust > these guidelines (or add new ones) based on your feedback once > consensus is reached on the list. We do ask, though, that the > comments/discussion be on the mailing list instead of the wiki page, > so that the archive of the mailing list preserves the full discussion. > > > Also, note that the guidelines reference both a documentation and > testing guidelines document. I have created pages for these documents > as well: > > Documentation: > http://astropy.wikispaces.com/Astropy+Documentation+Guidelines > Testing: http://astropy.wikispaces.com/Astropy+Testing+Guidelines > > These documents are not fleshed-out, as we wish to solicit ideas from > the community here. I will send additional messages to this mailing > list shortly to begin the discussion these topics, and ask again that > the discussion for each take place in their associated threads. > > > In the near future we (the committee) will also be sending out a > message on determining project hosting and choice of VCS, as well as a > set of priorities to guide development. Once we (the community) have > reached consensus on this items, we can begin the actual work in > earnest! > > Thanks in advance for your ideas and comments! > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Fri Jul 15 14:47:49 2011 From: perry at stsci.edu (Perry Greenfield) Date: Fri, 15 Jul 2011 14:47:49 -0400 Subject: [AstroPy] proposal for astropy version control software and respository Message-ID: Given discussions at scipy (consisting primarily of a good number of people from STScI, CFA/Chandra, and Gemini) it seemed clear to most of us there that the most straightforward choice for the astropy version control software and repository was git and github. This is what most of the science-based Python projects are using and they generally seem quite happy with it. Since we are encouraging distributed contributions, it also would seem we should favor a distributed vcs over a non-distributed one (i.e., svn). So we (Erik, Tom and me) are proposing that we simply put this particular choice to a yes/no vote. Here is a link to the poll: http://astropy.wikispaces.com/VCS+and+hosting+poll Should there be a significant level of objection to this, we'll look at seeing if there is a stronger consensus for something else. Perry From laidler at stsci.edu Fri Jul 15 16:24:15 2011 From: laidler at stsci.edu (Victoria G. Laidler) Date: Fri, 15 Jul 2011 16:24:15 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: Message-ID: <4E20A1EF.50207@stsci.edu> For those of us who are not very familiar with git and github, can someone suggest a source for additional information? I'd also like to note that in the discussion of the previous "simple yes/no poll", several people raised objections that this was too coarse a metric and that there is value in being able to signify "yes but" or "don't care" & so on. I'd like to second those objections and request that the Coordinating Committee formulate polls that allow more nuance. Perry Greenfield wrote: > Given discussions at scipy (consisting primarily of a good number of > people from STScI, CFA/Chandra, and Gemini) it seemed clear to most of > us there that the most straightforward choice for the astropy version > control software and repository was git and github. This is what most > of the science-based Python projects are using and they generally seem > quite happy with it. Since we are encouraging distributed > contributions, it also would seem we should favor a distributed vcs > over a non-distributed one (i.e., svn). So we (Erik, Tom and me) are > proposing that we simply put this particular choice to a yes/no vote. > Here is a link to the poll: > > http://astropy.wikispaces.com/VCS+and+hosting+poll > > Should there be a significant level of objection to this, we'll look > at seeing if there is a stronger consensus for something else. > > Perry > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From msk at astro.umd.edu Fri Jul 15 17:43:11 2011 From: msk at astro.umd.edu (Michael S. Kelley) Date: Fri, 15 Jul 2011 17:43:11 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <4E20A1EF.50207@stsci.edu> References: <4E20A1EF.50207@stsci.edu> Message-ID: Hello, On Fri, Jul 15, 2011 at 4:24 PM, Victoria G. Laidler wrote: > For those of us who are not very familiar with git and github, can someone > suggest a source for additional information? > I've only recently started to understand VCSs and Git, and I found The Git Community Book helpful: http://book.git-scm.com/ Cheers, Mike -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Michael S Kelley Department of Astronomy University of Maryland -- College Park 301-405-3796 From sienkiew at stsci.edu Fri Jul 15 17:50:25 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Fri, 15 Jul 2011 17:50:25 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: Message-ID: <4E20B621.4030105@stsci.edu> Perry Greenfield wrote: > Given discussions at scipy (consisting primarily of a good number of > people from STScI, CFA/Chandra, and Gemini) it seemed clear to most of > us there that the most straightforward choice for the astropy version > control software and repository was git and github. I was corresponding off-list (IIRC) about using git, but I'll repeat my major points here for everyone: - If the project will use git, the git advocates have a responsibility to present a clearly-defined work flow for each role in the project. Motivation: git implements several mutually-incompatible work flows. If you don't say which one you are using, you can get really screwed up. Also, what you do with git and why anybody would want to do that are not obvious to those of us who are not already converts. I am your audience: one of those subversion users who "look at git's features with confusion or disinterest". b.t.w. I hear that ipython has a work flow document that we can adapt from. - DVCS advocates in general seem to care deeply about the DVCSness of the system. I don't need anything from git that I can't get from subversion, but the git advocates are so insistent that I might as well just give up and say "ok, use git". Basically, they care, I don't, and I can probably make their favorite system work. Assuming they explain the work flow. But it would sure suck to lose the revision numbers that subversion has (but git does not) and not get something pretty intensely useful in return. I hope you git guys know what that is. Mark S. From erik.tollerud at gmail.com Fri Jul 15 20:54:19 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Fri, 15 Jul 2011 17:54:19 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <4E20A1EF.50207@stsci.edu> References: <4E20A1EF.50207@stsci.edu> Message-ID: > For those of us who are not very familiar with git and github, can someone > suggest a source for additional information? I'd say the github help page ( http://help.github.com/ ) is probably the best resources out there in terms of getting started quickly, along with the community book as suggested by Michael. If you're on OS X, there's also a few different git GUI tools - github has an app (get it from their web site), and gitx (http://gitx.frim.nl/) is a nice free history viewer. There are some nicer ones that can replace the command line interface completely if you're willing to pay: the best seem to be Tower ( http://www.git-tower.com/ ) and SourceTree ( http://www.sourcetreeapp.com/ ). There are a variety of open source options for linux (git cola, gitk, gitg, giggle), although most linux people are happy with the command line. There also exist some for windows, but I don't really know much about those. > I'd also like to note that in the discussion of the previous "simple > yes/no poll", several people raised objections that this was too coarse > a metric and that there is value in being able to signify "yes but" or > "don't care" & so on. I'd like to second those objections and request > that the Coordinating Committee formulate polls that allow more nuance. This is a very good point - I had been thinking we should do this for future documents, but it didn't occur to me that it's equally relevant for this poll... In the future, would "Yes", "Yes, but with reservations", and "No" be an acceptable set of options, or would you prefer something more fine-grained than that? > - If the project will use git, the git advocates have a responsibility > to present a clearly-defined work flow for each role in the project. Absolutely agreed - if the poll passes muster, we will certainly do this ASAP. Probably the easiest thing is to follow the standard workflow used by ipython, numpy, and scipy - it's based on a customizable template document called "gitwash", an example of which you can see at http://matthew-brett.github.com/pydagogue/gitwash_build.html > But it would sure suck to lose the revision numbers that subversion has > (but git does not) and not get something pretty intensely useful in > return. ?I hope you git guys know what that is. As an FYI, there are actually ways to get revision numbers in git (as long as it's understood that those "numbers" aren't unique across different forks of a repo), although it isn't quite as easy as svn, of course. -- Erik Tollerud From erwin at mpe.mpg.de Sat Jul 16 06:52:20 2011 From: erwin at mpe.mpg.de (Peter Erwin) Date: Sat, 16 Jul 2011 12:52:20 +0200 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: Message-ID: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> I'm concerned that this poll is perhaps a little premature and not sufficiently well justified. It feels a trifle too much like "A small group of us got together and decided what to do; we're not really interested in your input or in explaining our reasons or in having a wider discussion first, so just vote on it." (I realize I'm way overdoing the atrribution of sinister motives here, and I apologize; but this is in rather stark constrast with the ongoing discussion over documentation, which seems to be progressing well in evaluating the various alternatives and answering questions raised....) So... unpack, please! * Why git, and not mercurial (or even bazaar)? (My understanding is that mercurial -- which I've used -- is closer in syntax and commands to subversion than is the case for git, which argues for mercurial as possibly more accessible to people who haven't worked with either before, but who have worked with svn or CVS...) * Why github, and not, say Bitbucket or Google Code? (I'm not saying that this has to be a long, drawn-out discussion by any means; but at the moment the haste and lack of justification tempts me to vote "No" largely as method of protest.) -- Peter On Jul 15, 2011, at 8:47 PM, Perry Greenfield wrote: > Given discussions at scipy (consisting primarily of a good number of > people from STScI, CFA/Chandra, and Gemini) it seemed clear to most of > us there that the most straightforward choice for the astropy version > control software and repository was git and github. This is what most > of the science-based Python projects are using and they generally seem > quite happy with it. Since we are encouraging distributed > contributions, it also would seem we should favor a distributed vcs > over a non-distributed one (i.e., svn). So we (Erik, Tom and me) are > proposing that we simply put this particular choice to a yes/no vote. > Here is a link to the poll: > > http://astropy.wikispaces.com/VCS+and+hosting+poll > > Should there be a significant level of objection to this, we'll look > at seeing if there is a stronger consensus for something else. > > Perry > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)89 30000 3695 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From aldcroft at head.cfa.harvard.edu Sat Jul 16 07:32:32 2011 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Sat, 16 Jul 2011 06:32:32 -0500 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> Message-ID: On Sat, Jul 16, 2011 at 5:52 AM, Peter Erwin wrote: > I'm concerned that this poll is perhaps a little premature and not > sufficiently well justified. It feels a trifle too much like "A small > group of us got together and decided what to do; we're not really > interested in your input or in explaining our reasons or in having a > wider discussion first, so just vote on it." > > (I realize I'm way overdoing the atrribution of sinister motives here, > and I apologize; but this is in rather stark constrast with the ongoing > discussion over documentation, which seems to be progressing well in > evaluating the various alternatives and answering questions raised....) > > > So... unpack, please! > > * Why git, and not mercurial (or even bazaar)? > > (My understanding is that mercurial -- which I've used -- is closer > in syntax and commands to subversion than is the case for git, which > argues for mercurial as possibly more accessible to people who > haven't worked with either before, but who have worked with svn or > CVS...) > > * Why github, and not, say Bitbucket or Google Code? Based on what was evident at the SciPy conference, essentially everyone that is using DVCS is using github. Including Numpy, Scipy, IPython. I talked to people that have migrated away from google code (didn't ask precisely why, but the answer that you don't find a ton of Python projects there is probably a good one). During the talk on the amazing work on on IPython 0.11 over the last year, Fernando said that one huge factor in their success was using github. They did over 2000 commits and didn't totally hose things up. Granted they were coming from bzr/launchpad which has lots of problems, but the point is that github's UI is significantly better than the rest. Things like code review and cross-repo compares are amazing. A colleague of mine who is hardcore mercurial did a close comparison of github vs. bitbucket and decided that github is way better (and decided to use hg-git to basically take his hg repos and host on github as needed). > > > (I'm not saying that this has to be a long, drawn-out discussion by any > means; but at the moment the haste and lack of justification tempts me > to vote "No" largely as method of protest.) > > ? -- Peter > > On Jul 15, 2011, at 8:47 PM, Perry Greenfield wrote: > >> Given discussions at scipy (consisting primarily of a good number of >> people from STScI, CFA/Chandra, and Gemini) it seemed clear to most of >> us there that the most straightforward choice for the astropy version >> control software and repository was git and github. This is what most >> of the science-based Python projects are using and they generally seem >> quite happy with it. Since we are encouraging distributed >> contributions, it also would seem we should favor a distributed vcs >> over a non-distributed one (i.e., svn). So we (Erik, Tom and me) are >> proposing that we simply put this particular choice to a yes/no vote. >> Here is a link to the poll: >> >> http://astropy.wikispaces.com/VCS+and+hosting+poll >> >> Should there be a significant level of objection to this, we'll look >> at seeing if there is a stronger consensus for something else. >> >> Perry >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > ============================================================= > Peter Erwin ? ? ? ? ? ? ? ? ? Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de ? ? ? ? ? ? ?Physics, Giessenbachstrasse > tel. +49 (0)89 30000 3695 ? ? 85748 Garching, Germany > fax ?+49 (0)89 30000 3495 ? ? http://www.mpe.mpg.de/~erwin > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > From thomas.robitaille at gmail.com Sat Jul 16 09:20:08 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sat, 16 Jul 2011 09:20:08 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> Message-ID: Hi Peter, I understand your concerns that this might seem premature - just to explain our motives behind a yes/no poll at this stage: we got the impression from previous discussions on this list and from the discussions and success stories at scipy that this was something that most people would either agree with, or not have a strong opinion about. We can have had a lengthy debate about the detailed advantages of git vs hg vs svn vs other if needed, but we wanted to see whether that debate could be avoided in the first place if most people agreed anyway, hence the early poll (think of it as testing the waters). But we do want people to voice concerns or questions if you have any! For me, the fact that numpy, scipy, matplotlib, and ipython all went through the pain of migrating to git and github (which did not happen overnight) should be reassuring, and you can be sure that they all had lengthy discussions before doing this. This is not a case of 'the cool kids are doing it, so we should do it too', but I sincerely believe from previous experience that github is a great platform for reaching out to a community and getting as many people involved in the coding. I hope this clarifies things a little! Cheers, Tom On 16 July 2011 06:52, Peter Erwin wrote: > I'm concerned that this poll is perhaps a little premature and not > sufficiently well justified. It feels a trifle too much like "A small > group of us got together and decided what to do; we're not really > interested in your input or in explaining our reasons or in having a > wider discussion first, so just vote on it." > > (I realize I'm way overdoing the atrribution of sinister motives here, > and I apologize; but this is in rather stark constrast with the ongoing > discussion over documentation, which seems to be progressing well in > evaluating the various alternatives and answering questions raised....) > > > So... unpack, please! > > * Why git, and not mercurial (or even bazaar)? > > (My understanding is that mercurial -- which I've used -- is closer > in syntax and commands to subversion than is the case for git, which > argues for mercurial as possibly more accessible to people who > haven't worked with either before, but who have worked with svn or > CVS...) > > * Why github, and not, say Bitbucket or Google Code? > > > (I'm not saying that this has to be a long, drawn-out discussion by any > means; but at the moment the haste and lack of justification tempts me > to vote "No" largely as method of protest.) > > ? -- Peter > > On Jul 15, 2011, at 8:47 PM, Perry Greenfield wrote: > >> Given discussions at scipy (consisting primarily of a good number of >> people from STScI, CFA/Chandra, and Gemini) it seemed clear to most of >> us there that the most straightforward choice for the astropy version >> control software and repository was git and github. This is what most >> of the science-based Python projects are using and they generally seem >> quite happy with it. Since we are encouraging distributed >> contributions, it also would seem we should favor a distributed vcs >> over a non-distributed one (i.e., svn). So we (Erik, Tom and me) are >> proposing that we simply put this particular choice to a yes/no vote. >> Here is a link to the poll: >> >> http://astropy.wikispaces.com/VCS+and+hosting+poll >> >> Should there be a significant level of objection to this, we'll look >> at seeing if there is a stronger consensus for something else. >> >> Perry >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > ============================================================= > Peter Erwin ? ? ? ? ? ? ? ? ? Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de ? ? ? ? ? ? ?Physics, Giessenbachstrasse > tel. +49 (0)89 30000 3695 ? ? 85748 Garching, Germany > fax ?+49 (0)89 30000 3495 ? ? http://www.mpe.mpg.de/~erwin > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From perry at stsci.edu Sat Jul 16 09:35:25 2011 From: perry at stsci.edu (Perry Greenfield) Date: Sat, 16 Jul 2011 09:35:25 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <4E20B621.4030105@stsci.edu> References: <4E20B621.4030105@stsci.edu> Message-ID: <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> On Jul 15, 2011, at 5:50 PM, Mark Sienkiewicz wrote: > Perry Greenfield wrote: >> Given discussions at scipy (consisting primarily of a good number of >> people from STScI, CFA/Chandra, and Gemini) it seemed clear to most >> of >> us there that the most straightforward choice for the astropy version >> control software and repository was git and github. > > > I was corresponding off-list (IIRC) about using git, but I'll repeat > my > major points here for everyone: > > - If the project will use git, the git advocates have a responsibility > to present a clearly-defined work flow for each role in the project. > At some point this is necessary. I don't think it is necessary before deciding to use it. More on that later. > Motivation: git implements several mutually-incompatible work flows. > If you don't say which one you are using, you can get really screwed > up. Also, what you do with git and why anybody would want to do that > are not obvious to those of us who are not already converts. I am > your > audience: one of those subversion users who "look at git's features > with > confusion or disinterest". > Because we (STScI) don't need most of those features for our internal work (and I think from what I understand, we generally don't) doesn't mean they are useless for astropy though. > b.t.w. I hear that ipython has a work flow document that we can > adapt from. > > > - DVCS advocates in general seem to care deeply about the DVCSness of > the system. I don't need anything from git that I can't get from > subversion, but the git advocates are so insistent that I might as > well > just give up and say "ok, use git". Basically, they care, I don't, > and > I can probably make their favorite system work. Assuming they explain > the work flow. > > But it would sure suck to lose the revision numbers that subversion > has > (but git does not) and not get something pretty intensely useful in > return. I hope you git guys know what that is. I'm not sure what you mean here. Revision numbers in what sense? Git does track versions by numbers (if very large ones...). Do you mean numbers that have some sequential properties or something like that? But let's step back a bit here. I'd venture to say that a lot more git users have used svn than the other way around. They are probably going to be more knowledgeable about the comparative advantages of the two systems than most svn users. Secondly, yes, there are lots of different possible workflows that one could use with git, and one could, in principle, make a mess of things. But the fact is that a lot of projects are using git very nicely. If it often caused an unholy mess, I don't think they are such religious zealots that they would overlook that aspect. They generally are making it work one way or another. Lastly, astropy, at least in principle could take advantage of the DVCS features much more than we do internally. So no, I don't think all the details need to be spelled out ahead of time before a poll is taken. Those details can be worked out if there appears to a sizable majority willing to go that way. Given that most of the other python science projects are using it is a good reason not to buck the trend. Perry From beaumont at hawaii.edu Sat Jul 16 10:12:59 2011 From: beaumont at hawaii.edu (Chris Beaumont) Date: Sat, 16 Jul 2011 10:12:59 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: Hi Geert, In my limited experience, the one exception to "avoid multiple inheritance" in Python is when the methods and attributes of the two (or more) superclasses do not overlap each other. An example might be an interactive plotting class that supports multithreading, so that a time-consuming rendering step doesn't bog down the rest of the application. That class might inherit from "BasePlottingClass" to gain all of the plotting code, and "MultiThreadedClass" that helps take care of setting up, running, and destroying threads. If the two classes are completely different, then there is no ambiguity regarding which method overrides which class. I believe examples like this are why Java forbids multiple inheritance, but _does_ allow classes to implement multiple "interfaces" (one of which, e.g., is the "Runnable" interface for multi-threaded classes). cheers, Chris On Fri, Jul 15, 2011 at 12:14 PM, Geert Barentsen wrote: > Hi Erik et al, > > Many thanks for your efforts on astropy, this seems very worthwile ! > > My suggestion for the coding guidelines would be to urge people not to use > multiple inheritance altogether. Your guideline to avoid "super()" does not > remove ambiguity in the case where a subclass does -not- override a method > that is implemented by 2 or more superclasses (in this case the > super-mechanism still works its magic... unless you demand that every > subclass in the whole chain overrides -every- method?) > > I don't think it is ever possible to use multiple inheritance without > understanding the somewhat complex details. Simply having code guidelines > which say "do not inherit from more than one class at a time" are far less > scary than the complex story about super(). I'm happy to change my opinion > if someone comes up with a good example where multiple inheritance makes > life better :-) ... when I studied computer science in university the > consensus was they are rare! > > Geert > > > > On 10 July 2011 06:39, Erik Tollerud wrote: > >> Hello everyone, >> >> Now that the vision and name for astropy have been ratified, the >> Coordinating Committee (Thomas, Perry, and I) have been working on a >> draft of coding guidelines/requirements for the astropy package. The >> intent is to use these items to determine if code in affiliated >> packages can be merged with the core package (as outlined in the >> vision). I've posted our draft of these guidelines to the wikispaces >> page: >> >> http://astropy.wikispaces.com/Astropy+Coding+Guidelines >> >> Feel free to ask questions about or comment on any of these items. We >> want to make sure the community is on board with a common set of >> guidelines, so we have tried to take into account discussions that >> have already occurred on this list in this draft. Many of these items >> have not yet been discussed, however, so we will certainly adjust >> these guidelines (or add new ones) based on your feedback once >> consensus is reached on the list. We do ask, though, that the >> comments/discussion be on the mailing list instead of the wiki page, >> so that the archive of the mailing list preserves the full discussion. >> >> >> Also, note that the guidelines reference both a documentation and >> testing guidelines document. I have created pages for these documents >> as well: >> >> Documentation: >> http://astropy.wikispaces.com/Astropy+Documentation+Guidelines >> Testing: http://astropy.wikispaces.com/Astropy+Testing+Guidelines >> >> These documents are not fleshed-out, as we wish to solicit ideas from >> the community here. I will send additional messages to this mailing >> list shortly to begin the discussion these topics, and ask again that >> the discussion for each take place in their associated threads. >> >> >> In the near future we (the committee) will also be sending out a >> message on determining project hosting and choice of VCS, as well as a >> set of priorities to guide development. Once we (the community) have >> reached consensus on this items, we can begin the actual work in >> earnest! >> >> Thanks in advance for your ideas and comments! >> >> -- >> Erik Tollerud >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- ************************************ Chris Beaumont Graduate Student Institute for Astronomy University of Hawaii at Manoa 2680 Woodlawn Drive Honolulu, HI 96822 www.ifa.hawaii.edu/~beaumont ************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mperrin at stsci.edu Sat Jul 16 11:10:14 2011 From: mperrin at stsci.edu (Marshall Perrin) Date: Sat, 16 Jul 2011 15:10:14 +0000 Subject: [AstroPy] Voting procedures In-Reply-To: References: <4E20A1EF.50207@stsci.edu> Message-ID: <0AC21776-85F0-4B71-8AAC-126EFD8754CC@stsci.edu> On Jul 15, 2011, at 8:54 PM, Erik Tollerud wrote: Vicki Laidler wrote: I'd also like to note that in the discussion of the previous "simple yes/no poll", several people raised objections that this was too coarse a metric and that there is value in being able to signify "yes but" or "don't care" & so on. I'd like to second those objections and request that the Coordinating Committee formulate polls that allow more nuance. This is a very good point - I had been thinking we should do this for future documents, but it didn't occur to me that it's equally relevant for this poll... In the future, would "Yes", "Yes, but with reservations", and "No" be an acceptable set of options, or would you prefer something more fine-grained than that? I'd like to second Vicki's request for a 'Don't care' option. In the current case at hand, for instance, I have no strong preference in version control systems... - Marshall -------------- next part -------------- An HTML attachment was scrubbed... URL: From erwin at mpe.mpg.de Sat Jul 16 14:18:25 2011 From: erwin at mpe.mpg.de (Peter Erwin) Date: Sat, 16 Jul 2011 20:18:25 +0200 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> Message-ID: <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> On Jul 16, 2011, at 1:32 PM, Tom Aldcroft wrote: > On Sat, Jul 16, 2011 at 5:52 AM, Peter Erwin wrote: >> >> * Why github, and not, say Bitbucket or Google Code? > > Based on what was evident at the SciPy conference, essentially > everyone that is using DVCS is using github. Including Numpy, Scipy, > IPython. I talked to people that have migrated away from google code > (didn't ask precisely why, but the answer that you don't find a ton of > Python projects there is probably a good one). > > During the talk on the amazing work on on IPython 0.11 over the last > year, Fernando said that one huge factor in their success was using > github. They did over 2000 commits and didn't totally hose things up. > Granted they were coming from bzr/launchpad which has lots of > problems, but the point is that github's UI is significantly better > than the rest. Things like code review and cross-repo compares are > amazing. A colleague of mine who is hardcore mercurial did a close > comparison of github vs. bitbucket and decided that github is way > better (and decided to use hg-git to basically take his hg repos and > host on github as needed). That's actually quite helpful; thanks. (And thanks, too, to Tom Robtaille for his expansion on this.) Would I be correct, then, in guessing that the main argument in favor of github + git is really something like "github seems to be the best place for hosting this sort of project, and git is required to use github"? (Rather than something like, e.g., "git is so much better than hg because X, Y, and Z, and github is a natural place for hosting git-based projects".) cheers, Peter ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)89 30000 3695 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From erik.tollerud at gmail.com Sat Jul 16 18:55:07 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 16 Jul 2011 15:55:07 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> Message-ID: > I'm not sure what you mean here. Revision numbers in what sense? Git > does track versions by numbers (if very large ones...). Do you mean > numbers that have some sequential properties or something like that? git's revision numbers (like hg and bzr), are hashes, so they aren't sequential - they uniquely identify a *commit*, but you can't use them to get a sense of ordering. Most importantly (at least, for me, when I've encountered this problem), if you want to have preview releases and/or documentation labelled based on the commit, the distutils/setuptools/distribute won't work correctly if versions aren't in sequential order. Fortunately, the "git describe" command can get around this, as long as you use "git tag" whenever you have a new version number. > Would I be correct, then, in guessing that the main argument in favor of > github + git is really something like "github seems to be the best place > for hosting this sort of project, and git is required to use github"? > (Rather than something like, e.g., "git is so much better than hg > because X, Y, and Z, and github is a natural place for hosting git-based > projects".) I'd say this is part of it (probably the most compelling reason to prefer git over hg), but that's partly because some of the git design choices make it easier to implement/use the things that make github really nice (maily this is lightweight branches, and slightly more flexible history). Honestly, I'd say there's not a big difference between git and hg though - git used to be more user-hostile, but that's changed recently (better docs and github, mainly). Most of the things that can be done in one can also be done in the other, but it's just sometimes harder in one than the other. Generally, git has more support for doing somewhat arcane things, though, which we may need when we join affiliated projects here in astropy. There's a much bigger difference between hg/git and non-distributed VCSs, though. You couldn't implement anything like github or bitbucket on a non-D VCS, because of the poor/non-existant support for many local commits, many trees, remote branches, etc. -- Erik Tollerud From mrdavis at stsci.edu Sat Jul 16 21:00:45 2011 From: mrdavis at stsci.edu (Matt Davis) Date: Sun, 17 Jul 2011 01:00:45 +0000 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> Message-ID: Folks interested in learning more about git and how it's used by numpy might be interested in links from the "Contributing to Numpy" page: http://docs.scipy.org/doc/numpy/dev/ Here is the same page from iPython, which looks pretty much the same: http://ipython.scipy.org/doc/manual/html/development/gitwash/index.html Best, Matt Davis -------------- next part -------------- An HTML attachment was scrubbed... URL: From wkerzendorf at googlemail.com Sat Jul 16 21:30:51 2011 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Sun, 17 Jul 2011 11:30:51 +1000 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> Message-ID: <4E223B4B.2040700@gmail.com> When I started developing a package with a friend we had a look around and tried mercurial first. We later switched to git and it seemed much easier. In the process we also found this webpage: http://whygitisbetterthanx.com/ . Which gives an interesting (I'm sure in someway biased) overview. But for all its arguments it has data. Just my two cents, W On 17/07/11 4:18 AM, Peter Erwin wrote: > On Jul 16, 2011, at 1:32 PM, Tom Aldcroft wrote: > >> On Sat, Jul 16, 2011 at 5:52 AM, Peter Erwin wrote: >>> * Why github, and not, say Bitbucket or Google Code? >> Based on what was evident at the SciPy conference, essentially >> everyone that is using DVCS is using github. Including Numpy, Scipy, >> IPython. I talked to people that have migrated away from google code >> (didn't ask precisely why, but the answer that you don't find a ton of >> Python projects there is probably a good one). >> >> During the talk on the amazing work on on IPython 0.11 over the last >> year, Fernando said that one huge factor in their success was using >> github. They did over 2000 commits and didn't totally hose things up. >> Granted they were coming from bzr/launchpad which has lots of >> problems, but the point is that github's UI is significantly better >> than the rest. Things like code review and cross-repo compares are >> amazing. A colleague of mine who is hardcore mercurial did a close >> comparison of github vs. bitbucket and decided that github is way >> better (and decided to use hg-git to basically take his hg repos and >> host on github as needed). > > That's actually quite helpful; thanks. (And thanks, too, to Tom Robtaille > for his expansion on this.) > > Would I be correct, then, in guessing that the main argument in favor of > github + git is really something like "github seems to be the best place > for hosting this sort of project, and git is required to use github"? > (Rather than something like, e.g., "git is so much better than hg > because X, Y, and Z, and github is a natural place for hosting git-based > projects".) > > cheers, > > Peter > > ============================================================= > Peter Erwin Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de Physics, Giessenbachstrasse > tel. +49 (0)89 30000 3695 85748 Garching, Germany > fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From erik.tollerud at gmail.com Sun Jul 17 00:00:42 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 16 Jul 2011 21:00:42 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> <57870E39-7AE5-407B-A0CE-DAD7D7746C2A@mpe.mpg.de> Message-ID: Right these two are the same because it's based on gitwash, which we will probably adopt into the astropy documentation (with modifications given our slightly different needs when merging affiliated packages). On Sat, Jul 16, 2011 at 6:00 PM, Matt Davis wrote: > Folks interested in learning more about git and how it's used by numpy might > be interested in links from the "Contributing to Numpy" > page:?http://docs.scipy.org/doc/numpy/dev/ > Here is the same page from iPython, which looks pretty much the same: > http://ipython.scipy.org/doc/manual/html/development/gitwash/index.html > Best, > Matt Davis > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- Erik Tollerud From erik.tollerud at gmail.com Mon Jul 18 07:33:57 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 18 Jul 2011 04:33:57 -0700 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E1B18B1.4060803@stsci.edu> References: <4E1B18B1.4060803@stsci.edu> Message-ID: Thanks for the careful reading - I'm compiling a list of suggested changes to the guidelines themselves, and will update the document itself once things are pretty much converged (example fixes you should see now). > " > # The package should be importable with no dependencies other than the > Python Standard Library (v2.6), NumPy, SciPy, matplotlib, and components > already in the core Astronomy library." > > I was wondering which "Astronomy" library this refers to and what part > is "core". > > I think it means "core Astropy", and I think later it defines "core > Astropy" as any package that has already been accepted into astropy as > astropy.whatever; ?If this is correct, I recommend "in the core > Astronomy library" -> "already in the Astropy package". You're right, that was a typo from a version before we'd settled on the astropy name - it's supposed to be "already in the Astropy core package." Thanks for catching that. > I recommend that the template package include some small amount of C > code and the Cython example for how to use it. That makes a lot of sense, and should be quite straightforward. > I note that the Cython recommendation implies an additional external > dependency for somebody. ?Is this for anybody who installs Astropy from > source, or is it a preprocessor that you only run occasionally like > yacc/lex? It can be done either way, actually. You can have Cython be a requirement for the build process, in which case Cython builds the C-code before compiling. numpy/scipy mostly use an approach where they always include the Cython source files, but also run Cython on those files and include the generated .c files in the repository. I'd say our best approach would be to have developers use the first method when they're actively working on the cython code, but whenever they've reached a significant milestone, they generate and commit the .c file, as well (and of course anytime there's a release). I think this is pretty much what numpy/scipy do, although I don't know if there's actually a fixed policy or just sort of an understanding. > ? ?* If an external C library is needed, the source code for the > ? ? ?library should be bundled with the astropy core. Additionally, the > ? ? ?package must be compatible with using a system-installed library > ? ? ?instead of the version included in astropy." > > > add "... using the mechanism shown in the template package." > > We want everybody to provide the same flags, same interface, etc, so we > show one way to do it and ask everybody to do it that way. Good point - will add this. > If you want to require PEP 8, I suggest that you make your examples PEP > 8 compliant. Fair enough... we'll probably end up moving these in some form to the example package where we'll hopefully be more careful, but setting a good example even on the wiki page does make sense, so I've updated it with pep8-checked version (as much as possible on wiki, anyway). > I think we should explicitly recommend the new relative import mechanism > any time you import another module/package that is in the same package > you are part of. > > Say, for example, that you want to debug a package and you would like to > run both the original and a modified copy at the same time. ?With > relative imports, you don't need to edit every import line in the whole > package to make your modified copy. > > This is not compliant with PEP 8, which recommends the bad practice of > always using absolute imports. ?It made sense when relative imports were > ambiguous, but the Python language no longer has that particular weakness. Actually, there was a recent discussion of this by some of the python committers ( http://mail.python.org/pipermail/python-committers/2011-February/001451.html ). It looks like there's a variety of opinions even among them. I personally agree with you given the fact that relative imports now actually work right... Does anyone else have a strong opinion on this? If not, I'll add an exception to PEP 8 along these lines. > The property/get/set example is weak: ?1) Take out the functions that > are not relevant to get/set; you don't need them cluttering the > example. ?2) Show a user interacting with the object: > > ? ? x = star() > > ? ? # use this > ? ? x.color = 100 > ? ? print x.color > > ? ? # not this > ? ? x.set_color(100) > ? ? print x.get_color() I see your point here... The intent was to show how the property is different from direct attribute access - that's not obvious from the example you're showing here. But then again, when I think about it, we will definitely have an example lick this in the example packages to illustrate how this is implemented, anyway. So I've updated the wiki with the simpler example. > Realistically, I think people are likely to want to use the > get_color()/set_color() approach because it is more natural. ?It also > makes it more clear what is happening. ?I know that hard core OO guys > think properties are cool, but I've never seen the attraction myself. I don't find get_color/set_color to be natural at all - the property/attribute access seems much more so to me... although I don't think it's too hard to adapt from one style to the other (in either direction) for this case. The advantage of properties in this context is that they encourage a natural flow between simple attribute access (which should be used when nothing else is needed) to get/set methods (properties look like attribute access). On top of that, "star.color = 0.4" is more readable, and is a bit faster to type than "star.set_color(.4)". Properties in the context given here are also suggested by PEP8 (mostly for these reasons). > The super() example actually shows more instances of doing it the wrong > way than the right way, and it does not clearly mark which one to use. > > It should show one complete implementation using super() [marked "don't > do this"] and another complete implementation using direct calls [marked > "do this"]. The point of this example was to show what happens when you mix super and direct calls, rather than to illustrate the correct way. But I see your point, so I've added a final code block for the direct call. > One of the common cases for wanting to call into the superclass is to > invoke the __init__() method. ?The example should show how to do this, > including the case of multiple inheritance. This is now folded into the example, although it admittedly shows why super really is sometimes useful if you're doing multiple inheritance (i.e. the double-call to A.__init__). > In the "import *" example, "__init__" is shown as "init" (two places). > It looks like it needs a wiki escape. Good catch - corrected. -- Erik Tollerud From perry at stsci.edu Mon Jul 18 07:44:03 2011 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 18 Jul 2011 07:44:03 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: <4E1B18B1.4060803@stsci.edu> Message-ID: <36EC37E5-6587-449D-B75C-C41662CE3DD9@stsci.edu> On Jul 18, 2011, at 7:33 AM, Erik Tollerud wrote: > >> Realistically, I think people are likely to want to use the >> get_color()/set_color() approach because it is more natural. It also >> makes it more clear what is happening. I know that hard core OO guys >> think properties are cool, but I've never seen the attraction myself. > > I don't find get_color/set_color to be natural at all - the > property/attribute access seems much more so to me... although I don't > think it's too hard to adapt from one style to the other (in either > direction) for this case. The advantage of properties in this context > is that they encourage a natural flow between simple attribute access > (which should be used when nothing else is needed) to get/set methods > (properties look like attribute access). On top of that, "star.color > = 0.4" is more readable, and is a bit faster to type than > "star.set_color(.4)". Properties in the context given here are also > suggested by PEP8 (mostly for these reasons). I agree with Erik on this point. If something looks like an attribute (even if it is computed, or needs to call a function when set), then the attribute interface is the most natural. As Erik also mentions, it allows starting with a simple attribute and not requiring a change to the interface to use a method (why other languages think that all attributes always should use setters and getters when they don't have a properties mechanism). Perry From erik.tollerud at gmail.com Mon Jul 18 07:43:59 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 18 Jul 2011 04:43:59 -0700 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: I agree with the view that "multiple inheritance should be avoided when possible" is a pretty good guideline. But Chris identifies the fact that there really is call for separate classes - the use case Chris identifies seem to generally go by the name of "mixins." Fortunately, these are exactly the cases where super() is unnecessary even in __init__; if the classes are unrelated in functionality, you can freely just do "SuperClass1.__init__self()" and "SuperClass2.__init__(self)" without worrying that they will interact some way. So with this in mind, it's not clear to me what's best to put in the guidelines (if anything) on the topic. Perhaps "multiple inheritance should be avoided, aside from use of mixins",and define mixins as superclasses that provide extra functionality or support without overlapping on the other superclasses? Or "multiple inheritance where superclasses are derived from the same higher superclass should be avoided" (i.e. no diamond-inheritance structures, trusting that they won't overlap if they don't have the same supers)? On Sat, Jul 16, 2011 at 7:12 AM, Chris Beaumont wrote: > Hi Geert, > > In my limited experience, the one exception to "avoid multiple inheritance" > in Python is when the methods and attributes of the two (or more) > superclasses do not overlap each other. > > An example might be an interactive plotting class that supports > multithreading, so that a time-consuming rendering step doesn't bog down the > rest of the application. That class might inherit from "BasePlottingClass" > to gain all of the plotting code, and "MultiThreadedClass" that helps take > care of setting up, running, and destroying threads. If the two classes are > completely different, then there is no ambiguity regarding which method > overrides which class. > > I believe examples like this are why Java forbids multiple inheritance, but > _does_ allow classes to implement multiple "interfaces" (one of which, e.g., > is the "Runnable" interface for multi-threaded classes). > > cheers, > Chris > > On Fri, Jul 15, 2011 at 12:14 PM, Geert Barentsen > wrote: >> >> Hi Erik et al, >> >> Many thanks for your efforts on astropy, this seems very worthwile ! >> >> My suggestion for the coding guidelines would be to urge people not to use >> multiple inheritance altogether. Your guideline to avoid "super()" does not >> remove ambiguity in the case where a subclass does -not- override a method >> that is implemented by 2 or more superclasses (in this case the >> super-mechanism still works its magic... unless you demand that every >> subclass in the whole chain overrides -every- method?) >> >> I don't think it is ever possible to use multiple inheritance without >> understanding the somewhat complex details. Simply having code guidelines >> which say "do not inherit from more than one class at a time" are far less >> scary than the complex story about super(). I'm happy to change my opinion >> if someone comes up with a good example where multiple inheritance makes >> life better :-)? ... when I studied computer science in university the >> consensus was they are rare! >> >> Geert >> >> >> >> On 10 July 2011 06:39, Erik Tollerud wrote: >>> >>> Hello everyone, >>> >>> Now that the vision and name for astropy have been ratified, the >>> Coordinating Committee (Thomas, Perry, and I) have been working on a >>> draft of coding guidelines/requirements for the astropy package. ?The >>> intent is to use these items to determine if code in affiliated >>> packages can be merged with the core package (as outlined in the >>> vision). ?I've posted our draft of these guidelines to the wikispaces >>> page: >>> >>> http://astropy.wikispaces.com/Astropy+Coding+Guidelines >>> >>> Feel free to ask questions about or comment on any of these items. ?We >>> want to make sure the community is on board with a common set of >>> guidelines, so we have tried to take into account discussions that >>> have already occurred on this list in this draft. ?Many of these items >>> have not yet been discussed, ?however, so we will certainly adjust >>> these guidelines (or add new ones) based on your feedback once >>> consensus is reached on the list. ?We do ask, though, that the >>> comments/discussion be on the mailing list instead of the wiki page, >>> so that the archive of the mailing list preserves the full discussion. >>> >>> >>> Also, note that the guidelines reference both a documentation and >>> testing guidelines document. ?I have created pages for these documents >>> as well: >>> >>> Documentation: >>> http://astropy.wikispaces.com/Astropy+Documentation+Guidelines >>> Testing: http://astropy.wikispaces.com/Astropy+Testing+Guidelines >>> >>> These documents are not fleshed-out, as we wish to solicit ideas from >>> the community here. ?I will send additional messages to this mailing >>> list shortly to begin the discussion these topics, and ask again that >>> the discussion for each take place in their associated threads. >>> >>> >>> In the near future we (the committee) will also be sending out a >>> message on determining project hosting and choice of VCS, as well as a >>> set of priorities to guide development. ?Once we ?(the community) have >>> reached consensus on this items, we can begin the actual work in >>> earnest! >>> >>> Thanks in advance for your ideas and comments! >>> >>> -- >>> Erik Tollerud >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > > > -- > ************************************ > Chris Beaumont > Graduate Student > Institute for Astronomy > University of Hawaii at Manoa > 2680 Woodlawn Drive > Honolulu, HI 96822 > www.ifa.hawaii.edu/~beaumont > ************************************ > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- Erik Tollerud From hodge at stsci.edu Mon Jul 18 08:56:55 2011 From: hodge at stsci.edu (Phil Hodge) Date: Mon, 18 Jul 2011 08:56:55 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <820F7BD0-F29C-4FBD-A040-5D0BA0BDBFC5@mpe.mpg.de> Message-ID: <4E242D97.5010502@stsci.edu> On 07/16/2011 09:20 AM, Thomas Robitaille wrote: > Hi Peter, > > I understand your concerns that this might seem premature - just to > explain our motives behind a yes/no poll at this stage: we got the > impression from previous discussions on this list and from the > discussions and success stories at scipy that this was something that > most people would either agree with, or not have a strong opinion > about. We can have had a lengthy debate about the detailed advantages > of git vs hg vs svn vs other if needed, but we wanted to see whether > that debate could be avoided in the first place if most people agreed > anyway, hence the early poll (think of it as testing the waters). But > we do want people to voice concerns or questions if you have any! ... Testing the waters? If 10 people vote (only 10), and all vote for git/github, would that be interpreted as meaning our toes are wet, or we all (not just the ones who voted) jumped in head first? Phil From sienkiew at stsci.edu Mon Jul 18 10:23:45 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Mon, 18 Jul 2011 10:23:45 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> Message-ID: <4E2441F1.4000406@stsci.edu> >> >> But it would sure suck to lose the revision numbers that subversion has >> (but git does not) and not get something pretty intensely useful in >> return. I hope you git guys know what that is. > > I'm not sure what you mean here. Revision numbers in what sense? Git > does track versions by numbers (if very large ones...). Do you mean > numbers that have some sequential properties or something like that? Yes, subversion has revision numbers are sequential. They are also small integers. You can easily read/write/speak the revision numbers. You can easily include them in a note, and recognize them when you read them in another note. Git hashes have none of these features. They are 160 bit opaque data items. A computer can test them for equality, but it is not practical for a human to do that routinely. You would not write a git hash on a sticky note. So, git must contain some feature to compensate for this weakness. For example, if git has real tags (which subversion does not), you could tag a version when you need to talk about it. You might even make a convention for tag names that are reserved for that kind of use. It wouldn't be quite as convenient as revision numbers, but it would come close. Or maybe there is some non-obvious way of using git where you don't miss having the subversion revision numbers. I don't know. > Secondly, yes, there are lots of different possible workflows that one > could use with git, and one could, in principle, make a mess of > things. But the fact is that a lot of projects are using git very > nicely. If it often caused an unholy mess, I don't think they are such > religious zealots that they would overlook that aspect. They generally > are making it work one way or another. I think it does not cause an unholy mess when everybody involved is a true believer. The failure mode I want to avoid is where git-advocates have no trouble working with the system, but the git-inexperienced cannot use it effectively. > So no, I don't think all the details need to be spelled out ahead of > time before a poll is taken. Those details can be worked out if there > appears to a sizable majority willing to go that way. Yes, that is one approach. You are asking for people to accept an as-yet-unselected choice from a class of possible work flows. I'm confident that at least one of them can be made to work, so I agree that the details do not need to be spelled out in advance. All I'm asking for is a commitment that somebody will spell out the details after git is selected. I've had that in off-list discussions, but I wanted make the case publicly. Mark From embray at stsci.edu Mon Jul 18 10:42:31 2011 From: embray at stsci.edu (Erik Bray) Date: Mon, 18 Jul 2011 10:42:31 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: <4E244657.20707@stsci.edu> On 07/10/2011 01:39 AM, Erik Tollerud wrote: > Hello everyone, > > Now that the vision and name for astropy have been ratified, the > Coordinating Committee (Thomas, Perry, and I) have been working on a > draft of coding guidelines/requirements for the astropy package. The > intent is to use these items to determine if code in affiliated > packages can be merged with the core package (as outlined in the > vision). I've posted our draft of these guidelines to the wikispaces > page: > > http://astropy.wikispaces.com/Astropy+Coding+Guidelines I'm late to this discussion, so forgive me if it's already been discussed (I'm still catching up). But I'm strongly -1 on the discouragement of super() use. First of all, as Mark pointed out, the examples given are confusing. The first super() example is showing an incorrect use. And the text afterwards basically reads "You shouldn't use super() because this example will break" when in fact the given example *should* break. If `class C` in the example used super() as well, the example would work as expected. When in doubt about the method resolution order in a complex multiple inheritance case one can always call the .mro() on the class to check. In my experience, most examples of super() being bad seem to be due to people not using it correctly or consistently. A better policy would be to *require* super() so that when it's needed it's more likely to work. There *are* cases I've seen where super() is not desired, but they're the exception. Otherwise, I would stick to Raymond Hettinger's advice in the linked article--he gets it right. Erik From embray at stsci.edu Mon Jul 18 10:46:26 2011 From: embray at stsci.edu (Erik Bray) Date: Mon, 18 Jul 2011 10:46:26 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: <4E244742.8030608@stsci.edu> On 07/10/2011 01:39 AM, Erik Tollerud wrote: > Hello everyone, > > Now that the vision and name for astropy have been ratified, the > Coordinating Committee (Thomas, Perry, and I) have been working on a > draft of coding guidelines/requirements for the astropy package. The > intent is to use these items to determine if code in affiliated > packages can be merged with the core package (as outlined in the > vision). I've posted our draft of these guidelines to the wikispaces > page: > > http://astropy.wikispaces.com/Astropy+Coding+Guidelines And my other small suggestion for the coding guidelines draft: The example of correct use of `from foo import *` should also encourage that submodule1 and submodule2 define the __all__ global, so that import * only imports variables that are desired to be added to the flattened namespace. Thanks, Erik From embray at stsci.edu Mon Jul 18 12:00:22 2011 From: embray at stsci.edu (Erik Bray) Date: Mon, 18 Jul 2011 12:00:22 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: <4E245896.3020601@stsci.edu> On 07/18/2011 07:43 AM, Erik Tollerud wrote: > I agree with the view that "multiple inheritance should be avoided > when possible" is a pretty good guideline. But Chris identifies the > fact that there really is call for separate classes - the use case > Chris identifies seem to generally go by the name of "mixins." > Fortunately, these are exactly the cases where super() is unnecessary > even in __init__; if the classes are unrelated in functionality, you > can freely just do "SuperClass1.__init__self()" and > "SuperClass2.__init__(self)" without worrying that they will interact > some way. I also agree with the "multiple inheritance should be avoided when possible" guideline. But what does this really mean? It's possible I could rewrite my project to not use object-oriented patterns at all, thus avoiding any "need" for multiple inheritance, so it's not really a useful guideline. > So with this in mind, it's not clear to me what's best to put in the > guidelines (if anything) on the topic. Perhaps "multiple inheritance > should be avoided, aside from use of mixins",and define mixins as > superclasses that provide extra functionality or support without > overlapping on the other superclasses? Or "multiple inheritance where > superclasses are derived from the same higher superclass should be > avoided" (i.e. no diamond-inheritance structures, trusting that they > won't overlap if they don't have the same supers)? Mixins are probably the best use case for multiple inheritance, but they're not the only one. PyFITS uses multiple inheritance extensively. I'll admit that in some places it gets confusing, and I would like to try to simplify it more than I already have. But the existing implementation tracks with the FITS specification itself, and is working pretty well. Here's an example of how it's used in PyFITS: FITS has a notion of 'extension HDUs' that have some different properties from non-extension HDUs, though most HDUs in a FITS file are extensions. All extension HDU classes inherit from an ExtensionHDU class which has its own __init__() and a few other methods. There are two types of HDUs in FITS that can contain an image: a Primary HDU and an Image Extension HDU. The former is *not* an extension HDU, though they both work mostly the same with respect to handling their image data. PyFITS implements all image-related code in an ImageBaseHDU class. The PrimaryHDU class inherits only from ImageBaseHDU, while the ImageHDU class inherits from both ImageBaseHDU and ExtensionHDU. So this part of the class hierarchy looks basically like this: BaseHDU / \ / \ / \ / \ ImageBaseHDU ExtensionHDU / \ / / \ / PrimaryHDU \ / \ / ImageHDU So there's a good example of diamond inheritance structure in real life that's working, with additional branches off of it. A similar hierarchy exists for table-like HDUs (GroupHDUs, AsciiTableHDUs, and BinTableHDUs). Furthermore, not all of these classes take the same arguments in their __init__()s, but this is handled by accepting **kwds in all the __init__()s. So while I agree that multiple inheritance should be avoided, the question is how strong is this statement intended to be? I would say that it should be avoided unless you _really_ know what you're doing, which is a statement I would make about multiple inheritance in any OO language. Thanks, Erik From mrdavis at stsci.edu Mon Jul 18 12:04:14 2011 From: mrdavis at stsci.edu (Matt Davis) Date: Mon, 18 Jul 2011 12:04:14 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <4E2441F1.4000406@stsci.edu> References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: <965636D5-3ED9-486F-B7B3-F1F8EE016A41@stsci.edu> > I think it does not cause an unholy mess when everybody involved is a > true believer. The failure mode I want to avoid is where git-advocates > have no trouble working with the system, but the git-inexperienced > cannot use it effectively. The groups "git-advocates" and "git-inexperienced" are not exclusive. I don't have experience with git/github but based on its popularity with similar projects and the numpy/ipython documentation I've read it seems like an excellent solution. Since we'd likely adopt something not super different from the numpy/ipython work flows their documentation is a great place to get started learning about git/github. http://docs.scipy.org/doc/numpy/dev/ http://ipython.scipy.org/doc/manual/html/development/gitwash/index.html - Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Mon Jul 18 13:37:31 2011 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 18 Jul 2011 13:37:31 -0400 Subject: [AstroPy] poll closing date: Re: proposal for astropy version control software and respository In-Reply-To: References: Message-ID: <506F2991-40F9-47CE-A0D2-D1FEAA53F3BC@stsci.edu> We'll take results until 9 pm EST Tuesday... On Jul 15, 2011, at 2:47 PM, Perry Greenfield wrote: > Given discussions at scipy (consisting primarily of a good number of > people from STScI, CFA/Chandra, and Gemini) it seemed clear to most > of us there that the most straightforward choice for the astropy > version control software and repository was git and github. This is > what most of the science-based Python projects are using and they > generally seem quite happy with it. Since we are encouraging > distributed contributions, it also would seem we should favor a > distributed vcs over a non-distributed one (i.e., svn). So we (Erik, > Tom and me) are proposing that we simply put this particular choice > to a yes/no vote. Here is a link to the poll: > > http://astropy.wikispaces.com/VCS+and+hosting+poll > > Should there be a significant level of objection to this, we'll look > at seeing if there is a stronger consensus for something else. > > Perry From laidler at stsci.edu Mon Jul 18 14:37:22 2011 From: laidler at stsci.edu (Victoria G. Laidler) Date: Mon, 18 Jul 2011 14:37:22 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <4E20A1EF.50207@stsci.edu> Message-ID: <4E247D62.8090607@stsci.edu> >> For those of us who are not very familiar with git and github, can someone >> suggest a source for additional information? >> Thanks to all who posted with links to further information. Very helpful. >> I'd also like to note that in the discussion of the previous "simple >> yes/no poll", several people raised objections that this was too coarse >> a metric and that there is value in being able to signify "yes but" or >> "don't care" & so on. I'd like to second those objections and request >> that the Coordinating Committee formulate polls that allow more nuance. >> > > This is a very good point - I had been thinking we should do this for > future documents, but it didn't occur to me that it's equally relevant > for this poll... In the future, would "Yes", "Yes, but with > reservations", and "No" be an acceptable set of options, or would you > prefer something more fine-grained than that? > I'd certainly add "Don't care". I considered "No, but I might be convinced", but I think that belongs in the email list rather than in the poll. I might add "Abstain/protest", that one could use if one feels it is too soon to vote, or the poll is malformed, or so forth. This would be a way to say "I oppose _this_ specific poll" without indicating "I'm in disagreement on the underlying issue." One problem with this poll that I haven't seen mentioned yet is that it conflates use of git as the VCS with use of github as the hosting site. These are in principle independent. The hosting site is not just a place to stash the repository; it also provides related tools that are critical to planning & managing workflow, like tickets/issues, history/timeline, nice tools to view diff, history, & so on. From erik.tollerud at gmail.com Mon Jul 18 14:59:32 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 18 Jul 2011 11:59:32 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: <4E2441F1.4000406@stsci.edu> References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: > Yes, subversion has revision numbers are sequential. ?They are also > small integers. ?You can easily read/write/speak the revision numbers. > You can easily include them in a note, and recognize them when you read > them in another note. > > Git hashes have none of these features. ?They are 160 bit opaque data > items. ?A computer can test them for equality, but it is not practical > for a human to do that routinely. ?You would not write a git hash on a > sticky note. Well, in git you can use only the first 4 characters of a hash and it will work the way a revision number does - that's something you *can* write on a sticky note. Occasionally its ambiguous if multiple commits have the same first 4 chars, but that's pretty uncommon. > So, git must contain some feature to compensate for this weakness. ?For > example, if git has real tags (which subversion does not), you could tag > a version when you need to talk about it. ?You might even make a > convention for tag names that are reserved for that kind of use. ?It > wouldn't be quite as convenient as revision numbers, but it would come > close. > > Or maybe there is some non-obvious way of using git where you don't miss > having the subversion revision numbers. ?I don't know. I'm not sure what you mean by "real" tags in subversion. Git has tags that are as simple as "git tag " (the current commit - in git parlance, HEAD) or "git tag " (tag some other commit). These can be local to you or you can push them up to the repository. Tags can also be "lightweight", or annotated - the latter is actually a commit object and thus can have associated messages and be signed and the like. Generally the convention is to use tags locally as shorthand for some commit you often go back to, and publically tag version numbers. Then the "git describe" command basically gives you all the information you need. > All I'm asking for is a commitment that somebody will spell out the > details after git is selected. ?I've had that in off-list discussions, > but I wanted make the case publicly. I would guess we all agree on this, although I'd say it is not really even a git-specific point (although you're right that it is somewhat *more* important in a DVCS due to the flexibility). If this isn't done to your satisfaction once we actually have the repository up and running, hopefully it can then be corrected, as there will then be something specific to work on. -- Erik Tollerud From chris.h.devries at gmail.com Mon Jul 18 15:08:44 2011 From: chris.h.devries at gmail.com (Christopher De Vries) Date: Mon, 18 Jul 2011 12:08:44 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: I think the argument regarding revision numbers is a bit of a red herring. In client/server version control a single master repository was the rule and checkins were supposed to be well vetted by the person authorized to use the server. Distributed version control means that people will be working in their own repositories on various bits, creating new hashes regularly, which may or may not be uploaded back to the github repository in some particular order, so the hashes in git and revision numbers in subversion or cvs serve different purposes. Anyway, what I wanted to point out, for mercurial lovers like me, is that there is a git plugin for mercurial ( http://hg-git.github.com/ ) which allows mercurial users to participate in the git ecosystem. I have used it for checkouts, though I also find that using git is fairly straightforward too. I do not, however, believe there is a good plugin for git users to use mercurial repositories, so that would be another point in favor of git and github. Chris On Mon, Jul 18, 2011 at 11:59 AM, Erik Tollerud wrote: >> Yes, subversion has revision numbers are sequential. ?They are also >> small integers. ?You can easily read/write/speak the revision numbers. >> You can easily include them in a note, and recognize them when you read >> them in another note. >> >> Git hashes have none of these features. ?They are 160 bit opaque data >> items. ?A computer can test them for equality, but it is not practical >> for a human to do that routinely. ?You would not write a git hash on a >> sticky note. > > Well, in git you can use only the first 4 characters of a hash and it > will work the way a revision number does - that's something you *can* > write on a sticky note. ?Occasionally its ambiguous if multiple > commits have the same first 4 chars, but that's pretty uncommon. > > >> So, git must contain some feature to compensate for this weakness. ?For >> example, if git has real tags (which subversion does not), you could tag >> a version when you need to talk about it. ?You might even make a >> convention for tag names that are reserved for that kind of use. ?It >> wouldn't be quite as convenient as revision numbers, but it would come >> close. >> >> Or maybe there is some non-obvious way of using git where you don't miss >> having the subversion revision numbers. ?I don't know. > > I'm not sure what you mean by "real" tags in subversion. ?Git has tags > that are as simple as "git tag " (the current commit - in git > parlance, HEAD) or "git tag " (tag some other > commit). ?These can be local to you or you can push them up to the > repository. ?Tags can also be "lightweight", or annotated - the latter > is actually a commit object and thus can have associated messages and > be signed and the like. > > Generally the convention is to use tags locally as shorthand for some > commit you often go back to, and publically tag version numbers. ?Then > the "git describe" command basically gives you all the information you > need. > > >> All I'm asking for is a commitment that somebody will spell out the >> details after git is selected. ?I've had that in off-list discussions, >> but I wanted make the case publicly. > > I would guess we all agree on this, although I'd say it is not really > even a git-specific point (although you're right that it is somewhat > *more* important in a DVCS due to the flexibility). ?If this isn't > done to your satisfaction once we actually have the repository up and > running, hopefully it can then be corrected, as there will then be > something specific to work on. > > > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From sienkiew at stsci.edu Mon Jul 18 17:51:30 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Mon, 18 Jul 2011 17:51:30 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: <4E24AAE2.6020004@stsci.edu> > > I'm not sure what you mean by "real" tags in subversion. Git has tags > that are as simple as "git tag " (the current commit - in git > parlance, HEAD) or "git tag " (tag some other > commit). And presumably I can ask git to "update" or whatever to a specific tag name. Once I update to a tag, a commit does not change the tagged data -- it just makes a new revision. Yes, that would be a real tag. Subversion does not have tags or branches at all, but we pretend it does. By convention, you have a directory named /trunk where your code lives; to make a branch or a tag, you copy that directory to /branches/xxx or /tags/xxx. Since there is no difference between /trunk and /tags/xxx except the name, nothing prevents you from committing to the tagged code. This is relatively easy to do by mistake, compared with CVS where you would have to explicitly do a commit and then issue tag-altering commands to change the content of a tag. > Generally the convention is to use tags locally as shorthand for some > commit you often go back to, and publically tag version numbers. Then > the "git describe" command basically gives you all the information you > need. > So, for example, if you have a continuous integration system that does builds/tests every day, it would start by tagging the source code it is about to build, instead of just noting a revision number in the logs. As a user, I can see what source code the CI system used by telling git "get me this tag", where I know the name of the tag that the CI assigned. From sienkiew at stsci.edu Mon Jul 18 18:15:32 2011 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Mon, 18 Jul 2011 18:15:32 -0400 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: <4E24B084.7010406@stsci.edu> Christopher De Vries wrote: > I think the argument regarding revision numbers is a bit of a red > herring. It's not an argument. It is an example. The point is not that git is somehow "wrong", but that it is "different", and that difference means there is a different work flow. Do not think that a discussion of the differences necessarily implies an adversarial relationship. The only thing potentially adversarial here is that I expect the git advocates to explain how to use it effectively, rather than expecting me to reverse-engineer the proper work flow. Since that has, in fact, been agreed, there is nothing to argue about. Mark S. From fred.grollier at gmail.com Tue Jul 19 05:03:40 2011 From: fred.grollier at gmail.com (=?iso-8859-1?Q?Fr=E9d=E9ric?= Grollier) Date: Tue, 19 Jul 2011 11:03:40 +0200 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: Message-ID: <20110719090340.GA12953@free.fr> Hello, On Sat, Jul 09, 2011 at 10:39:46PM -0700, Erik Tollerud wrote: > http://astropy.wikispaces.com/Astropy+Coding+Guidelines > > Feel free to ask questions about or comment on any of these items. As long as C extensions are allowed, perhaps PEP7 should be mentioned somewhere. Maybe it's too early for this, but I think some version numbers should be added regarding numpy, scipy and matplotlib compatibility, at least to assure some consistency between the various core components. Fred. From matthewturk at gmail.com Tue Jul 19 09:49:49 2011 From: matthewturk at gmail.com (Matthew Turk) Date: Tue, 19 Jul 2011 06:49:49 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: Hi all, For what it is worth, enzo and yt are both astrophysics projects and both use and love mercurial; the buy-in feels to be much lower than that of git, it's much more straightforward, and extensions are very, very easy to write. The primary hg hosting solution, BitBucket, has substantially improved pull requests and other collaborative features recently, and this has I believe brought it in line with GitHub in many ways. One point I feel very strongly about is that under no circumstances should a scientific code make a habit of 'rewriting history' or changing the hash numbers of existing revisions. This would, for instance, be part of a rebase-based workflow with git or on github. hg discourages rewriting history, but it seems quite prevalent with git. So even if git/github is chosen as a platform, I sincerely hope that a workflow that does not change existing revision numbers in any published repository is selected. Best, Matt On Mon, Jul 18, 2011 at 12:08 PM, Christopher De Vries wrote: > I think the argument regarding revision numbers is a bit of a red > herring. In client/server version control a single master repository > was the rule and checkins were supposed to be well vetted by the > person authorized to use the server. Distributed version control means > that people will be working in their own repositories on various bits, > creating new hashes regularly, which may or may not be uploaded back > to the github repository in some particular order, so the hashes in > git and revision numbers in subversion or cvs serve different > purposes. > > Anyway, what I wanted to point out, for mercurial lovers like me, is > that there is a git plugin for mercurial ( http://hg-git.github.com/ ) > which allows mercurial users to participate in the git ecosystem. I > have used it for checkouts, though I also find that using git is > fairly straightforward too. I do not, however, believe there is a good > plugin for git users to use mercurial repositories, so that would be > another point in favor of git and github. > > Chris > > On Mon, Jul 18, 2011 at 11:59 AM, Erik Tollerud wrote: >>> Yes, subversion has revision numbers are sequential. ?They are also >>> small integers. ?You can easily read/write/speak the revision numbers. >>> You can easily include them in a note, and recognize them when you read >>> them in another note. >>> >>> Git hashes have none of these features. ?They are 160 bit opaque data >>> items. ?A computer can test them for equality, but it is not practical >>> for a human to do that routinely. ?You would not write a git hash on a >>> sticky note. >> >> Well, in git you can use only the first 4 characters of a hash and it >> will work the way a revision number does - that's something you *can* >> write on a sticky note. ?Occasionally its ambiguous if multiple >> commits have the same first 4 chars, but that's pretty uncommon. >> >> >>> So, git must contain some feature to compensate for this weakness. ?For >>> example, if git has real tags (which subversion does not), you could tag >>> a version when you need to talk about it. ?You might even make a >>> convention for tag names that are reserved for that kind of use. ?It >>> wouldn't be quite as convenient as revision numbers, but it would come >>> close. >>> >>> Or maybe there is some non-obvious way of using git where you don't miss >>> having the subversion revision numbers. ?I don't know. >> >> I'm not sure what you mean by "real" tags in subversion. ?Git has tags >> that are as simple as "git tag " (the current commit - in git >> parlance, HEAD) or "git tag " (tag some other >> commit). ?These can be local to you or you can push them up to the >> repository. ?Tags can also be "lightweight", or annotated - the latter >> is actually a commit object and thus can have associated messages and >> be signed and the like. >> >> Generally the convention is to use tags locally as shorthand for some >> commit you often go back to, and publically tag version numbers. ?Then >> the "git describe" command basically gives you all the information you >> need. >> >> >>> All I'm asking for is a commitment that somebody will spell out the >>> details after git is selected. ?I've had that in off-list discussions, >>> but I wanted make the case publicly. >> >> I would guess we all agree on this, although I'd say it is not really >> even a git-specific point (although you're right that it is somewhat >> *more* important in a DVCS due to the flexibility). ?If this isn't >> done to your satisfaction once we actually have the repository up and >> running, hopefully it can then be corrected, as there will then be >> something specific to work on. >> >> >> >> -- >> Erik Tollerud >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From tevaughan at gmail.com Tue Jul 19 16:45:55 2011 From: tevaughan at gmail.com (Thomas Vaughan) Date: Tue, 19 Jul 2011 14:45:55 -0600 Subject: [AstroPy] Help with vegamag. Message-ID: I'm trying to do some calibration work for a star tracker that uses Iota Persei as the standard. --- --- What I Start With --- I have vt_iper = 4.107, which is the Tycho-2 V_T magnitude of Iota Persei. And I have vt_vega = 0.058, which is the Tycho-2 V_T magnitude of Vega (Maiz-Apellaniz, 2005) And I have v_vega = 0.026, which is the Johnson V magnitude of Vega (Bohlin & Gilliland, 2004) --- --- What Doesn't Seem To Work --- The definition of vegamag suggests that I do something like this: vm_vt_iper = vt_iper - vt_vega # V_T vegamag of Iota Persei new_spec = spec.renorm( vm_vt_iper, 'vegamag', bp_vt ) # Tycho V_T bandpass Then, for a sanity check, I do v_obs = psyn.Observation( new_spec, bp_jv ) # Johnson V bandpass v = v_obs.effstim( 'vegamag' ) + v_vega When I do this, though, the V magnitude comes out really off, like around 4.0 instead of 4.05. --- --- What Seems To Work But Doesn't Make Sense --- To get an answer that looks right, I have to do reverse the signs as follows: vm_vt_iper = vt_iper + vt_vega Then, at the end, v = v_obs.effstim( 'vegamag' ) - v_vega This gives an answer around the published value of 4.05, but the reversal doesn't make sense to me by the definition of vegamag. --- --- The Question --- Am I missing something obvious? -- Thomas E. Vaughan From thomas.robitaille at gmail.com Wed Jul 20 00:51:30 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Wed, 20 Jul 2011 00:51:30 -0400 Subject: [AstroPy] Version Control for AstroPy Message-ID: Hi everyone, At the end of last week we asked whether you would approve of using the git version control system with hosting on GitHub for the core AstroPy package, and the final results are 49 in favor (74%) and 17 against (26%). Given that a strong majority is in favor of the proposition, that the poll numbers imply that a majority of people have voted, and that it is unlikely that any version control system would get unanimous approval, we have decided to go with the results of the poll and recommend that we use git with GitHub. However, we don?t want to ignore those of you who voted against, and we want to address the concerns raised on the list, including from people who voted negatively in the poll. Therefore, we want to clarify the following points: - Future polls will be preceded by at least a few days of discussion before being created, and polls will be more nuance than simply ?yes?/?no? (allowing, for example, ?don?t care,? ?don?t know,? or ?abstain/protest?). - Clear instructions and assistance will be provided so that anyone not familiar with git and/or github will have no problems contributing, and that no one will be left out. Instructions will also be provided for how svn and hg can be used to talk to git repositories. - With your help, we will write a clear document outlining a strict workflow for contributing to the core package via git. This will likely be based on the gitwash/numpy/scipy guidelines. - Importantly, affiliated packages are under no obligation to use git/github until they are merged with the core package. Even once they are merged with the core package, future development on the component can still technically be done with hg or svn, so long as the changes are ultimately submitted via git. We hope these points will be acceptable to people who voiced concerns on these topics, and that we still have you all on board. If not, please let us know how we can address your concerns. We also welcome other feedback on any of these points. We are really excited that things are moving forward, where previous discussions in the past have failed, and the core astronomy library now seems a tangible goal more than ever! Cheers, Thomas, Erik, and Perry From tevaughan at gmail.com Wed Jul 20 11:35:28 2011 From: tevaughan at gmail.com (Thomas Vaughan) Date: Wed, 20 Jul 2011 09:35:28 -0600 Subject: [AstroPy] Help with pysynphot and vegamag. Message-ID: My original post didn't mention that I'm trying to use pysynphot. The short of it is that I'm wondering whether my original use of vegamag is correct below. On Tue, Jul 19, 2011 at 14:45, Thomas Vaughan wrote: > I'm trying to do some calibration work for a star tracker that uses > Iota Persei as the standard. > > --- > --- What I Start With > --- > > I have vt_iper = 4.107, which is the Tycho-2 V_T magnitude of Iota Persei. > And I have vt_vega = 0.058, which is the Tycho-2 V_T magnitude of Vega > (Maiz-Apellaniz, 2005) > And I have v_vega = 0.026, which is the Johnson V magnitude of Vega > (Bohlin & Gilliland, 2004) > > --- > --- What Doesn't Seem To Work > --- > > The definition of vegamag suggests that I do something like this: > > vm_vt_iper = vt_iper - vt_vega ?# V_T vegamag of Iota Persei > new_spec = spec.renorm( vm_vt_iper, 'vegamag', bp_vt ) # Tycho V_T bandpass > > Then, for a sanity check, I do > > v_obs = psyn.Observation( new_spec, bp_jv ) # Johnson V bandpass > v = v_obs.effstim( 'vegamag' ) + v_vega > > When I do this, though, the V magnitude comes out really off, like > around 4.0 instead of 4.05. > > --- > --- What Seems To Work But Doesn't Make Sense > --- > > To get an answer that looks right, I have to do reverse the signs as follows: > > vm_vt_iper = vt_iper + vt_vega > > Then, at the end, > > v = v_obs.effstim( 'vegamag' ) - v_vega > > This gives an answer around the published value of 4.05, but the > reversal doesn't make sense to me by the definition of vegamag. > > --- > --- The Question > --- > > Am I missing something obvious? -- Thomas E. Vaughan From moritz.guenther at gmx.de Wed Jul 20 13:22:33 2011 From: moritz.guenther at gmx.de (Hans Moritz Guenther) Date: Wed, 20 Jul 2011 13:22:33 -0400 Subject: [AstroPy] Documentation Guidelines In-Reply-To: References: Message-ID: <4E270ED9.1060800@gmx.de> +1 for numpy/scipy style Mainly to keep compatible with numpy/scipy as a standard. We should avoid to reinvent the wheel and none of the points raised in the previous discussion convinced me that the numpy/scipy format has any significant disadvantages. I also think it's easy to read in plain text and html. Moritz From stefan.czesla at hs.uni-hamburg.de Thu Jul 21 08:01:55 2011 From: stefan.czesla at hs.uni-hamburg.de (Stefan Czesla) Date: Thu, 21 Jul 2011 14:01:55 +0200 Subject: [AstroPy] Documentation Guidelines In-Reply-To: <4E1CAB8C.1040201@du.edu> References: <4E1C8BC1.7060103@stsci.edu> <4E1CAB8C.1040201@du.edu> Message-ID: <4E281533.8050409@hs.uni-hamburg.de> I also support adopting the Numpy/Scipy docstring format. What about putting this issue up for a vote? The discussion on this seems to be quite mature. The yes/no thing has bot been accepted too well, so the range of options should be broader this time. I would suggest that the doc guidelines contain an encouragement for writing examples/tutorial. I know its in the coding guidelines, but this seems to be a natural place to put it, too. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: stefan_czesla.vcf Type: text/x-vcard Size: 224 bytes Desc: not available URL: From jturner at gemini.edu Fri Jul 22 15:43:28 2011 From: jturner at gemini.edu (James Turner) Date: Fri, 22 Jul 2011 15:43:28 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E245896.3020601@stsci.edu> References: <4E245896.3020601@stsci.edu> Message-ID: <4E29D2E0.6030708@gemini.edu> It's helpful to have concrete examples like this amidst the fairly abstract discussion, thanks. I'm aware that complex class design is said to require a lot of experience, but (by definition) don't have a very good feel for the pitfalls. I've found a simple OO pattern well suited for many astronomical data processing tasks, but in IRAF I was fighting the language just to define class-like structures at all (in SPP). Erik's PyFITS example seems a natural one that I might have come up with, so it's good to have this kind of guidance on the list from those with more CS background (we also get some of that internally from Craig Allen). Like Chris B, I'd have thought the case where superclasses are "orthogonal bases" (I think that's what you mean by mixins) would not present problems. Cheers, James. On 18/07/11 12:00, Erik Bray wrote: > On 07/18/2011 07:43 AM, Erik Tollerud wrote: >> I agree with the view that "multiple inheritance should be avoided >> when possible" is a pretty good guideline. But Chris identifies the >> fact that there really is call for separate classes - the use case >> Chris identifies seem to generally go by the name of "mixins." >> Fortunately, these are exactly the cases where super() is unnecessary >> even in __init__; if the classes are unrelated in functionality, you >> can freely just do "SuperClass1.__init__self()" and >> "SuperClass2.__init__(self)" without worrying that they will interact >> some way. > > I also agree with the "multiple inheritance should be avoided when > possible" guideline. But what does this really mean? It's possible I > could rewrite my project to not use object-oriented patterns at all, > thus avoiding any "need" for multiple inheritance, so it's not really a > useful guideline. > >> So with this in mind, it's not clear to me what's best to put in the >> guidelines (if anything) on the topic. Perhaps "multiple inheritance >> should be avoided, aside from use of mixins",and define mixins as >> superclasses that provide extra functionality or support without >> overlapping on the other superclasses? Or "multiple inheritance where >> superclasses are derived from the same higher superclass should be >> avoided" (i.e. no diamond-inheritance structures, trusting that they >> won't overlap if they don't have the same supers)? > > Mixins are probably the best use case for multiple inheritance, but > they're not the only one. PyFITS uses multiple inheritance extensively. > I'll admit that in some places it gets confusing, and I would like to > try to simplify it more than I already have. But the existing > implementation tracks with the FITS specification itself, and is working > pretty well. Here's an example of how it's used in PyFITS: > > FITS has a notion of 'extension HDUs' that have some different > properties from non-extension HDUs, though most HDUs in a FITS file are > extensions. All extension HDU classes inherit from an ExtensionHDU > class which has its own __init__() and a few other methods. > > There are two types of HDUs in FITS that can contain an image: a Primary > HDU and an Image Extension HDU. The former is *not* an extension HDU, > though they both work mostly the same with respect to handling their > image data. PyFITS implements all image-related code in an ImageBaseHDU > class. The PrimaryHDU class inherits only from ImageBaseHDU, while the > ImageHDU class inherits from both ImageBaseHDU and ExtensionHDU. So > this part of the class hierarchy looks basically like this: > > BaseHDU > / \ > / \ > / \ > / \ > ImageBaseHDU ExtensionHDU > / \ / > / \ / > PrimaryHDU \ / > \ / > ImageHDU > > So there's a good example of diamond inheritance structure in real life > that's working, with additional branches off of it. A similar hierarchy > exists for table-like HDUs (GroupHDUs, AsciiTableHDUs, and > BinTableHDUs). Furthermore, not all of these classes take the same > arguments in their __init__()s, but this is handled by accepting **kwds > in all the __init__()s. > > So while I agree that multiple inheritance should be avoided, the > question is how strong is this statement intended to be? I would say > that it should be avoided unless you _really_ know what you're doing, > which is a statement I would make about multiple inheritance in any OO > language. > > Thanks, > Erik > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From erik.tollerud at gmail.com Sat Jul 23 23:31:38 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 23 Jul 2011 20:31:38 -0700 Subject: [AstroPy] proposal for astropy version control software and respository In-Reply-To: References: <4E20B621.4030105@stsci.edu> <1E77E068-8BF9-41F0-B04F-292ABFAA4485@stsci.edu> <4E2441F1.4000406@stsci.edu> Message-ID: Some of this was discussed in Tom's e-mail a few days back, but I just wanted to address a couple points that had been left dangling here. On Mon, Jul 18, 2011 at 11:37 AM, Victoria G. Laidler wrote: > > One problem with this poll that I haven't seen mentioned yet is that it > conflates use of git as the VCS with use of github as the hosting site. > These are in principle independent. The hosting site is not just a place to > stash the repository; it also provides related tools that are critical to > planning & managing workflow, like tickets/issues, history/timeline, nice > tools to view diff, history, & so on. You're absolutely right that these are in principal independent, and for some VCSs there are number of rather different competitive options. But in the particular case of git, there's really nothing even approaching github in terms of functionality, particularly for workflows that involve regularly merging other projects (as the vision here calls for). In fact, for a number of people I know that use it, github the major selling point of git, as opposed to the other way around. On Tue, Jul 19, 2011 at 6:49 AM, Matthew Turk wrote: > For what it is worth, enzo and yt are both astrophysics projects and > both use and love mercurial; the buy-in feels to be much lower than > that of git, it's much more straightforward, and extensions are very, > very easy to write. ?The primary hg hosting solution, BitBucket, has > substantially improved pull requests and other collaborative features > recently, and this has I believe brought it in line with GitHub in > many ways. Admittedly this is partly a matter of personal taste... I've used both and I actually prefer mercurial/bitbucket for small/private projects, but for to me it has seemed like git works for more complicated needs (like we certainly have here). In addition, most of the base python scientific packages are now on github (not counting more domain-specific tools like enzo and yt, of course!), so it seemed like the more natural choice. The other thing is that there's a hg-git plugin that lets you use mercurial to interact with a git repo, and as far as I can tell, it works flawlessly. So there's actually nothing stopping you from using hg to both push to and pull from a github repository. > One point I feel very strongly about is that under no circumstances > should a scientific code make a habit of 'rewriting history' or > changing the hash numbers of existing revisions. ?This would, for > instance, be part of a rebase-based workflow with git or on github. > hg discourages rewriting history, but it seems quite prevalent with > git. ?So even if git/github is chosen as a platform, I sincerely hope > that a workflow that does not change existing revision numbers in any > published repository is selected. Actually, github specifically says that you should not go back and alter anything actually pushed to a main repo (see the warning at the top of http://help.github.com/rebase/ ) for exactly the reasons you point out. So I 100% agree that altering the history of anything that's actually been put out there as part of the master branch or certainly for any release, is a huge no-no. Rebase is, however, useful for squashing commits you make as part of the process of fixing a problem/implementing a feature branch. This can make the history much more readable as it isn't full of "fixed tiny typo" sorts of commits. Additionally, it's good for merging patches that were merged with the master branch somewhere along the way, as this greatly simplifies the merge network. IPython actually has a policy of doing this for all significant patches to be merged as long as it is reasonably possible. So you're completely right that re-writing history is a thing to be *very* careful of, and to only use in certain restricted cases, but I think the base mercurial philosophy of basically no re-writing ever is a bit extreme (and in fact, that's why plugins like the rebase plugin were eventually developed for mercurial). -- Erik Tollerud From erik.tollerud at gmail.com Sun Jul 24 05:39:38 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sun, 24 Jul 2011 02:39:38 -0700 Subject: [AstroPy] Documentation Guidelines In-Reply-To: <4E281533.8050409@hs.uni-hamburg.de> References: <4E1C8BC1.7060103@stsci.edu> <4E1CAB8C.1040201@du.edu> <4E281533.8050409@hs.uni-hamburg.de> Message-ID: After the discussion on the list and some further experimentation, I think my concerns about numpydoc for processing docstrings have been addressed. I'm pretty sure I was the only one who had any concerns about that area, so I'm not sure there's even need for a vote, as it appears to me that we have reached consensus that numpy/scipy docstring style is the way to go. Given this, does anyone specifically want to see a vote on this matter? If not, I will update the next version of the guidelines to state that docstrings should follow the numpy/scipy style. On Thu, Jul 21, 2011 at 5:01 AM, Stefan Czesla wrote: > I also support adopting the Numpy/Scipy docstring format. > What about putting this issue up for a vote? The discussion > on this ?seems to be quite mature. > The yes/no thing has bot been accepted too well, so the range of > options should be broader this time. > > I would suggest that the doc guidelines contain an encouragement > for writing examples/tutorial. I know its in the coding guidelines, > but this seems to be a natural place to put it, too. > > Stefan > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > -- Erik Tollerud From geert at barentsen.be Sun Jul 24 17:48:47 2011 From: geert at barentsen.be (Geert Barentsen) Date: Sun, 24 Jul 2011 22:48:47 +0100 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: <4E29D2E0.6030708@gemini.edu> References: <4E245896.3020601@stsci.edu> <4E29D2E0.6030708@gemini.edu> Message-ID: > > Like Chris B, I'd have thought the case where superclasses are > "orthogonal bases" (I think that's what you mean by mixins) would > not present problems. > One could argue that the 'orthogonal bases' might become non-orthogonal by accident at some point in the future (i.e. someone adding a new feature), which could lead to an obscure bug. This is particularly true for a big project or library where you might not be aware of all subclasses. On the other hand, PyFITS demonstrates that multiple inheritance does not impede a library to be excellent and very useful :-)) Certainly seems fine to be pragmatic on this one. The main issue is that beginning OO programmers tend to be drawn to multiple inheritance when designing their first projects, while single inheritance would do fine in most cases. So, a useful guideline may be "Avoid multiple inheritance unless you understand the dirty details" :-) Geert -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Mon Jul 25 08:09:37 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 25 Jul 2011 05:09:37 -0700 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: <4E245896.3020601@stsci.edu> <4E29D2E0.6030708@gemini.edu> Message-ID: I have made a new version of the coding guidelines and posted them to: http://astropy.org/development/codeguide.html This version incorporates most of the feedback we've gotten so far on this thread, so if you've suggested something, you might want to look to make sure what's in there is what you had in mind. In particular, Michael, you may want to look at the new phrasing for the data file limit. Additionally, a few of you may want to look over the new phrasing for when C extensions are acceptable. One thing I have *not* yet altered is the section on super() and multiple inheritance, as the discussion does not seem to have concluded. Finally, a few specific items: > Maybe it's too early for this, but I think some version numbers should > be added regarding numpy, scipy and matplotlib compatibility, at least > to assure some consistency between the various core components. This definitely needs to be addressed, but I think the better way to address this is to include a specific version in the astropy setup.py script, and just say that the "supported version is that specified as required by setup.py". That makes it easier to keep things up to date, but of course we wouldn't bump those version numbers without good reason. I've included this in the new draft, but if anyone disagrees with this approach, I can change it. > What about weave? ?I've fallen in love with using inline C codes for > the obvious performance advantage when the stock NumPy/SciPy routines > don't cut it... ?just for quick and tiny hacks and such. ?It's much > less cumbersome than writing full blown C extensions for small tasks. My experience with weave is that it is great for doing quick small hacks and the like, but it can be very tricky to get it to deploy well - in many cases weird compiler version-specific glitches come up if you're not careful. I've also heard that weave is having trouble finding someone to be its lead developer, as the original author is no longer (although that may have changed). Additionally, I think there's no way to package the final product if the code includes weave - weave by nature requires a compiler matched to scipy, as far as I understand it (although I could be wrong about that). So given that you can do the same things with Cython, I'm inclined to say weave is too much of a distribution challenge to be allowed, despite the very nice features it offers. Any other opinions on the matter? -- Erik Tollerud From thomas.robitaille at gmail.com Mon Jul 25 12:36:04 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 25 Jul 2011 12:36:04 -0400 Subject: [AstroPy] Using MacPorts for Python Message-ID: Hi Everyone, Earlier this year I was given commit privileges to the MacPorts (http://www.macports.org/) package index, and I've already added a few Astronomy Python packages. After moved between various install methods in the last couple of years (system Python, EPD, source install, python.org distribution, ...) I've now settled on using the Python distribution from MacPorts. One of my main reasons for doing this is that I needed some complex packages (Qt and GTK+) which are not included in EPD, and by far the easiest way to install them has been through MacPorts. However, I believe a MacPorts Python distribution can still be very useful even if you don't need these packages, so I've compiled a list of simple instructions to set up a full Python distribution using MacPorts on Mac: http://astrofrog.github.com/macports-python/ Updating packages is easy, and dependencies are automatically taken care of. For now I recommend using Python 2.7 as indicated in the instructions. I would welcome any feedback, especially if you run into issues, and would also welcome suggestions for other packages to add. For new packages you can either email me directly or open a ticket on MacPorts and cc robitaille at macports.org on the ticket. To report issues with the instructions, open an issue at: https://github.com/astrofrog/macports-python Note that it's possible/easy to install several Python versions with MacPorts at the same time and you can switch which one is the default with 'port select', or call the version you want directly with ipython-3.1 for example. This makes it ideal if you want to be able to test your code using different python versions or want to experiment with Python 3. Cheers, Thomas From embray at stsci.edu Mon Jul 25 16:45:20 2011 From: embray at stsci.edu (Erik Bray) Date: Mon, 25 Jul 2011 16:45:20 -0400 Subject: [AstroPy] Coding Guidelines draft (comments encouraged) In-Reply-To: References: <4E245896.3020601@stsci.edu> <4E29D2E0.6030708@gemini.edu> Message-ID: <4E2DD5E0.5070404@stsci.edu> On 07/25/2011 08:09 AM, Erik Tollerud wrote: > I have made a new version of the coding guidelines and posted them to: > http://astropy.org/development/codeguide.html > > This version incorporates most of the feedback we've gotten so far on > this thread, so if you've suggested something, you might want to look > to make sure what's in there is what you had in mind. In particular, > Michael, you may want to look at the new phrasing for the data file > limit. Additionally, a few of you may want to look over the new > phrasing for when C extensions are acceptable. > > > One thing I have *not* yet altered is the section on super() and > multiple inheritance, as the discussion does not seem to have > concluded. Allow me to add a little more fuel to the discussion then :) First, for some additional motivating background, here's a post by Guido on the history of class method resolution orders (MROs) in Python, and the rationale for the current solution: http://python-history.blogspot.com/2010/06/method-resolution-order.html It's easier reading than any PEP on the topic, so I would consider it worth reading for anyone who would like more background. The recommendation given in the current Coding Guidelines completely breaks the intended MRO for __init__(). In this example A.__init__() is called twice (as well as object.__init__(), though this does nothing)! The effective MRO is D->B->A->object->C->A->object. You can see this for yourself if you take the original example and add some print statements: >>> class A(object): ... inits = 0 ... def __init__(self): ... self.inits += 1 ... if self.inits > 1: ... print 'Calling A.__init__() again!' ... else: ... print 'Calling A.__init__() for the first time.' ... >>> class B(A): ... def __init__(self): ... print 'Calling B.__init__().' ... A.__init__(self) ... >>> class C(A): ... def __init__(self): ... print 'Calling C.__init__().' ... A.__init__(self) ... >>> class D(C, B): ... def __init__(self): ... print 'Calling D.__init__().' ... B.__init__(self) ... C.__init__(self) >>> D() Calling D.__init__(). Calling B.__init__(). Calling A.__init__() for the first time. Calling C.__init__(). Calling A.__init__() again! <__main__.D object at 0x2afb22dd7750> While this may be harmless in some cases, it's most likely not what the developer really wants. And it's completely different from what Python would otherwise do which is D->C->B->A->object. This is a much more sensible MRO (and based on a well thought out algorithm that handles numerous corner cases that you might not think of immediately). This is also very difficult to do without cooperative super() calls. Having some classes *not* using super() will break any code that does try to use them as part of a cooperative super() call. So the better policy is to *require* the use of super(), especially in __init__() which is the most common case--I can't emphasize this enough :). The problems with this, I think, are overstated: In the simplest and most common case of single-inheritance using super(Foo, self).__init__() is equivalent to Foo.__init__(self). In the case of inheriting from two classes with no overlapping functionality--orthogonal bases as James put it--again it's a moot point. super() *only* adds complexity and/or confusion in diamond-like or even more complex class hierarchies. But in these cases super() will always make sure that the "right" thing is done. What Python's algorithm considers "right" may not always be exactly what the developer wants, but in the case of multiple inheritance Python follows its Zen by having One Right Way to do it. And for a complex and confusing subject like multiple inheritance that's a good thing. Admittedly, the semantics for using super() in Python 2.x are cumbersome. Python 3 at least allows calling super() with no arguments, which is the same as super(__class__, self)--the most common way of calling it. But that's just Python 3 :) Erik From thomas.robitaille at gmail.com Tue Jul 26 18:38:03 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 26 Jul 2011 18:38:03 -0400 Subject: [AstroPy] AstroPy Coordination Meeting this fall Message-ID: Hi everyone, Following the poll on a coordination meeting this fall, it looks like the CfA is convenient for the most people, so Tom Aldcroft and myself have been looking into when/how we could organize this. We are envisaging a two-day AstroPy development coordination meeting for AstroPy (not a general Python in Astronomy meeting, which could happen as a AAS splinter for example). We've come up with a couple of dates that would be best in terms of logistics, namely September 8-9th and October 12-13th. Note that this would be more of an informal meeting than a conference (no 'official' catering for example). We're planning on booking a meeting room big enough for 30-40 people with a projector, tables, and (of course) internet. We will also try and make sure that we can hold meetings in smaller sub-groups for coordination relating to specific components on one of the days. We've created a poll at the following URL where you can indicate your name and preferences: http://www.doodle.com/7ea9i6frm95tekv9 If neither of the dates work, please indicate it so we can discuss this further! Thanks! Thomas ps: In case this is relevant to some of you, the AAS HEAD meeting is 7-10th September in Rhode Island, which would conflict with the first set of dates. From thomas.robitaille at gmail.com Tue Jul 26 20:40:16 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 26 Jul 2011 20:40:16 -0400 Subject: [AstroPy] AstroPy Coordination Meeting this fall In-Reply-To: References: Message-ID: Just a quick note - it looks like Doodle will be down from 1 to 4pm EST on July 27th, so if you can't access the poll during those hours, try again later. Thomas On 26 July 2011 18:38, Thomas Robitaille wrote: > Hi everyone, > > Following the poll on a coordination meeting this fall, it looks like > the CfA is convenient for the most people, so Tom Aldcroft and myself > have been looking into when/how we could organize this. We are > envisaging a two-day AstroPy development coordination meeting for > AstroPy (not a general Python in Astronomy meeting, which could happen > as a AAS splinter for example). We've come up with a couple of dates > that would be best in terms of logistics, namely September 8-9th and > October 12-13th. > > Note that this would be more of an informal meeting than a conference > (no 'official' catering for example). We're planning on booking a > meeting room big enough for 30-40 people with a projector, tables, and > (of course) internet. We will also try and make sure that we can hold > meetings in smaller sub-groups for coordination relating to specific > components on one of the days. > > We've created a poll at the following URL where you can indicate your > name and preferences: > > http://www.doodle.com/7ea9i6frm95tekv9 > > If neither of the dates work, please indicate it so we can discuss this further! > > Thanks! > > Thomas > > ps: In case this is relevant to some of you, the AAS HEAD meeting is > 7-10th September in Rhode Island, which would conflict with the first > set of dates. > From vanderplas at astro.washington.edu Wed Jul 27 14:11:52 2011 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Wed, 27 Jul 2011 11:11:52 -0700 Subject: [AstroPy] AstroPy Coordination Meeting this fall In-Reply-To: References: Message-ID: <4E3054E8.6000707@astro.washington.edu> I'm not sure if there are many DES people on this list, but there will be a DES collaboration meeting at Penn from October 8-11. The October 12-13 option might allow some to attend the coordination meeting who would otherwise be unable to justify the trip out east. -Jake From jturner at gemini.edu Fri Jul 29 20:20:14 2011 From: jturner at gemini.edu (James Turner) Date: Fri, 29 Jul 2011 20:20:14 -0400 Subject: [AstroPy] AstroPy Coordination Meeting this fall In-Reply-To: References: Message-ID: <4E334E3E.1090104@gemini.edu> Hi Thomas, (Sorry for the delay -- our telescope has had problems this week). Two days would be quite a short time to fly all the way from Chile for. It might make more sense to try to connect remotely if that's the plan. Were you planning on setting up video links? I had been hoping we'd get together and get people contributing to the repository on the spot, but maybe that's optimistic. Personally, if I do come out, October would be easier as I have to renew my US visa around Sept, but that's just me (will comment on the poll once we figure out about the remote connections). Cheers, James. > Hi everyone, > > Following the poll on a coordination meeting this fall, it looks like > the CfA is convenient for the most people, so Tom Aldcroft and myself > have been looking into when/how we could organize this. We are > envisaging a two-day AstroPy development coordination meeting for > AstroPy (not a general Python in Astronomy meeting, which could happen > as a AAS splinter for example). We've come up with a couple of dates > that would be best in terms of logistics, namely September 8-9th and > October 12-13th. > > Note that this would be more of an informal meeting than a conference > (no 'official' catering for example). We're planning on booking a > meeting room big enough for 30-40 people with a projector, tables, and > (of course) internet. We will also try and make sure that we can hold > meetings in smaller sub-groups for coordination relating to specific > components on one of the days. > > We've created a poll at the following URL where you can indicate your > name and preferences: > > http://www.doodle.com/7ea9i6frm95tekv9 > > If neither of the dates work, please indicate it so we can discuss this further! > > Thanks! > > Thomas > > ps: In case this is relevant to some of you, the AAS HEAD meeting is > 7-10th September in Rhode Island, which would conflict with the first > set of dates. > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From thomas.robitaille at gmail.com Sat Jul 30 00:21:38 2011 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sat, 30 Jul 2011 00:21:38 -0400 Subject: [AstroPy] Testing Guidelines In-Reply-To: References: <00945222-7EE6-421B-AB46-A534A4E4A733@astro.physik.uni-goettingen.de> Message-ID: Let's keep the discussion going on testing frameworks. In the past, I've mainly used the standard unittest framework built-in to Python to write tests. I was wondering if those of you familiar with nose or py.test could explain what these packages add to the standard framework? What is it that *we* need on top of the most basic testing functionality unittest provides? Since py.test and nose are so similar, what are the *differences*? I think if we know this, it will be easier for those of us not too familiar with either framework to weigh in on the issue. By the way, it looks like there's been a lot of work on unittest in python 2.7 (aka unittest2) so how does that compare to nose and py.test? Cheers, Tom On 13 July 2011 18:54, Erik Tollerud wrote: >> b.t.w. ?I suggest that you differentiate the _purpose_ of a test from >> the _mechanism_ of the test. > > This is a very good point - when I think about it more, it makes sense > to have the same test-runner framework. ?With either nose or py.test > it seems reasonable to divide the testing into tests which always > depend only on the core modules, and a separate suite that is more > general, potentially using e.g. pysofa to test coordinate conversions > and the like. ?As Mark says, there isn't necessarily a need (at least > at this stage) to carefully divide tests into categories, as long as > testing coverage is good for the actual purpose of the module. > >> What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion. >> I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly ?developed or revised code, also to encourage test-driven development practices. > > Absolutely agreed - the plan is to make it clear in the example > package that everything should be unit-tested if it can be, as that's > what new packages will presumably use. > > > Now the nose vs. py.test question... From Mark's summary, it sounds > like it's difficult to choose between them because they're too > similar! ?Given Mark and Victoria's comments (and looking over the > docs in more detail myself), I'm beginning to think that py.test has > more potential. ?Especially important is that fact that it appears > that py.test can directly use tests written for nose, while the > converse is not true... this would make integrate already-existing > projects with nose much easier and lower the re-learning penalty to > pretty much nothing. ?Anyone else have experience with py.test that > may be of relevance? > > >> >> Cheers, >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek >> >>> On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz wrote: >>>> >>>>>> As for the testing framework, we were thinking nose >>>>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). ? Nose is >>>>>> very nice for a project like this because it's super easy to run and >>>>>> also easy to write tests with, but is also compatible with the stdlib >>>>>> unittests. >>>> >>>> >>>> "use nose" is a little ambiguous. ?numpy "uses nose" but if you look >>>> inside numpy.test() it looks like of like they implemented their own >>>> testing system using nose as a library. ?I've never succeeded in using >>>> nosetests to run the numpy tests, for example. >>>> >>>> If you choose nose, my preference is to say: ?Tests come in a separate >>>> directory from the source. ?You can run all the tests with the command >>>> "nosetests tests/". >>>> >>>> This is not to exclude the possibility of installing some of the tests >>>> or of providing a test() function, but I also want to run the tests >>>> directly with nosetests. >>>> >>>> I actually have an ulterior motive here: ?I use a nose plugin that feeds >>>> a report generator. ?If you only provide a foo.test() function, I can't >>>> necessarily get it to use the plugin even if you base the whole thing on >>>> nose. ?You might not think that was a big deal, but it gets important >>>> when you have the test volume that I regularly deal with: > 100 000 >>>> tests per night, sometimes > 10 000 fail, depending what the developers >>>> are up to. >>>> >>>> b.t.w. ?This doesn't mean I have any attachment to nose; it just happens >>>> to be one of a small number of test runners that I use right now. >>>> >>> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > > > -- > Erik Tollerud > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From erik.tollerud at gmail.com Sat Jul 30 06:07:10 2011 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 30 Jul 2011 03:07:10 -0700 Subject: [AstroPy] Testing Guidelines In-Reply-To: References: <00945222-7EE6-421B-AB46-A534A4E4A733@astro.physik.uni-goettingen.de> Message-ID: (To summarize, if this message is TL;DR: After looking more closely, I'm thinking py.test fits our purposes because it seems to do be able to accept syntax from unittest, doctest, nose, and its own all at once, while the others are more limited. It also may be better for future plugins, and it doesn't have be installed for the test suit to be run by users, unlike nose.) > Let's keep the discussion going on testing frameworks. In the past, > I've mainly used the standard unittest framework built-in to Python to > write tests. I was wondering if those of you familiar with nose or > py.test could explain what these packages add to the standard > framework? What is it that *we* need on top of the most basic testing > functionality unittest provides? Basically, both nose and py.test's biggest improvements over unittest are much simpler testing syntax, and they gather and run whole test suites more easily. Compare the "basic example" section for unittest (http://docs.python.org/library/unittest.html#basic-example) to the basic examples for nose (http://packages.python.org/nose/writing_tests.html). Unittest closely follows the JUnit technique, which is based on java-style programming, and hence has a quite verbose syntax requiring a class for basically every test, and yet another class if you want a whole test suite. With nose and py.test, on the other hand, you implement simply by adding a "test_modname.py" file, and write functions in that file of the form "def test_thisfeature()". Then, when you run the command line tools, it usually finds all your tests and runs them, issuing a final report of passes and failures. (Also, "python setup.py test" from distrbute/setuptools can be set to do the same thing, and it seems to require quite a bit of configuration in unittest). nose and py.test both also extend this functionality quite a bit and provide additional bells and whistles, but this is really the main advantage, in my view: they both can do everything unittest does (in fact, by default they even call unittest tests), but with much simpler syntax, and without all the (at least, to me) repetitive boilerplate code you see in unittest. > Since py.test and nose are so > similar, what are the *differences*? I think if we know this, it will > be easier for those of us not too familiar with either framework to > weigh in on the issue. They have mostly the same basic features, but a lot of the more complicated tricks and the plugin libraries are slightly different in their syntax and capabilities. It appears that both nose and py.test can run unittest and doctest tests, and py.test has explicit support for a number of nose's unique features, but nose doesn't support py.test syntax beyond where the two of them match. In my mind this is the best reason to pick py.test : it is the least restrictive because people can continue writing tests in whatever they're used to, and py.test should run it without problems. Victoria pointed out that nose is apparently undergoing a development hiccup as the main author is backing off, and the community is slowly (which may mean never) picking up the slack. py.test, on the other hand, is currently being actively improved. This may not matter too much because both are already quite stable, but it might be a concern for the future. It appears both work on python 3, but it seems like it has been more recently added for nose (it claims to be fully supported, although 3rd party plugins are not necessarily). After hearing from a few people (on-list and elsewhere) and glancing over the docs, it looks to me that py.test is somewhat easier to write plugins for than nose, partly because of a cleaner plugin architecture (py.test is actually implemented internally as a bunch of plugins), and more complete documentation. This may be relevant for us because we probably will need to do some of this if we want conformance testing of some modules against "known-good" external libraries (although py.test seems to have a builtin plugin that does that sort of thing already?). The only thing that I see as useful that only one of them has is that py.test can generate a script that can be used to do all py.test actions. If we did theis, users don't even have to have py.test installed! So that would allow us to include the py.test runner script in the source code, and people wouldn't need to install an external package to collect/run the test suite like they do with nose. In principle, the similarities between nose and py.test mean we wouldn't really have to make a decision until we need features that differ between them- developers could run whichever they prefer... Except there's one problem: the configuration files are formatted differently... We want to use configuration settings for just one of the packages so that the suite always runs the same. So what we're really deciding is which one to write configuration files for, and request that people use as the "standard" test suite. Given all the points here, I now think py.test is a mildly better choice... but either py.test or nose would probably suit our needs. > By the way, it looks like there's been a lot of work on unittest in > python 2.7 (aka unittest2) so how does that compare to nose and > py.test? The biggest change I see is that it seems to now include a test discovery tool, but it is much less documented (a few paragraphs rather than many pages each for the other two), and seems to lack most of the nose and py.test runners' customization (like configuring in the setup.cfg file, easy plugins, etc.). It's an improvement, but it's still based around the more complex syntax of unittest, so most of what I said above as far as I can tell is still true. Regardless, we couldn't use that anyway if we're planning to stay 2.6-compatible. > > Cheers, > Tom .+++ > > On 13 July 2011 18:54, Erik Tollerud wrote: >>> b.t.w. ?I suggest that you differentiate the _purpose_ of a test from >>> the _mechanism_ of the test. >> >> This is a very good point - when I think about it more, it makes sense >> to have the same test-runner framework. ?With either nose or py.test >> it seems reasonable to divide the testing into tests which always >> depend only on the core modules, and a separate suite that is more >> general, potentially using e.g. pysofa to test coordinate conversions >> and the like. ?As Mark says, there isn't necessarily a need (at least >> at this stage) to carefully divide tests into categories, as long as >> testing coverage is good for the actual purpose of the module. >> >>> What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion. >>> I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly ?developed or revised code, also to encourage test-driven development practices. >> >> Absolutely agreed - the plan is to make it clear in the example >> package that everything should be unit-tested if it can be, as that's >> what new packages will presumably use. >> >> >> Now the nose vs. py.test question... From Mark's summary, it sounds >> like it's difficult to choose between them because they're too >> similar! ?Given Mark and Victoria's comments (and looking over the >> docs in more detail myself), I'm beginning to think that py.test has >> more potential. ?Especially important is that fact that it appears >> that py.test can directly use tests written for nose, while the >> converse is not true... this would make integrate already-existing >> projects with nose much easier and lower the re-learning penalty to >> pretty much nothing. ?Anyone else have experience with py.test that >> may be of relevance? >> >> >>> >>> Cheers, >>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek >>> >>>> On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz wrote: >>>>> >>>>>>> As for the testing framework, we were thinking nose >>>>>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). ? Nose is >>>>>>> very nice for a project like this because it's super easy to run and >>>>>>> also easy to write tests with, but is also compatible with the stdlib >>>>>>> unittests. >>>>> >>>>> >>>>> "use nose" is a little ambiguous. ?numpy "uses nose" but if you look >>>>> inside numpy.test() it looks like of like they implemented their own >>>>> testing system using nose as a library. ?I've never succeeded in using >>>>> nosetests to run the numpy tests, for example. >>>>> >>>>> If you choose nose, my preference is to say: ?Tests come in a separate >>>>> directory from the source. ?You can run all the tests with the command >>>>> "nosetests tests/". >>>>> >>>>> This is not to exclude the possibility of installing some of the tests >>>>> or of providing a test() function, but I also want to run the tests >>>>> directly with nosetests. >>>>> >>>>> I actually have an ulterior motive here: ?I use a nose plugin that feeds >>>>> a report generator. ?If you only provide a foo.test() function, I can't >>>>> necessarily get it to use the plugin even if you base the whole thing on >>>>> nose. ?You might not think that was a big deal, but it gets important >>>>> when you have the test volume that I regularly deal with: > 100 000 >>>>> tests per night, sometimes > 10 000 fail, depending what the developers >>>>> are up to. >>>>> >>>>> b.t.w. ?This doesn't mean I have any attachment to nose; it just happens >>>>> to be one of a small number of test runners that I use right now. >>>>> >>>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >>> >> >> >> >> -- >> Erik Tollerud >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud