From barrett at stsci.edu  Tue Sep  7 15:09:37 2004
From: barrett at stsci.edu (Paul Barrett)
Date: Tue, 07 Sep 2004 15:09:37 -0400
Subject: [AstroPy] Python version of IDL Astron library
Message-ID: <413E0771.1010705@stsci.edu>

<x-flowed>
Dear Astro-pythoneers,

As part of my functional duties at STScI, I have been tasked with developing a 
Python version of the IDL Astron Library, at least the most relevant parts of 
it. I began with a rewrite of the IDLDB module last winter, however I have not 
been able to complete it due to other higher priority projects, namely 
matplotlib development and CALSTIS maintainance.  While at the SciPy 2004 
Conference at CalTech last week (Sep 2-3), Lee Rottler and Joe Harrington 
suggested that a good mechanism to get astronomers to migrate from IDL to Python 
would be a Python version of the IDL Astron library .  After further discussion, 
it was decided that I would solicit help from the AstroPy community and would 
coordinate such a project.

The IDL Astronomy User's Library (http://idlastro.gsfc.nasa.gov/contents.html) 
web page lists 18 software categories, however in my opinion, only 8 of them 
have functions or procedures that are of current interest to astronomers and 
have not already been duplicated by other Python modules.  The FITS procedures 
are a good example, since most, if not all, of this functionality is duplicated 
in the pyfits module.  I have identified the following categories as requiring 
attention:

  o Astronomical Utilities
  o DAOPHOT-Type Photometry Procedures
  o Database Procedures
  o Disk I/O (MIDAS, IRAF files)
  o Image Manipulation
  o Math and Statistics
  o Robust Statistics Procedures
  o Miscellaneous (Non-Astronomy) Procedures

I have refrained from listing the procedures in each category, since there are a 
large number of them.  Instead, I will add a page to the AstroPy web site in the 
next few days, if there is sufficient interest in the project by the AstroPy 
community.  I do not expect developers to devote are large percentage of their 
time to this project, unless they want to of course, but to contribute two or 
three procedures that are of interest to them over the next year.  I will 
coordinate the integration of these procedures into a Python package and check 
that the necessary documentation and test code has been included.

Any takers?

  -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218
_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From William.T.Bridgman.1 at gsfc.nasa.gov  Tue Sep  7 15:52:01 2004
From: William.T.Bridgman.1 at gsfc.nasa.gov (W.T. Bridgman)
Date: Tue, 7 Sep 2004 15:52:01 -0400
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <413E0771.1010705@stsci.edu>
References: <413E0771.1010705@stsci.edu>
Message-ID: <626117F6-0107-11D9-967A-0030657B87AC@gsfc.nasa.gov>

<x-flowed>I have some coordinate transformation stuff - though not entirely 
complete/consistent - that I use for my Sun-Earth Connection work 
(geocentric/heliocentric/ecliptic).  I've been experimenting with a 
python wrapper for the cxform coordinate conversion library (in C) 
which partially works for Sun-Earth coordinate systems.  I believe I 
may even have an AITOFF re-projection routine from some experiments 
with BATSE data(!!).  More recently, I've been cleaning up my UTCclass 
for managing UTC times.

Are we looking for pure-python solutions or are mixed python/C 
solutions acceptable?

Big trick would be trying to 'normalize' them so they work together 
well, calling other components of the module when needed.

My $0.02 worth,
Tom

On Sep 7, 2004, at 3:09 PM, Paul Barrett wrote:

>
> Dear Astro-pythoneers,
>
> As part of my functional duties at STScI, I have been tasked with 
> developing a Python version of the IDL Astron Library, at least the 
> most relevant parts of it. I began with a rewrite of the IDLDB module 
> last winter, however I have not been able to complete it due to other 
> higher priority projects, namely matplotlib development and CALSTIS 
> maintainance.  While at the SciPy 2004 Conference at CalTech last week 
> (Sep 2-3), Lee Rottler and Joe Harrington suggested that a good 
> mechanism to get astronomers to migrate from IDL to Python would be a 
> Python version of the IDL Astron library .  After further discussion, 
> it was decided that I would solicit help from the AstroPy community 
> and would coordinate such a project.
>
> The IDL Astronomy User's Library 
> (http://idlastro.gsfc.nasa.gov/contents.html) web page lists 18 
> software categories, however in my opinion, only 8 of them have 
> functions or procedures that are of current interest to astronomers 
> and have not already been duplicated by other Python modules.  The 
> FITS procedures are a good example, since most, if not all, of this 
> functionality is duplicated in the pyfits module.  I have identified 
> the following categories as requiring attention:
>
>  o Astronomical Utilities
>  o DAOPHOT-Type Photometry Procedures
>  o Database Procedures
>  o Disk I/O (MIDAS, IRAF files)
>  o Image Manipulation
>  o Math and Statistics
>  o Robust Statistics Procedures
>  o Miscellaneous (Non-Astronomy) Procedures
>
> I have refrained from listing the procedures in each category, since 
> there are a large number of them.  Instead, I will add a page to the 
> AstroPy web site in the next few days, if there is sufficient interest 
> in the project by the AstroPy community.  I do not expect developers 
> to devote are large percentage of their time to this project, unless 
> they want to of course, but to contribute two or three procedures that 
> are of interest to them over the next year.  I will coordinate the 
> integration of these procedures into a Python package and check that 
> the necessary documentation and test code has been included.
>
> Any takers?
>
>  -- Paul
>
> -- 
> Paul Barrett, PhD      Space Telescope Science Institute
> Phone: 410-338-4475    ESS/Science Software Branch
> FAX:   410-338-4767    Baltimore, MD 21218
> _________________________________________________
> AstroPy mailing list     -      astropy at stsci.edu
> http://www.astro.washington.edu/owen/AstroPy.html
>
>
--
Dr. William T."Tom" Bridgman               Scientific Visualization 
Studio
Global Science & Technology, Inc.          NASA/Goddard Space Flight 
Center
Email: William.T.Bridgman.1 at gsfc.nasa.gov  Code 935
Phone: 301-286-1346                        Greenbelt, MD 20771
FAX:   TBD                                 http://svs.gsfc.nasa.gov/

_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From mccannwj at pha.jhu.edu  Tue Sep  7 16:54:36 2004
From: mccannwj at pha.jhu.edu (William Jon McCann)
Date: Tue, 07 Sep 2004 16:54:36 -0400
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <413E0771.1010705@stsci.edu>
References: <413E0771.1010705@stsci.edu>
Message-ID: <413E200C.5040103@pha.jhu.edu>

<x-flowed>(sent this earlier with attachments but it didn't seem to get through)

Hi Paul,

Paul Barrett wrote:
> As part of my functional duties at STScI, I have been tasked with 
> developing a Python version of the IDL Astron Library, at least the most 
> relevant parts of it. I began with a rewrite of the IDLDB module last 
> winter, however I have not been able to complete it due to other higher 
> priority projects, namely matplotlib development and CALSTIS 
> maintainance.  While at the SciPy 2004 Conference at CalTech last week 

I wrote this a few years ago and haven't looked at it since - I may not
want to...  Anyway, it might be useful.

It was designed to mimic the IDLDB API as closely as possible and as a
result probably doesn't do things in a very Pythonic way.

We used this to run DB queries from the web during ACS ground calibration.

http://acs.pha.jhu.edu/instrument/calibration/software/zdbase/

Cheers,
Jon

_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From rays at blue-cove.com  Tue Sep  7 17:07:59 2004
From: rays at blue-cove.com (Ray S)
Date: Tue, 07 Sep 2004 14:07:59 -0700
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <413E0771.1010705@stsci.edu>
Message-ID: <5.2.0.4.2.20040907132928.09effb80@blue-cove.com>

<x-flowed>Speaking of migrating from IDL to Python, I would like to see some 
discussion of the issues/problems others have uncovered in their ports.

I have been trying to port Hongjie Xie's IDL/ENVI Implementation of the FFT 
Based Algorithm for Automatic Image Registration
http://www.nmt.edu/%7Ehjxie/xie-paper.pdf PDF
ftp://ftp.iamg.org/VOL29/v29-08-10.zip sources
which calculates shift, scale and rotation for 2d arrays, to 
Python/numarray/Numeric and running into some subtle problems.

As small examples, the IDL ROT() requires some additional steps to emulate, 
and the MAX(array, position) as well.
I'm not at all familiar with the correspondence of the FFT libraries... 
even the simple shift-only routine of mine
http://rjs.org/astro/1004x/Python/register/shift_idl.py
produces different (incorrect) x shift values than Xie's IDL.

My current code
http://rjs.org/astro/1004x/Python/register/
conforms to Xie's flow as much as possible for now, but should and will be 
re-factored when functional. His LogPolar function, to convert rectangular 
coordinate arrays to polar, is particularly slow.

Dr. Xie said that he is available for specific assistance, but Python is 
not part of his repertoire (although C is).

I was recently informed that IDL has a usable trial, so I am trying it out 
this week, but if anyone has an interest in this general array 
registration, please let me know... I think that the algorithm would make a 
nice standalone module, or perhaps something for nd_image.


At 03:09 PM 9/7/2004 -0400, Paul Barrett wrote:

>I have refrained from listing the procedures in each category, since there 
>are a large number of them.  Instead, I will add a page to the AstroPy web 
>site in the next few days, if there is sufficient interest in the project 
>by the AstroPy community.  I do not expect developers to devote are large 
>percentage of their time to this project, unless they want to of course, 
>but to contribute two or three procedures that are of interest to them 
>over the next year.

>Any takers?

I am interested in developing registration methods, at least. I might be 
able to enlist the help of some SANPYT members as well; Dr. Chris Seberino 
was at SciPy 2004, and I'd only heard some tidbits from him as yet.


_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From barrett at stsci.edu  Wed Sep  8 10:17:57 2004
From: barrett at stsci.edu (Paul Barrett)
Date: Wed, 08 Sep 2004 10:17:57 -0400
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <20040908125826.2c6c729b.federico.nati@tiscalinet.it>
References: <413E0771.1010705@stsci.edu>	<200409072156.16675.jaime@iaa.es>	<7FE03BA0-010A-11D9-9B91-000A95B68E50@stsci.edu>	<413E3105.70404@stsci.edu> <20040908125826.2c6c729b.federico.nati@tiscalinet.it>
Message-ID: <413F1495.3040807@stsci.edu>

<x-flowed>Federico Nati wrote:
> Hi,
> 
> very good idea.
> if this project is just a Python version of IDL Astron library then I
> think this can help all of us who use python to have some useful
> routines not yet included in other packages. Everyone of us has his own
> small program or module to do that coords conversion or that file
> format handling and so on, and this project can help in joining the
> efforts.
> I will be happy to help this project.
> 
> But I think another issue is making people migrate from IDL to
> Python.
> 
> The question is: why people (I mean astronomers) use IDL? Or, better,
> which features make IDL more suitable for astronomers than Python? 
> I think this is beacause:
> 
> 1- they have Astronomy User's Library
> 2- they have one, only one, standard, easy (and nice!) way to make plots
> 3- they have specific library and tool written for IDL, one example is
>    Healpix:  http://www.eso.org/science/healpix/
> 
> and I would add that they are used to it... but there is no solution for
> this...
> 
> This project should help in point 1, but if we really want to complete
> the task we should solve also the other points.

Yes.  My intention for this project is to make the transition easy for IDL 
users, hence the desire for similar functionality and behavior.  Though later we 
may find that some routines can be replaced by better ones in future.

> IMHO point 2 can be solved (or maybe is already almost solved) by using
> Gnuplot and Python's interface to it, eventually extending it to make it
> slightly easier for users.

I think this point is solved by matplotlib 
(http://sourceforge.net/projects/matplotlib/).  In my opinion, the general 
consensus at ScyPy '04 is that matplotlib solves or is very close to solving 
this problem. Take a look at it and tell us what you think.

> Point 3 requires specific efforts.

Well we have pyfits as a start.  We just need an high-energy astronomy to take 
the lead in developing a healpix module.  Want to take the lead?

  -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218
_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From jh at oobleck.astro.cornell.edu  Wed Sep  8 10:49:49 2004
From: jh at oobleck.astro.cornell.edu (Joe Harrington)
Date: Wed, 8 Sep 2004 10:49:49 -0400
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <034F5E36-0193-11D9-B782-0030657B87AC@gsfc.nasa.gov>
	(William.T.Bridgman.1@gsfc.nasa.gov)
References: <413E0771.1010705@stsci.edu> <626117F6-0107-11D9-967A-0030657B87AC@gsfc.nasa.gov> <p0611040bbd63cd559e8d@[128.95.99.44]> <034F5E36-0193-11D9-B782-0030657B87AC@gsfc.nasa.gov>
Message-ID: <200409081449.i88EnnF1024479@oobleck.astro.cornell.edu>

Hi all,

A Python version of the IDLastro library is a great idea, and crucial
for the success of Python in the astronomy community.  The main issues
raised so far:

underlying packages (numeric vs. Numarray, plotting)
Python, C, C++, FORTRAN?
closeness of implementation to IDL library

to which I'll add (interspersed):

documentation
code conversion
coordination

UNDERLYING PACKAGES

There isn't much of a standard in the IDL library, except for
consistently following the IDL doc header format.  However, Python's
richness and the fact that this won't be a first implementation by a
few people opens us up to greater risk than the IDL developers faced.
I think that we should develop a set of standards.  At SciPy '04, we
came up with the following:

1. Numarray for all new code, and to teach to new users.  New docs
   teach only Numarray.  Soon Numarray vs. numeric will be a run-time
   choice, and hopefully in a year or two there won't be a compelling
   reason to use numeric over Numarray.  Right now the issues are
   mainly performance issues for small arrays (less than 1000
   elements), which matter more to mathematicians and biologists than
   to astronomers.

2. Matplotlib for all plotting.  Try it.  You'll like it!  It has a
   matlab-like interface, but that's just a front end to the full OO
   plotting functionality.  You don't have to learn the matlab-like
   interface if you don't want to.  It's interactive so you can adjust
   your axis scaling with the mouse, etc.

3. Ipython.  This is a wonderful and 100% compatible improvement to
   the interactive parser.  It doesn't affect programming at all, so
   it's not very relevant to the current topic.

4. All of the above to be provided in binary packages as well as
   source packages and tarballs.  Each currently-popular platform must
   be supported by its native packager:
	Fedora		RPM/YUM
	Debian		Debian Package Manager/APT
	Sun		tar+checkinstall?  or is there something better now?
	PC		I forget the name, but there is one now
	MAC		I forget the name, but there is one now

5. A community effort to (re)write all needed documentation.

6. A community effort to create an excellent web site at scipy.org,
   with (links to) discipline-specific pages that collect, organize,
   rate, and distribute topical application software.  The goal here
   is to proactively take the user to where he or she needs to go, not
   just to throw a lot of information around and baffle people.

I have sent Perry a draft of a foundational document for the project,
which I will post here for comment when we've agreed on it among
ourselves.  However, the basic points above were consensus (or nearly
so) at the conference.

PYTHON/C/C++/FORTRAN

Here's how this affects AstroPy (or PyAstro, or whatever Paul wants to
call it).  First, the choice of what underlying code to use (pure
Python, C, C++, FORTRAN) should be the developer's, so long as that
person can get it working in scipy_distutils on all platforms.  I was
told at the conference that Python, C, and C++ are all easy to get
working, but that FORTRAN is problematic.  However, there was a
presentation at the conference of a new build utility (it doesn't have
a name, though I jokingly called it "makeover") that claims to have
cataloged all the options for every FORTRAN compiler in existence,
including Borland's older ones.  So, that may become the new standard,
or that code may be borrowed by distutils.  Stay tuned on that one.

There are strong benefits to writing in a compiled language, the main
one being portability to later interactive languages.  Python will not
last forever.  A good run for an interpreted language is 15 years.  In
contrast, FORTRAN and C are 2-3 times as old and going strong.  Python
may grow and last longer, or may not.  When we move on to the next
great thing, we will again go through an agonizing process of
rewriting and conversion.  The stuff that gets rewritten first will be
the stuff that's just wrappers around C, C++, and FORTRAN.  Wouldn't
it be great if that's nearly everything?  If you look at SciPy now,
the majority of it is wrapped compiled libraries that implement
practically all of the basic numerical routines.  Imagine where we'd
be if we had to *write* that stuff in Python, rather than wrap it.
Imagine how hard the next switch in interactive languages would be if
we did that.

There is also a strong argument against rewriting code that already
works.  We have a lot of work to do, so if you have working compiled
code, please just wrap it.  If you are writing new code, it might make
sense for large, monolithic algorithms to be written in C or C++ and
to be wrapped for Python.  Smaller routines (the majority, by number
at least) are probably best done in Python directly.  These won't be
hard to redo in the next language.  In any case, if you don't want to
learn distutils and won't be providing distutils-building code, please
stick to Python, or hook up with a distutils hacker.

DOCS

Documentation is a strong suit of IDL's and we need to be as good.  I
am not aware of a standard doc header for Python.  If there is one, we
should use it.  If not, I suggest we essentially copy IDL's, with some
modifications to rationalize it and to make it work for Python.  We
also need a code that will extract the doc pages from a package and
will collect the docs, turn them into HTML and PDF, and put them in a
searchable database.  This code will be central to SciPy as a whole so
upstream coordination with the soon-to-be-born doc effort would be
great.

The source language of docs is another issue.  It has to be open and
functional on all platforms.  It has to handle simple markup and
inline figures.  It has to produce PDF and HTML.  MSWord is out.
Research Structured Text or LaTeX are my votes.  Some like LyX and
TeXmacs.  We'll see what the community wants.

SYNTACTIC CLOSENESS

Syntactic closeness to the IDL library would be nice but isn't crucial
from my perspective.  Where things are reasonable, keep them.  Where
they are not (what does "sxpar" mean, and to whom?), make them so.  We
could provide a compatibility wrapper for fans of sxpar.

CODE CONVERTER

One idea that has been tossed around is a code converter.  There are
two approaches to this, which I call the 80% solution and the 100%
solution.  The 80% solution converts all the procedural code on a
line-by-line basis.  It would do the gruntwork, and would ensure that
array access, etc., is converted correctly (otherwise subtle bugs will
creep in).  The developer would still have some gruntwork to do to get
it to actually work, but not a lot, particularly if the code was
simple and didn't depend on a lot of IDL library routines.  The 100%
solution would be 100 times harder to write.  It would convert all the
code, including the OO parts, and would also implement the IDL
library, generally in terms of a set of wrapper routines that called
SciPy routines.  It would actually be a re-implementation of IDL.  I
claim this is unnecessary and not even very desirable.  We want people
to switch to Python and to contribute new code to the community.  I
have no personal interest in providing them a free IDL.

Anyway, if we had a converter, this entire project, as well as the
project of getting astronomers to switch, would be *much* easier!  It
would be safest from a legal point of view for the converter authors
NEVER to have run IDL nor to have read RSI's manuals.  The IDL license
prohibits reverse-engineering, and most of us have agreed to it.  I
don't believe it's reverse engineering to write a code converter from
a language that you know into another language that you know that has
the same capability.  However, I'm not a lawyer, and I don't have the
means to fight that battle in court.  IDL is simple enough that
someone who doesn't know it can get a commercial book on IDL, such as
Gumley's Practical IDL Programming, that provides all the information
that's needed.  Any of us experienced IDL users can then publish a FAQ
on the web that answers any questions the programmers might have,
without our actually writing or looking at any code.  This should keep
the whole effort legally above reproach.

COORDINATION

Finally, to address coordination issues, I think community testing and
review is key.  Let's ask people to post their development plans in
advance for community design review and to update a public-read CVS
often.  Let's have public review and testing of all new code before
accepting it into the library.  That way we'll address
interoperability issues and catch bugs early.  If we're a successful
community project, Paul's greatest contribution will be the
coordination of the effort and the moderation of the review process,
rather than the bits of code he produces himself.

As always, I welcome your comments.

--jh--
Joe Harrington
326 Space Sciences Building
Cornell University
Ithaca, NY 14853-6801
(607) 254-8960 office
(607) 255-9002 fax
jhmail at oobleck.astro.cornell.edu
_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html

From perry at stsci.edu  Wed Sep  8 17:53:11 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 8 Sep 2004 17:53:11 -0400
Subject: [AstroPy] Python version of IDL Astron library
Message-ID: <7A5B095D-01E1-11D9-B878-000A95B68E50@stsci.edu>

<x-flowed>
On Sep 8, 2004, at 10:49 AM, Joe Harrington wrote:

> Hi all,
>
> A Python version of the IDLastro library is a great idea, and crucial
> for the success of Python in the astronomy community.  The main issues
> raised so far:
>
> underlying packages (numeric vs. Numarray, plotting)
> Python, C, C++, FORTRAN?
> closeness of implementation to IDL library
>
> to which I'll add (interspersed):
>
> documentation
> code conversion
> coordination
>
> UNDERLYING PACKAGES
>
> There isn't much of a standard in the IDL library, except for
> consistently following the IDL doc header format.  However, Python's
> richness and the fact that this won't be a first implementation by a
> few people opens us up to greater risk than the IDL developers faced.
> I think that we should develop a set of standards.  At SciPy '04, we
> came up with the following:
>
> 1. Numarray for all new code, and to teach to new users.  New docs
>    teach only Numarray.  Soon Numarray vs. numeric will be a run-time
>    choice, and hopefully in a year or two there won't be a compelling
>    reason to use numeric over Numarray.  Right now the issues are
>    mainly performance issues for small arrays (less than 1000
>    elements), which matter more to mathematicians and biologists than
>    to astronomers.
>
I'll agree with that (no surprise).

> 2. Matplotlib for all plotting.  Try it.  You'll like it!  It has a
>    matlab-like interface, but that's just a front end to the full OO
>    plotting functionality.  You don't have to learn the matlab-like
>    interface if you don't want to.  It's interactive so you can adjust
>    your axis scaling with the mouse, etc.
>
I'll just add that it needs some more documentation. John Hunter is 
developing
a user guide which looks to be more than half done now. There are still 
some
rough edges but it is pretty functional at this point. Nothing come 
close to
it in terms of cross platform support and cross GUI support.

> 3. Ipython.  This is a wonderful and 100% compatible improvement to
>    the interactive parser.  It doesn't affect programming at all, so
>    it's not very relevant to the current topic.
>
True, but it is really a cool tool that all should be aware of.

> 4. All of the above to be provided in binary packages as well as
>    source packages and tarballs.  Each currently-popular platform must
>    be supported by its native packager:
> 	Fedora		RPM/YUM
> 	Debian		Debian Package Manager/APT
> 	Sun		tar+checkinstall?  or is there something better now?
> 	PC		I forget the name, but there is one now
> 	MAC		I forget the name, but there is one now
>
> 5. A community effort to (re)write all needed documentation.
>
> 6. A community effort to create an excellent web site at scipy.org,
>    with (links to) discipline-specific pages that collect, organize,
>    rate, and distribute topical application software.  The goal here
>    is to proactively take the user to where he or she needs to go, not
>    just to throw a lot of information around and baffle people.
>
Well, it raises the question of whether it should all be hosted at 
scipy.org
or not. I think of an astron equivalent as being primarily 
astronomy-oriented
in scope; to the extent that scipy.org will have areas for specialties 
such as
astronomy, that makes sense (something we should pursue with them).
  If not, an astropy site for astron-type tools and scipy.org for 
general science
and engineering tools.

Would you volunteer to manage the top-level scipy web page?
>
> PYTHON/C/C++/FORTRAN
>
> Here's how this affects AstroPy (or PyAstro, or whatever Paul wants to
> call it).  First, the choice of what underlying code to use (pure
> Python, C, C++, FORTRAN) should be the developer's, so long as that
> person can get it working in scipy_distutils on all platforms.  I was
> told at the conference that Python, C, and C++ are all easy to get
> working, but that FORTRAN is problematic.  However, there was a
> presentation at the conference of a new build utility (it doesn't have
> a name, though I jokingly called it "makeover") that claims to have
> cataloged all the options for every FORTRAN compiler in existence,
> including Borland's older ones.  So, that may become the new standard,
> or that code may be borrowed by distutils.  Stay tuned on that one.
>
Since there is a lot of good Fortran code, this would be very nice.
Much of scipy is based on Fortran code (and the source of most of the
build and installation hassles).

> There are strong benefits to writing in a compiled language, the main
> one being portability to later interactive languages.  Python will not
> last forever.  A good run for an interpreted language is 15 years.  In
> contrast, FORTRAN and C are 2-3 times as old and going strong.  Python
> may grow and last longer, or may not.  When we move on to the next
> great thing, we will again go through an agonizing process of
> rewriting and conversion.  The stuff that gets rewritten first will be
> the stuff that's just wrappers around C, C++, and FORTRAN.  Wouldn't
> it be great if that's nearly everything?  If you look at SciPy now,
> the majority of it is wrapped compiled libraries that implement
> practically all of the basic numerical routines.  Imagine where we'd
> be if we had to *write* that stuff in Python, rather than wrap it.
> Imagine how hard the next switch in interactive languages would be if
> we did that.
>
> There is also a strong argument against rewriting code that already
> works.  We have a lot of work to do, so if you have working compiled
> code, please just wrap it.  If you are writing new code, it might make
> sense for large, monolithic algorithms to be written in C or C++ and
> to be wrapped for Python.  Smaller routines (the majority, by number
> at least) are probably best done in Python directly.  These won't be
> hard to redo in the next language.  In any case, if you don't want to
> learn distutils and won't be providing distutils-building code, please
> stick to Python, or hook up with a distutils hacker.
>
On the other hand, I'd argue that things that are currently written in 
IDL
are probably best mapped to python given that the effort to write them
in C or another compiled language is going to be substantially larger.
It's not clear to me that writing a library in C that could be done just
as well (i.e., run efficiently) in Python should be written in C. Sure, 
it
can be used with other languages, including future scripting languages, 
but
if it entails 5 times a much work, one always has to ask if the 
tradeoff is
worth it. Often it won't be (for much the same reason it wasn't for 
IDL).

I'll argue that Python will be around considerably longer than 15 years
(it is almost that old now and Perl certainly is). But it won't last 
forever.
It may be that there will be good tools for translating Python into new
languages in the future though.

> DOCS
>
> Documentation is a strong suit of IDL's and we need to be as good.  I
> am not aware of a standard doc header for Python.  If there is one, we
> should use it.  If not, I suggest we essentially copy IDL's, with some
> modifications to rationalize it and to make it work for Python.  We
> also need a code that will extract the doc pages from a package and
> will collect the docs, turn them into HTML and PDF, and put them in a
> searchable database.  This code will be central to SciPy as a whole so
> upstream coordination with the soon-to-be-born doc effort would be
> great.
>
Epydoc looked promising. It was presented at pycon (link to paper:
http://www.python.org/pycon/dc2004/papers/37/epydoc.pdf ). We haven't
used it but it looked good as the first thing to try.

While we are speaking of such things, I'd suggest use of doctest as
the default testing framework. It's true that some kinds of tests are
better handled using unittest, but for simplicity and transparency, it's
hard to beat doctest.

> The source language of docs is another issue.  It has to be open and
> functional on all platforms.  It has to handle simple markup and
> inline figures.  It has to produce PDF and HTML.  MSWord is out.
> Research Structured Text or LaTeX are my votes.  Some like LyX and
> TeXmacs.  We'll see what the community wants.
>
Some have suggested openoffice as a portable and free alternative.
Did you mean ReStructured Text? That's generally good, but has problems
handling equations well, and for this application that seems like a
killer. LaTeX is clumsy, but does equations well.

> SYNTACTIC CLOSENESS
>
> Syntactic closeness to the IDL library would be nice but isn't crucial
> from my perspective.  Where things are reasonable, keep them.  Where
> they are not (what does "sxpar" mean, and to whom?), make them so.  We
> could provide a compatibility wrapper for fans of sxpar.
>
I'd agree (and a dual interface probably makes sense for some things).
As for the FITS examples, I'd argue we would be doing a disservice by
trying to retain compatibility. The IDL interface is generally harder to
use than PyFITS now I'd say.

> CODE CONVERTER
>
> One idea that has been tossed around is a code converter.  There are
> two approaches to this, which I call the 80% solution and the 100%
> solution.  The 80% solution converts all the procedural code on a
> line-by-line basis.  It would do the gruntwork, and would ensure that
> array access, etc., is converted correctly (otherwise subtle bugs will
> creep in).  The developer would still have some gruntwork to do to get
> it to actually work, but not a lot, particularly if the code was
> simple and didn't depend on a lot of IDL library routines.  The 100%
> solution would be 100 times harder to write.  It would convert all the
> code, including the OO parts, and would also implement the IDL
> library, generally in terms of a set of wrapper routines that called
> SciPy routines.  It would actually be a re-implementation of IDL.  I
> claim this is unnecessary and not even very desirable.  We want people
> to switch to Python and to contribute new code to the community.  I
> have no personal interest in providing them a free IDL.
>
> Anyway, if we had a converter, this entire project, as well as the
> project of getting astronomers to switch, would be *much* easier!  It
> would be safest from a legal point of view for the converter authors
> NEVER to have run IDL nor to have read RSI's manuals.  The IDL license
> prohibits reverse-engineering, and most of us have agreed to it.  I
> don't believe it's reverse engineering to write a code converter from
> a language that you know into another language that you know that has
> the same capability.  However, I'm not a lawyer, and I don't have the
> means to fight that battle in court.  IDL is simple enough that
> someone who doesn't know it can get a commercial book on IDL, such as
> Gumley's Practical IDL Programming, that provides all the information
> that's needed.  Any of us experienced IDL users can then publish a FAQ
> on the web that answers any questions the programmers might have,
> without our actually writing or looking at any code.  This should keep
> the whole effort legally above reproach.
>
I guess I would like some experience trying the brute force approach 
before
tackling code conversion for this (to get a little experience anyway).
Besides, I've used IDL previously (does the fact that a company or 
university
has purchased a License mean that all that work for it, whether or not 
they
have used it, have effectively agreed to the license?)

This certainly would be useful for persuading IDL users so that they
could more easily convert their code.

> COORDINATION
>
> Finally, to address coordination issues, I think community testing and
> review is key.  Let's ask people to post their development plans in
> advance for community design review and to update a public-read CVS
> often.  Let's have public review and testing of all new code before
> accepting it into the library.  That way we'll address
> interoperability issues and catch bugs early.  If we're a successful
> community project, Paul's greatest contribution will be the
> coordination of the effort and the moderation of the review process,
> rather than the bits of code he produces himself.
>
> As always, I welcome your comments.

Paul mentioned that you or someone suggested that the mailing list move 
to a
scipy supported one (same name? scipy-astro?). I'd say we should do
that as soon as possible to get the features of mailman (archives in
particular).

Perry

_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From oliphant at ee.byu.edu  Mon Sep 13 20:20:28 2004
From: oliphant at ee.byu.edu (Travis E. Oliphant)
Date: Mon, 13 Sep 2004 18:20:28 -0600
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu>
References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu>
Message-ID: <4146394C.5090407@ee.byu.edu>

<x-flowed>Since Tom invited me into the discussion, I will comment, despite the 
fact that I am not an astronomer.

> Hi folks-
> 
> Sorry to have missed all of the interesting discussion on this; I was
> at the HEAD meeting (on a panel on astrostatistics software, among
> other things---lots of folks are interested in Python, by the way).
> 
> Joe's summary of the main issues was very good.  I agree on many
> points; here are some possible disagreements.
> 
> 
>>UNDERLYING PACKAGES
>>
>>1. Numarray for all new code, and to teach to new users.

The reality is that Numarray is not ready as a drop-in replacement for 
Numeric but people keep introducing it to new users as if it were ready. 
   This is a real problem in the community right now.

These issues have been discussed before and are on the scipy.org wiki 
that Eric has started.  Basically the issues preventing a drop-in 
replacement are 1) small arrays are too slow, 2) no backward-compatible 
ufunc C-API --- nothing like the compatibility API for arrays exists for 
ufuncs.  3) migrating legacy code is way too time consuming and effort 
filled for almost no gain.

While not optimal the current best solution is to make Numarray more 
aware of Numeric arrays and vice versa, so that unnecessary memory 
copying is not happening.

> 
> 
>>2. Matplotlib for all plotting.  Try it.  You'll like it!
> 

Chaco is nearly there.  In some ways I feel bad that John Hunter didn't 
use his energy to making easy plots using Chaco/ Tk instead of 
developing yet another infrastructure.  It is true that matplotlib got 
to a functional state more quickly, but chaco has a much more sturdy 
foundation for development, I think.

The Enable library that Enthought has put together allows amazing 
possibilities and improved portability.  Their motives are completely 
pure, though their time is limited.   It would be nice if Enthought 
published the location of their chaco SVN server so that users could see 
the current code and not the legacy stuff in SciPy.  I think the reason 
they haven't is more of a logistic issue (not being ready to support 
inquiries) and perhaps separating it from other code bases.  But, if 
Eric is reading this, I would suggest to him, that not publishing and 
promoting the SVN site creates more problems and discourages potential 
users in just the way Tom is expressing.

> 
> Do either of these packages (matplotlib, Chaco) allow one to interact
> with plots in a way that lets Python easily know where on a plot the
> user is clicking?  I.e., is there the basis for making "interactive"
> plots, not merely in the sense of panning and zooming, but in actually
> manipulating points on the plots as controls (maybe "dynamic" would be
> a better adjective)?  I have in mind something like SysQuake
> (http://www.calerga.com/).
> 

I believe chaco/enable makes this kind of thing very possible.

> g77 is free and trivial to install on OS X.  In my experience
> Fortran on linux and the Mac isn't problematical.  Is this a
> Windows problem?  Are there really many astronomers using Windows for
> scientific calculation?

g77 also works under Windows. This should not be a problem.  SciPy is 
compiled under Windows using g77 constantly.

> 
>>DOCS
>>
>>Documentation is a strong suit of IDL's and we need to be as good.  I
>>am not aware of a standard doc header for Python.  
> 
> 
> What little there is that is standard is outlined by Guido:
> 
> http://www.python.org/doc/essays/styleguide.html
> 
> 
>>The source language of docs is another issue.  It has to be open and
>>functional on all platforms.  It has to handle simple markup and
>>inline figures.  It has to produce PDF and HTML.  MSWord is out.
>>Research Structured Text or LaTeX are my votes.  Some like LyX and
>>TeXmacs.  We'll see what the community wants.
> 
> 
> I'd love to see a format that allows including equations somehow, or
> some "web"-like framework (referring to Knuth's WEB, not the world-wide
> one) that allows one to jointly maintain fairly extensive technical
> documentation with the code.

I say, use LaTeX format in equations in doc strings if necessary and use 
LyX as a format for documentation.   The file format is structured text 
that with the LyX editor becomes very easy to manipulate.


> 
> I believe Bill Press and Saul Teukolsky wrote software (or had it
> written) for maintaining Numerical Recipes code within LaTeX for
> their book.  Perhaps we can find out about what they did to see
> if it might be adaptable to our purposes.
> 
> 
>>COORDINATION
> 
> 
> Regarding coordination with SciPy, it may be worth pointing out
> that the astrostat project I have funded by AISR supports Travis 
> Oliphant for a month a year to support the project.  He's offered
> to host our Inference package at SciPy.org.  Perhaps I can persuade
> him to make that just a part of a larger set of packages, and get
> his help behind hosting our effort at SciPy.org (in this last year of 
> funding!).
> 

I hope that the AstroPy coordinates with SciPy.  A lot of build issues 
have been worked out in SciPy.  SciPy has been explicitly designed to 
work with subpackages.  I see no reason that much of what astropy wants 
to have available can't be a sub-package of SciPy.  If there are 
reasons, then we need to fix them on the SciPy end, or the stated 
purpose of SciPy is not being fulfilled.

Now, I can also see the blossoming of very special-purpose packages, but 
the line between special-purpose and general is changing all the time. 
However, if there are general-purpose algorithms that apply beyond 
astronomy, they really should be in SciPy and not in a different 
location.  SciPy is supposed to be a community effort.  There is room 
for many more developers in the SciPy world.

-Travis Oliphant



_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From oliphant at ee.byu.edu  Mon Sep 13 20:27:29 2004
From: oliphant at ee.byu.edu (Travis E. Oliphant)
Date: Mon, 13 Sep 2004 18:27:29 -0600
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu>
References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu>
Message-ID: <41463AF1.1090902@ee.byu.edu>

<x-flowed>
> 
> Regarding coordination with SciPy, it may be worth pointing out
> that the astrostat project I have funded by AISR supports Travis 
> Oliphant for a month a year to support the project.  He's offered
> to host our Inference package at SciPy.org.  Perhaps I can persuade
> him to make that just a part of a larger set of packages, and get
> his help behind hosting our effort at SciPy.org (in this last year of 
> funding!).

You can easily persuade me to help making anything that is deemed 
acceptable a part of the scipy package.  We want SciPy to be general 
purpose and easily available.

I am anxious to hear about what I can do to help get your inference 
routines into SciPy.

-Travis

_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From perry at stsci.edu  Mon Sep 13 20:31:41 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Mon, 13 Sep 2004 20:31:41 -0400
Subject: [AstroPy] Python version of IDL Astron library
In-Reply-To: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu>
References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu>
Message-ID: <72E7CCD1-05E5-11D9-B225-000393989D66@stsci.edu>

<x-flowed>
On Sep 13, 2004, at 6:04 PM, Tom Loredo wrote:

>
> Hi folks-
>
> Sorry to have missed all of the interesting discussion on this; I was
> at the HEAD meeting (on a panel on astrostatistics software, among
> other things---lots of folks are interested in Python, by the way).
>
> Joe's summary of the main issues was very good.  I agree on many
> points; here are some possible disagreements.
>
>> UNDERLYING PACKAGES
>>
>> 1. Numarray for all new code, and to teach to new users.
>
> There was recently discussion of Numeric vs. numarray on the SciPy
> list, and my impression from it was that SciPy will *not* be switching
> to numarray anytime soon (i.e., within say a year) due to issues of
> small array performance and some different conventions.  Is this an
> accurate impression?  I think it is a big pain in the neck that all the
> useful computing is done with a great package (SciPy) that uses
> Numeric, but that astronomical i/o is with a great package (pyfits)
> that uses numarray .  I think anything that can be done to make Numeric
> a package of the past would be good; having two packages for array
> calculations is certainly an obstacle for new users.
>
I addressed this at scipy. The point I made was that it was clear that 
there
was going to have to be coexistance for at least a while. What we would 
like
to do is make it possible for 3rd party packages to support both 
Numeric and
numarray so the problem you mention goes away.

>> 2. Matplotlib for all plotting.  Try it.  You'll like it!
>
> I'm just getting into it, and though I like it overall, I think
> its quality and completeness have been exaggerated from a Mac OS X
> user perspective.  I bounce back and forth every day between linux
> and OS X, and it definitely has some warts on OS X.  But it is
> probably "inheriting" these from Tk and/or wx.  If there is ever
> an Aqua version of gtk, that might solve things, but my searching
> along those lines has not been encouraging.
>
It would be nice to know what they are (at least from the Tk 
perspective,
we support that backend).

> The Chaco situation has me feeling a little miffed.  If I recall
> correctly, it was built with NASA funding.  But it was incomplete at
> the end of the funding period.  Since then Enthought has gone on
> improving it, and it sounds like neat things have been done, but it has
> either been inaccessible or not easily accessible.  It gives the
> impression that NASA funded a small business startup that has left
> little of lasting value to astronomers in return for the funding (in so
> far as Chaco by itself is concerned).  This reflects rather poorly on
> Enthought and provides a negative example to NASA of what has come of
> funding Python development.
>
I think this summary gives a misleading impression of what happened.
NASA didn't directly decide to fund Enthought to do this, STScI did
as part of its work on Python infrastructure (and it was my idea to
do so, so it's really my responsibility, not NASA's). While I certainly
regret that it didn't result in a successful graphics package for our
purposes (though it did result in a useful spinoff, Traits) and it was
not an insignificant amount of money, it was an order of magnitude less
than is spent on a typical NASA AISR grant (many of which also don't
result in much useful in the way of results; I hope yours is different 
;-)
And the amount of money Enthought got was never critical to their
success (they had already much more income from other sources).
If you are going to be miffed, I'm the proper target.

Enthought is still using chaco internally. I'd say that it really is
not in any state useful for most projects, mainly because it is changing
so rapidly and there is little documentation to describe the current 
state
(and even if it did exist, it would likely become rapidly obsolete).
Furthermore, the current development is only being done on MS Windows
using WxPython. Using it with other GUI's or platforms will likely take
substantial work. I'm guessing it's still at least 6 months to a year 
away
from being useful for others. We gave up on it because it was becoming 
clear
that it was too ambitious and complex for our most immediate needs.

> Do either of these packages (matplotlib, Chaco) allow one to interact
> with plots in a way that lets Python easily know where on a plot the

Yes. In fact this is what makes Chaco more complex. It has a lot of 
built
in machinery to handle interaction including selecting and manipulating
graphics elements (you could drag a plotted curve to a new location if 
you
set it up properly. Matplotlib does have means to obtain a clicked x,y
position, but the user has to handle what is done as a result of that 
click.

> user is clicking?  I.e., is there the basis for making "interactive"
> plots, not merely in the sense of panning and zooming, but in actually
> manipulating points on the plots as controls (maybe "dynamic" would be
> a better adjective)?  I have in mind something like SysQuake
> (http://www.calerga.com/).
>
Sounds like you are talking about chaco-like features. But you'll have
to wait. And it will be more complex to use (one of the reasons we 
discounted
the value of such capabilities is that after all the interactive 
changes one
makes, there is no easy means of applying the same changes to your next 
plot
whereas with a plot script there is an easy means to do so. It isn't as 
sexy,
but I'll argue it is generally more useful for most astronomers. Adding 
such
capabilities to an interactive system isn't impossible, but it isn't 
currently
in the Chaco plan, and such things are generally tricky to implement 
and less
than obvious to use and apply. But I may be misreading what you are 
asking
for. I'm hoping that matplotlib doesn't try to do too much in this 
area, at
least for a while.

>> PYTHON/C/C++/FORTRAN
>>
>> I was
>> told at the conference that Python, C, and C++ are all easy to get
>> working, but that FORTRAN is problematic.
>
> g77 is free and trivial to install on OS X.  In my experience
> Fortran on linux and the Mac isn't problematical.  Is this a
> Windows problem?  Are there really many astronomers using Windows for
> scientific calculation?
>
Engineering, certainly. Many of scipy downloads are for windows. It may 
not
be for astronomy, but scipy has to worry about a bigger community. It's
not just windows that is the problem. One has to devise a distribution 
system
that works easily for all platforms. Just because you have few problems 
on
Mac OS building certain applications doesn't mean that there aren't 
glitches
in building the same applications on other platforms in a way that 
doesn't
require twiddling with make files and such. With fortran, the 
interlanguage
conventions change from platform to platform, then there are the linking
issues. Things are much more standard with C.

>> There are strong benefits to writing in a compiled language, the main
>> one being portability to later interactive languages.
>
> I think folks should follow the typical Python development path:
> Develop in Python, with just the time-intensive parts rewritten in
> C, etc., as necessary.  I think the emphasis should be on a good
> interface and getting the algorithm available quickly.  Developing
> as much as possible in Python I think will make things available
> more quickly.
>
I'll agree with you here (so long as existing code in C or Fortran
can't do the job; it makes little sense to rewrite well tested libraries
so long as they are suitably portable).

> In the last few years Python very quickly got to the "75-80% level"
> as a scientific computing tool.  But the pace has really dragged
> (to my perception) in the last year or so for closing the gap
> (and I'm partly to blame in regard to a stats package I'm writing).
> That last 20% or so is crucial; though one encounters the missing
> pieces only a fraction of the time, a *lot* of time gets wasted
> working around them, and it discourages use.  So I think this
> community project Joe et al. has started is a great idea, but let's
> focus on getting stuff done quickly and well, and worry about
> performance (in the sense of writing lots in C/C++/Fortran) later.
>
Well, for us the drag was with plotting. Without a plotting, it wasn't
going to go anywhere. I think things are looking much brighter now.

Perry

_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>

From perry at stsci.edu  Mon Sep 13 20:43:20 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Mon, 13 Sep 2004 20:43:20 -0400
Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library
In-Reply-To: <4146394C.5090407@ee.byu.edu>
References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu>
Message-ID: <131B68FF-05E7-11D9-B225-000393989D66@stsci.edu>

<x-flowed>
On Sep 13, 2004, at 8:20 PM, Travis E. Oliphant wrote:
>
> The reality is that Numarray is not ready as a drop-in replacement for 
> Numeric but people keep introducing it to new users as if it were 
> ready.   This is a real problem in the community right now.
>
> These issues have been discussed before and are on the scipy.org wiki 
> that Eric has started.  Basically the issues preventing a drop-in 
> replacement are 1) small arrays are too slow, 2) no 
> backward-compatible ufunc C-API --- nothing like the compatibility API 
> for arrays exists for ufuncs.  3) migrating legacy code is way too 
> time consuming and effort filled for almost no gain.
>
> While not optimal the current best solution is to make Numarray more 
> aware of Numeric arrays and vice versa, so that unnecessary memory 
> copying is not happening.
>
Well, we want to go beyond that to the point that scipy and other 
Numeric packages can support numarray and Numeric simultaneously. We 
intend to do most of the work to make it so with scipy. f2py already 
has been adapted to work with numarray, and we would
also take a look at the ufunc api issue to deal with that. I think we 
should start
on this fairly soon (certainly within a month or so). This is pretty 
much our
highest priority now that numarray 1.1 is out.

Perry

_________________________________________________
AstroPy mailing list     -      astropy at stsci.edu
http://www.astro.washington.edu/owen/AstroPy.html
</x-flowed>