[AstroPy] Deployment and packaging
luigi at lambrate.inaf.it
Thu Jun 16 05:55:21 EDT 2011
I agree with Matt, Robitaille and Prasanth.
I see four different points raised during this huge thread:
1. Build up one (and just one) package (or set of non duplicated
packages) containing the most complete and interoperable Python software
2. With respect to 1, how to handle third party dependencies, in
particular if it is not Python stuff, e.g. legacy software like
SExtractor, Galfit, IRAF, etc.;
3. For the astronomical Python package(s), which packaging adopt;
4. Whether or not make an effort to produce a reference Python & C.
distribution (maybe binary) like EPD etc.
With respect to point 1. I certainly think that it would be great, but I
would prefer having several small and well focused interoperable
packages than a single huge package all comprehensive. Provided that all
packages are developed by the same community (but each single package by
a smaller team), an packaged and developed following the same
"astronomical framework", then, in my opinion, with this approach the
target would be centred in a more flexible and effective way,
With respect to point 2, I think that for the time being we can assume
that the third party software is installed separately as a requirement,
and we just provide the Python bindings (by the way, I have already
produced a Python wrapper to SExtractor as a demo within FASE project,
but I assume that SExtractor is installed and in the PATH). In any case
I think that, once we have this Python framework well defined and
stable, then we can try to collaborate with the legacy software
developers in order to find a shared strategy for the software distribution.
Point 3... no doubts: distutils, pip and PyPI (or a separate
implementation of PyPI).
With respect to point 4. I bring to you my experience along several
years distributing Python software within the PANDORA initiative (R.I.P.).
We started producing software with mixed Python and C code. There were a
lot of dependencies with third party libraries and we decided to build
up a detailed documentation of all requirements. We didn't distribute
the binaries of our software, but just the source to be compiled.
Result: most of the end users were discouraged at the first sight of
such documentation and gave up; those balder that decided to give a try,
frequently had a lot of installation problems with the third party
software, and in those rare case where they didn't give up, they asked
us to solve their problems (sometimes leaving to us their PC!!!); The
remaining (very few) users that arrived to install our software, usually
had a lot of problems 'cause there wasn't a complete compatibility
between the libraries they installed and our software.
Then we decided to select the versions of the dependences and build up a
package installing all the third party software we needed (the add-ons,
which is more or less the same idea of EPD, Sage, etc.). The add-ons
solved the problem of incompatibility between our software and the third
party libraries, but two new problems raised:
a) since we distributed the add-ons as sources which were automatically
locally compiled by a script we set up, there were still a lot of
compiling problems of the add-ons, and the users still asked us a lot of
support, up to giving us their computers (again)!
b) after some years the add-ons were no more compatible with the most
recent platforms and then we had to bug fix the third party software in
order to solve their incompatibility (!!!) or upgrade it, with the
unpleasant consequence of running back all the following
incompatibilities in our software.
Presently my opinion is that there cannot be a final solution, but, at
least to simplify the end-user life, it would be nice having an open and
free reference Python & C. binary distribution (one for platform) a la
EPD/CASA/etc., and developing the astro-python-packages taking into
account this reference distribution, and leaving as a recommendation to
support other contexts. It is clear that producing and maintaining such
a reference distribution is a job by itself, and it requires a specific
team of developers and maintainers willing to do this dirty job.
Moreover, it is clear that this reference distribution must be upgraded
periodically in order to support the new OSs, and this will implicate a
periodical upgrade of the astro-python-packages (where needed).
Sorry for this long mail.
Il 06/16/11 10:24, Prasanth ha scritto:
> I strongly agree with Matt Davis and Thomas Robitaille here.
> Many of the packages like asciidata, vo.table, ATpy, APLpy, stsci_python
> etc., are easier to install, if we have a distribution that satisfies
> the basic requirements.
> Right now I think the easiest way for a non expert person with no root
> access to quickly get started with using Python, is by first downloading
> EPD/Python(x,y) and then the astronomy package of interest. If there is
> an EPD like free package with most of the core dependencies, not
> specifically astronomy related, then it might be easier for a wider
> range of non expert users to get started easily.
> As Matt points out, this could possibly help other user communities as well.
> On Thu, Jun 16, 2011 at 7:00 AM, Matt Davis <mrdavis at stsci.edu
> <mailto:mrdavis at stsci.edu>> wrote:
> I can see that there seems to be substantial interest in producing
> an easy Python installation usable by astronomers. Something like
> EPD but free, or maybe like Python(x,y) for Windows. In fact,
> there's no reason something like that needs to be specific to
> astronomy. All the tools you list would be great for someone in
> geoscience, medicine, engineering, or many other fields. I think it
> would be fantastic if we could fill this need, but I don't think
> it's a need specific to astronomy and for that reason it should be a
> separate project from AstroPy (with substantial staff overlap, to be
> sure). I have worked in Earth sciences and they face the same
> challenges to Python adoption that astronomers do. And there is much
> to be gained by helping other fields join the Python fold,
> especially when it comes to things like handling large data sets,
> statistics, pipelines, parallel processing, imaging, etc.
> AstroPy, I think, should be a project separate from being packaged
> with anything else. It can have dependencies, of course, and some
> modules may have more dependencies than others, but the fact would
> be that someone could install AstroPy tools into a Python
> environment they made themselves, into stsci_python, into EPD, into
> some Python distribution we (as a community) make, or into any other
> Python environment that supports our distribution method.
> - Matt
More information about the AstroPy