[AstroPy] Deployment and packaging

Thomas Robitaille thomas.robitaille at gmail.com
Wed Jun 15 08:53:21 EDT 2011

Just to add a small comment, I think this discussion should not be confused with the astrolib/astropy discussion (which is why Tom A. started this in a different thread). I think it's obvious that any new package we create should be available as a 'standard' python package that can be manually installed into any Python distribution. The issue of packaging/bundling is separate, because it applies regardless of whether or not we manage to achieve this common package.


On Wednesday, June 15, 2011 at 2:29 AM, Erik Tollerud wrote:

> I also agree that another python distribution is not the way to go, at
> least for a time, and that we should be aiming for "expert" users. In
> fact, I think most people working in astronomy are in some weird
> middle ground between users and developers, so they're primarily in
> that category.
> I also really don't think a *directly* EPD-based solution is a good
> thing to do. It should definitely be kept compatible with EPD, but if
> someone wants to by hand install and compile only the packages that
> are needed for the astronomy package, that should also be an option.
> That's the whole point of packaging, after all, and it would be
> foolish to try to do it ourselves when all the tools that have been
> mentioned already are available and work just fine (well,
> sometimes...).
> Larger scale, I see two different problems that are being discussed here:
> 1) What's the best way to package up a new python library with it's
> python dependencies (possibly with standalone C-extensions)?
> To me the answer is clear: PyPI with standard python tools. If
> desired, it's easy to plug into those tools, bootstrapping the library
> to install dependencies as needed.
> 2) How can we provide everything anyone using a python library might
> want, even if it's not in python?
> This is a *much* harder problem, and while it would be great to solve
> this problem, for now I think the resources should be devoted mostly
> to solving #1, given that there are other package management tools
> exist. It's fairly straightforward to make the library link into
> whatever extensions are present, and present helpful messages as to
> where to get them if they are not.
> PyPI and it's associated tools pretty much work, and it's quite easy
> to write tools to make use of it to download and install arbitrary
> packages. I've done exactly this as part of astropysics, and it has
> worked fine on Mac and two different Linux distros (although perhaps
> there are Windows complications?)... If the astro library goes in a
> direction of a "scikits" approach, having everything on PyPI will make
> it much easier to make tools that auto-install whatever kits are
> desired (something scipy/scikits seems to have trouble with, because
> the kit concept was tacked on later). Furthermore, a lot of the
> packaging people who work on the big packaging projects use PyPI as an
> easy way to monitor and update for new releases, so it's in some sense
> using PyPI is two birds with one stone.
> PyPI also handles binaries, although admittedly not as cleanly. But
> aren't most of the "expert" users going to have compilers? In that
> case tools like pip easily install anything with a C-extension from
> source. If the concern is about binary *dependencies* that aren't in
> the python ecosystem at all, the dizzying array of possible platforms
> and versions to support makes me think that it's a problem that we
> don't have the people-power to solve.
> If someone wants the astronomy python library to call SExtractor, some
> random C-code with a command line interface, or even some IDL routine,
> it's actually quite easy to do that using inter-process communication
> (subprocess.Popen and the like), and then it's left to the user to
> install the outside requirement however it wants to be installed.
> I've used this approach multiple times, and it has always worked fine
> (even across different platforms if you do the python code right, as
> long as the 2nd package works on said platform). If a direct link to
> some C-extension is needed, that is probably something to be avoided
> as much as possible (or rendered optional a la the way the GUI and
> OpenGL pieces of Traits are optional).
> On Wed, Jun 15, 2011 at 12:50 AM, Vicki Laidler <laidler at stsci.edu (mailto:laidler at stsci.edu)> wrote:
> > I too think that the goal of this effort should neither start with yet another python distribution, nor produce a mixed general-purpose astronomy library containing IRAF, SExtractor, or whathaveyou. Both those items produce significant barriers at both ends (the developer/deployment end, and the user/installation end).
> > 
> > Let's at least start out by shooting for a scikit-like python library (C extensions allowed(?)) so that people who really just want a coherent python astronomy library (a la the Astron library in IDL) can have one, and people who want to make a more general astronomy library (like the unified release) can include the resulting product as part of it.
> > 
> > Meanwhile, we can work on collecting some survey data from our potential users, segmented by the various user communities of interest -- I thought Andrew's point that it's the more expert users we want to attract to the project was a very good one; and if need be, revisit the question when we have actual measured data rather than anecdata on which to base the decision.
> > 
> > Vicki
> > 
> > ________________________________________
> > From: astropy-bounces at scipy.org (mailto:astropy-bounces at scipy.org) [astropy-bounces at scipy.org (mailto:astropy-bounces at scipy.org)] on behalf of Andrew Williams [andrew at physics.uwa.edu.au (mailto:andrew at physics.uwa.edu.au)]
> > Sent: Wednesday, June 15, 2011 12:01 AM
> > To: astropy at scipy.org (mailto:astropy at scipy.org)
> > Subject: Re: [AstroPy] Deployment and packaging
> > 
> > On 15/06/2011 10:43 AM, Tommy Grav wrote:
> > > 
> > > On Jun 14, 2011, at 10:11 PM, Perry Greenfield wrote:
> > > > IRAF is only one example. SExtractor is another. And there are
> > > > others. There isn't yet enough to replace these, and won't be for
> > > > some years (if ever, it is probably silly to think everything we
> > > > need will be in Python, or that there won't be some very useful
> > > > things out there not written in Python).
> > > 
> > > My opinion is that those tools belong in as separate packages. To me
> > > they just don't
> > 
> > I strongly agree - people who work with tools like these already will be
> > _less_ likely to adopt an 'astropy' package if it also includes an extra
> > copy of SExtractor, IRAF, and every other tool they might possibly want,
> > generating conflicts with anything that's already installed.
> > 
> > 'All in one' setups like this make life easier for people setting up new
> > computers for training workshops or for new postgrads, but they make
> > life a lot harder for people who already have the tools they need set
> > up, and don't want to break them by installing some monolithic
> > collection of loosely coupled packages. It's that latter set of expert
> > users who are the ones you most want to get involved using and
> > contributing to the collection...
> > 
> > For the same reason, I'm also very much against the idea of turning this
> > into another complete python distribution. Apart from Windows users,
> > most people already have Python installed with the operating system.
> > Installing a second copy (and getting all the other libraries you want
> > to use installed under _both_ Python versions) is a real pain.
> > 
> > There's nothing to stop people re-packaging 'astropy', or whatever the
> > Python-only collection gets called, along with SExtractor, IRAF, numpy,
> > and everything else, if there is demand for an 'all in one' collection.
> > 
> > I'm more used to catering for the other end of the user base - the
> > people who, at worst, _only_ have access to software that's part of the
> > default OS install, or at best, packages maintained in the ubuntu/fedora
> > software systems. For example, I wrote a pyfits-like FITS file library
> > in pure Python for people who couldn't get pyfits installed on the
> > machine they had to use...
> > 
> > Another point - if this 'astropy' collection is maintained as a
> > ubuntu/fedora package (plus the OSX equivalent, I forget the name), it
> > will expand the available user base hugely - an awful lot of people use
> > computers they don't have root access for. It's really, really hard to
> > convince most sysadmins to download some random piece of software off
> > the internet and install it on a machine unless it's maintained in the
> > software packaging system for that OS.
> > 
> > The more things you add, like SExtractor and numpy (which are both
> > already distinct ubuntu packages), the harder it will be to package it
> > properly that way.
> > 
> > Andrew
> > 
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org (mailto:AstroPy at scipy.org)
> > http://mail.scipy.org/mailman/listinfo/astropy
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org (mailto:AstroPy at scipy.org)
> > http://mail.scipy.org/mailman/listinfo/astropy
> -- 
> Erik Tollerud
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org (mailto:AstroPy at scipy.org)
> http://mail.scipy.org/mailman/listinfo/astropy

More information about the AstroPy mailing list