[AstroPy] Deployment and packaging
embray at stsci.edu
Wed Jun 15 11:00:10 EDT 2011
I'm just getting caught up on this discussion, though Andrew's message
already pretty much encapsulates my thoughts on this thread so far. I
do have some additional notes inlined below. But first I'd just like to
add that packaging/deployment is of particular interest to me, and it's
something I have a bit of experience in.
I've pulled together some large-ish projects from disjoint pieces and
made them work quite nicely on different platforms. This includes pure
Python projects, as well as projects that include other big system
dependencies including libraries, web servers, etc. (Actually, to be
specific, one such project involved my own Ubuntu fork based on a
specific set of packages, so I'm quite familiar with
Ubuntu/Debian/apt/etc.) Unfortunately Macs are a weak point for me.
I also have pretty deep knowledge of Python's packaging mechanisms,
including the new 'packaging' module (formerly known as distutils2). So
I think I can contribute there.
I agree with Andrew that an AstroPy effort should focus on being based
on one or more "pure" Python packages. By "pure" I don't mean to
exclude C extensions of course; and pre-built binary packages should be
maintained for as many platforms and Python versions as possible, to
make things easier for users who don't have development tools on their
systems. This is, of course, especially important for Windows. But
it's important for all other platforms too--just the other day I
couldn't get matplotlib to build due to a dependency on libpng, for
example. Numpy can also be tricky. But fortunately Numpy and
matplotlib are already available as binary packages for most platforms.
As for packages that have external, non-Python dependencies (PyRAF for
example), I do not by any means think they should be excluded
altogether. But they should be able to work on a multitude of system
configurations, and should be excluded from any installation where their
dependencies are not bet. At any rate, where IRAF is concerned the goal
should be to gradually replace its pieces with Python modules. PyRAF is
invaluable in the meantime, but in a "perfect" world it wouldn't have to
exist at all :)
Now if there was some desire to work on a Python + numpy + IRAF +
SExtractor + ... all encompassing distribution (as I know there is) that
*could* also be organized under the AstroPy banner. But I think it
should consist of one or more separate sub-projects, loosely coupled so
that the best practices can be followed for different target platforms
(i.e. .deb distributions, rpm distributions). The pure Python part may
have certain dependencies, but it should not be targeted for any one
specific system layout.
On 06/15/2011 12:01 AM, Andrew Williams wrote:
> 'All in one' setups like this make life easier for people setting up new
> computers for training workshops or for new postgrads, but they make
> life a lot harder for people who already have the tools they need set
> up, and don't want to break them by installing some monolithic
> collection of loosely coupled packages. It's that latter set of expert
> users who are the ones you most want to get involved using and
> contributing to the collection...
Exactly. We use such 'all in one' setups here, but I never use them for
development because they're installed as root! I have my own copies of
them to hack on, but most of the time it's inconvenient.
> For the same reason, I'm also very much against the idea of turning this
> into another complete python distribution. Apart from Windows users,
> most people already have Python installed with the operating system.
> Installing a second copy (and getting all the other libraries you want
> to use installed under _both_ Python versions) is a real pain.
> There's nothing to stop people re-packaging 'astropy', or whatever the
> Python-only collection gets called, along with SExtractor, IRAF, numpy,
> and everything else, if there is demand for an 'all in one' collection.
On Ubuntu/Debian this could be a meta-package that installs all
requirements at the system level. I definitely think this should be
considered, but it should not be required either.
> I'm more used to catering for the other end of the user base - the
> people who, at worst, _only_ have access to software that's part of the
> default OS install, or at best, packages maintained in the ubuntu/fedora
> software systems. For example, I wrote a pyfits-like FITS file library
> in pure Python for people who couldn't get pyfits installed on the
> machine they had to use...
> Another point - if this 'astropy' collection is maintained as a
> ubuntu/fedora package (plus the OSX equivalent, I forget the name), it
> will expand the available user base hugely - an awful lot of people use
> computers they don't have root access for. It's really, really hard to
> convince most sysadmins to download some random piece of software off
> the internet and install it on a machine unless it's maintained in the
> software packaging system for that OS.
> The more things you add, like SExtractor and numpy (which are both
> already distinct ubuntu packages), the harder it will be to package it
> properly that way.
Which is why I think any software collections should be designed for
specific platforms in mind. If we're talking Ubuntu, most of the
required packages already exist. So the only work would be making
packages for those requirements that don't already exist, and managing
the dependencies. On a different platform, say, Windows, it might be
necessary to include more software in the distribution. But I think
it's futile to make one exactly common base of installed software for
all possible platforms, and to me this kind of defeats the purpose of
basing our efforts around Python (where most of the platform
interoperability issues should be managed at the Python level).
More information about the AstroPy