[AstroPy] Deployment and packaging

Wed Jun 15 13:39:07 EDT 2011

Argh, can't keep up with this thread :-).

I agree that packaging/distributing Python dependencies is a separate
problem from collaborating on a common library of modules (except that
the two things could be associated in future and maybe even share a
name like AstroPy -- but only if we can agree on a solution for
packaging, which we're nowhere near doing yet).

I think the community includes some pretty strongly opposing views
on how to tackle packaging and I'm sure there will be room for more
than one answer. For my part, I intend to remain focused on getting
our Sage-like/based distribution out of the door. We (Gemini and
STScI) only barely have the resources to complete that project and
any change of direction would certainly derail it. Maybe the community
will accept our solution and find it helpful and maybe it won't (more
likely the preference will be split), but at least we will learn
something concrete and I have no doubt that it will be useful
internally and for some users either way. It sounds like putting stuff
on PyPI will also be an important / easy to do part of the solution.

Regarding EPD, we did look briefly at using that for our effort but
had a few significant concerns. Licencing is an obvious one. Some of
us (eg. observatories) don't even qualify for the academic licence.
That's not just a question of cost, but also practicality (eg. can't
give it to someone and say "try this"). The distribution comes pre-
assembled which makes it harder to modify for our own needs. There
were also a couple of issues like it requiring special privileges to
install on a Mac (any user will be able to install what we're
working on).

Cheers,

James.

On 15/06/11 08:53, Thomas Robitaille wrote:
> Just to add a small comment, I think this discussion should not be confused with the astrolib/astropy discussion (which is why Tom A. started this in a different thread). I think it's obvious that any new package we create should be available as a 'standard' python package that can be manually installed into any Python distribution. The issue of packaging/bundling is separate, because it applies regardless of whether or not we manage to achieve this common package.
>
> Cheers,
> Tom
>
>
> On Wednesday, June 15, 2011 at 2:29 AM, Erik Tollerud wrote:
>
>> I also agree that another python distribution is not the way to go, at
>> least for a time, and that we should be aiming for "expert" users. In
>> fact, I think most people working in astronomy are in some weird
>> middle ground between users and developers, so they're primarily in
>> that category.
>>
>> I also really don't think a *directly* EPD-based solution is a good
>> thing to do. It should definitely be kept compatible with EPD, but if
>> someone wants to by hand install and compile only the packages that
>> are needed for the astronomy package, that should also be an option.
>> That's the whole point of packaging, after all, and it would be
>> foolish to try to do it ourselves when all the tools that have been
>> mentioned already are available and work just fine (well,
>> sometimes...).
>>
>>
>> Larger scale, I see two different problems that are being discussed here:
>>
>> 1) What's the best way to package up a new python library with it's
>> python dependencies (possibly with standalone C-extensions)?
>> To me the answer is clear: PyPI with standard python tools. If
>> desired, it's easy to plug into those tools, bootstrapping the library
>> to install dependencies as needed.
>>
>> 2) How can we provide everything anyone using a python library might
>> want, even if it's not in python?
>> This is a *much* harder problem, and while it would be great to solve
>> this problem, for now I think the resources should be devoted mostly
>> to solving #1, given that there are other package management tools
>> exist. It's fairly straightforward to make the library link into
>> whatever extensions are present, and present helpful messages as to
>> where to get them if they are not.
>>
>>
>> PyPI and it's associated tools pretty much work, and it's quite easy
>> to write tools to make use of it to download and install arbitrary
>> packages. I've done exactly this as part of astropysics, and it has
>> worked fine on Mac and two different Linux distros (although perhaps
>> there are Windows complications?)... If the astro library goes in a
>> direction of a "scikits" approach, having everything on PyPI will make
>> it much easier to make tools that auto-install whatever kits are
>> desired (something scipy/scikits seems to have trouble with, because
>> the kit concept was tacked on later). Furthermore, a lot of the
>> packaging people who work on the big packaging projects use PyPI as an
>> easy way to monitor and update for new releases, so it's in some sense
>> using PyPI is two birds with one stone.
>>
>> PyPI also handles binaries, although admittedly not as cleanly. But
>> aren't most of the "expert" users going to have compilers? In that
>> case tools like pip easily install anything with a C-extension from
>> source. If the concern is about binary *dependencies* that aren't in
>> the python ecosystem at all, the dizzying array of possible platforms
>> and versions to support makes me think that it's a problem that we
>> don't have the people-power to solve.
>>
>> If someone wants the astronomy python library to call SExtractor, some
>> random C-code with a command line interface, or even some IDL routine,
>> it's actually quite easy to do that using inter-process communication
>> (subprocess.Popen and the like), and then it's left to the user to
>> install the outside requirement however it wants to be installed.
>> I've used this approach multiple times, and it has always worked fine
>> (even across different platforms if you do the python code right, as
>> long as the 2nd package works on said platform). If a direct link to
>> some C-extension is needed, that is probably something to be avoided
>> as much as possible (or rendered optional a la the way the GUI and
>> OpenGL pieces of Traits are optional).
>>
>>
>>
>>
>>
>>
>> On Wed, Jun 15, 2011 at 12:50 AM, Vicki Laidler<laidler at stsci.edu (mailto:laidler at stsci.edu)>  wrote:
>>> I too think that the goal of this effort should neither start with yet another python distribution, nor produce a mixed general-purpose astronomy library containing IRAF, SExtractor, or whathaveyou. Both those items produce significant barriers at both ends (the developer/deployment end, and the user/installation end).
>>>
>>> Let's at least start out by shooting for a scikit-like python library (C extensions allowed(?)) so that people who really just want a coherent python astronomy library (a la the Astron library in IDL) can have one, and people who want to make a more general astronomy library (like the unified release) can include the resulting product as part of it.
>>>
>>> Meanwhile, we can work on collecting some survey data from our potential users, segmented by the various user communities of interest -- I thought Andrew's point that it's the more expert users we want to attract to the project was a very good one; and if need be, revisit the question when we have actual measured data rather than anecdata on which to base the decision.
>>>
>>> Vicki
>>>
>>> ________________________________________
>>> From: astropy-bounces at scipy.org (mailto:astropy-bounces at scipy.org) [astropy-bounces at scipy.org (mailto:astropy-bounces at scipy.org)] on behalf of Andrew Williams [andrew at physics.uwa.edu.au (mailto:andrew at physics.uwa.edu.au)]
>>> Sent: Wednesday, June 15, 2011 12:01 AM
>>> To: astropy at scipy.org (mailto:astropy at scipy.org)
>>> Subject: Re: [AstroPy] Deployment and packaging
>>>
>>> On 15/06/2011 10:43 AM, Tommy Grav wrote:
>>>>
>>>> On Jun 14, 2011, at 10:11 PM, Perry Greenfield wrote:
>>>>> IRAF is only one example. SExtractor is another. And there are
>>>>> others. There isn't yet enough to replace these, and won't be for
>>>>> some years (if ever, it is probably silly to think everything we
>>>>> need will be in Python, or that there won't be some very useful
>>>>> things out there not written in Python).
>>>>
>>>> My opinion is that those tools belong in as separate packages. To me
>>>> they just don't
>>>
>>> I strongly agree - people who work with tools like these already will be
>>> _less_ likely to adopt an 'astropy' package if it also includes an extra
>>> copy of SExtractor, IRAF, and every other tool they might possibly want,
>>> generating conflicts with anything that's already installed.
>>>
>>> 'All in one' setups like this make life easier for people setting up new
>>> computers for training workshops or for new postgrads, but they make
>>> life a lot harder for people who already have the tools they need set
>>> up, and don't want to break them by installing some monolithic
>>> collection of loosely coupled packages. It's that latter set of expert
>>> users who are the ones you most want to get involved using and
>>> contributing to the collection...
>>>
>>> For the same reason, I'm also very much against the idea of turning this
>>> into another complete python distribution. Apart from Windows users,
>>> most people already have Python installed with the operating system.
>>> Installing a second copy (and getting all the other libraries you want
>>> to use installed under _both_ Python versions) is a real pain.
>>>
>>> There's nothing to stop people re-packaging 'astropy', or whatever the
>>> Python-only collection gets called, along with SExtractor, IRAF, numpy,
>>> and everything else, if there is demand for an 'all in one' collection.
>>>
>>> I'm more used to catering for the other end of the user base - the
>>> people who, at worst, _only_ have access to software that's part of the
>>> default OS install, or at best, packages maintained in the ubuntu/fedora
>>> software systems. For example, I wrote a pyfits-like FITS file library
>>> in pure Python for people who couldn't get pyfits installed on the
>>> machine they had to use...
>>>
>>> Another point - if this 'astropy' collection is maintained as a
>>> ubuntu/fedora package (plus the OSX equivalent, I forget the name), it
>>> will expand the available user base hugely - an awful lot of people use
>>> computers they don't have root access for. It's really, really hard to
>>> convince most sysadmins to download some random piece of software off
>>> the internet and install it on a machine unless it's maintained in the
>>> software packaging system for that OS.
>>>
>>> The more things you add, like SExtractor and numpy (which are both
>>> already distinct ubuntu packages), the harder it will be to package it
>>> properly that way.
>>>
>>> Andrew
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org (mailto:AstroPy at scipy.org)
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org (mailto:AstroPy at scipy.org)
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>>
>> --
>> Erik Tollerud
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org (mailto:AstroPy at scipy.org)
>> http://mail.scipy.org/mailman/listinfo/astropy
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy