[Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification

[Gael Varoquaux]

On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote:
> Developers shouldn't use system packages, it just doesn't 
> make any sense to have that intermediation. 

I don't agree. I work as a developer/scientist in a lab. 

I develop algorithms that sometimes will never be used on another box
than mine (that's when they are bad). We happen to be standardised on a
Linux distro that is a bit old, and with little packages. I can tell you
it really annoys me to have to install manually a package each time I want
to use it. The pure-Python ones, with no other dependencies are not hard,
easy-install does the trick, but these actually represent a small
fraction of the important ones (wxpython, numpy, pytables, matplotlib,
elementtree, pyobjects, cython, ets, pyvtk). 

So I lose a lot of time installing packages on my box. But the real
nightmare begins when I want to share the algorithms and applications I
developed. Across our institute we have NFS-shared drives on which we
have to compile all the dependencies, for all three platforms we support.
And that's only for distributing software in the institute, when we want
to distribute outside we end up shipping big binary blobs with
everything, from our own version of Python, to numpy, wxPython, ATLAS
(that's bad). If only we could rely on package managers to provide us
with base packages (and API-stability of these packages to be sure they
work).

> Users don't use Python modules, they use applications.  Users only care
> that their applications work, that they can install applications
> without unnecessary conflicts, that the applications don't break based
> on unintentional environment changes (e.g., the value of PYTHONPATH).

That is true, as long as you ship "the world", ie everything you'll ever
need for your extensible application, including the c libraries that go
behind. This means putting a huge burden on your development team, and if
you think about out, this is exactly the work of a distribution. So you
are duplicating this effort.

I addition, one of the great things about Python, is that it is easy to
read and to code. Advanced users will want to write plugins, extend there
programs, and programs should be written for this: 

    "Hey, I have this cool application that does brain segmentation, and
    visualization of the tissues, and I have this cool other app that has an
    algorithm to identify tumorous tissues, I want to plug them together, and
    because the APIs are well-written, it should be trivial...

    Darn, the two apps have confined Pythons, and they can't import from
    each-others. No big deal, I'll unbundle the whole thing...

    Darn, one has been build with UCS-2, the other one with UCS-4, and
    there C libraries are incompatible...

    So now the easy task has become a long effort of building the two
    apps, and I can't easily share my work."

I can hardly believe this happens only in the scientific computing world.
I have fought (and I still fighting) to deploy a Django-based
application. It didn't get along with the server's apache version.

Gaël