[Catalog-sig] [Distutils] PyPI - Evolve our own or reuse existing package systems?

[Jeff Shell]

On 8/14/07, Bjørn Stabell <bjorn at exoweb.net> wrote:
> There seems to be two issues:
>
> 1) Should Python have its own package management system (with
> dependencies etc) in parallel with what's already on many platforms
> (at least Linux and OS X)?  Anyone that has worked with two parallel
> package management systems knows that dependencies are hellish.
>
>    * If you mix and match you often end up with two of everything.
>
>    * It'll be incomplete because you can't easily specify
> dependencies to non-Python packages.

On that second bullet, tools like 'buildout' seem better equipped for
handling those situations. Yesterday I saw a `buildout.cfg` for
building and testing `lxml` against fresh downloads and builds of
libxml2 and libxslt. It downloaded and built those two things locally
before getting, building, and installing the `lxml` egg locally.

Platform package management terrifies me. I work in Python far more
than I work in a particular operating system (even though our office
is pretty much Mac OS X and FreeBSD based). It's very easy for our
servers to get stuck at a particular FreeBSD version, while the ports
move on. Eventually they get so out of date that ports are pretty much
unusable.

> 2) If we agree Python should have a package management system, should
> we build or repurpose some other one?
>
> [snip]
>
>    * With a Python version we could relatively easily innovate beyond
> what traditional packaging systems do; ports and apt are pretty much
> stagnated.  I think RubyGems seems to have some cool features,
> features that probably wouldn't have happened if they were using
> ports or apt-get (but then they could piggyback on innovations in
> those tools, I guess).  If it works for them, why shouldn't it work
> for us?

I agree.

>    * It would have to be as portable as Python is; many packaging
> systems are by nature relatively platform-specific.

You could change "have to" to "gets to" there. :). This is a big plus
-- I know how easy_install and `gem` work as I use them far more
frequently on both my desktop and various servers than any
platform-specific packaging system.

>    * Packaging systems are useful for mega frameworks like Zope,
> TurboGears, and Django, and slightly less so for projects you roll on
> your own, to manage distribution and installation of plugins and
> addons.  Relying on platform-specific packaging systems for these may
> not work that well.  (But I could be wrong about that.)

Personally, I think packaging systems are worse here. But I just may
be a control freak... And I've had the luxury of Zope being a big self
contained package for quite some time. Now that it's breaking into
smaller pieces, it gets a bit more complex, but the combination of
`setuptools` and `buildout` seem to be doing their jobs admirably.
Relatively admirably.

Once you have Ruby and Gems, Ruby on Rails installs with just one line::

    gem install rails --include-dependencies

I think Pylons and/or Turbogears does just about the same..? It's been
a while since I looked at either of them. But that one line is a lot
easier to work with than::

    If running Debian, run ``apt-get ....``

    If running RedHat or RPM system, ....

    If running Mac OS X with MacPorts, run ...

    If running ... then ...

> That said, it might be possible to do some kind of hybrid, for PyPI
> to be a "meta package" repository that can easily feed into platform
> specific packaging systems.  And to perhaps also have a client-side
> "meta package manager" that will call upon the platform-specific
> package manager to install stuff.

For my own experience, that sounds worse. However, it would be nice if
'egg' could detect that certain things were installed by a non-egg
system (ie, having `py-sqlite` from MacPorts) and not installing it.
This goes into a deeper frustration I've had in the past: I installed
MySQL on my desktop (Mac OS X) using a disk image / .pkg installer
downloaded from MySQL's web site. Then I think I tried installing a
python package from MacPorts (maybe just the mysql bindings?) that had
a MySQL dependency. It didn't detect that I already had MySQL
installed, and MacPorts then tried installing it on its own. At that
point, I stopped using ports for just about anything Python related,
aside from getting Python and Py-Readline. It was easier to use
easy_install or regular distutils and the like. The dependencies were
met, but not advertised in a way that was friendly to the packaging
system in question.

> It looks like, for example, ports have targets to build to other
> systems, e.g., pkg, mpkg, dmg, rpm, srpm, dpkg.  So maintaining
> package information in (or compatible with) ports could make it easy
> to feed packages into other package systems.
>
>    * Benefit: We're working with other package systems, just making
> it easier to get Python packages into them.
>
>    * Drawback: They may not want to include all packages, at the
> speed at which we want, or the way we want to.  (I.e., there may
> still be packages you'd want that are only available on PyPI.)

Or packages you only want internally. Or packages you don't want
available on PyPI because they're very specific to a large
framework/toolkit like Zope 3.

>    * Drawback: Some systems don't have package systems.

And some administrators don't use them beyond (maybe) initially
setting up the system.

I also don't know how well those package systems deal with concepts
like local-installs. Not just local to a single user, but local to a
single package. `zc.buildout` is good about this, almost to a fault.

There is a rough balance there between desktop and personal machine
global-install ease of use and being able to set up fine-tuned
self-contained setups.

Anyways, I'd vote pure-python. Even on the most barren of machines,
it's relatively easy to build and install Python from source. Even on
a fairly old installation, it's easy to build an install a new version
of Python from source - probably far easier than wrestling with the
package manager about updating its database and then updating package
after package after package after package that one doesn't want.

I think that Python should be all that you need in order to get other
Python packages. `easy_install` pretty much gives us this today. There
are improvements I'd love to see - reports of what I have installed,
what's active, what's taking precedence in my environment, etc. Your
tool may do this, I haven't had time to look yet. Ruby's 'gem' command
does this beautifully. And I hardly ever touch Ruby or gems; it was
just very easy to use for the few things I've wanted to try.

-- 
Jeff Shell