[Distutils] PyPI - Evolve our own or reuse existing package systems?

[Arve Knudsen]

Hei Bjørn :)

These are some interesting points you are making.
I have in fact been developing a general software deployment system,
Conduit<http://conduit.simula.no>, in Python for some time, that is
capable of supporting several major platforms (at the moment: Linux,
Windows and OS X). It's not reached any widespread use, but we (at the
Simula Research Laboratory) are using it to distribute
 software to students attending some courses at the university of Oslo.
Right now we are in the middle of preparing for the semester start, which is
next week.

The system is designed to be general, both with regard to the target
platform and the deployable software. I've solved the distribution problem
by constructing an XML-RPC Web service, that serves information about
software projects in RDF (based on the DOAP format). This distribution
service is general and independent of the installation system, which acts as
a client of the latter.

If this sounds interesting to you I'd love if you checked it out and gave me
some feedback. It is an experimental project, and as such we are definitely
interested in ideas/help from others.

Arve

On 8/15/07, Bjørn Stabell <bjorn at exoweb.net> wrote:
>
> (Since my email was a bit long and wide I'm trying to update the
> subject when the response is rather focused.)
>
> On Aug 15, 2007, at 06:37, Paul Boddie wrote:
> > Bjørn Stabell wrote:
> [...]
> > I've been moderately negative about evolving a parallel
> > infrastructure to
> > other package and dependency management systems in the past, and
> > I'm not
> > enthusiastic about things like CPAN or language-specific
> > equivalents. The
> > first thing most people using a GNU/Linux or *BSD distribution are
> > likely to
> > wonder is, "Where are the Python packages in my package selector?"
> >
> > There are exceptions, of course. Some people may be sufficiently
> > indoctrinated
> > in the ways of Python, which I doubt is the case for a lot of
> > people looking
> > for packages. Others may be working in restricted environments
> > where system
> > package management tools don't really help. And people coming from
> > Perl might
> > wonder where the CPAN equivalent is, but they should also remind
> > themselves
> > what the system provides - they have manpages for Perl, after all.
> [...]
> > I've read through the text that I've mercilessly cut from this
> > response, and I
> > admire the scope of this effort, but I do wonder whether we
> > couldn't make use
> > of existing projects (as others have noted), and not only at the
> > Python-specific level, especially since the user interface to the
> > "egg" tool
> > seems to strongly resemble other established tools - as you seem to
> > admit in
> > this and later messages, Bjørn.
> [...]
> > I was thinking of re-using the Debian indexing strategy. It's very
> > simple,
> > perhaps almost quaintly so, but a lot of the problems revealed with
> > the
> > current strategies around PyPI (not exactly mitigated by bizarre
> > tool-related
> > constraints) could be solved by adopting existing well-worn
> > techniques.
> [...]
> > If I recall correctly, the PEP concerned just "bailed" on the version
> > numbering and dependency management issue, despite seeming to be
> > inspired by
> > Debian or RPM-style syntax.
> [...]
> > As I've said before, it's arguably best to work with whatever is
> > already
> > there, particularly because of the "interface" issue you mention with
> > non-Python packages. I suppose the apparent lack of an open and
> > widespread
> > package/dependency management system on Windows (and some UNIX
> > flavours) can
> > be used as a justification to write something entirely new, but I
> > imagine
> > that only very specific tools need writing in order to make existing
> > distribution mechanisms work with Windows - there's no need to
> > duplicate
> > existing work from end to end "just because".
> [...]
> > Agreed. And by adopting existing mechanisms, we can hopefully avoid
> > having to
> > reinvent their feature sets, too.
> >
> > P.S. Sorry if this sounds a bit negative, but I've been reading the
> > archives
> > of the catalog-sig for a while now, and it's a bit painful reading
> > about how
> > sensitive various projects are to downtime in PyPI, how various
> > workarounds
> > have been devised with accompanying whisper campaigns to tell
> > people where
> > unofficial mirrors are, all whilst the business of package
> > distribution
> > continues uninterrupted in numerous other communities.
> >
> > If I had a critical need to get Python packages directly from their
> > authors to
> > run on a Windows machine, for example, I'd want to know how to do
> > so via a
> > Debian package channel or something like that. This isn't original
> > thought:
> > I'm sure that Ximian Red Carpet and Red Hat Network address many
> > related
> > issues.
>
> There seems to be two issues:
>
> 1) Should Python have its own package management system (with
> dependencies etc) in parallel with what's already on many platforms
> (at least Linux and OS X)?  Anyone that has worked with two parallel
> package management systems knows that dependencies are hellish.
>
>    * If you mix and match you often end up with two of everything.
>
>    * It'll be incomplete because you can't easily specify
> dependencies to non-Python packages.
>
>
> 2) If we agree Python should have a package management system, should
> we build or repurpose some other one?
>
>    * I think it's a matter of pride and proof of concept to have one
> written in Python.  That doesn't mean we can't get ideas from others.
>
>    * It's also not that hard to do.  The prototype I threw up took
> one weekend + half a day, and consists of about 500 lines of new
> code.  It could be refactored and made smaller, but even if a
> complete version is ten times the size of that, it's still not a huge
> undertaking.
>
>    * With a Python version we could relatively easily innovate beyond
> what traditional packaging systems do; ports and apt are pretty much
> stagnated.  I think RubyGems seems to have some cool features,
> features that probably wouldn't have happened if they were using
> ports or apt-get (but then they could piggyback on innovations in
> those tools, I guess).  If it works for them, why shouldn't it work
> for us?
>
>    * It would have to be as portable as Python is; many packaging
> systems are by nature relatively platform-specific.
>
>    * If we don't build our own, doesn't that mean we throw out eggs?
>
>    * Packaging systems are useful for mega frameworks like Zope,
> TurboGears, and Django, and slightly less so for projects you roll on
> your own, to manage distribution and installation of plugins and
> addons.  Relying on platform-specific packaging systems for these may
> not work that well.  (But I could be wrong about that.)
>
>
> That said, it might be possible to do some kind of hybrid, for PyPI
> to be a "meta package" repository that can easily feed into platform
> specific packaging systems.  And to perhaps also have a client-side
> "meta package manager" that will call upon the platform-specific
> package manager to install stuff.
>
> It looks like, for example, ports have targets to build to other
> systems, e.g., pkg, mpkg, dmg, rpm, srpm, dpkg.  So maintaining
> package information in (or compatible with) ports could make it easy
> to feed packages into other package systems.
>
>    * Benefit: We're working with other package systems, just making
> it easier to get Python packages into them.
>
>    * Drawback: They may not want to include all packages, at the
> speed at which we want, or the way we want to.  (I.e., there may
> still be packages you'd want that are only available on PyPI.)
>
>    * Drawback: Some systems don't have package systems.
>
>
> Which brings me to: If we're just distributing source files why don't
> we use a source control system such as svn, bzr, or hg?  The package
> developers have trunk, PyPI is a branch, the platform-specific
> package maintainers have a branch, and what's installed onto your
> system is in the end a branch (serially connected).  Some systems,
> like Subversion, can also include externals like I did with cliutils
> on the egg package.  Just a thought.
>
>
> Rgds,
> Bjorn
> _______________________________________________
> Distutils-SIG maillist  -   Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070815/c33a2a13/attachment.htm