Need better packaging (was Re: [Python-Dev] SQLite module for Python 2.5)

[Jim Fulton]

M.-A. Lemburg wrote:
...
> Maybe we should have this discussion on more general grounds:
> do we really want a fat Python distribution or should we focus
> more on making installation of third-party tools easier ?

I think you hit the nail on the head.  For myself, I think we want to
make it *much* easier to install 3rd-party tools.  I also think we need
to make it *much* easier to *update* the modules that come with Python
without having to wait for a new Python release (much as it is easy to
update packages in a linux distribution).

While I sympathise with the "batteries included" philosophy, it has a
number of drawbacks:

- It makes technical decisions very risky. After all, once something
   is added to the Python distribution, it's had to take it out and it's
   hard to change it (see below).

- It can favor some technologies unfairly

- It can cause things to be included that shouldn't be.  My favorite
   examples of this are asyncore and the Berkeley DB extensions.
   The former is no-longer supported by it's original author and causes
   an undue burden on the Python developers.  The later is inadequately
   supported and causes instability.  (For example, I can't
   use "make test" in a CVS checkout or the beta release, as Python
   seg-faults when it gets to the bdb tests.)

- It actually stifles development of the library.  It's hard to be
   motivated to improve library modules when the time between releases
   is sooooo long.  Either:

   - You develop features that you won't be able to use for a year or more.
     (Large systems like Zope and Twisted can't rely on current versions
     of Python.)

     or

   - You have to use copies of future library modules.  We are doing this
     now with doctest.  We've had to do this in the past, with considerable
     difficulty, for CPickle and ayncore.

   Part of the problem is that library modules evolve at a different rate
   than the language.  Often this is because library modules are new. Newer
   systems typically evolve faster than mature systems.  Sometimes, mature
   modules, like doctest, suddenly experience a growth spurt.

Finally, I think that the *real* needs that drive "batteries included"
would be better served by a packaging system.  A packaging system (ala
cpan or rpm) would make it much easier for people to get the featureful
Python's they need than the current system.

(It could be argues that we should just use native packaging systems,
like RPM.  This approach has 2 serious problems.  First, it doen't work on
Windows.  Second, it raises the packaging bar much higher, as packagers
have to create separate packages for each target system.  A Python-based
packaging system would allow people to create packages that are usable on
any platform that Python runs on.)

IMO, one of the (if not *the*) most important Python development project is
the development of a packaging system.  I think there are a number of good
starts toward this, such as PEP 262 and various efforts such as the mac-based
packaging system and some work Fred Drake has done here at Zope Corp.  It would be
great to follow through with this and get to the point where its us far less
important what happens to be included in a distribution. I'm committed to making
this happen.  ZC will eventually build something if no one else does.  But it
can happen *much* sooner if we all work together.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org