[Python-Dev] Thoughts on stdlib evolvement

[Stephen J. Turnbull]

>>>>> "Skip" == Skip Montanaro <skip at pobox.com> writes:

    Skip> If you provide the necessary namespace structure for them to
    Skip> nestle into, I suspect most of them could be maintained
    Skip> outside the stdlib just fine.

FWIW, this has worked well for XEmacs; it's one of our most popular
features vis-a-vis GNU Emacs.  Users like it because we do provide a
centralized directory of modules (with many mirrors) that they can
download updates from, as well as a "batteries included" library
distribution (which we call "SUMO tarballs" for historical reasons).
Package developers like it because they can maintain closer control,
yet have up-to-date versions distributed from our repository.  Finally
the core maintainers like it because there's a clear delegation of
responsibility.

We provide a centralized repository, which is mirrored by a couple
score sites; I'm not sure how much harder it would be to make it
distributed.  The essential feature is the centralized directory.

We do have problems.  One is the megapackage issue that Phillip Eby
referred in his reply to Bob Ippolito
<5.1.1.6.0.20050606182742.01f7d118 at mail.telecommunity.com>.  Our
packaging system doesn't work well for them.  Some of the problems are
our fault, but I think it's actually a hard problem; we could fix up
some details, but not really address the core issues of megapackages.

The other one is that our initial division into "core" and "package"
was pretty radical, and several core developers still want to pull
certain packages back into core.  But this creates backward
compatibility problems.  I think this would be less of a problem for
Python.  Creating the package repository could be more conservative:
Python's standard library has plenty of modules that would not present
such problems.  In any case, Python has better facilities for dealing
with backward compatibility issues (such as deprecation warnings and
the like) than we do, and the language itself is simply more suited to
preserving modularity than Emacs Lisp is (with its single dynamically
scoped namespace).

Our package namespace structure is trivial (it's basically flat); what
is more important than the namespace structure, I think, is provision
of package organization and directory utilities as Phillip Eby
mentions.  If they're reasonably flexible and Python invests some
effort in helping package maintainers with different current
distribution methods to adapt, I think this would work well for Python
too.  One thing we have found is that it takes only a small amount of
incremental work to maintain the package directory and repository as
such, even as it has grown dramatically.  By contrast, I find the Zope
wiki-based organization of Products quite annoying to work with.

Of course the packaged code itself can and does bitrot if the external
maintainer goes AWOL, or simply doesn't like hanging out with XEmacs
people.  In such cases we've found that users will often step up to
help with the nitty-gritty management of "their" package, such as
tracking bug reports, liaison with an "unsociable" upstream,
regression testing, and integrating changes (anybody can run make and
patch), even if they have to ask for help with code-related work.  Ie,
the fact that there is a well-defined package requiring specific
management tasks encourages them to take on the work they can handle.

For XEmacs the effort of separating out the packages was moderately
large; it took about a year, with three or four developers spending a
substantial fraction of their time on it, either organizing the
repository or creating infrastructure.  It was a very worthwhile
investment.  Python's investment probably would be proportionately
smaller, as much of the necessary infrastructure seems to be already
available.

For XEmacs, I think the main returns come from creating well-defined
chunks that people can "own" and take responsibility for.  This helps
to mitigate the "dormant bug" problem, and generally keeps the
packaged code fresher than it was in the old days, even though we have
a much larger volume of code and probably less manpower now.


-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.