[Distutils] Re: CPAN functionality for python - requirements

[Moore, Paul]

From: Sean Reifschneider [mailto:jafo at tummy.com]
> On Tue, Feb 27, 2001 at 09:30:13AM -0700, Evelyn Mitchell wrote:
> >But it will also discover and resolve dependences in your perl
> >site-packages, and automatically fetch them from your closest
> >CPAN archive.
> 
> Not according to my tests the night before last.  I did a test CPAN
> install of "News::Newsrc", which failed because the "make test" was
> failing.  I then installed the "Set::BitSet" (?  Something like that)
> module and tried News::Newsrc again and it worked...
> 
> Maybe this was just a fluke and News::Newsrc is the exception and/or
> isn't used enough that people have gotten the prereqs right yet.  If
> anyone knows for sure, I'm curious.

There are basically a number of aspects to "CPAN", which need separating
out.

MakeMaker
---------

This is a perl module which implements a build process. You write a
Makefile.PL, which calls MakeMaker defining the module's structure and
metadata. Thanks to MakeMaker, the process for building a Perl module is
(nearly always) simply

    perl Makefile.PL
    make
    make test      <-- optional, but pretty much standard - runs module unit
tests
    make install   <-- installs the module

This is, in both concept and functionality, almost identical to Distutils.
There are some areas in which distutils is better (building of
platform-specific installers, for instance) and some where MakeMaker is
better (it hooks into a very standard structure for modules, generated by
thye h2xs program, which practically forces module authors to write test
suites and documentation in a standard format), but these are details.

We can consider this covered. (Although the distutils-sig could still
usefully learn from MakeMaker).

The system of FTP sites and mirrors
-----------------------------------

Frankly, this is nothing special. It has some nice features (automated
uploads for module authors, plus quite nice indexing and server multiplexing
features), but it isn't rocket science as far as I know. We could quite
happily start with a simple FTP site for this (that's what CPAN did - the
mirroring came later as popularity grew).

CPAN.pm
-------

This is a Perl module, which automates the process of downloading and
installing modules. I don't use this personally, for a number of reasons.
Frankly, I find that manual downloading and running the 4 lines noted above
is fine.

It relies for things like dependency tracking on metadata added into the
Makefile.PL which is not necessary for the straight 4-line build above. As
such, the level to which that metadata is added by module authors is
variable (to be polite). In practice, I wouldn't rely on it - accurate
dependency data seems to be the exception rather than the rule.

I *don't* regard CPAN.pm to be important to the overall CPAN "phenomenon".
But some people like it. Writing something "at least as good as" CPAN.pm
shouldn't be hard in Python - not least because the standard library is rich
enough that things like FTP client support is available out of the box
(whereas CPAN.pm won't work until you manually install libnet, and possibly
some other modules, I forget which...)

But writing a "perfect" utility for automated download-and-install, with
dependency tracking, etc etc, is going to be VERY HARD. Don't get misled -
Perl doesn't have such a beast. And We won't even have what Perl has if we
focus on perfection rather than practicality.

The h2xs program
----------------

This is VERY important. The idea is that when you write a Perl module,
either pure perl or a C (XS) extension, you run h2xs first, to generate a
template build directory. It automatically includes

    * The perl module, with some basic template code and embedded POD
      documentation
    * The XS extension, with template code (if requested)
    * A Makefile.PL shell
    * A basic test script - all it does is test that the module loads,
      but it includes a placeholder for your own tests

Essentially, h2xs forces a standard structure on all Perl modules. This is
important for Perl, where modules have to conform to some standards in order
to work at all. However, it brings HUGE benefits in standardisation of all
the "other" parts of the process (documentation, tests, etc).

Python is at a disadvantage here, precisely because writing a Python module
involves so little in the way of a specific structure. So people will likely
rebel against having a structure "imposed"...

A social structure
------------------

This is a bit of a chicken and egg issue, but Perl developers expect to
write modules using h2xs and MakeMaker, they expect to write tests (even if
they are minimal), they expect to fill in the sections in the POD
documentation, and they expect to submit their modules to CPAN. So this all
"just works".

Interestingly, developers probably don't "expect" to have to include
dependency information, and hence many don't - resulting in the problems you
hit. But then again, Perl users don't "expect" to be totally shielded from
dependency issues.

Python is VERY far behind here. This is a maturity issue - distutils is
still (relatively) new, and so there are LOTS of packages which don't come
with a setup.py yet. Often, adding one isn't hard, but it is still to
happen. And when you are distributing a pure python module, as a single .py
file, it's hard to see the benefit of changing that into a .tar.gz file
containing the module, plus a setup.py. (Once you start adding test suites
and documentation, the point of the whole archive bit is clearer, but we're
not even close to that stage yet).

Things are looking better with Python 2.1, though. Included with 2.1, it
looks like there will be a standard unit testing module, and a new "pydoc"
package, which will extract documentation from module docstrings. So the
infrastructure will be there, it just becomes a case of getting developers
to use it consistently. Distutils can help with this, by "just working" if
people follow a standard structure.

Sorry, this wasn't meant to get so long...
Paul.