[Distutils] The "dist" command

[Greg Ward]

I realized last night that I can't honestly release Distutils 0.1 until
it can distribute itself, i.e. until there's a "dist" command to create
a source distribution.  So I sat down and did a brain dump on what's
needed to create source distributions.  If you enjoy reading braindumps, 
please read this one and post your reactions.

<braindump>
Possible modes of operation:
  - require an explicit manifest that lists every single file (presumably
    along with a way to auto-generate the manifest)
  - require an explicit manifest, but allow it to have globs or
    filename patterns of some kind (and also have auto-generation)
  - allow an explict manifest, but automatically augment it at runtime
    with the source files mentioned in 'packages', 'py_modules', and
    'ext_modules' (and any other such things that might come along)

I'm liking the third way.  Possible gotchas:
  - redundant specification: 'packages' includes 'foo' and manifest
    includes 'foo/*.py'
  - obvious conflict: 'packages' includes 'foo' and manifest
    includes '! foo/*.py' (can't imagine why you'd want this)
  - subtle conflict:  'packages' includes 'foo' and manifest
    includes '! foo/bar.py' (this could well be desired: eg. exclude
    an experimental module from distribution)

Syntax for the manifest file:
  - each line is by default a Unix-style glob specifying files to include
  - if the line starts with '!', then it's an exclude pattern
  - if the line just names a directory in the distribution, that directory
    is included recursively
  - if the first word of a line names a directory, then the remaining
    words are include/exclude patterns that are applied recursively
    under that directory

example (ignoring auto-augmentation!):
  distutils/*.py
  distutils/command/*.py
  ! distutils/bleeding_edge.py
  examples/*.py
  examples/README

smarter way (that *will* include distutils/command/bleeding_edge.py!)
  distutils *.py
  ! distutils/bleeding_edge.py
  examples !*~ !*.py[co]
  test !*~ !*.py[co]
  README
  setup.py

The actual Distutils manifest (don't need to mention source files,
README, setup.py -- they're automatically distributed!):
  examples !*~ !*.py[co]
  test !*~ !*.py[co]

The algorithm that will make it work:
  files = stuff from 'packages', 'py_modules', 'ext_modules',
    plus README, setup.py, ... ?
  foreach spec in manifest file:
    if simple-include-spec:         # "distutils/*.py"
      files.append (glob (spec))
    elif simple-exclude-spec:       # "! distutils/foo*"
      xfiles = glob (spec)
      remove all xfiles from files
    elif dir-spec:                  # "examples" (just a directory name)
      dir_specs = rest-of-words-on-line
      if dir_specs:
        for dspec in dir_specs:
          if include-spec:
            dir_files.append (recursive_glob (dir, dspec))
          elif exclude-spec:
            xfiles = recursive_glob (dir, dspec)
            remove all xfiles from dir_files
      else:
        dir_files = recursive_glob (dir, '*')


Other things we need to look for in creating a source distribution:
  - make sure there's a README
  - make sure the distribution meta-info is supplied and non-empty
    (*must* have name, version, ((author and author_email) or
    (maintainer and maintainer_email)), url

Frills:
  - make sure the setup script is called "setup.py"
  - make sure the README refers to "setup.py" (ie. has a line matching
    /^\s*python\s+setup\.py/)

A crazy idea that conflicts with having/requiring 'version' in setup.py:
  - make sure there's a version number in the "main file" (main file
    is __init__.py of first package, or the first module if no packages,
    or the first extension module if no pure Python modules)
  - XXX how do we look for __version__ in an extension module?
  - XXX do we import and look for __version__? or just scan source for
    /^__version__\s*=\s*"[^"]+"/ ?
  - what about 'version_from' as an alternative to 'version' -- then
    we know just where to search for the version -- no guessing about
    what the "main file" is
</braindump>

If you have comments, better get 'em in quick -- I've already started
implementing this.

        Greg

-- 
Greg Ward - software developer                    gward@cnri.reston.va.us
Corporation for National Research Initiatives    
1895 Preston White Drive                           voice: +1-703-620-8990
Reston, Virginia, USA  20191-5434                    fax: +1-703-620-0913