[Distutils] Fixing the mess in sdist/egg_info

[Tarek Ziadé]

Hi,

I am back on that problem with the code that builds the file list. The
current Distutils trunk isn't working anymore with setuptools because
of a recursive loop:

distutils.sdist.run() -> setuptools.build_py.data_files ->
setuptools.egg_info.run() -> distutils.sdist.add_defaults() ->
setuptools.build_py.data_files -> etc

The mess is introduced by the fact that build_py is patched by
setuptools in order to inject the files that are provided by the
(D)VCS.
But it also uses some APIs provided by sdist to complete the file list.

In order to solve this, we need to define clearly what is the role of
each command, and make sure it's a distinct role.

which is not the case right now :

1/ distutils.sdist  = this command is used to build a source distribution
2/ setuptools.egg_info = this command is used to build an .egg-info
directory but *also* injects the files founded with (D)VCS in the
MANIFEST
3/ distutils.build_py = this command is used to build pure python
module but *also* to find all the .py files to include in a
distribution (used by sdist).   In fact, it plays a central role in
sdist to get the files to include.

Here's a first thaught to discuss:

what about introducing a new simple command called "manifest" that
could be used by sdist or any other command, and that would be
responsible of collecting the files that are part of the distribution
(and nothing else). This command would generate the MANIFEST file and
also provide the APIs to get the files included into MANIFEST.

This command would introduce and use a simple plugin system similar to
the "setuptools.file_finders" entry points setuptools has for (D)VCS
files collecting. But with the files list being built as an argument
to the plugin. (partly described here
http://wiki.python.org/moin/Distutils/ManifestPluginSystem) so
Distutils, setuptools or any third party tool can register some code
that add or remove file in this file list.

The manifest command would provide default plugins, and setuptools
could refactor part of its code to use the manifest command rather
than calling sdist APIs. The goal would be to have the same result at
the end, but make it simpler to extend, and avoid command
interdependencies like what we have today (and that makes it hard to
maitain and make evolve).

For instance the MANIFEST.in templating system would be one of the
default plugin provided by Distutils.

The initial work would consist of refactoring the current code that
gets the files, using different strategies, into plugins for the new
manifest command, then make sdist uses this command as a subcommand.

The second phase would consist of working at setuptools level to use
the same technique. Setuptools would be able to make existing
"setuptools.file_finders" entry points work with Distutils manifest
command registery by providing a bridge.

Now for the plugin system, I see two options so far:

1/ create a simple ad-hoc plugin system in Distutils, and declare the
plugins in a new section in distutils.cfg / pydistutils.cfg for
loading them (setuptools existing entry points would be collected
through a unique plugin declared that would act like a bridge)

2/ use entry points (so add them into Distutils) and define a entry
point name based on the command name, maybe
"distutils:metadata.file_finders"  so the plugin system could be used
elsewhere in distutils.

Opinions ?

Tarek

-- 
Tarek Ziadé | http://ziade.org