Tarek Ziadé wrote:
Hi,
I am back on that problem with the code that builds the file list. The current Distutils trunk isn't working anymore with setuptools because of a recursive loop:
distutils.sdist.run() -> setuptools.build_py.data_files -> setuptools.egg_info.run() -> distutils.sdist.add_defaults() -> setuptools.build_py.data_files -> etc
The mess is introduced by the fact that build_py is patched by setuptools in order to inject the files that are provided by the (D)VCS. But it also uses some APIs provided by sdist to complete the file list.
In order to solve this, we need to define clearly what is the role of each command, and make sure it's a distinct role.
which is not the case right now :
1/ distutils.sdist = this command is used to build a source distribution 2/ setuptools.egg_info = this command is used to build an .egg-info directory but *also* injects the files founded with (D)VCS in the MANIFEST 3/ distutils.build_py = this command is used to build pure python module but *also* to find all the .py files to include in a distribution (used by sdist). In fact, it plays a central role in sdist to get the files to include.
Here's a first thaught to discuss:
what about introducing a new simple command called "manifest" that could be used by sdist or any other command, and that would be responsible of collecting the files that are part of the distribution (and nothing else). This command would generate the MANIFEST file and also provide the APIs to get the files included into MANIFEST.
This command would introduce and use a simple plugin system similar to the "setuptools.file_finders" entry points setuptools has for (D)VCS files collecting. But with the files list being built as an argument to the plugin. (partly described here http://wiki.python.org/moin/Distutils/ManifestPluginSystem) so Distutils, setuptools or any third party tool can register some code that add or remove file in this file list.
The manifest command would provide default plugins, and setuptools could refactor part of its code to use the manifest command rather than calling sdist APIs. The goal would be to have the same result at the end, but make it simpler to extend, and avoid command interdependencies like what we have today (and that makes it hard to maitain and make evolve).
For instance the MANIFEST.in templating system would be one of the default plugin provided by Distutils.
The initial work would consist of refactoring the current code that gets the files, using different strategies, into plugins for the new manifest command, then make sdist uses this command as a subcommand.
I think this is good stuff. You might find that the pieces that collect files for the MANIFEST are useful in more than creating the MANIFEST, though. Unless I've missed a message saying that we're going to collapse sdist and package-installed-files together.
The second phase would consist of working at setuptools level to use the same technique. Setuptools would be able to make existing "setuptools.file_finders" entry points work with Distutils manifest command registery by providing a bridge.
Now for the plugin system, I see two options so far:
1/ create a simple ad-hoc plugin system in Distutils, and declare the plugins in a new section in distutils.cfg / pydistutils.cfg for loading them (setuptools existing entry points would be collected through a unique plugin declared that would act like a bridge)
2/ use entry points (so add them into Distutils) and define a entry point name based on the command name, maybe "distutils:metadata.file_finders" so the plugin system could be used elsewhere in distutils.
Having a library that makes creating plugins easy is a good general purpose thing. Whatever plugin system is adopted/created, it would be good for it to not be internal to distutils. I've always hated that distutils option handling is different from the option handling that a coder can use from the standard library. -Toshio