[Distutils] Fixing the mess in sdist/egg_info

[Toshio Kuratomi]

Tarek Ziadé wrote:
> Hi,
> 
> I am back on that problem with the code that builds the file list. The
> current Distutils trunk isn't working anymore with setuptools because
> of a recursive loop:
> 
> distutils.sdist.run() -> setuptools.build_py.data_files ->
> setuptools.egg_info.run() -> distutils.sdist.add_defaults() ->
> setuptools.build_py.data_files -> etc
> 
> The mess is introduced by the fact that build_py is patched by
> setuptools in order to inject the files that are provided by the
> (D)VCS.
> But it also uses some APIs provided by sdist to complete the file list.
> 
> In order to solve this, we need to define clearly what is the role of
> each command, and make sure it's a distinct role.
> 
> which is not the case right now :
> 
> 1/ distutils.sdist  = this command is used to build a source distribution
> 2/ setuptools.egg_info = this command is used to build an .egg-info
> directory but *also* injects the files founded with (D)VCS in the
> MANIFEST
> 3/ distutils.build_py = this command is used to build pure python
> module but *also* to find all the .py files to include in a
> distribution (used by sdist).   In fact, it plays a central role in
> sdist to get the files to include.
> 
> Here's a first thaught to discuss:
> 
> what about introducing a new simple command called "manifest" that
> could be used by sdist or any other command, and that would be
> responsible of collecting the files that are part of the distribution
> (and nothing else). This command would generate the MANIFEST file and
> also provide the APIs to get the files included into MANIFEST.
> 
> This command would introduce and use a simple plugin system similar to
> the "setuptools.file_finders" entry points setuptools has for (D)VCS
> files collecting. But with the files list being built as an argument
> to the plugin. (partly described here
> http://wiki.python.org/moin/Distutils/ManifestPluginSystem) so
> Distutils, setuptools or any third party tool can register some code
> that add or remove file in this file list.
> 
> The manifest command would provide default plugins, and setuptools
> could refactor part of its code to use the manifest command rather
> than calling sdist APIs. The goal would be to have the same result at
> the end, but make it simpler to extend, and avoid command
> interdependencies like what we have today (and that makes it hard to
> maitain and make evolve).
> 
> For instance the MANIFEST.in templating system would be one of the
> default plugin provided by Distutils.
> 
> The initial work would consist of refactoring the current code that
> gets the files, using different strategies, into plugins for the new
> manifest command, then make sdist uses this command as a subcommand.
> 
I think this is good stuff.  You might find that the pieces that collect
files for the MANIFEST are useful in more than creating the MANIFEST,
though.  Unless I've missed a message saying that we're going to
collapse sdist and package-installed-files together.

> The second phase would consist of working at setuptools level to use
> the same technique. Setuptools would be able to make existing
> "setuptools.file_finders" entry points work with Distutils manifest
> command registery by providing a bridge.
> 
> Now for the plugin system, I see two options so far:
> 
> 1/ create a simple ad-hoc plugin system in Distutils, and declare the
> plugins in a new section in distutils.cfg / pydistutils.cfg for
> loading them (setuptools existing entry points would be collected
> through a unique plugin declared that would act like a bridge)
> 
> 2/ use entry points (so add them into Distutils) and define a entry
> point name based on the command name, maybe
> "distutils:metadata.file_finders"  so the plugin system could be used
> elsewhere in distutils.
> 
Having a library that makes creating plugins easy is a good general
purpose thing.  Whatever plugin system is adopted/created, it would be
good for it to not be internal to distutils.  I've always hated that
distutils option handling is different from the option handling that a
coder can use from the standard library.

-Toshio

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20090416/a4f57a3a/attachment.pgp>