[Distutils] MANIFEST destiny :)

Phillip J. Eby pje at telecommunity.com
Thu Nov 17 01:28:52 CET 2005

At 12:19 AM 11/17/2005 +0100, M.-A. Lemburg wrote:
> > Thus, I find these super-custom processes you're talking about highly
> > implausible, because the only people who could implement them are the
> > people with a strong knowledge of the distutils -- an incredibly rare
> > breed of person, in other words.
>I was only talking about special ways to build the MANIFEST
>files, not "super-custom" processes. No idea where you
>got that impression from.

Anything that involves a user knowing about MANIFEST at *all* (let alone 
changing its name) is a "super custom" process.  Which unfortunately means 
that even for a fairly basic sdist, it's an issue.

>I agree that this feature is underdocumented, but changing
>the framework won't help with this: documentation patches
>is what we *really* need !

That's certainly part of it; but it's certainly not all of it.  Mostly, 
it's the difference between being a tool and a framework.  The distutils 
provides all the mechanism you like, and none of the policy.  That was a 
brilliant political move, as it dodges all the infighting that otherwise 
would have stopped it getting off the ground.  However, there's no longer 
any reason *not* to have One Official Python Way To Do It, and to 
streamline the tools around that philosophy.

One minor example of the problems with the distutils, is that their 
flexibility and inconsistent documentation causes people to create wildly 
different project source layouts, just because they can.  You call that 
flexibility, I call that bad design for the developers who don't want to 
have to think about all these options (i.e., most of them).

>BTW, not many people need to have these MANIFEST files
>at all - distutils uses a built-in file finder based on these
>           - README or README.txt
>           - setup.py
>           - test/test*.py
>           - all pure Python modules mentioned in setup script
>           - all C sources listed as part of extensions or C libraries
>             in the setup script (doesn't catch C headers!)

It also leaves out documentation and package data, for starters.

>Carelessly overwriting hand-edited or otherwise generated
>files in a build process is simply bad design.

Not if it's a file *owned* by the distutils.  If MANIFEST is assumed to be 
some sort of possibly user-generated file, the distutils should've picked a 
different name for *its* manifest file, and maybe provided the ability to 
read other files to generate its manifest.

This is my point: there should be a manifest file that is wholly owned and 
controlled by the distutils with *no user-serviceable parts 
inside*.  Having a file that is *partly* owned by the distutils and partly 
by the user is a recipe for disaster and the real bad design in this case.

>MANIFEST is only used for source code distrbutions. I don't
>see how you can use it for anything else.

Because if you have package data files as part of your *source*, then the 
MANIFEST has to list them in order to have a valid source 
distribution.  So, for some projects, it is equally valid to say, "be sure 
to install any package data files listed in the MANIFEST", and that is what 
the 'install_package_data' option would do.  (Or maybe 
'use_manifest_package_data' would be a better name, but you get the general 

>I don't buy this: on one hand you are talking about simple
>packages (which don't need the MANIFEST files in the
>first place),

Not true!  You run into a problem the moment you have any documentation 
besides README, or if you have any package data files.

>on the other about hooks to adjust distutils'
>build process, something I'd group under more complex

My point is that a well-designed system that offers customization hooks 
should not offer you *meaningless* choices between different ways to 
customize the same thing.

>So your point is to make your life as setuptools author
>easier ?

No, if that was my point I'd have just done whatever the heck I wanted and 
not bothered posting to the list.  :)  My point is to get the *community* 
better tools, and part of that is lowering the entry barriers to creating 
them.  If it was only me I cared about, I'd not have embarked on the entire 
venture to begin with.  Sheesh.

>Why don't you just disable all these options in your
>setuptools front-end and hard-code the MANIFEST file
>names ?

I plan to!  The point of my posting was to find out whether I was going to 
be introducing any meaningful barriers to adoption of setuptools.  For 
example, if there were some bdist* command out there in widespread use that 
would break as a result of these changes.  Yes, I ranted a little about 
making any "distutils 2" less flexible, but my practical present-day 
assumption is that setuptools actually modifying the distutils is still a 
long time coming.  I don't even plan to *propose* that idea myself, 
certainly not any time soon.

>distutils is a loosly coupled framework of components.
>In such a framework, a basic design principle is to
>be able to decouple and recouple existing components.
>The only way to implement this is by making the components
>suitably independent and this is what was done in

You're confusing a usability issue with a technical design issue.  One 
calls for TMTOWTDI (There's More Than One Way To Do It), the other for 
TSBO-APOO-OWTDI (There Should Be One - And Preferably Only One - Obvious 
Way To Do It).

>My point is that you shouldn't try to strip down distutils
>itself just because you think it's hard work to support
>all these features in setuptools. It's not needed to strip
>down distutils for this reason as you can easily disable
>these options for anyone using your setuptools.
>As a result, both users of setuptools and straight
>distutils are happy.

This was always my plan; the question was to find out whether there were 
any holes in that plan, for *setuptools*.  I'm sorry if that wasn't clear; 
I only talked about removing these things in distutils under a hypothetical 
"distutils 2" scenario.  *None* of these changes were being proposed for a 
"pure" distutils today.

That doesn't mean I don't think that the distutils design is too flexible 
in terms of offering meaningless choices, or more precisely, offering lots 
of choices to either shoot yourself in the foot.  :)  It just means that I 
realistically understand that today's distutils are unlikely to be changeable.

Rather, I'm looking at the eventual transmutation of setuptools into 
"distutils 2", and want to make sure that late adopters moving to it in the 
distant future don't run into issues that haven't been accomodated in some way.

More information about the Distutils-SIG mailing list