[Distutils] MANIFEST destiny :)
Phillip J. Eby
pje at telecommunity.com
Thu Nov 17 01:28:52 CET 2005
At 12:19 AM 11/17/2005 +0100, M.-A. Lemburg wrote:
> > Thus, I find these super-custom processes you're talking about highly
> > implausible, because the only people who could implement them are the
> > people with a strong knowledge of the distutils -- an incredibly rare
> > breed of person, in other words.
>I was only talking about special ways to build the MANIFEST
>files, not "super-custom" processes. No idea where you
>got that impression from.
Anything that involves a user knowing about MANIFEST at *all* (let alone
changing its name) is a "super custom" process. Which unfortunately means
that even for a fairly basic sdist, it's an issue.
>I agree that this feature is underdocumented, but changing
>the framework won't help with this: documentation patches
>is what we *really* need !
That's certainly part of it; but it's certainly not all of it. Mostly,
it's the difference between being a tool and a framework. The distutils
provides all the mechanism you like, and none of the policy. That was a
brilliant political move, as it dodges all the infighting that otherwise
would have stopped it getting off the ground. However, there's no longer
any reason *not* to have One Official Python Way To Do It, and to
streamline the tools around that philosophy.
One minor example of the problems with the distutils, is that their
flexibility and inconsistent documentation causes people to create wildly
different project source layouts, just because they can. You call that
flexibility, I call that bad design for the developers who don't want to
have to think about all these options (i.e., most of them).
>BTW, not many people need to have these MANIFEST files
>at all - distutils uses a built-in file finder based on these
> - README or README.txt
> - setup.py
> - test/test*.py
> - all pure Python modules mentioned in setup script
> - all C sources listed as part of extensions or C libraries
> in the setup script (doesn't catch C headers!)
It also leaves out documentation and package data, for starters.
>Carelessly overwriting hand-edited or otherwise generated
>files in a build process is simply bad design.
Not if it's a file *owned* by the distutils. If MANIFEST is assumed to be
some sort of possibly user-generated file, the distutils should've picked a
different name for *its* manifest file, and maybe provided the ability to
read other files to generate its manifest.
This is my point: there should be a manifest file that is wholly owned and
controlled by the distutils with *no user-serviceable parts
inside*. Having a file that is *partly* owned by the distutils and partly
by the user is a recipe for disaster and the real bad design in this case.
>MANIFEST is only used for source code distrbutions. I don't
>see how you can use it for anything else.
Because if you have package data files as part of your *source*, then the
MANIFEST has to list them in order to have a valid source
distribution. So, for some projects, it is equally valid to say, "be sure
to install any package data files listed in the MANIFEST", and that is what
the 'install_package_data' option would do. (Or maybe
'use_manifest_package_data' would be a better name, but you get the general
>I don't buy this: on one hand you are talking about simple
>packages (which don't need the MANIFEST files in the
Not true! You run into a problem the moment you have any documentation
besides README, or if you have any package data files.
>on the other about hooks to adjust distutils'
>build process, something I'd group under more complex
My point is that a well-designed system that offers customization hooks
should not offer you *meaningless* choices between different ways to
customize the same thing.
>So your point is to make your life as setuptools author
No, if that was my point I'd have just done whatever the heck I wanted and
not bothered posting to the list. :) My point is to get the *community*
better tools, and part of that is lowering the entry barriers to creating
them. If it was only me I cared about, I'd not have embarked on the entire
venture to begin with. Sheesh.
>Why don't you just disable all these options in your
>setuptools front-end and hard-code the MANIFEST file
I plan to! The point of my posting was to find out whether I was going to
be introducing any meaningful barriers to adoption of setuptools. For
example, if there were some bdist* command out there in widespread use that
would break as a result of these changes. Yes, I ranted a little about
making any "distutils 2" less flexible, but my practical present-day
assumption is that setuptools actually modifying the distutils is still a
long time coming. I don't even plan to *propose* that idea myself,
certainly not any time soon.
>distutils is a loosly coupled framework of components.
>In such a framework, a basic design principle is to
>be able to decouple and recouple existing components.
>The only way to implement this is by making the components
>suitably independent and this is what was done in
You're confusing a usability issue with a technical design issue. One
calls for TMTOWTDI (There's More Than One Way To Do It), the other for
TSBO-APOO-OWTDI (There Should Be One - And Preferably Only One - Obvious
Way To Do It).
>My point is that you shouldn't try to strip down distutils
>itself just because you think it's hard work to support
>all these features in setuptools. It's not needed to strip
>down distutils for this reason as you can easily disable
>these options for anyone using your setuptools.
>As a result, both users of setuptools and straight
>distutils are happy.
This was always my plan; the question was to find out whether there were
any holes in that plan, for *setuptools*. I'm sorry if that wasn't clear;
I only talked about removing these things in distutils under a hypothetical
"distutils 2" scenario. *None* of these changes were being proposed for a
"pure" distutils today.
That doesn't mean I don't think that the distutils design is too flexible
in terms of offering meaningless choices, or more precisely, offering lots
of choices to either shoot yourself in the foot. :) It just means that I
realistically understand that today's distutils are unlikely to be changeable.
Rather, I'm looking at the eventual transmutation of setuptools into
"distutils 2", and want to make sure that late adopters moving to it in the
distant future don't run into issues that haven't been accomodated in some way.
More information about the Distutils-SIG