[Distutils] Python people want CPAN and how the latter came about

David Cournapeau cournape at gmail.com
Tue Dec 22 14:02:12 CET 2009


On Tue, Dec 22, 2009 at 4:43 PM, Lennart Regebro <regebro at gmail.com> wrote:
> On Mon, Dec 21, 2009 at 23:48, David Cournapeau <cournape at gmail.com> wrote:
>> mind". Reliable packaging requires explicit handling, where the whole
>> python stack for packaging relies a lot on implicit behavior.
>
> Can you give examples of this implicit behaviour?

>From the top of my head:
 - how data files are included
 - how sdist is produced (a lot of sdist-generated tarballs contain
lot of junk like *.pyc or egg-info). I think we already had this
discussion, and that I was the only one who thought this was a
problem.
 - how files are installed: distutils just copy whatever is in the
build directory for at least some files, so you cannot easily check
which files are installed - hence my proposal on a build manifest to
make this step explicit. This is maybe the number one issue we have in
numpy/scipy where people unfamiliar with distutils warts try to
build/install numpy/scipy.

> It's not worse, it's the same.But it's admittedly a big lack of
> feature in the current Python situation, yes. But it's also not a
> question of "CPAN", is it? CPAN doesn't uninstall things.

Yes, CPAN does not uninstall things, which is why I was making the
difference betwen explicitely uninstally things and (implicitely)
uninstalling beforere reinstalling things. The former would not be
done though CPAN, the later would.

>
> Up until this, you wore concrete and could explain exactly what you
> didn't like. You are partly incorrect and for some reason very angry,
> which is nonconstructive but beside the point. Your claim that it's
> not of features and concrete problems but a "state of mind" was
> countered by your concrete answers. ;-)

Features only help as much as long as you agree on the purpose (or
"state of mind" as I put it). A lot of things I consider misfeatures
in the whole packaging ecosystem in python are justified by being
convenience to the developer, at the cost of reliability (at least in
my opinion).

Data files is my favorite example, as it is deceptively simple
feature-wise, and yet so confusing. We had 2 emails in the recent days
on this ML on the topic, and even people very familiar with distutils
got it wrong. Just considering numpy.distutils/distutils/setuptools,
there are 5 ways to handle data files at least (data_files,
package_data, MANIFEST[.in], VCS, add_data_dir). None of them makes it
easy to install a data file in arbitrary location. I think
distutils/setuptools lacks a robust way of including data files, but
it is not as much a feature problem (which could be solved relatively
easily), as more a problem of what people want.

The devil is really in the details for packaging, which is why
explicit is a very desirable feature. Someone noticed earlier that
distutils was the exact contrary of the python zen on some core
issues: more than one way to do it, lack of simplicity, often
implicit. I think this describe the problem quite well, and I am
deeply convinced that that's the root cause of the failure of Pypi for
a subset of the community.

David


More information about the Distutils-SIG mailing list