[Distutils] Python people want CPAN and how the latter came about

[Tarek Ziadé]

On Mon, Dec 21, 2009 at 11:48 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro <regebro at gmail.com> wrote:
>> What nobody still fails to explain in this discussion is what CPAN
>> "is" and Why Python doesn't already have it.
>
> That's not the right question to ask. The problem is not much a
> feature problem as much as a fundamental implementation and "state of
> mind". Reliable packaging requires explicit handling, where the whole
> python stack for packaging relies a lot on implicit behavior. I don't
> know much about CPAN, but both CRAN ("CPAN for R") and hackage ("CPAN
> for haskell") work reliably where Pypi often fails, and the reasons
> are easy to see:
>
>  - distutils (and most tools on top) throw away metadata, and then
> other tools try to guess them back. That alone is source of numerous
> errors, weird hacks, and unexplainable issues. Generally, when there
> is a design error at some level, instead of fixing it at this level,
> tools try to add another level of complexity to circumvent it. The
> fact that python has something like 5 or 6 tools to deploy things
> where most other languages have only one or two is quite striking.

I think you are quite confused here. The "metadata", as defined in PEP 314,
are not thrown away by Distutils, but properly uploaded at PyPI.

Other tools like Setuptools created their own metadata extension, that were
added in extra files when distributions are created and uploaded.

This is not a *design error*, this is called *innovation*.

And PEP 345 (the new metadata v1.2) is here to catch up with these changes
that are widely used by the community, and will eventually make it simpler for
all tools to get these new metadata.

http://www.python.org/dev/peps/pep-0345/


>  - Linked to the above, metadata are not enforced in pypi. This just
> cannot work. Most other systems in existence enforce strict rules.

I am not sure what you mean by enforcing strict rules. You need to be
more precise.

What do you want to do ? reject any package that doesn't meet some QA tests ?
Is PyPI is a community repository, or an entreprise repository ?


>  - Most crucial stages (install by distutils/setuptools, retrieving
> metadata, file formats like eggs) are not specified, and
> implemented/executed in an ad-hoc fashion. As a result, easy_install
> often fails to install things correctly, and the errors are hard to
> recover for people not familiar with python. For example, when you
> install a package, the previous installation is not removed. Worse,
> there is no way to correctly uninstall things, because the way
> distutils install things is dumb (dumping whatever is in the build
> directories instead of building a precise list of what is to be
> installed). This alone is a constant source of issues for us with
> numpy and scipy.

I would have understood this frustration a year ago, but now, that's
just a free rant/flame  :)

You can't ignore the work we are doing to address these issues

   http://www.python.org/dev/peps/pep-0376/

Now, what would really help, is having you (or anyone else) help around in these
PEPs, rather than ranting about packaging like that from time to time.

Regards
Tarek