[Distutils] Python people want CPAN and how the latter came about

[David Cournapeau]

On Tue, Dec 22, 2009 at 9:04 AM, Tarek Ziadé <ziade.tarek at gmail.com> wrote:
> On Mon, Dec 21, 2009 at 11:48 PM, David Cournapeau <cournape at gmail.com> wrote:
>> On Mon, Dec 21, 2009 at 7:13 PM, Lennart Regebro <regebro at gmail.com> wrote:
>>> What nobody still fails to explain in this discussion is what CPAN
>>> "is" and Why Python doesn't already have it.
>>
>> That's not the right question to ask. The problem is not much a
>> feature problem as much as a fundamental implementation and "state of
>> mind". Reliable packaging requires explicit handling, where the whole
>> python stack for packaging relies a lot on implicit behavior. I don't
>> know much about CPAN, but both CRAN ("CPAN for R") and hackage ("CPAN
>> for haskell") work reliably where Pypi often fails, and the reasons
>> are easy to see:
>>
>>  - distutils (and most tools on top) throw away metadata, and then
>> other tools try to guess them back. That alone is source of numerous
>> errors, weird hacks, and unexplainable issues. Generally, when there
>> is a design error at some level, instead of fixing it at this level,
>> tools try to add another level of complexity to circumvent it. The
>> fact that python has something like 5 or 6 tools to deploy things
>> where most other languages have only one or two is quite striking.
>
> I think you are quite confused here.

I really don't think I am.

> The "metadata", as defined in PEP 314,
> are not thrown away by Distutils, but properly uploaded at PyPI.

But there is much more than what PEP 314 defines. Think about
everything which is in an egg, for example. How are files locations
are described, how are files outside site-packages described, etc...
As a concrete example, there is not enough metadata in every installer
so that you can convert from one to the other, even though this would
be very useful and definitely possible.

So it is not just metadata at the "input" stage (although there is
certainly a lot of things lost there too), but how they are
represented internally, and how they are represented in the various
installers. This matters a lot for people who extend distutils.

>
>>  - Linked to the above, metadata are not enforced in pypi. This just
>> cannot work. Most other systems in existence enforce strict rules.
>
> I am not sure what you mean by enforcing strict rules. You need to be
> more precise.

For example, you can upload a distribution which does not even have a
name, or a version, easy_install tries to find tarballs by scrapping
webpages, etc...

>
> What do you want to do ? reject any package that doesn't meet some QA tests ?
> Is PyPI is a community repository, or an entreprise repository ?

That's common sense to reject packages which do not define a basic set
of metadata, and this has nothing to do with enterprise vs community.
Debian, for example, enforces metadata in its package, and it does not
qualify as enterprise software to me if the term is meant as the
opposite of community.

Those metada are needed, because other tools do use them (e.g.
easy_install, and I guess pip as well).

> I would have understood this frustration a year ago, but now, that's
> just a free rant/flame  :)

I have tried to explain several times about describing how files are
installed where, and everytime I was shut down (not by you). In
particular, I think I have explained already why I believe PEP 376
does not solve the issue of interoperation with other installers
because it loses source/target directory information.

David