[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Vinay Sajip]

> Correct me if I'm wrong, but I've a feeling you once said you'd tested

> distil against all the packages on PyPI (which is a mammoth task, so I
> could easily be wrong...) 

Not fully tested in the sense you mean - that *would* be a mammoth task :-)

However, I have tried to make declarative metadata from all releases on PyPI :-) and then distil uses that metadata to build / package to wheel, install etc. I use distil wherever possible, and outside of NumPy related work, it has worked as well as pip on the modest selection of PyPI projects that I use, and better than pip in the sense that I can see all dependencies needed before downloading anything.

Many packages presented no problems during smoke testing - e.g. Django (lots of package data), Jinja2, hiredis (C extension), Assimulo (Fortran extensions). By "no problems" I mean that the same files were installed by distil in the same places as was done by pip (IIRC - it was a while ago). But there are a lot of important packages that distil can't handle, such as NumPy, because they e.g. generate sources in their setup.py, which means that a purely declarative approach doesn't currently work. That's not to say that such an approach *couldn't* work - it most definitely could, with a sensible hook mechanism where needed for publisher-defined code, and a better approach for build metadata (still to be addressed) and installation metadata (which we have now got a good handle on). People have gone with setup.py as that's all there was, but that's just circumstance rather than a technically necessary approach. There are migration issues, of course, but I
 would guess that for the vast majority of packages, the auto-generated metadata approach that distil uses is good enough. Of course, more complex / important packages are exceptions, and would need more work (corresponding to their complexity) to fit into a declarative approach.

Regards,

Vinay Sajip