[Catalog-sig] Deprecation of External Urls, Statistics

[M.-A. Lemburg]

On 08.03.2013 02:40, Donald Stufft wrote:
> So I updated my script (had to remove eventlet) and I believe it's now accurate. The total time was ~54 hours so this is hardly scientific but it should give a good idea what sort of impact we are talking about.
> 
> This is a list of versions that pip's PackageFinder (what it uses to locate packages to install) could find that were not available on PyPI.
> 
> The results and script is available at: https://gist.github.com/dstufft/5088915
> 
> Some statistics:
> 
>     Projects affected (with dev): 2269
>     Versions affected (with dev): 8006
> 
>     Projects affected (without dev): 1880
>     Versions affected (without dev): 7586
> 
> These numbers are if all external urls were immediately removed from PyPI, so this would be the total affected. This does not test if the actual package is installable, just if pip is able to locate an url that it thinks represents a version for that project.

Thanks for running the test.

About 10% of all packages. The numbers are already impressive,
but if you factor in the popularity of some of those
packages, the situation becomes worse.

I'm beginning to wonder whether caching the external link content
on the PyPI CDN wouldn't be a better idea.

We'd have to make that legally waterproof and also have an opt-out
mechanism, but it would get us from here to there a lot faster.

Together with the added hash tag on the download file URLs (*),
this would solve the availability and the security aspects.
Instead of deprecating external links altogether, we could then
deprecate non-compliant download links and get an overall
very flexible system for Python package distribution.

(*) Yes, I know, I still have to deliver the updated proposal -
been working on getting our indexes ready to serve as example :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 07 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/