[Distutils] Deprecating little used file types/extensions on PyPI?

Wes Turner wes.turner at gmail.com
Tue Aug 23 12:45:16 EDT 2016

On Tuesday, August 23, 2016, Donald Stufft <donald at stufft.io
<javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:

> It’s hard to get per project numbers for these (or at least, it takes a
> more
> complex query than I can manage with my head here). However, I think it’s
> pretty telling that when you start looking at other formats, not only is
> the
> primary consumer tools that just indiscriminately download everything from
> PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those
> mirrors who
> follow vastly different usage patterns than what we see on PyPI itself,
> the primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume
> disk space
> and bandwidth via the mirroring infrastructure.

Other ways to reduce mirroring bandwidth:

Caching proxies:

- devpi (proxy cache only what is needed)
- "pypi proxy" https://www.google.com/search?q=pypiproxy

- nginx caching proxy
  - https://gist.github.com/dctrwatson/5785638
    - this would need to be updated for the pypi.org warehouse routes
    - it may be worth linking to these in the/some mirroring infrastructure

Pip --offline switch

- Internet was down the other day and i knew the packages were locally
cached but I couldn't figure out how to make pip work w/ local offline
dependency resolution

More efficient mirroring implementations:

- rsync
- bup

... delta downloads would be more efficient here:
"Add API endpoint to get latest version of all projects"

> I’d also like to note, that the numbers above are conservative on what they
> consider to be a “mirroring client”. For instance, devpi used to use the
> default
> requests user-agent, and we see downloads via the requests user agent, but
> did not
> count them as mirroring clients because it could be some other script
> doing the
> downloading.
>> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/ca53e2e3/attachment.html>

More information about the Distutils-SIG mailing list