[Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

[holger krekel]

On Tue, Mar 12, 2013 at 12:18 -0600, Carl Meyer wrote:
> It seems to me that there's a remarkable level of consensus developing
> here (though it may not look like it), and a small set of remaining open
> questions.
> 
> The consensus (as I see it):
> 
> - Migrate away from scraping external HTML pages, with package owners in
> control of the migration but a deadline for a forced switch, as outlined
> in Holger's PEP (with all appropriate caution and testing).
> 
> - In some way, migrate to a situation where the popular installer tools
> install only release files from PyPI by default, but are capable of
> installing from other locations if the user provides an option.
> 
> The open question is basically how to implement the latter portion. I
> see two options proposed:
> 
> A) Leave external links in the PyPI simple index, but migrate the major
> tools to not use external links by default (i.e. Philip's plan to make
> allow-hosts=pypi the default in a future setuptools), with an option to
> turn them back on.
> 
> or
> 
> B) Do a second PyPI migration, again with a per-package toggle and
> package owners in control, to a "no external links in simple index" setting.
> 
> Consider for a moment how similar the end state here is with either A or
> B. In either case, by default users install only from PyPI, but by
> providing a special option they can install from some external source.
> (In B, that special option would be something like --find-links with a
> URL). In either case, we can continue to allow packages to register
> themselves on PyPI, be found in searches, etc, without uploading release
> files to PyPI if they prefer not to; they'll just have to provide
> special installation instructions to their users in that case.
> 
> Here are some differences:
> 
> 1) With B, we can provide a gentler migration for package owners, where
> they are in control of when the switch happens. With A, regardless of
> how it's done at some point some package owners are likely to start
> getting "hey, i can't install your stuff anymore" reports from users,
> and they can't control when that starts happening.
> 
> 2) With B, all end users benefit from the new defaults, not only end
> users who update to the latest and greatest tools.
> 
> 3) With B (and probably some forms of A as well), end users clearly
> state which external sources they would like to trust and install from,
> rather than having a global "trust everything!" flag, which is less
> secure and less sensible.
> 
> It seems to me that option B (a controlled, per-package, PyPI migration
> to no-external-links in simple index) is a better migration path than A
> (leaving it up to external tools), and the end result either way is very
> similar.

Thanks for outlining this so well.  I agree with the B approach and
suggest to introduce three per-package hosting-states then:

- pypi-only: only pypi-hosted files and all #egg links are served via simple/
  (#egg links are necccessary and a special case for installing
  development snapshots - we should not leave them out i think)

- pypi-nocrawl: all links as of know but without rel-attribution (i.e.
  all description links are served and also the homepage/download ones but
  without rel-attribution)
   
- pypi-crawl: all links as of know

The automatic transition of the hosting-mode for most packages (with
pre-announcements) specified in the PEP will need to differentiate
between switching to pypi-only and pypi-nocrawl.  

And as discussed elsewhere, the implementation of the underlying
analysis script and the PYPI changes certainly needs to be ready 
before the PEP can be finally accepted.

Am open to an according PR to the PEP-draft :)

holger


> 
> Carl
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>