[Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

[holger krekel]

On Tue, Mar 12, 2013 at 15:21 -0400, PJ Eby wrote:
> On Tue, Mar 12, 2013 at 2:18 PM, Carl Meyer <carl at oddbird.net> wrote:
> > It seems to me that there's a remarkable level of consensus developing
> > here (though it may not look like it), and a small set of remaining open
> > questions.
> >
> > The consensus (as I see it):
> >
> > - Migrate away from scraping external HTML pages, with package owners in
> > control of the migration but a deadline for a forced switch, as outlined
> > in Holger's PEP (with all appropriate caution and testing).
> >
> > - In some way, migrate to a situation where the popular installer tools
> > install only release files from PyPI by default, but are capable of
> > installing from other locations if the user provides an option.
> 
> Perhaps I'm confused, but ISTM that every time I've said this, Donald
> and Lennart argue that it should not be possible to provide such an
> option -- or to be more specific, that PyPI should not publish the
> information that makes that option possible.
> 
> If that's *not* the position they're taking, it'd be good to know,
> because we could totally stop arguing about it in that case.

I don't know.  At least the pre-PEP doesn't take the position
to unconditionally ban external links.  Maybe Lennart or Donald can they
whether they oppose the moves outlined in the PEP.  I'd hope
that the perceived "perfect" doesn't become the enemy 
of the good here :)

> > A) Leave external links in the PyPI simple index, but migrate the major
> > tools to not use external links by default (i.e. Philip's plan to make
> > allow-hosts=pypi the default in a future setuptools), with an option to
> > turn them back on.
> 
> I don't know who has proposed this option, but it's not me.  You seem
> to be confusing external links and HTML-scraped links (rel=""
> attributed links in /simple).

The suggested behaviour of installers is not fully formulated yet in
the PEP.  We should improve that.

> I was the first person to propose disabling HTML-scraped links from
> PyPI *ASAP*.

Yes, and thanks for pushing us in this direction. 

> I still want them gone.  That won't require tool
> changes, it just requires a rollout plan.  Holger has one, let's work
> on that.
>
> The second thing I proposed is that new tools be developed to *assist*
> package authors in moving their files onto PyPI, so that future tool
> changes wouldn't result in widespread instances of people needing to
> set their tools to insecure settings just to get anything done.  We
> need to get people's files moving onto PyPI *first*, in order to make
> changing the tool defaults practical.

Indeed, it's a good idea to require the "re-hosting" or "transfer" tool ready
before installers change their defaults.

> The *only* thing I object to is the part where some people want to ban
> external links from /simple, always and forever, regardless of the
> package authors' choice in the matter.

I agree the package author should have a choice about the serving of links
for their package.  And installers should change defaults so that install-users have a choice as well, eventually, to control whether they are fine with
crawling or using external links.
 
> > B) Do a second PyPI migration, again with a per-package toggle and
> > package owners in control, to a "no external links in simple index" setting.
> >
> > Consider for a moment how similar the end state here is with either A or
> > B. In either case, by default users install only from PyPI, but by
> > providing a special option they can install from some external source.
> > (In B, that special option would be something like --find-links with a
> > URL). In either case, we can continue to allow packages to register
> > themselves on PyPI, be found in searches, etc, without uploading release
> > files to PyPI if they prefer not to; they'll just have to provide
> > special installation instructions to their users in that case.
> 
> Not true: approach B means that you won't know what values to pass to
> the option.

Yes and no: in the one case you need to specify "--crawl" or 
"--use-external-links" and in the other "--find-links https://..." 
The latter requires reading the homepage for the correct URL or 
long_description of a package so is less obvious to install-users.

> It's also confused about an important point.  All the links that
> appear in /simple are *already* completely under the package author's
> control.  No new switches are required to remove external links - you
> can simply remove them from your releases' descriptions.  This process
> could be made more transparent or easy, sure -- but it's a mistake to
> say that this is granting the package owners control that they don't
> already have.

Right.  I think allowing a package maintainer to say "actually, please don't
serve external links for my package" (hosting mode "pypi-only") is an
easy expressive way to exert this control.

> What they lack control over is the rel="" attributes, short of
> removing those links entirely.  That's why I've proposed having a
> switch for that , as reflected in Holger's pre-PEP.
> 
> 
> > 1) With B, we can provide a gentler migration for package owners, where
> > they are in control of when the switch happens.
> >
> > 2) With B, all end users benefit from the new defaults, not only end
> > users who update to the latest and greatest tools.
> >
> > 3) With B (and probably some forms of A as well), end users clearly
> > state which external sources they would like to trust and install from,
> > rather than having a global "trust everything!" flag, which is less
> > secure and less sensible.
> 
> These 3 statements all mischaracterize things substantially, because
> none of those benefits are exclusive to A, and nobody has proposed a

i guess you mean "B" here.

> "trust everything" flag.  Removing rel="" attributes also benefits
> everyone right away, *without* new tools.

Right.  I don't see much overall disagreement however ...
let's re-check once the next PEP draft is out :)

holger

> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>