[Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting

[holger krekel]

On Sat, Oct 04, 2014 at 00:24 +1000, Nick Coghlan wrote:
> On 3 October 2014 22:02, Donald Stufft <donald at stufft.io> wrote:
>
> > As far as simplication goes, I don't believe it simplifies the implementation
> > of PyPI at all, it just shuffles things around and creates work on my part
> > in order to get PyPI supporting the new stuff. It does however let installers
> > become simpler and it enables installers to present accurate error information
> > that actually helps determine the root cause of a failure instead of the
> > current silent failure with a confusing error message model.
> >
> > I look forward to your suggestions, but I'm not hopeful. I've been thus far
> > unable to determine a way to improve the current solution in a way that isn't
> > just papering over one problem without solving the fundamental issue.
> 
> Donald's perspective here matches my own. 

I don't see the "the fundamental issue" that PEP470 tries to solve.
The first para of the abstract says it wants to substitute the existing
mechanism for registering external indexes with another one.  It doesn't
say why.  And it doesn't say why this can't be done in a backward
compatible manner which would be preferable (i hope we agree there).

And because the PEP doesn't precisely say what "fundamental issue"
it solves it's a bit hard to present an alternative.  If it's about
focusing on "multi-repository operations" and simplifying installer UI
it could be done with full backward compat:

- add PyPI maintainer UI to add external indexes along with a message
  
- change pip to disallow crawling to an external index it finds
  but rather present a message that you need to add the index 
  manually to your installer invocation. (pip already finds external
  crawl URLs and it can also find the "new" ones - no need for
  any breakage).
  
- tell all project maintainers which have "explicit file urls" 
  that they need to move their release files to an offsite
  own external index (or to pypi itself) within N months. 
  Then disable the file urls (after examination of how many people
  are effected) and remove related un-needed options in pip.

Of course, i leave out some details but overall think it's pretty much
doable.  With this strategy, both old and new versions of pip wold work
fine with the changed PyPI.  It also wouldn't introduce very complicated
transition phases or communication steps.

I postpone other issues with respect to clarity and security of
PEP/multi-repo operations to first get clarity on the backward compat
issue and general strategy.

best,
holger

P.S.: Nick, i think my rough draft above satisfies all of your points
below, although they only partly relate to what we discuss in the PEP
IMHO.


> I'll be interested to hear alternative proposals, but they should aim
> to address at least the following user experience expectations:
> 
> 1. Easily allow external hosting to "just work" when appropriately
> configured at the system, user or virtual environment level (pip
> already supports this at the user level, and will support it at the
> system and environment level in the next version).
>
> 2. Easily allow package authors to tell PyPI "my releases are hosted
> <here>" and have that advertised in such a way that tools can clearly
> communicate it to users, without silently introducing unexpected
> dependencies on third party services.
> 
> 3. Eliminate any and all references to the confusing "verifiable
> external" and "unverifiable external" distinction from the user
> experience (both when installing and when releasing packages).
> 
> 4. The repository aspects of PyPI should become *just* the default
> package hosting location (i.e. the only one that is treated as opt-out
> rather than opt-in by most client tools in their default
> configuration). Aside from that aspect, hosting on PyPI should not
> otherwise provide an enhanced user experience over hosting your own
> package repository.
> 
> 5. Do all of the above while providing default behaviour that is
> secure against most attackers below the nation state adversary level.
> 
> In my view, the most debatable part of Donald's latest proposal would
> be the handling of projects that don't get updated to properly
> register an external URL before the link spidering support is removed
> from the client applications. That aspect should arguably include a
> step where the decision on whether or not to disable that support is
> based on *looking at the numbers again* before turning the feature off
> on the server, and perhaps also monitoring for user complaints for a
> period after it is first turned off, before the feature is removed
> from the clients.
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>