[Distutils] some questions about PEP470

[holger krekel]

On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote:
> On 12 October 2014 09:49, Donald Stufft <donald at stufft.io> wrote:
> >
> > On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> >> On 12 October 2014 04:29, Donald Stufft <donald at stufft.io> wrote:
> >>>> I plan to put the external repositories (and the commands needed to use them)
> >>>> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more
> >>>> focused on defining the API differences and the changes.
> >>>
> >>> I forgot to mention, one other thing I thought about, we could move the current
> >>> external links to external repositories automatically. I’m not sure if this is
> >>> a good idea because on some projects that will end up with a giant list of things
> >>> that a user would be suggested to install. Perhaps it’d work to scan them for
> >>> installable files and only move the ones where an installable file was discovered,
> >>> though that could be error prone (for example right now bytereef.org is down, so
> >>> we’d not discover any files there).
> >>
> >> If I recall Holger's suggestion correctly, it was just to move the
> >> current "Download URL" links over, rather than the scraped links.
> >
> > Hmm, I thought his suggestion was to just leave the existing links alone so that the
> > existing pip/easy_installs would continue to function as they do now?
> 
> Sure, we could do that too. Regardless, I don't believe we should let
> the conversion of the silent security failures into noisy exceptions
> block the other usability enhancements - if decoupling the two
> decisions will achieve that, then let's go down that path.
> 
> I think I suggested at one point that the PEP could potentially commit
> to making the backwards compatible changes on the server side, and
> then for the *in*compatible changes, only commit to *looking at the
> numbers again* in a few months time after folks have had a chance to
> experience the replacement model. Client applications would be
> officially granted permission to ignore the PEP 438 external hosting
> mechanism in favour of the new PEP 470 external index advertisements.
> 
> The sequence of events from a compatibility break/user experience
> enhancement perspective would then be:
> 
> * new model added to PyPI (no compatibility break)
> * old model dropped from pip (client side compatibility break, error
> handling improvements)
> * old model dropped from PyPI (conditional, based on changes in server metrics)

I like this strategy and would add that I guess it's then only a matter of
when not if the old model is dropped (and the PEP could encode that).

The question is how to best do the "(almost) no compat break" for
a reasonable definition of "almost".  I am mostly concerned about
the verified externals and "crawl urls".  If they keep showing up as
"rel=download" links on the simple page that should be fine for existing
tools, right?

But with PEP470 they might get an additional attribute so that a newer
pip can recognize it and advise accordingly.  How pypi/warehouse
internally keeps the state during/post PEP470 is less clear to me but it
might involve a one-time conversion from the existing information to the
new one.

I am actually wondering if a new pip could change its UI even without
any changes to pypi.python.org? For example, when "pip install pil"
happens it would say:

     Could not find any release file for "Pil" on https://pypi.python.org 
     itself but the project has historically had additional external pages 
     which linked to release files:
     
        http://effbot.org/zon/pil-changes-115.html/
        http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz
        http://www.pythonware.com/products/pil

    Please manully check these links if they directly link to or contain
    release files for "Pil" and then add an appropriate one like this::
    
        pip install --find-links URL [other options] Pil

    (See <link> for more info ...)

and pip could deduce this information from the current https://pypi.python.org/simple/pil/ page.  (The pip message above may not be precisely right but
you get the idea).

FWIW this strategy of advising to use newer/other options but keeping
the old working (undocumneted or documented with a warning)
is how i generally try to handle things with pytest.  It allows
people to use the new way and incrementally migrate their setups.
And if someone complains about problems with the old ways, point them to
the new ways so there is no need to maintain the old ways too much.

Even then, it would probably still makes sense to advance the server
side and allow explicit registration of external index pages i guess
(and maybe findlinks pages so you can really just use apache/nginx
directly?) and then have pip provide a more precise message.

> That changes the dynamic from "the updated user experience will be
> better, trust us" to "if you prefer the old user experience, you can
> keep using the old pip, if you prefer the new user experience,
> upgrade".

I think it's very much fine to say "this is the future" as long as
old-tool/current-server instances keep working for a longer while.

Python packaging infra is a bit like a big container ship these days --
which is both a good and a bad thing i guess -- and people build
all kinds of funny containers on top.

best,
holger

> The eventual decision on when to drop the legacy model from PyPI can
> then by made based on a richer set of input data, that includes the
> level of adoption of the new version of pip that treats all missing
> link targets as potential errors.
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>