[Distutils] PEP470, backward compat is a ...

[Donald Stufft]

On May 16, 2014, at 2:38 PM, Carl Meyer <carl at oddbird.net> wrote:

> On 05/16/2014 12:10 PM, Donald Stufft wrote:
>>> 2. Add a deprecation path for --allow-unverified; can describe it in
>>> general terms as "the PEP 438 installer flag allowing installation of
>>> unverified external packages" if you don't want to be pip-specific.
>>> Currently PEP 470 has no mention of this, but I think letting a
>>> deprecation of --allow-unverified fully run its course _before_ making
>>> breaking changes on the PyPI side is a critical part of making this
>>> transition in a user-friendlier way.
>> 
>> Perhaps I wasn't being as obvious as I thought! My goal was to nail down what
>> the final destination looked like, and then once we figured out an end goal
>> deprecate everything that doesn't match that end goal (and ultimately remove
>> it).
>> 
>> One thing I don't want to do is add another intermediary fix with it's own
>> combination of flags that will do the thing that the user wants to do. We
>> already have:
>> 
>> 1) No additional flags
>> 2) --allow-external + --allow-insecure
>> 3) --allow-external + --allow-unverified
>> 4) --allow-unverified
>> 
>> Depending on what version of pip you're using[1]. I really really want the 5th
>> incantation of this to be the final incantation. That was one of the reasons
>> why I went the way I did in PEP 470. I don't believe it's possible to move
>> away from these without a break in compatibility for <=7% of projects,
>> *however* a key benefit of PEP 470 is that the mechanisms for allowing
>> additional locations has existed in pip for a long time. We can have a singular
>> clear message that says "If you want to do X then use these flags" and it
>> doesn't matter what version you're on. I vastly prefer that to the current
>> situation (and the "just let the deprecation run it's course" proposal) where
>> you have to pick the right combination of flags based on pip version.
> 
> I guess the key thing I don't understand yet is, why would we assume
> that any package that hasn't already switched to verified-external-links
> since PEP 438, given a one-year window so far to do so, is any more
> likely to populate this new discoverable-index metadata from PEP 470,
> given a six-month window?
> 
> It seems like PEP 470 places a lot of weight on an assumption of active
> maintainers, when really the core problem is a significant chunk of
> packages that (from the evidence of PEP 438 uptake) don't have active
> maintainers.
> 
> So if we conclude that the bulk of the problematic legacy packages will
> probably never populate this new discoverable-index metadata either, at
> what stage in the process would their users get any useful clue as to
> how to continue to install that package?

Sometimes the answer is "it breaks".

To be specific I don't believe most of these packages are in active use.
It is my belief that a good chunk of them are vestigial appendages which a fear
of breaking them is preventing forward progress. Given that PyPI is a web
service and is version-less that mandates that sometimes things break. The
alternative is trying to maintain a union of every feature (and in some cases
every bug) that has ever existed from now into perpetuity.

The questions that need to be asked are:

1) Is this a feature that we want supported long term?
2) How many projects is this going to affect?
3) How many users is this going to affect?
4) Is there a way we can maintain some semblance of compatibility?
5) Is the answer to #4 worth it?

For me it's:

1) Not in this form, in the form of PEP 470 yes.
2) ~7% is the maximum possible impact, the real number will be lower.
3) I'm attempting to get some sort of "activity" measurement for this now.
4) Yesish
5) See below.

> 
> One option is Holger's solution: scraping the current links and
> populating them as verified external links. We both don't like this
> because it involves PyPI misleading users about the status of those
> links, and you also don't like it because you want to deprecate verified
> external links too.

Correct.

> 
> Another option is if the deprecation message for --allow-unverified also
> gives the user the exact --find-links URL(s) they need to install that
> same package/version (which I think is implementable in pip today
> without any changes in PyPI). The downside here is that it really
> doesn't improve the current UX for these legacy packages, it just
> replaces --allow-unverified with explicit --find-links: but at least the
> latter is more explicit and places the responsibility clearly on the
> external host, not PyPI.

It also doesn’t allow us to stop the crawling on PyPI (in fact it makes
it worse because we have to crawl to discover these links, instead
of being able to skip crawling if —allow-unverified isn’t selected).

> 
> Or, thirdly, Paul's proposal could solve this, if PyPI automatically
> generated an "external legacy index" for any packages that haven't
> generated their own external index URL by a certain date. Really in a
> way this is similar to Holger's proposal, except it uses
> external-indexes instead of verified-external-URLs, and is again a bit
> more explicit about what's going on (at the cost of requiring more
> adjustment from users).

It’s an interesting idea. I’d have to think about it. There is of course nothing
stopping anyone from doing this and shoving it on pythonhosted.org.

> 
> Basically, I think some acknowledgment of this problem of packages
> without active maintainers (and ideally a proposed solution to it)
> should be in PEP 470.

Right now the PEP's (and my) position is that it breaks because I believe that
the impact of this change is being overblown. I'm attempting to gather more
data now.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140516/f475f924/attachment-0001.sig>