[Distutils] PEP470, backward compat is a ...

[Carl Meyer]

On 05/16/2014 02:15 PM, Donald Stufft wrote:
>> I guess the key thing I don't understand yet is, why would we assume
>> that any package that hasn't already switched to verified-external-links
>> since PEP 438, given a one-year window so far to do so, is any more
>> likely to populate this new discoverable-index metadata from PEP 470,
>> given a six-month window?
>>
>> It seems like PEP 470 places a lot of weight on an assumption of active
>> maintainers, when really the core problem is a significant chunk of
>> packages that (from the evidence of PEP 438 uptake) don't have active
>> maintainers.
>>
>> So if we conclude that the bulk of the problematic legacy packages will
>> probably never populate this new discoverable-index metadata either, at
>> what stage in the process would their users get any useful clue as to
>> how to continue to install that package?
> 
> Sometimes the answer is "it breaks".

Depends what you mean by that. I think "it breaks at some point" is a
fine answer for these packages. But I think we can do better than "it
breaks without the user of the package ever getting a clue what is
happening, or how to continue to install that package in some other way".

> To be specific I don't believe most of these packages are in active use.
> It is my belief that a good chunk of them are vestigial appendages which a fear
> of breaking them is preventing forward progress. Given that PyPI is a web
> service and is version-less that mandates that sometimes things break. The
> alternative is trying to maintain a union of every feature (and in some cases
> every bug) that has ever existed from now into perpetuity.
> 
> The questions that need to be asked are:
> 
> 1) Is this a feature that we want supported long term?
> 2) How many projects is this going to affect?
> 3) How many users is this going to affect?
> 4) Is there a way we can maintain some semblance of compatibility?
> 5) Is the answer to #4 worth it?
> 
> For me it's:
> 
> 1) Not in this form, in the form of PEP 470 yes.
> 2) ~7% is the maximum possible impact, the real number will be lower.
> 3) I'm attempting to get some sort of "activity" measurement for this now.
> 4) Yesish
> 5) See below.
> 
>>
>> One option is Holger's solution: scraping the current links and
>> populating them as verified external links. We both don't like this
>> because it involves PyPI misleading users about the status of those
>> links, and you also don't like it because you want to deprecate verified
>> external links too.
> 
> Correct.
> 
>>
>> Another option is if the deprecation message for --allow-unverified also
>> gives the user the exact --find-links URL(s) they need to install that
>> same package/version (which I think is implementable in pip today
>> without any changes in PyPI). The downside here is that it really
>> doesn't improve the current UX for these legacy packages, it just
>> replaces --allow-unverified with explicit --find-links: but at least the
>> latter is more explicit and places the responsibility clearly on the
>> external host, not PyPI.
> 
> It also doesn’t allow us to stop the crawling on PyPI (in fact it makes
> it worse because we have to crawl to discover these links, instead
> of being able to skip crawling if —allow-unverified isn’t selected).

I don't think it does make it worse (or I wouldn't have proposed it).
Pip would only show a deprecation message for --allow-unverified if
--allow-unverified was used, meaning the crawling is already being done
anyway and pip already has those links. So I don't think this would
introduce any crawling that isn't happening now.

I'm not sure what you mean by "the crawling on PyPI" -- do you mean the
link-scraping done by PyPI itself, or the crawling that pip does at
install time? It's true that PyPI's link-scraping should continue to be
supported (for legacy projects only) until good uptake of a version of
pip that fully removed --allow-unverified, after the deprecation path.
But that's true regardless.

>> Or, thirdly, Paul's proposal could solve this, if PyPI automatically
>> generated an "external legacy index" for any packages that haven't
>> generated their own external index URL by a certain date. Really in a
>> way this is similar to Holger's proposal, except it uses
>> external-indexes instead of verified-external-URLs, and is again a bit
>> more explicit about what's going on (at the cost of requiring more
>> adjustment from users).
> 
> It’s an interesting idea. I’d have to think about it. There is of course nothing
> stopping anyone from doing this and shoving it on pythonhosted.org.

The part that not anyone could do would be auto-populating the
discoverable external-index-url metadata with this auto-generated index
url, for inactive projects. That would require PyPI admin intervention.
That part is key, because it's the only way the user of such a package
ever finds out about this new external index for it.

>> Basically, I think some acknowledgment of this problem of packages
>> without active maintainers (and ideally a proposed solution to it)
>> should be in PEP 470.
> 
> Right now the PEP's (and my) position is that it breaks because I believe that
> the impact of this change is being overblown. I'm attempting to gather more
> data now.

You could be right. More data would certainly be good.

Thanks for all your work on all this stuff!

Carl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140516/7e4657a5/attachment.sig>