[Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

[Donald Stufft]

On August 27, 2015 at 5:00:27 PM, M.-A. Lemburg (mal at egenix.com) wrote:
>  
> Uhm, no :-) This would be a more user friendly way of doing it:

Well, except we’d have to throw away all of the work we’ve done for discovering things up until that point so we’d need to essentially restart the entire process anyway, I don’t think that there has ever been any effort to make setuptools or pip re-rentrant in that regards. The mechanism in the older PEP 470 also supported more than one index to support people who wanted to host binary builds of their project in cases where the compatibility information in Wheel aren’t enough to adequately differentiate when wheels are compatible or not. If we’re going to support a discoverable index like that then we should support the other major reason people might not host on PyPI too, but that means we can’t automagically add things to the list of repositories because we don’t know which list of repositories is accurate.

The other problem is that end users should really be adding the configuration to their requirements.txt or other files in the situations they are using those so that they work in situations where they don’t have an interactive console to prompt them (for example, deploying to Heroku). If we’re automagically adding it to the list on a prompt then we make it less obvious they need to do anything else and we just push the pain off until they attempt to deploy their project.

Finally, additional repositories are similar to additional CAs in the browser ecosystem, you want to carefully consider which ones you trust because they get the ability to essentially execute whatever arbitrary code they want on your system. There *should* be some level of deliberation and thought behind a user adding a new repository. Allowing a new one with a simple prompt is as dangerous as a browser running into a HTTPS certificate it doesn’t trust and going “Well I see you don’t trust this CA, do you want to add it to your list and reload?”, a UI that most (or all) browsers are moving away from and hiding as much as possible.

>  
> All Linux distros I know and use have repositories distributed all
> over the planet, and many also provide official and less official
> ones, for the users to choose from, so there is more than enough evidence
> that a federated system for software distribution works better than a
> centralized one.
>  
> I wonder why we can't we agree on this ?

Sure, and people are more than welcome to not host on PyPI and all of the tools support a federated system. However those tools also don’t have any sort of meta links between repositories that will automatically suggest that you add additional repositories to your list. What you have configured is what you have configured. The latest update to PEP 470 represents moving to the exact same system that all the Linux distros you know and use have.

>  
> I'm happy to help write a PEP for the discovery feature and I'd also
> love to help with the implementation. My problem is that no one is
> paying me to work on this and so my time investment into this has
> to stay way behind of what I'd like to invest.

Sure, I mean I don’t expect other people to have near the amount of time I do, since my entire job is working on Python packaging. Part of why I’m bringing this up now instead of closer to when Warehouse is ready to launch is to give plenty of time for discussion, implementation, and migration.

>  
> No matter how much you try to get people to host everything
> on pypi.python.org, there are always going to be some which
> don't want to do this and rather stick with their own PyPI
> index server for whatever reason.

I don’t care if people host things off of PyPI, I just don’t think we need or should complicate the API to try and provide a seamless experience for people hosting things outside of PyPI. If you’re going off on your own you should expect there to be some level of “not on by default”-ness.

Honestly, If someone wanted to set up an additional repositor(y|ies) I wouldn’t even be personally opposed to adding it to the default list of repositories in pip assuming some basic guidelines/rules were followed. I don’t speak for all of the other maintainers so they might be opposed to it but I’d think something like:

* Is being operated by a known and trusted entity (e.g. Joe Nobody doesn’t get to do this).
* Agrees to consider PyPI the central authority for who owns a particular name (e.g. just because you host a repository doesn’t mean you get to make Django releases).
* Some plan for how they plan to operate it in regards to how they’ll keep the uptime high.

> I'd just remove the whole section. Splitting the user base into US
> and non-US users, even if just to explain that you cannot cover all
> non-US views or requirements is not something we should put into
> an official Python document.

Okay.

>  
> Since PyPI is legally run by the PSF, the PSF board will have to
> approve the new terms.
>  
> Having you on board for the WG, would certainly be very useful,
> since there may well be technical details that come into play.
>

Ok, sure sign me up.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA