[Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

Thu Aug 27 23:00:13 CEST 2015

On 27.08.2015 13:51, Donald Stufft wrote:
> On August 27, 2015 at 6:57:17 AM, M.-A. Lemburg (mal at egenix.com) wrote:
>> This feature was part of a compromise to reach consensus on the removal
>> of external hosting. While I don't think the details of the repository discovery
>> need to be part of PEP 470, I do believe that the PEP should continue
>> to support the idea of having a way for package managers to easily
>> find external indexes for a particular package and not outright reject it.
>>  
>> Instead, the PEP should highlight this compromise and defer it to a
>> separate PEP.
> 
> I’ve never thought that particular API was actually a good idea, I think it’s a poor end user experience because it invokes the same sort of “well if you knew what I needed to type to make it work, why didn’t you just do it for me” reaction as PEP 438 does. The user experience will be something like:
> 
> $ pip install something-not-hosted-on-pypi
> ...
> ERROR: Can not find something-not-hosted-on-pypi, it is not hosted on PyPI, it's author has indicated that it found at:
> 
> * https://pypi.example.com/all/ : All Platforms
> * https://pypi.example.com/ubuntu-trust/ : Ubuntu Trusty
> 
> To enable, please invoke pip by added --extra-index-url <An URL from Above>

Uhm, no :-) This would be a more user friendly way of doing it:

$ pip install something-not-hosted-on-pypi
...
I'm sorry, but we cannot find something-not-hosted-on-pypi on the available
configured trusted indexes:

* https://pypi.python.org/

However, the author has indicated that it can be found at:

* https://pypi.example.com/

Should we add this PyPI index to the list of trusted indexes ? (y/n)
>>> y

Thank you. We added https://pypi.example.com/ to the list of indexes. You
are currently using these indexes as trusted indexes:

* https://pypi.python.org/
* https://pypi.example.com/

We will now retry the installation.

...

something-not-hosted-on-pypi installed successfully.
$

> $ pip install --extra-index-url https://pypi.example.com/all/ something-not-hosted-on-pypi
> 
> This leaves the user feeling annoyed that we didn’t just search those locations by default. I truly think it is a bad experience and I only ever added it because I wanted the discussion to be over with and I was trying to placate people by giving them a bad feature.

There's nothing bad in the above UI. A user will go through the discovery
process once and the next time around, everything will just work.

> Instead, I think that we can design a solution that works by default and will work without the end user needing to do anything at all. However, I’m not an expert in the US laws (the country I live in and have lived in all my life) and I’m especially not an expert in the laws of countries other than the US. This means I don’t fully understand the issue that needs to be solved. In addition to that, I only have so many hours in the day and I need to find a way to prioritize what I’m working on, the data sovereignty issue may only affect people who do not live in the US, however it does not affect everyone who is outside of the US. Most projects from authors outside of the US are perfectly fine with hosting their projects within the US and it is a minority of projects who cannot or will not for one reason or another.

All Linux distros I know and use have repositories distributed all
over the planet, and many also provide official and less official
ones, for the users to choose from, so there is more than enough evidence
that a federated system for software distribution works better than a
centralized one.

I wonder why we can't we agree on this ?

> I am happy to work with someone impacted by the removal of offsite to design and implement a solution to these issues that provides an experience to those people that matches the experience for people willing or able to host in the US. If the PSF wants to hire someone to do this instead of relying on someone affected to volunteer, I’m also happy to work with them. However, I do not think it’s fair to everyone else, inside and outside of the US, to continue to confuse and infuriate them while we wait for someone to step forward. I’m one person and I’m the only person who gets paid dedicated time to work on Python’s packaging, but I’m spread thin and I have a backlog a mile long, if I don’t prioritize the things that affect most people over the things that affect a small number of people, and leave it up to the people who need an edge case feature things that are already blocked on me are going to languish even further.

I'm happy to help write a PEP for the discovery feature and I'd also
love to help with the implementation. My problem is that no one is
paying me to work on this and so my time investment into this has
to stay way behind of what I'd like to invest.

> Finally, I wasn’t sure if this should be a new PEP or if it should continue as PEP 470, I talked to Nick and he suggested it would be fine to just continue on using PEP 470 for this.

If you don't feel comfortable with the discovery feature,
I think it's better to split it off into a separate PEP.

>> More comments:
>>  
>> * The user experience argument you give in the PEP 470 for rejecting the
>> idea is not really sound: the purpose of the discovery feature is to
>> provide a *better user experience* than an error message saying that a package
>> cannot be found and requiring the user to do research on the web to find
>> the right URLs. Package managers can use the information about the
>> other indexes they receive from PyPI to either present them to the user
>> to use/install them or to even directly go there to find the packages.
> 
> It’s a slightly better user experience than a flat out error yes, but it’s a much worse user experience than what people using PyPI deserve. If we’re going to solve a problem, I’d much rather do it correctly in a way that doesn’t frustrate end users and gives them a great experience rather than something that is going to annoy them and be a “death of a thousand cuts” to people hosting off of PyPI. I think that the compromise feature I added in PEP 470 will be the same sort of compromise feature we had in PEP 438, something that on the tin looks like a compromise, enough to get the folks who need/want it to think the PEP is supporting their use case, but in reality is just another cut in a “death of a thousand cuts” to hosting outside of the US (or off of PyPI).

I think you are forgetting that the worst user experience is one where
users are left without installable Python packages :-)

No matter how much you try to get people to host everything
on pypi.python.org, there are always going to be some which
don't want to do this and rather stick with their own PyPI
index server for whatever reason.

> I don’t want to continue to implement half solutions that I know are going to be painful to end users with the idea in mind that I know the pain is going to drive people away from those solutions and towards the one good option we have currently. I’d rather be honest to everyone involved about what is truly supported and focus on making that a great experience for everyone.
>
>>  
>> * The section on data sovereignty should really be removed or reworded.
>> PEPs should be neutral and not imply political views, in particular not
>> make it look like the needs of US users of PyPI are more important then
>> those of non-US users. Using "poor user experience" as argument here is
>> really not appropriate.
> 
> I’m perhaps not great at wording things. I don’t think it’s US users vs Non-US users, since plenty of people outside of the US are perfectly happy and able to upload their projects to PyPI. I think it’s more of “the needs of the many outweigh the needs of the few”, but with an explicit callout that if one of those few want to come forward and work with me, we can get something in place that really solves that problem in a user friendly way.
> 
> Perhaps you could suggest a rewording that you think says the above? I don’t see a political view being implied nor do I see the needs of US users being prioritized over the needs of non-US users.

I'd just remove the whole section. Splitting the user base into US
and non-US users, even if just to explain that you cannot cover all
non-US views or requirements is not something we should put into
an official Python document.

>> PyPI is a central part of the Python community infrastructure and
>> should be regarded as a resource for the world-wide community.
>> There is no reason to assume that we cannot have several PyPI
>> installations around the world to address any such issues.
> 
> I don’t assume that we can’t do something like that, one of my ideas for solving this issue looks something like that in fact. However without someone who cares willing to step forward and bring to the table an expertise in what will satisfy those legalities or not and with a willingness to pitch it and contribute to help make such a solution a reality I don’t feel comfortable spending any time a solution that may not even actually solve the problem at hand.
> 
> I don’t think the solution that was in the PEP is that solution though. I think it was a poison pill that I fully expected to have a terrible experience which would just force people to host on PyPI or have their project suffer. Repositories hosted by random people end up making people’s installs extremely unreliable. We’ve made great strides in making it so that ``pip install <foo>`` rarely fails because something is down, and I think blessing a feature or continuing to support one that doesn’t aid in making that more the case is harmful to the community as a whole.
> 
> Honestly, if someone comes forward now-ish or the PSF tells me now-ish they will hire someone to figure out the intricacies of this matter, I’ll even put this PEP back on hold while we design this feature. I have no particularly animosity towards hosting outside of the US, I just don’t have the expertise or time to design and implement it on my own.

Fully understood, alas, I don't have more cycles to spare to help
with designing a full blown distributed PyPI system on my
free time.

As for the PSF: the situation is somewhat similar. The money
is available for such projects, but we don't have enough people
to provide management and oversight :-( While we're slowly working
on changing this, it won't happen over night.

>> * It is rather unusual to have a PEP switch from a compromise solution
>> to a rejection of the compromise this late in the process.
>>  
>> I will soon be starting a PSF working group to address some of
>> the reasons why people cannot upload packages to PyPI. The intent
>> is to work on the PyPI terms to make them more package author
>> friendly. Anyone interested to join ?
> 
> I don’t have any particular insight into the ToS nor do I really care what it says as long as it grants PyPI everything it needs to function. I should probably be a part of the WG though since it involves PyPI and I’m really the only person working on PyPI. I wouldn’t want to adopt a ToS for PyPI without Van’s approval though.

Since PyPI is legally run by the PSF, the PSF board will have to
approve the new terms.

Having you on board for the WG, would certainly be very useful,
since there may well be technical details that come into play.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 27 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-08-27: Released eGenix mx Base 3.2.9 ...     http://egenix.com/go83
2015-08-19: Released mxODBC 3.3.5 ...             http://egenix.com/go82

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/