On May 9, 2014, at 4:12 AM, M.-A. Lemburg email@example.com wrote:
On 08.05.2014 23:22, Donald Stufft wrote:
On a personal note, I'm uncomfortable with the way this change is perceived as a case of pip enforcing a behaviour that the pip developers feel should be required. I actually don't like this change particularly. So having pip implement the behaviour required by that PEP is to me simply a case of compliance with the agreed standard. But now, as a pip developer, being held responsible for the resulting user pain, and being expected to defend it, does not make me happy.
I think the pain is being overrepresented and the positives are being ignored. The problem is the benefits of this PEP are much like the benefits of TLS too. For the vast majority of people they don’t notice anything different except installing things is faster and more reliable. They don’t attribute that to the PEP or this decision, they just internalize it as the new norm. However the people who this does affect will seek out why it broke and raise an issue citing that thing specifically. This creates a perception of lots of pain for no gain when the reality is not that.
Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago.
Didn’t mean to imply that I thought it was otherwise.
However, I find it troubling that we as Python developers are forcing the whole Python world to put their code into PyPI.
Forcing is a strong word. As of right now we’re "forcing" you to put it onto PyPI if you want to install it without any flags to pip. You're more then capable of hosting it externally if you wish, and pip will even tell the people who try to install it what they need to enable in order to install it. It even allows people to say "I don't care about any of this crap, just make all the external stuff work".
Even if pip removed the external link handling all together and required you to do something like:
$ pip install --extra-index-url https://example.com/our-packages/ foo $ pip install --find-links https://example.com/our-packages/ foo
We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that.
There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details).
I re-read the reasons, I'm not sure I really agree with most (or all?) of them however if people really want to do it, then there is nothing stopping them. It's their choice and people on the other side of that who are installing these packages also get to make a choice if they want to allow it or not.
Accordingly, we should respect those reasons make it possible for Python packages to live elsewhere, without having our tools put those packages into a bad light or making it harder for Python users to install such packages than needed.
I'm not sure what "put it into a bad light" is supposed to mean here. Is it about the warning? I've already removed that completely.
As far as making it harder than it "needs" to be, that's a hard line to draw. I personally know people for whom things that are hosted externally have caused them a lot of pain to the point where they have to automatically spider PyPI for their own dependencies every hour or so in order to isolate themselves from the failure of random dependencies suddenly no longer being installable.
This breakage is going to happen for basically any project that installs their dependencies with any frequency. One of the big projects that I'm aware of that has had this problem is Openstack who, in their CI, installs things several hundred times a day. If a service they depend on goes down that causes significant disruption for them. Now they've solved it by the above solution of hosting their own mirror of PyPI that includes spidering for externally installable things. They have one or two packages left that don't host directly on PyPI at which point they'll no longer need that afaik.
I don't think telling every downstream of us of any size that in order to get reliable installs they are going to have to work around PyPI, and in some cases even develop custom software in order to do so effectively, is a very user friendly position.
With the checksum uploaded to PyPI, the only argument against fetching packages from other places on the Internet is reliability, not security.
I've never said differently. There are some minor privacy things in there but primarily it's about reliability and that people should generally be aware where their packages are coming from.
PyPI is not the only reliable file hosting system on the Internet, so this argument is rather weak.
It actually doesn't matter how reliable the other systems are. Reliability wise the best possible outcome is the external host has 100% (extremely unlikely) uptime and it has no affect on reliability. The realistic outcome is that the external hosts will not have 100% uptime and will infact decrease the total availability of the install sets on PyPI.
This isn't an opinion it is fact.
But in reality reliability also includes more things than just the server temporarily going down. It also includes things like the maintainer has quit maintaining that package, left Python all together, died even. There are a decent number of packages on PyPI that people use which have no active maintainer at all. If people rely on a package that isn't hosted on PyPI they take a larger risk that their dependencies are going to become uninstallable due to the above. When scanning the unverifiable links it is amazing how many of those links are now 404s or 500s or even "nodename not founds". All of these cause issues for people and are not hypothetical, I've dealt with people complaining about one or another of them.
When things are hosted on PyPI then we (we being the infra team) are able to
ensure that the likelihood of
pip install <something> actually works
continues to work. When things are hosted externally the best we can do is
say sorry. At that point the only major thing outside of our control that can
cause someones installations to suddenly start failing is if the package
author willfully removes their things from PyPI. That is unfortunate but it
is also their right and there's nothing to be done for that. Thankfully that
case is far less common than the case of "an external site just isn't there".
I think it's important to point out that one of the driving factors that caused
me to finally push for changes and what lead to PEP438 being created was that
Mercurial's external hosted was being extremely flaky. I can't remember the
exact details but I want to say that over the span of a week or two I was
getting massive numbers of users complaining that
pip install Mercurial
was suddenly failing. This isn't to knock on the Mercurial folks or anything
but to simply point out that these problems aren't things that just happen to
(under|un)maintained software nor are they hypothetical. This PEP was born of
the frustration that was being relayed to me by end users of PyPI/pip.
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA