On May 9, 2014, at 9:58 AM, M.-A. Lemburg firstname.lastname@example.org wrote:
On 09.05.2014 13:44, Donald Stufft wrote:
On May 9, 2014, at 4:12 AM, M.-A. Lemburg email@example.com wrote:
Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago.
Didn’t mean to imply that I thought it was otherwise.
However, I find it troubling that we as Python developers are forcing the whole Python world to put their code into PyPI.
Forcing is a strong word. As of right now we’re "forcing" you to put it onto PyPI if you want to install it without any flags to pip.
Which is what most people expect to be able to do.
Sure, but sometimes it’s better to make an informed choice about things being installed instead of having it be installed by default and sometimes it’s better to disallow doing something at all.
Further most people don’t expect an install to touch any server host other than PyPI. As far as I’m aware none of the other package repositories feature this, and even with the fact we support it barely anyone even cares to utilize it.
You're more then capable of hosting it externally if you wish, and pip will even tell the people who try to install it what they need to enable in order to install it. It even allows people to say "I don't care about any of this crap, just make all the external stuff work".
Even if pip removed the external link handling all together and required you to do something like:
We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that.
This is all true, but in reality, you are making externally hosted packages second class citizens in Python land by requiring extra options even for package listings that are perfectly safe to download from other servers.
As mentioned before: I can understand that you want to make downloads safe for users, but if a package is hosted on some other reliable servers and pip can check that it's a valid package, there's little to argue for not allowing such downloads.
Except the fact that the only people I’ve ever seen happy that people can host packages externally are a handful (<5) of people (tbh primarily you and Stefan) and the feedback I get from every other person around the web in unequivocally yes please get rid of externally hosted files, they’ve done nothing but cause me pain.
It’s not reasonable to allow a minority of users to negatively impact the majority.
There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details).
I re-read the reasons, I'm not sure I really agree with most (or all?) of them however if people really want to do it, then there is nothing stopping them. It's their choice and people on the other side of that who are installing these packages also get to make a choice if they want to allow it or not.
You don't have to agree with those reasons. Fact is, they exist, prevent package authors from uploading to PyPI and we as Python developers should respect those reasons.
No I disagree that they are reasons at all. Half of the ones you enumerated are nonsensical or irrelevant, the other half can be fixed by adding features to or fixing PyPI. One or two read like reasons why someone might actually decide not to upload something to PyPI but which that reason isn’t really all that reasonable and finally a grand total of one or two of them look like an actual legit reasons and it only applies to Crypto software.
I mean your reasoning included things like:
PyPI doesn’t let you upload two files with the same name, you gave the example of UC2 vs UC4
The thing being if PyPI did allow you to do that then we’d have no way to determine which one is the right one and half of the people would just get randomly failing installs because we guessed the wrong one.
They don’t know it’s possible to upload to PyPI
I don’t even know how to response to this one.
PyPI used to be unreliable but isn’t now and they don’t know about it
Nor this one.
Not wanting people to download a package at all and instead check out from a VCS repo
This’ll randomly fail for people depending on if they happen to have that VCS installed. Also I don’t see any evidence that people are actually doing this outside of supporting a foo==dev install which pip won’t install by default either.
Not wanting to add latency for yourself for downloading from PyPI
Instead you’re going to inflict extra latency on everyone else when you could just run a mirror or upload the packages to your own server.
File types that PyPI doesn’t support and which utilize a third party package format, PyPM being a given example.
Seriously? Should an apt repo support external links because you can’t install RPMs with apt?
Wanting people to have to read something before installing the package because it relies on some additional setup.
This is completely nonsensical if they want people to have to read something before they attempt to pip install it then they don’t want pip to discover the external file at all then.
It currently takes too long uploading that many files to PyPI. This causes a problem, since in order to start the upload, we have to register the release on PyPI, which tools will then immediately find. However, during the upload time, they won't necessarily find the right files to download and then fail.
So if this is an actual problem file a bug and we can make it so PyPI lets you upload things as “drafts” and then you can publish them all at once or some such thing.
having a strong need to know the number of downloads per package and associated statistics such as downloads per country, per year/month/day/hour
If additional statistics are desired for PyPI file a bug and we can add them.
not wanting to give up access to the download log files
This appears to be repeating the last reason with different wording.
the PyPI uploads not being compatible to their release process
I’m not sure how it’s possible to have a release process where it isn’t compatible to upload to an additional server. Maybe ones that don’t currently do it but ones that are fundamentally incompatible?
Almost all of your “reasons” read like someone who maybe had one corner case where it might have made sense but didn’t want to admit that it was a corner case so tried to think up any random reason they could think of in order to make the list look longer and more impressive than it actually was.
With the new infrastructure, it's far more attractive to host packages on PyPI than it was before, so people who do not host on PyPI will have carefully thought about this.
Accordingly, we should respect those reasons make it possible for Python packages to live elsewhere, without having our tools put those packages into a bad light or making it harder for Python users to install such packages than needed.
I'm not sure what "put it into a bad light" is supposed to mean here. Is it about the warning? I've already removed that completely.
It's using the same unfortunate strategy that setuptools used by requiring an option called --single-version-externally-managed to be able to install a package without all the .pth and egg logic (an option that pip now uses per default, BTW ;-)).
I snipped the rest of the discussion and reliability, using unmaintained packages and projects using their own mirrors (which should really be the standard, not an exceptional case), because it's not really leading anywhere:
Using your own mirror shouldn’t be the standard if all you’re doing is automatically updating that mirror. It’s a hack to get around unreliability and it should be seen of as a sign of a failure to provide a service that people can rely on and that’s how I see it. People depend on this service and it’s irresponsible to not treat it as a critical piece of infrastructure.
At the time we discussed the PEP, security was the main concern, not reliability.
The main concern, not the only concern.
Now that there is a secure way to reference external distribution files, I think we should revisit the defaults decision in the light of the new possibilities.
Re-evaluate all you want. If they change it will be over my strenuous objections. The feedback I get from users is overwhelmly positive. Even the ones who get struck by needing to add extra flags tell me how they are glad that pip and PyPI are moving towards a more reliable model.
BTW: The PEP mentions re-hosting tools to upload their packages to PyPI and also says "This re-hosting tool MUST be available before automated hosting-mode changes are announced to package maintainers." I am not aware of such tools.
It appears this got missed. I’ll add it to my stack to remedy this.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, May 09 2014)
::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA