[Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

[Donald Stufft]

On May 9, 2014, at 9:58 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 09.05.2014 13:44, Donald Stufft wrote:
>> 
>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> Donald: I don't think anyone is arguing that hosting packages on
>>> PyPI is a bad thing and PyPI as a service has gotten a lot better
>>> than it was a few years ago.
>> 
>> Didn’t mean to imply that I thought it was otherwise.
>> 
>>> 
>>> However, I find it troubling that we as Python developers are *forcing*
>>> the whole Python world to put their code into PyPI.
>> 
>> Forcing is a strong word. As of right now we’re "forcing" you to put it onto
>> PyPI if you want to install it without *any* flags to pip. 
> 
> Which is what most people expect to be able to do.

Sure, but sometimes it’s better to make an informed choice about things being
installed instead of having it be installed by default and sometimes it’s better
to disallow doing something at all.

Further most people don’t expect an install to touch any server host other
than PyPI. As far as I’m aware none of the other package repositories
feature this, and even with the fact we support it barely anyone even cares to
utilize it.

> 
>> You're more then
>> capable of hosting it externally if you wish, and pip will even tell the people
>> who try to install it what they need to enable in order to install it. It even
>> allows people to say "I don't care about any of this crap, just make all the
>> external stuff work".
>> 
>> Even if pip removed the external link handling all together and required you
>> to do something like:
>> 
>>    $ pip install --extra-index-url https://example.com/our-packages/ foo
>>    $ pip install --find-links https://example.com/our-packages/ foo
>> 
>> We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
>> discouraging people from not hosting on PyPI and providing incentives to doing
>> that.
> 
> This is all true, but in reality, you are making externally hosted
> packages second class citizens in Python land by requiring
> extra options even for package listings that are perfectly safe
> to download from other servers.
> 
> As mentioned before: I can understand that you want to make downloads
> safe for users, but if a package is hosted on some other reliable
> servers and pip can check that it's a valid package, there's little
> to argue for not allowing such downloads.

Except the fact that the only people I’ve ever seen *happy* that people can
host packages externally are a handful (<5) of people (tbh primarily you and
Stefan) and the feedback I get from every other person around the web
in unequivocally yes please get rid of externally hosted files, they’ve done
nothing but cause me pain.

It’s not reasonable to allow a minority of users to negatively impact the majority.

> 
>>> There are plenty good reasons not to do this, and sometimes it's
>>> even impossible if you want to stay legal (see the PEP for details).
>> 
>> I re-read the reasons, I'm not sure I really agree with most (or all?) of them
>> however if people really want to do it, then there is nothing stopping them.
>> It's their choice and people on the *other* side of that who are installing
>> these packages also get to make a choice if they want to allow it or not.
> 
> You don't have to agree with those reasons. Fact is, they exist,
> prevent package authors from uploading to PyPI and we as
> Python developers should respect those reasons.

No I disagree that they are reasons at all. Half of the ones you enumerated are
nonsensical or irrelevant, the other half can be fixed by adding features to or
fixing PyPI. One or two read like reasons why someone might actually decide
not to upload something to PyPI but which that reason isn’t really all that
reasonable and finally a grand total of one or two of them look like an actual legit
reasons and it only applies to Crypto software.

I mean your reasoning included things like:

> PyPI doesn’t let you upload two files with the same name, you gave the
> example of UC2 vs UC4

  The thing being if PyPI did allow you to do that
  then we’d have no way to determine which one is the right one and half
  of the people would just get randomly failing installs because we guessed
  the wrong one.

> They don’t know it’s possible to upload to PyPI

I don’t even know how to response to this one.

> PyPI used to be unreliable but isn’t now and they don’t know about it

Nor this one.

> Not wanting people to download a package at all and instead check out
> from a VCS repo

This’ll randomly fail for people depending on if they happen to have that VCS
installed. Also I don’t see any evidence that people are actually doing this
outside of supporting a foo==dev install which pip won’t install by default either.

> Not wanting to add latency for yourself for downloading from PyPI

Instead you’re going to inflict extra latency on everyone else when you could just
run a mirror or upload the packages to your own server.

> File types that PyPI doesn’t support and which utilize a third party package
> format, PyPM being a given example. 

Seriously? Should an apt repo support external links because you can’t install RPMs
with apt?

> Wanting people to have to read something before installing the package
> because it relies on some additional setup.

This is completely nonsensical if they want people to have to read something before
they attempt to pip install it then they don’t want pip to discover the external file at all
then.

> It currently takes too long uploading that many files to
> PyPI. This causes a problem, since in order to start the upload,
> we have to register the release on PyPI, which tools will then
> immediately find. However, during the upload time, they won't
> necessarily find the right files to download and then fail.

So if this is an actual problem file a bug and we can make it so PyPI lets you upload
things as “drafts” and then you can publish them all at once or some such thing.

> having a strong need to know the number of downloads per
> package and associated statistics such as downloads per
> country, per year/month/day/hour

If additional statistics are desired for PyPI file a bug and we can add them.

> not wanting to give up access to the download log files

This appears to be repeating the last reason with different wording.

> the PyPI uploads not being compatible to their release process

I’m not sure how it’s possible to have a release process where it isn’t
compatible to upload to an additional server. Maybe ones that don’t
currently do it but ones that are fundamentally incompatible?


Almost all of your “reasons” read like someone who maybe had one corner
case where it might have made sense but didn’t want to admit that it was
a corner case so tried to think up any random reason they could think of
in order to make the list look longer and more impressive than it actually was.

> 
> With the new infrastructure, it's far more attractive to
> host packages on PyPI than it was before, so people who do
> not host on PyPI will have carefully thought about this.
> 
>>> Accordingly, we should respect those reasons make it possible for
>>> Python packages to live elsewhere, without having our tools put
>>> those packages into a bad light or making it harder for Python
>>> users to install such packages than needed.
>> 
>> I'm not sure what "put it into a bad light" is supposed to mean here. Is it
>> about the warning? I've already removed that completely.
> 
> It's using the same unfortunate strategy that setuptools used by
> requiring an option called --single-version-externally-managed
> to be able to install a package without all the .pth and egg
> logic (an option that pip now uses per default, BTW ;-)).
> 
> I snipped the rest of the discussion and reliability, using
> unmaintained packages and projects using their own mirrors (which
> should really be the standard, not an exceptional case),
> because it's not really leading anywhere:

Using your own mirror shouldn’t be the standard if all you’re doing
is automatically updating that mirror. It’s a hack to get around
unreliability and it should be seen of as a sign of a failure to provide
a service that people can rely on and that’s how I see it. People
depend on this service and it’s irresponsible to not treat it as a
critical piece of infrastructure.

> 
> At the time we discussed the PEP, security was the main concern,
> not reliability.

The main concern, not the only concern.

> 
> Now that there is a secure way to reference external distribution
> files, I think we should revisit the defaults decision in the
> light of the new possibilities.

Re-evaluate all you want. If they change it will be over my strenuous
objections. The feedback I get from users is overwhelmly positive.
Even the ones who get struck by needing to add extra flags tell me
how they are glad that pip and PyPI are moving towards a more
reliable model.

> 
> 
> BTW: The PEP mentions re-hosting tools to upload their packages
> to PyPI and also says "This re-hosting tool MUST be available
> before automated hosting-mode changes are announced to package
> maintainers."  I am not aware of such tools.

It appears this got missed. I’ll add it to my stack to remedy this.

> 
> --
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, May 09 2014)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/9c21c74f/attachment.sig>