[Distutils] some questions about PEP470

Thu Oct 9 19:40:05 CEST 2014

> On Oct 9, 2014, at 12:41 PM, holger krekel <holger at merlinux.eu> wrote:
> 
> Hi Donald, Nick,
> 
> to change the somewhat unsuccessfull way how we were conversing about PEP470
> so far i'd like to kindly ask you a few questions related to the PEP. 
> This is to check if i am maybe barking up the wrong tree and also to
> enlarge the common ground/understanding that we are discussing on.
> 
> Please take your time, i'd appreciate if you give a joint
> answer rather than a quick one.
> 
> thanks,
> holger
> 
> 
> Numbers of users affected
> ---------------------------------
> 
> Do i see it right that the PEP470 changes would mean about 6-7 thousand 
> users (per day) need to change their installation options to use
> "--extra-index-url"?  If not, how many?  Is there a monthly figure?

It’s impossible to couch this in terms of “users” because we have no way
of correlating what we see on the PyPI side with users. On the single day
I selected to look at the logs (which was more or less the day before the
day I was compounding numbers) there were 6.6k total unique IP addresses
that hit a /simple/ page which belonged to one of the affected projects.

Beyond knowing how many IP addresses it’s difficult to determine how that
correlates into users, that could be a single user with 6.6k different EC2
machines, or it could be 6.6k individual users (or even more than that if
there is a transparent proxy at play). In all likelihood it is not a single
user and it is not 6.6k users but somewhere in between.

Important to point out that this number also includes people spinning up
bandersnatch mirrors, devpi mirrors, or any other automated fetching of
the /simple/ page for reasons other than “I’d like to install this project”.

> 
> And that the affected users can only do that if the respective
> maintainers of the projects offer an external index (or re-upload to PyPI)?

No and Yes.

Wherever pip/easy_install are currently finding the download from can serve
as the external index. This likely won’t be the most efficient repository
since often times these are regular web pages which have other content and
the like but it won’t be any worse than it is currently. For instance you
can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the
current page as a find-links repository with pip.

> 
> Do i see it right that up to a 1000 maintainers need to act and offer an
> external index if they want to keep their projects properly installable?

If their project is already installable, then they already have something
which is usable as either a simple or a find-links repository. The only
action required on their part is if they want the discovery affordances
in this PEP they would need to tell PyPI that.

> 
> What can users of (older or current) easy_install versions do if they
> want to install a project with external links in a post PEP470 world?

They can use —find-links.

> 
> How many people currently use easy_install/not-pip (numbers, not only
> percentages)?

Later I can create a similar graph with absolute numbers, but here’s
a graph showing percentages for all downloads from PyPI. This does not
include people strictly downloading external links since this is looking
specifically at the access logs for /packages/*, but it is likely to be
fairly representative.

http://d.stufft.io/image/0r0X3Y2M100p

> 
> I've understood you made these two statements during the discussion:
> 
> - PEP438 caused bad UI for dealing with pypi-external links -- 
>  many people are confused by it and we thus need to fix it.
> 
> - PEP470 breaking backward compatibility for pypi-external links is
>  not a big deal because it affects only a tiny fraction of the users.
> 
> Could you choose which one of them you consider is true?

I consider them both to be true.

The PEP 438 UX is confusing, out of the people who have had to use it I
have seem a fairly high percentage of those completely confused by it. It,
especially right when pip 1.5 was released, was one of our most reported
issues. The total number of people who need to use it has gone down over
time, however I still believe that percentage wise most people who need to
use it are confused by it.

I do not believe that PEP 470 breaking backwards compatability for pypi-external
links to be a terrible burden because it only affects a small percentage of the
total users of PyPI.

I think perhaps the reason you think both of them can’t be true is you’re
assuming that I’m talking about percentages of the same total population?

> 
> 
> Recommendation of "--extra-index-url"
> --------------------------------------
> 
> In your mind and forgetting about PEP470, in what situations exactly is
> "pip install --extra-index-url" a safe option for users?

The answer to this isn’t really related to —extra-index-url, ``pip install foo``
is “safe” (given the threat model we operate under) if, and only if, you trust
the operators of all of the repositories you have configured (by default, via
—index-url, via —extra-index-url, via —find-links, and via —process-dependency-links),
to give you the correct files for “foo”. How the repositories have come to be
configured isn’t particularly meaningful.

> 
> 
> Interpretation of external link usage
> --------------------------------------------
> 
> In the main rationale you say:
> 
>    "While a large number of projects did ultimately decide to upload to
>    PyPI, some of them did so only because the UX around what PEP 438 was so
>    bad that they felt forced to do so."
> 
> Could you provide some tractable background (not just your strong opinion)
> for this interpretation?  Why can it not be that people nowadays just
> prefer to upload to PyPI without even considering alternative options?
> 

Well Stefan had voiced that complaint last time that he felt we were trying
to force him to upload to PyPI by making the UX so bad. I’ve had a few other
people say similar things to me in private.

> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA