Hi Donald, Nick, to change the somewhat unsuccessfull way how we were conversing about PEP470 so far i'd like to kindly ask you a few questions related to the PEP. This is to check if i am maybe barking up the wrong tree and also to enlarge the common ground/understanding that we are discussing on. Please take your time, i'd appreciate if you give a joint answer rather than a quick one. thanks, holger Numbers of users affected --------------------------------- Do i see it right that the PEP470 changes would mean about 6-7 thousand users (per day) need to change their installation options to use "--extra-index-url"? If not, how many? Is there a monthly figure? And that the affected users can only do that if the respective maintainers of the projects offer an external index (or re-upload to PyPI)? Do i see it right that up to a 1000 maintainers need to act and offer an external index if they want to keep their projects properly installable? What can users of (older or current) easy_install versions do if they want to install a project with external links in a post PEP470 world? How many people currently use easy_install/not-pip (numbers, not only percentages)? I've understood you made these two statements during the discussion: - PEP438 caused bad UI for dealing with pypi-external links -- many people are confused by it and we thus need to fix it. - PEP470 breaking backward compatibility for pypi-external links is not a big deal because it affects only a tiny fraction of the users. Could you choose which one of them you consider is true? Recommendation of "--extra-index-url" -------------------------------------- In your mind and forgetting about PEP470, in what situations exactly is "pip install --extra-index-url" a safe option for users? Interpretation of external link usage -------------------------------------------- In the main rationale you say: "While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so." Could you provide some tractable background (not just your strong opinion) for this interpretation? Why can it not be that people nowadays just prefer to upload to PyPI without even considering alternative options?
On Oct 9, 2014, at 12:41 PM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald, Nick,
to change the somewhat unsuccessfull way how we were conversing about PEP470 so far i'd like to kindly ask you a few questions related to the PEP. This is to check if i am maybe barking up the wrong tree and also to enlarge the common ground/understanding that we are discussing on.
Please take your time, i'd appreciate if you give a joint answer rather than a quick one.
thanks, holger
Numbers of users affected ---------------------------------
Do i see it right that the PEP470 changes would mean about 6-7 thousand users (per day) need to change their installation options to use "--extra-index-url"? If not, how many? Is there a monthly figure?
It’s impossible to couch this in terms of “users” because we have no way of correlating what we see on the PyPI side with users. On the single day I selected to look at the logs (which was more or less the day before the day I was compounding numbers) there were 6.6k total unique IP addresses that hit a /simple/ page which belonged to one of the affected projects. Beyond knowing how many IP addresses it’s difficult to determine how that correlates into users, that could be a single user with 6.6k different EC2 machines, or it could be 6.6k individual users (or even more than that if there is a transparent proxy at play). In all likelihood it is not a single user and it is not 6.6k users but somewhere in between. Important to point out that this number also includes people spinning up bandersnatch mirrors, devpi mirrors, or any other automated fetching of the /simple/ page for reasons other than “I’d like to install this project”.
And that the affected users can only do that if the respective maintainers of the projects offer an external index (or re-upload to PyPI)?
No and Yes. Wherever pip/easy_install are currently finding the download from can serve as the external index. This likely won’t be the most efficient repository since often times these are regular web pages which have other content and the like but it won’t be any worse than it is currently. For instance you can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the current page as a find-links repository with pip.
Do i see it right that up to a 1000 maintainers need to act and offer an external index if they want to keep their projects properly installable?
If their project is already installable, then they already have something which is usable as either a simple or a find-links repository. The only action required on their part is if they want the discovery affordances in this PEP they would need to tell PyPI that.
What can users of (older or current) easy_install versions do if they want to install a project with external links in a post PEP470 world?
They can use —find-links.
How many people currently use easy_install/not-pip (numbers, not only percentages)?
Later I can create a similar graph with absolute numbers, but here’s a graph showing percentages for all downloads from PyPI. This does not include people strictly downloading external links since this is looking specifically at the access logs for /packages/*, but it is likely to be fairly representative. http://d.stufft.io/image/0r0X3Y2M100p
I've understood you made these two statements during the discussion:
- PEP438 caused bad UI for dealing with pypi-external links -- many people are confused by it and we thus need to fix it.
- PEP470 breaking backward compatibility for pypi-external links is not a big deal because it affects only a tiny fraction of the users.
Could you choose which one of them you consider is true?
I consider them both to be true. The PEP 438 UX is confusing, out of the people who have had to use it I have seem a fairly high percentage of those completely confused by it. It, especially right when pip 1.5 was released, was one of our most reported issues. The total number of people who need to use it has gone down over time, however I still believe that percentage wise most people who need to use it are confused by it. I do not believe that PEP 470 breaking backwards compatability for pypi-external links to be a terrible burden because it only affects a small percentage of the total users of PyPI. I think perhaps the reason you think both of them can’t be true is you’re assuming that I’m talking about percentages of the same total population?
Recommendation of "--extra-index-url" --------------------------------------
In your mind and forgetting about PEP470, in what situations exactly is "pip install --extra-index-url" a safe option for users?
The answer to this isn’t really related to —extra-index-url, ``pip install foo`` is “safe” (given the threat model we operate under) if, and only if, you trust the operators of all of the repositories you have configured (by default, via —index-url, via —extra-index-url, via —find-links, and via —process-dependency-links), to give you the correct files for “foo”. How the repositories have come to be configured isn’t particularly meaningful.
Interpretation of external link usage --------------------------------------------
In the main rationale you say:
"While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so."
Could you provide some tractable background (not just your strong opinion) for this interpretation? Why can it not be that people nowadays just prefer to upload to PyPI without even considering alternative options?
Well Stefan had voiced that complaint last time that he felt we were trying to force him to upload to PyPI by making the UX so bad. I’ve had a few other people say similar things to me in private.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Hi Donald, many thanks for answering. A few follow up questions inline. On Thu, Oct 09, 2014 at 13:40 -0400, Donald Stufft wrote:
On Oct 9, 2014, at 12:41 PM, holger krekel <holger@merlinux.eu> wrote:
Numbers of users affected ---------------------------------
Do i see it right that the PEP470 changes would mean about 6-7 thousand users (per day) need to change their installation options to use "--extra-index-url"? If not, how many? Is there a monthly figure?
It’s impossible to couch this in terms of “users” because we have no way of correlating what we see on the PyPI side with users. On the single day I selected to look at the logs (which was more or less the day before the day I was compounding numbers) there were 6.6k total unique IP addresses that hit a /simple/ page which belonged to one of the affected projects.
Beyond knowing how many IP addresses it’s difficult to determine how that correlates into users, that could be a single user with 6.6k different EC2 machines, or it could be 6.6k individual users (or even more than that if there is a transparent proxy at play). In all likelihood it is not a single user and it is not 6.6k users but somewhere in between.
Important to point out that this number also includes people spinning up bandersnatch mirrors, devpi mirrors, or any other automated fetching of the /simple/ page for reasons other than “I’d like to install this project”.
I understand it's hard to get to somewhat sensible numbers and the number of unique IPs is probably only an upper bound. devpi and bandersnatch make even that fuzzy because more than one user may be behind each such instance. Anyway, can you provide a monthly number of unique IP addresses on the simple pages on projects with external links?
And that the affected users can only do that if the respective maintainers of the projects offer an external index (or re-upload to PyPI)?
No and Yes.
Wherever pip/easy_install are currently finding the download from can serve as the external index. This likely won’t be the most efficient repository since often times these are regular web pages which have other content and the like but it won’t be any worse than it is currently. For instance you can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the current page as a find-links repository with pip.
How can affected users discover they need to use this particular option and URL if they use today's pip/easy_install versions and a post PEP470 PyPI?
Do i see it right that up to a 1000 maintainers need to act and offer an external index if they want to keep their projects properly installable?
If their project is already installable, then they already have something which is usable as either a simple or a find-links repository. The only action required on their part is if they want the discovery affordances in this PEP they would need to tell PyPI that.
Is this true also for the (small but still) set of maintainers who registered external links with checksums? If maintainers don't act, will using a post PEP470 released pip help the users in any way?
I've understood you made these two statements during the discussion:
- PEP438 caused bad UI for dealing with pypi-external links -- many people are confused by it and we thus need to fix it.
- PEP470 breaking backward compatibility for pypi-external links is not a big deal because it affects only a tiny fraction of the users.
Could you choose which one of them you consider is true?
I consider them both to be true.
The PEP 438 UX is confusing, out of the people who have had to use it I have seem a fairly high percentage of those completely confused by it. It, especially right when pip 1.5 was released, was one of our most reported issues. The total number of people who need to use it has gone down over time, however I still believe that percentage wise most people who need to use it are confused by it.
I do not believe that PEP 470 breaking backwards compatability for pypi-external links to be a terrible burden because it only affects a small percentage of the total users of PyPI.
I think perhaps the reason you think both of them can’t be true is you’re assuming that I’m talking about percentages of the same total population?
Yes, i was assuming that for both statements the same basis group was used. So i understand know you are saying overall very few people depend on external links but out of those who do, many are confused and annoyed about how it works. Will the people who suffered from the current external linking options be the same ones who could be affected by backward compatibility issues (i.e. commands which now work can fail with a post-PEP470 PyPI server)? personal side question: do i remember correctly that when we discussed PEP438 you pushed for the current set of behaviours wrt to external links while i tried to keep it simpler because you put higher priority on protection against MITM attacks?
Recommendation of "--extra-index-url" --------------------------------------
In your mind and forgetting about PEP470, in what situations exactly is "pip install --extra-index-url" a safe option for users?
The answer to this isn’t really related to —extra-index-url, ``pip install foo`` is “safe” (given the threat model we operate under) if, and only if, you trust the operators of all of the repositories you have configured (by default, via —index-url, via —extra-index-url, via —find-links, and via —process-dependency-links), to give you the correct files for “foo”. How the repositories have come to be configured isn’t particularly meaningful.
I understand that as a fairly generic security statement. But I was trying to rather ask about use cases and scenarios where precisely the --extra-index-url option is useful and to be recommended. I'd be grateful if Nick or you could still describe use cases, especially outside PEP470 external links context (the option existed before so i presume there must be some use cases).
Interpretation of external link usage --------------------------------------------
In the main rationale you say:
"While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so."
Could you provide some tractable background (not just your strong opinion) for this interpretation? Why can it not be that people nowadays just prefer to upload to PyPI without even considering alternative options?
Well Stefan had voiced that complaint last time that he felt we were trying to force him to upload to PyPI by making the UX so bad. I’ve had a few other people say similar things to me in private.
I can sympathize. In fact, I think we didn't deliver the upload tools that we outlined with PEP438, particularly registration of externally verified links. My bad as well. best, holger
On Oct 11, 2014, at 2:31 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
many thanks for answering. A few follow up questions inline.
On Thu, Oct 09, 2014 at 13:40 -0400, Donald Stufft wrote:
On Oct 9, 2014, at 12:41 PM, holger krekel <holger@merlinux.eu> wrote:
Numbers of users affected ---------------------------------
Do i see it right that the PEP470 changes would mean about 6-7 thousand users (per day) need to change their installation options to use "--extra-index-url"? If not, how many? Is there a monthly figure?
It’s impossible to couch this in terms of “users” because we have no way of correlating what we see on the PyPI side with users. On the single day I selected to look at the logs (which was more or less the day before the day I was compounding numbers) there were 6.6k total unique IP addresses that hit a /simple/ page which belonged to one of the affected projects.
Beyond knowing how many IP addresses it’s difficult to determine how that correlates into users, that could be a single user with 6.6k different EC2 machines, or it could be 6.6k individual users (or even more than that if there is a transparent proxy at play). In all likelihood it is not a single user and it is not 6.6k users but somewhere in between.
Important to point out that this number also includes people spinning up bandersnatch mirrors, devpi mirrors, or any other automated fetching of the /simple/ page for reasons other than “I’d like to install this project”.
I understand it's hard to get to somewhat sensible numbers and the number of unique IPs is probably only an upper bound. devpi and bandersnatch make even that fuzzy because more than one user may be behind each such instance. Anyway, can you provide a monthly number of unique IP addresses on the simple pages on projects with external links?
I can do that, it’ll take a little while because the scripts I have to do it aren’t really designed to handle more than a single file and the files are quite large once decompressed (30 days is roughly 120GB of log files).
And that the affected users can only do that if the respective maintainers of the projects offer an external index (or re-upload to PyPI)?
No and Yes.
Wherever pip/easy_install are currently finding the download from can serve as the external index. This likely won’t be the most efficient repository since often times these are regular web pages which have other content and the like but it won’t be any worse than it is currently. For instance you can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the current page as a find-links repository with pip.
How can affected users discover they need to use this particular option and URL if they use today's pip/easy_install versions and a post PEP470 PyPI?
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
Do i see it right that up to a 1000 maintainers need to act and offer an external index if they want to keep their projects properly installable?
If their project is already installable, then they already have something which is usable as either a simple or a find-links repository. The only action required on their part is if they want the discovery affordances in this PEP they would need to tell PyPI that.
Is this true also for the (small but still) set of maintainers who registered external links with checksums?
A find-link repository can be as simple as a direct link to a file yes, it’s not the best idea since you can’t ever change the location of that file without breaking it for people but it can be done.
If maintainers don't act, will using a post PEP470 released pip help the users in any way?
Absolutely! A common complaint that I’ve seen from users is that when faced with network errors pip’s output is extremely unhelpful. For instance here’s an issue about how TLS errors are presented - https://github.com/pypa/pip/issues/1511. The primary reason for this is that because we don’t know if a particular link *should* be up, or if it’s expected to be down (xfail, in testing lingo) we have to assume that it should be down. Sometimes there can be many of these links so to avoid having lots of extraneous “error” looking messages in the output we just silently hide them. However if none of the links can be found they’ll just get a confusing “Sorry can’t find anything to install for that”. Worse yet, if we can find some things but not everything then we might install an older insecure version. This could actually be used to prevent people from getting a security update for a particular project if they’ve switched from hosting on PyPI to hosting externally and a MITM attacker is blocking their attempts to reach the external host. A post PEP 470 pip will be able to treat all URLs it fetches as mandatory, so it can raise proper error messages when it can’t locate an URL to communicate that to the end user and ideally make things far less confusing.
I've understood you made these two statements during the discussion:
- PEP438 caused bad UI for dealing with pypi-external links -- many people are confused by it and we thus need to fix it.
- PEP470 breaking backward compatibility for pypi-external links is not a big deal because it affects only a tiny fraction of the users.
Could you choose which one of them you consider is true?
I consider them both to be true.
The PEP 438 UX is confusing, out of the people who have had to use it I have seem a fairly high percentage of those completely confused by it. It, especially right when pip 1.5 was released, was one of our most reported issues. The total number of people who need to use it has gone down over time, however I still believe that percentage wise most people who need to use it are confused by it.
I do not believe that PEP 470 breaking backwards compatability for pypi-external links to be a terrible burden because it only affects a small percentage of the total users of PyPI.
I think perhaps the reason you think both of them can’t be true is you’re assuming that I’m talking about percentages of the same total population?
Yes, i was assuming that for both statements the same basis group was used. So i understand know you are saying overall very few people depend on external links but out of those who do, many are confused and annoyed about how it works.
Yes exactly.
Will the people who suffered from the current external linking options be the same ones who could be affected by backward compatibility issues (i.e. commands which now work can fail with a post-PEP470 PyPI server)?
Yes-ish. A lot of those people switched away from external hosting all together or chose to rely on different projects that didn’t need that. It would likely be a subset of those users.
personal side question: do i remember correctly that when we discussed PEP438 you pushed for the current set of behaviours wrt to external links while i tried to keep it simpler because you put higher priority on protection against MITM attacks?
I have very little recollection of the discussion around PEP 438, I have a bad memory for details like that. It’s entirely possible and sounds like me to be security focused.
Recommendation of "--extra-index-url" --------------------------------------
In your mind and forgetting about PEP470, in what situations exactly is "pip install --extra-index-url" a safe option for users?
The answer to this isn’t really related to —extra-index-url, ``pip install foo`` is “safe” (given the threat model we operate under) if, and only if, you trust the operators of all of the repositories you have configured (by default, via —index-url, via —extra-index-url, via —find-links, and via —process-dependency-links), to give you the correct files for “foo”. How the repositories have come to be configured isn’t particularly meaningful.
I understand that as a fairly generic security statement. But I was trying to rather ask about use cases and scenarios where precisely the --extra-index-url option is useful and to be recommended. I'd be grateful if Nick or you could still describe use cases, especially outside PEP470 external links context (the option existed before so i presume there must be some use cases).
I’m not sure exactly what you want here. It’s hard to come up with an exhaustive list of scenarios where it’s useful. One I’ve used in the past was: -i https://index.example.com/base/ —extra-index-url https://index.example.com/project-foo-overrides/
Interpretation of external link usage --------------------------------------------
In the main rationale you say:
"While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so."
Could you provide some tractable background (not just your strong opinion) for this interpretation? Why can it not be that people nowadays just prefer to upload to PyPI without even considering alternative options?
Well Stefan had voiced that complaint last time that he felt we were trying to force him to upload to PyPI by making the UX so bad. I’ve had a few other people say similar things to me in private.
I can sympathize. In fact, I think we didn't deliver the upload tools that we outlined with PEP438, particularly registration of externally verified links. My bad as well.
best, holger
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Oct 11, 2014, at 2:27 PM, Donald Stufft <donald@stufft.io> wrote:
And that the affected users can only do that if the respective maintainers of the projects offer an external index (or re-upload to PyPI)?
No and Yes.
Wherever pip/easy_install are currently finding the download from can serve as the external index. This likely won’t be the most efficient repository since often times these are regular web pages which have other content and the like but it won’t be any worse than it is currently. For instance you can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the current page as a find-links repository with pip.
How can affected users discover they need to use this particular option and URL if they use today's pip/easy_install versions and a post PEP470 PyPI?
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links.
Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now?
On 12 October 2014 09:49, Donald Stufft <donald@stufft.io> wrote:
On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links.
Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now?
Sure, we could do that too. Regardless, I don't believe we should let the conversion of the silent security failures into noisy exceptions block the other usability enhancements - if decoupling the two decisions will achieve that, then let's go down that path. I think I suggested at one point that the PEP could potentially commit to making the backwards compatible changes on the server side, and then for the *in*compatible changes, only commit to *looking at the numbers again* in a few months time after folks have had a chance to experience the replacement model. Client applications would be officially granted permission to ignore the PEP 438 external hosting mechanism in favour of the new PEP 470 external index advertisements. The sequence of events from a compatibility break/user experience enhancement perspective would then be: * new model added to PyPI (no compatibility break) * old model dropped from pip (client side compatibility break, error handling improvements) * old model dropped from PyPI (conditional, based on changes in server metrics) That changes the dynamic from "the updated user experience will be better, trust us" to "if you prefer the old user experience, you can keep using the old pip, if you prefer the new user experience, upgrade". The eventual decision on when to drop the legacy model from PyPI can then by made based on a richer set of input data, that includes the level of adoption of the new version of pip that treats all missing link targets as potential errors. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote:
On 12 October 2014 09:49, Donald Stufft <donald@stufft.io> wrote:
On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links.
Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now?
Sure, we could do that too. Regardless, I don't believe we should let the conversion of the silent security failures into noisy exceptions block the other usability enhancements - if decoupling the two decisions will achieve that, then let's go down that path.
I think I suggested at one point that the PEP could potentially commit to making the backwards compatible changes on the server side, and then for the *in*compatible changes, only commit to *looking at the numbers again* in a few months time after folks have had a chance to experience the replacement model. Client applications would be officially granted permission to ignore the PEP 438 external hosting mechanism in favour of the new PEP 470 external index advertisements.
The sequence of events from a compatibility break/user experience enhancement perspective would then be:
* new model added to PyPI (no compatibility break) * old model dropped from pip (client side compatibility break, error handling improvements) * old model dropped from PyPI (conditional, based on changes in server metrics)
I like this strategy and would add that I guess it's then only a matter of when not if the old model is dropped (and the PEP could encode that). The question is how to best do the "(almost) no compat break" for a reasonable definition of "almost". I am mostly concerned about the verified externals and "crawl urls". If they keep showing up as "rel=download" links on the simple page that should be fine for existing tools, right? But with PEP470 they might get an additional attribute so that a newer pip can recognize it and advise accordingly. How pypi/warehouse internally keeps the state during/post PEP470 is less clear to me but it might involve a one-time conversion from the existing information to the new one. I am actually wondering if a new pip could change its UI even without any changes to pypi.python.org? For example, when "pip install pil" happens it would say: Could not find any release file for "Pil" on https://pypi.python.org itself but the project has historically had additional external pages which linked to release files: http://effbot.org/zon/pil-changes-115.html/ http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz http://www.pythonware.com/products/pil Please manully check these links if they directly link to or contain release files for "Pil" and then add an appropriate one like this:: pip install --find-links URL [other options] Pil (See <link> for more info ...) and pip could deduce this information from the current https://pypi.python.org/simple/pil/ page. (The pip message above may not be precisely right but you get the idea). FWIW this strategy of advising to use newer/other options but keeping the old working (undocumneted or documented with a warning) is how i generally try to handle things with pytest. It allows people to use the new way and incrementally migrate their setups. And if someone complains about problems with the old ways, point them to the new ways so there is no need to maintain the old ways too much. Even then, it would probably still makes sense to advance the server side and allow explicit registration of external index pages i guess (and maybe findlinks pages so you can really just use apache/nginx directly?) and then have pip provide a more precise message.
That changes the dynamic from "the updated user experience will be better, trust us" to "if you prefer the old user experience, you can keep using the old pip, if you prefer the new user experience, upgrade".
I think it's very much fine to say "this is the future" as long as old-tool/current-server instances keep working for a longer while. Python packaging infra is a bit like a big container ship these days -- which is both a good and a bad thing i guess -- and people build all kinds of funny containers on top. best, holger
The eventual decision on when to drop the legacy model from PyPI can then by made based on a richer set of input data, that includes the level of adoption of the new version of pip that treats all missing link targets as potential errors.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Oct 13, 2014, at 7:41 AM, holger krekel <holger@merlinux.eu> wrote:
On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote:
On 12 October 2014 09:49, Donald Stufft <donald@stufft.io> wrote:
On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links.
Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now?
Sure, we could do that too. Regardless, I don't believe we should let the conversion of the silent security failures into noisy exceptions block the other usability enhancements - if decoupling the two decisions will achieve that, then let's go down that path.
I think I suggested at one point that the PEP could potentially commit to making the backwards compatible changes on the server side, and then for the *in*compatible changes, only commit to *looking at the numbers again* in a few months time after folks have had a chance to experience the replacement model. Client applications would be officially granted permission to ignore the PEP 438 external hosting mechanism in favour of the new PEP 470 external index advertisements.
The sequence of events from a compatibility break/user experience enhancement perspective would then be:
* new model added to PyPI (no compatibility break) * old model dropped from pip (client side compatibility break, error handling improvements) * old model dropped from PyPI (conditional, based on changes in server metrics)
I like this strategy and would add that I guess it's then only a matter of when not if the old model is dropped (and the PEP could encode that).
This is more or less the sequence of events that will happen with PEP 470 as it is except that PEP 470 doesn’t make the deprecation optional. I do not believe it is a good idea to make it optional because it is a silent security risk. The other main difference is that PEP 470 doesn’t allow new projects to ever register externally hosted files in the old way.
The question is how to best do the "(almost) no compat break" for a reasonable definition of "almost". I am mostly concerned about the verified externals and "crawl urls". If they keep showing up as "rel=download" links on the simple page that should be fine for existing tools, right?
Keeping backwards compatibility for unsafe things is completely off the table as far as I’m concerned. Specifically that means the crawl links. I think keeping backwards compatibility for the ~70 projects which are doing things safely today is a waste of resources in the long run, but something that should be phased in as PEP 470 does. More people were broken when PyPI went forced TLS and then again when PyPI switched from a sha1 certificate to a sha256 certificate. The numbers of safe installs is miniscule.
But with PEP470 they might get an additional attribute so that a newer pip can recognize it and advise accordingly. How pypi/warehouse internally keeps the state during/post PEP470 is less clear to me but it might involve a one-time conversion from the existing information to the new one.
I am actually wondering if a new pip could change its UI even without any changes to pypi.python.org? For example, when "pip install pil" happens it would say:
Could not find any release file for "Pil" on https://pypi.python.org itself but the project has historically had additional external pages which linked to release files:
http://effbot.org/zon/pil-changes-115.html/ http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz http://www.pythonware.com/products/pil
Please manully check these links if they directly link to or contain release files for "Pil" and then add an appropriate one like this::
pip install --find-links URL [other options] Pil
(See <link> for more info ...)
and pip could deduce this information from the current https://pypi.python.org/simple/pil/page. (The pip message above may not be precisely right but you get the idea).
FWIW this strategy of advising to use newer/other options but keeping the old working (undocumneted or documented with a warning) is how i generally try to handle things with pytest. It allows people to use the new way and incrementally migrate their setups. And if someone complains about problems with the old ways, point them to the new ways so there is no need to maintain the old ways too much.
It’s a fine way of doing thing, and we do it in pip too, when (and this is important), it’s not a security sensitive issue.
Even then, it would probably still makes sense to advance the server side and allow explicit registration of external index pages i guess (and maybe findlinks pages so you can really just use apache/nginx directly?) and then have pip provide a more precise message.
That changes the dynamic from "the updated user experience will be better, trust us" to "if you prefer the old user experience, you can keep using the old pip, if you prefer the new user experience, upgrade".
I think it's very much fine to say "this is the future" as long as old-tool/current-server instances keep working for a longer while.
Python packaging infra is a bit like a big container ship these days -- which is both a good and a bad thing i guess -- and people build all kinds of funny containers on top.
best, holger
The eventual decision on when to drop the legacy model from PyPI can then by made based on a richer set of input data, that includes the level of adoption of the new version of pip that treats all missing link targets as potential errors.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
(for example right now bytereef.org is down, so we’d not discover any files there).
Indeed. It was up reliably since 2005, down for maintenance on September 23rd (before ShellShock ...). Then I discovered that someone had put up m3-cdecimal on PyPI (presumably abusing PyPI as their private repo --- there are several m3-* packages now). This triggered some reflection on whether I would make a significant effort in the future to keep things running smoothly for an open source community where authors are largely viewed as expendable. Subsequently the downtime (again, the first one since 2005) was picked up for propagandistic purposes on Twitter and Reddit. Last year I would have felt an obligation to minimize the downtime to an hour at most. I no longer feel any such obligations and I'll do it when I have time. Stefan Krah
On Oct 12, 2014, at 10:29 AM, Stefan Krah <stefankrah@freenet.de> wrote:
(for example right now bytereef.org is down, so we’d not discover any files there).
Indeed. It was up reliably since 2005, down for maintenance on September 23rd (before ShellShock ...). Then I discovered that someone had put up m3-cdecimal on PyPI (presumably abusing PyPI as their private repo --- there are several m3-* packages now).
If you’re this upset over someone redistributing your work, then maybe Open Source Software is the wrong hobby for you. Nonetheless I told you how to get that remediated and as of yet I don’t see an open support request from you on it.
This triggered some reflection on whether I would make a significant effort in the future to keep things running smoothly for an open source community where authors are largely viewed as expendable.
Authors are not viewed as expendable. Placing constraints on a system is just that, placing constraints on a system.
Subsequently the downtime (again, the first one since 2005) was picked up for propagandistic purposes on Twitter and Reddit.
I think you might want to rethink this strategy if it’s your goal, unless the view point you’re trying to push is that I was right all along because there are a number of people* who previously didn’t think it was a big deal but now agree with me since they couldn’t install cdecimal because bytrereef.org was down. * That have personally reached out to me.
Last year I would have felt an obligation to minimize the downtime to an hour at most. I no longer feel any such obligations and I'll do it when I have time.
By all means! You don’t owe anyone your time, so if you want to wait until some other point go for it. Until then, well it’s making my case for me when people don’t understand why I care, so thanks :) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
If you're this upset over someone redistributing your work, then maybe Open Source Software is the wrong hobby for you.
Usually one does not tell a core developer that his contributions are "a hobby". I have contributed 40000+ lines of original, dense C code, backed by 100% code coverage and 30000+ lines of ACL2 proofs. These days it may be more productive to hack other people's brains and produce 10000+ tweets in order to have more political influence. It's great that my code is distributed with Python. Likewise, it was great to work with Matthias Klose and Hans-Peter Jansen to produce Debian, Ubuntu, and OpenSuse packages. cdecimal is also distributed by NetBSD, ActiveState, continuum.io, and others. The difference here is that the above persons and entities respect people. They respect software. The package maintainers are very competent and easy to work with.
Nonetheless I told you how to get that remediated and as of yet I don't see an open support request from you on it.
My interest in cleaning up PyPI is practically zero now. In the end, who cares what m3-cdecimal was supposed to accomplish: Maybe they wanted to teach me a lesson, maybe they were upset about a minor detail, maybe they have a zero-day exploit for tar and that's their delivery mode. All I know is that they didn't even run ``make distclean'' before packaging, so some user info is present in config.log. Some problems can only be fixed by a curated distribution run by neutral maintainers.
I think you might want to rethink this strategy if it's your goal, unless the view point you're trying to push is that I was right all along because there are a number of people* who previously didn't think it was a big deal but now agree with me since they couldn't install cdecimal because bytrereef.org was down.
Shrug. This is more loss of interest than a strike! Even it were a strike, the observation is not particularly interesting: Any strike (think railway) potentially alienates some users. Anyway, it will be kind of tough to force U.S. exceptionalism via the terms and conditions on an international body of authors if only uploaded packages are allowed. Stefan Krah
On Oct 14, 2014, at 8:50 PM, Stefan Krah <stefankrah@freenet.de> wrote:
Donald Stufft <donald <at> stufft.io> writes:
If you're this upset over someone redistributing your work, then maybe Open Source Software is the wrong hobby for you.
Usually one does not tell a core developer that his contributions are "a hobby". I have contributed 40000+ lines of original, dense C code, backed by 100% code coverage and 30000+ lines of ACL2 proofs.
Uhh, maybe you’re misunderstanding the word hobby, unless you’re getting paid for your OSS work you’re not doing it professionally. A hobby isn’t a negative thing, until last December my OSS work was entirely a hobby too, and it’s still a hobby in my spare time too.
These days it may be more productive to hack other people's brains and produce 10000+ tweets in order to have more political influence.
I’m not even sure what this is saying… Are you accusing me of hacking people’s brains? If so I’m kind of impressed by what power you think I have.
It's great that my code is distributed with Python. Likewise, it was great to work with Matthias Klose and Hans-Peter Jansen to produce Debian, Ubuntu, and OpenSuse packages. cdecimal is also distributed by NetBSD, ActiveState, continuum.io, and others.
The difference here is that the above persons and entities respect people. They respect software. The package maintainers are very competent and easy to work with.
If this is about m3-cdecimal, well your license doesn’t give it’s permissions out only if the person is nice to you.
Nonetheless I told you how to get that remediated and as of yet I don't see an open support request from you on it.
My interest in cleaning up PyPI is practically zero now. In the end, who cares what m3-cdecimal was supposed to accomplish: Maybe they wanted to teach me a lesson, maybe they were upset about a minor detail, maybe they have a zero-day exploit for tar and that's their delivery mode.
Ok, if you don’t care then I find it hard to care much about it either.
All I know is that they didn't even run ``make distclean'' before packaging, so some user info is present in config.log.
Some problems can only be fixed by a curated distribution run by neutral maintainers.
Sure.
I think you might want to rethink this strategy if it's your goal, unless the view point you're trying to push is that I was right all along because there are a number of people* who previously didn't think it was a big deal but now agree with me since they couldn't install cdecimal because bytrereef.org was down.
Shrug. This is more loss of interest than a strike! Even it were a strike, the observation is not particularly interesting: Any strike (think railway) potentially alienates some users.
Ok! Well a loss of interest makes for a good example too, so thanks!
Anyway, it will be kind of tough to force U.S. exceptionalism via the terms and conditions on an international body of authors if only uploaded packages are allowed.
I’m not even sure what this is trying to say… How are our pretty simple ToS some sort of US exceptionalism?
Stefan Krah
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Oct 14, 2014, at 8:50 PM, Stefan Krah <stefankrah@freenet.de> wrote:
Anyway, it will be kind of tough to force U.S. exceptionalism via the
terms
and conditions on an international body of authors if only uploaded
On 15 Oct 2014 11:16, "Donald Stufft" <donald@stufft.io> wrote: packages
are allowed.
I’m not even sure what this is trying to say… How are our pretty simple ToS some sort of US exceptionalism?
PyPI is hosted in the US, and thus covered by US export laws. I don't follow Stefan's objection, however, given that the objective of PEP 470 is to improve the user experience of external hosting, rather than to disallow it. We're also working with the TUF developers to make sure that the next draft of their PEP appropriately covers the external hosting use case. The only things we're actively trying to eliminate are the MITM vulnerability affecting the majority of current externally hosted packages, and the poor user experience that arises when the current link spidering mechanism leads to packaging clients feeling obliged to silently ignore unreachable URLs when looking for externally hosted packages. Regards, Nick.
Stefan Krah
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Nick Coghlan <ncoghlan <at> gmail.com> writes:
PyPI is hosted in the US, and thus covered by US export laws. I don't follow Stefan's objection, however, given that the objective of PEP 470 is to improve the user experience of external hosting, rather than to disallow it.
Sorry if it wasn't clear. This sub-thread was no longer about PEP 470 at all. Rough context: Mr. Stufft: cdecimal downtime means more people will want to enforce uploading. Me: Even if that is so, you cannot exploit it politically, since upload-only packages are an insult for international authors. Stefan Krah
At this point (and possibly before) you are just trolling and not worth any further correspondence. If some of your feigned surprise questions are actually genuine, I recommend walking away from the keyboard for a couple of weeks and reading some literature. Otherwise it is just a waste of time to attempt any further conversation.
On 15 October 2014 12:20, Stefan Krah <stefankrah@freenet.de> wrote:
At this point (and possibly before) you are just trolling and not worth any further correspondence. If some of your feigned surprise questions are actually genuine, I recommend walking away from the keyboard for a couple of weeks and reading some literature.
Otherwise it is just a waste of time to attempt any further conversation.
Stefan, I personally apologise if at any point you, or any other developer that chooses to use non-PyPI hosting, has ever been (or even felt) personally attacked over your hosting choices. There's no way we can hope to ensure that folks that are passionate about the end user experience offered by the Python packaging ecosystem first come up to speed on the subtleties of copyright licensing (and the concerns around the current clause in PyPI's conditions of use that ensures, amongst other things, the legal right to operate PyPI mirrors) or US export restrictions (and the fact that uploading to PyPI requires agreeing to abide by them, which may not be an acceptable condition for non-US based developers). However, balancing multiple competing viewpoints is *why* we have the Python Enhancement Proposal process, and this is why the accepted PEPs have always included external hosting support, even when the feature has been missing from some of the earlier draft proposals. You don't need to fight that battle any more - it was won a long time ago (and was never really in danger of being lost). At the same time, silently introducing additional points of failure into users' infrastructure, and especially introducing silent vulnerabilities to man-in-the-middle DNS hijacking attacks, is *not* OK. Eliminating those two problems has been the focus of many of the more controversial PyPI changes over the past year or more, so it is entirely understandable that developers that choose to use external hosting *have* felt personally attacked, especially when other developers have failed to understand why "just host your packages on PyPI" isn't always going to be an acceptable answer. The latest proposal in PEP 470 is designed to provide a mechanism which is completely explicit (so end users always know when they're depending on additional repositories beyond PyPI), while also relatively streamlined (so end users don't complain when a developer chooses to make use of the external hosting support). The previous design in PEP 438 ended up failing on both of those counts, which is why there is now this new proposal to replace it with a different mechanism that has been designed to address the existing usability challenges. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan <at> gmail.com> writes:
On 15 October 2014 12:20, Stefan Krah <stefankrah <at> freenet.de> wrote:
At this point (and possibly before) you are just trolling and not worth any further correspondence. If some of your feigned surprise questions are actually genuine, I recommend walking away from the keyboard for a couple of weeks and reading some literature. Otherwise it is just a waste of time to attempt any further conversation.
[In case anyone is using a non-threaded view: The above paragraph was not directed at Nick.]
Stefan, I personally apologise if at any point you, or any other developer that chooses to use non-PyPI hosting, has ever been (or even felt) personally attacked over your hosting choices.
I feel a bit embarrassed, since you have absolutely nothing to apologize for. It is a very nice gesture, so thanks!
There's no way we can hope to ensure that folks that are passionate about the end user experience offered by the Python packaging ecosystem first come up to speed on the subtleties of copyright licensing (and the concerns around the current clause in PyPI's conditions of use that ensures, amongst other things, the legal right to operate PyPI mirrors) or US export restrictions (and the fact that uploading to PyPI requires agreeing to abide by them, which may not be an acceptable condition for non-US based developers).
Some people who *should* know better have repeatedly engaged in off-list FUD and personal attacks. It is nowhere near the level that, say, Lennart Poettering had to deal with, but enough for me to lose quite a bit of interest in Python core development. Stefan Krah
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/14/2014 09:15 PM, Donald Stufft wrote:
On Oct 14, 2014, at 8:50 PM, Stefan Krah <stefankrah@freenet.de> wrote:
Donald Stufft <donald <at> stufft.io> writes:
If you're this upset over someone redistributing your work, then maybe Open Source Software is the wrong hobby for you.
Usually one does not tell a core developer that his contributions are "a hobby". I have contributed 40000+ lines of original, dense C code, backed by 100% code coverage and 30000+ lines of ACL2 proofs. Uhh, maybe you’re misunderstanding the word hobby, unless you’re getting paid for your OSS work you’re not doing it professionally. A hobby isn’t a negative thing, until last December my OSS work was entirely a hobby too, and it’s still a hobby in my spare time too.
That use of "hobby" vs. "professional" is completely irrelevant to the discussion, and is effectively condescending and ad hominem. In my experience, the quality and committment of a developer's open source work are often unrelated to whether she gets paid directly by an employer to do it. Getting paid *does* make it possible to devote more time, rather than passion, to one's projects. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlRAPOIACgkQ+gerLs4ltQ5vWACfRDCmT5yjkqfeBB+4xGAiBnAv n7MAoKag+7GkicRdZ9eSpJdz+HKml6aG =5Uxr -----END PGP SIGNATURE-----
Stefan Krah <stefankrah <at> freenet.de> writes:
(for example right now bytereef.org is down, so we’d not discover any files there).
Indeed. It was up reliably since 2005, down for maintenance on September 23rd (before ShellShock ...). Then I discovered that someone had put up m3-cdecimal on PyPI (presumably abusing PyPI as their private repo --- there are several m3-* packages now).
This triggered some reflection on whether I would make a significant effort in the future to keep things running smoothly for an open source community where authors are largely viewed as expendable.
I don't know what it means for "authors to be largely viewed as expendable", but half the point of hosting things on PyPI is that you *don't* need to do any work at all as an author for reliable delivery of your package.
Subsequently the downtime (again, the first one since 2005) was picked up for propagandistic purposes on Twitter and Reddit.
Ok, but you seem to be doing the other side's propaganda. Every single person I've spoken to agrees that this just underscores the need to encourage packages to be on PyPI.
Last year I would have felt an obligation to minimize the downtime to an hour at most. I no longer feel any such obligations and I'll do it when I have time.
Ok. The PyPI administrators still feel an obligation to their users, so I'll prefer packages under their care.
Stefan Krah
Cheers, Alex
Hi Holger, On 10/11/2014 12:31 AM, holger krekel wrote:
I understand that as a fairly generic security statement. But I was trying to rather ask about use cases and scenarios where precisely the --extra-index-url option is useful and to be recommended. I'd be grateful if Nick or you could still describe use cases, especially outside PEP470 external links context (the option existed before so i presume there must be some use cases).
I don't use it anymore (because these days for everything other than interactive playing around, I install only from a curated local index specifically limited to each project's dependencies using --no-index and --find-links), but I used to use it. My use case was this: generally dependencies were installed from PyPI, but occasionally I would need to patch a dependency, so I would create an sdist with a patched version number (e.g. if I patched 2.0.1, I would create an sdist for version 2.0.1.obc1, where "obc" is a tag based on my company name or the project) and add this patched sdist to my own index, which I would add to my installs with --extra-index-url. Because I used a patched version number and pinned all dependencies exactly, it didn't matter to me that both PyPI and my "extra index" were considered for installation; in fact that was convenient, since it meant I could very easily upgrade to a newer PyPI release. I never used it for "private" non-PyPI packages. Carl
On 10/11/2014 12:31 AM, holger krekel wrote:
I understand that as a fairly generic security statement. But I was trying to rather ask about use cases and scenarios where precisely the --extra-index-url option is useful and to be recommended. I'd be grateful if Nick or you could still describe use cases, especially outside PEP470 external links context (the option existed before so i presume there must be some use cases).
I use it to have a local repository of wheels for projects that do not (yet) host wheels on PyPI. I want the fallback to PyPI should newer versions be released. Paul
Hi Carl, Paul, all, On Sat, Oct 11, 2014 at 18:48 -0600, Carl Meyer wrote:
Hi Holger,
On 10/11/2014 12:31 AM, holger krekel wrote:
I understand that as a fairly generic security statement. But I was trying to rather ask about use cases and scenarios where precisely the --extra-index-url option is useful and to be recommended. I'd be grateful if Nick or you could still describe use cases, especially outside PEP470 external links context (the option existed before so i presume there must be some use cases).
I don't use it anymore (because these days for everything other than interactive playing around, I install only from a curated local index specifically limited to each project's dependencies using --no-index and --find-links), but I used to use it. My use case was this: generally dependencies were installed from PyPI, but occasionally I would need to patch a dependency, so I would create an sdist with a patched version number (e.g. if I patched 2.0.1, I would create an sdist for version 2.0.1.obc1, where "obc" is a tag based on my company name or the project) and add this patched sdist to my own index, which I would add to my installs with --extra-index-url.
Because I used a patched version number and pinned all dependencies exactly, it didn't matter to me that both PyPI and my "extra index" were considered for installation; in fact that was convenient, since it meant I could very easily upgrade to a newer PyPI release.
I never used it for "private" non-PyPI packages.
Right, makes sense and is in line with what Paul noted as his use case (adding wheels to existing pypi sdists): I think it's good and safe to use it when you are adding/patching things wrt existing projects on pypi.python.org. However, many people don't realize that using --extra-index-url to install private packages is a bad idea unless you register every private package as an empty pypi package. But the latter restriction is virtually never mentioned (and is an unrealistic recommendation in my opinion), see examples here: http://devcenter.gemfury.com/articles/pypi-server.html http://exhuma.github.io/mypi/index-config.html and I just noted that the very Python guide on packaging is advertising using plain --extra-index-url for private packages as well: http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi and, besides the need for fixing the various discussions/pages, i think that PEP470 should contribute to a more careful discussion of the feature (it's fine for the actual external linking to existing pypi projects usecase, mind you). And i guess pip should have a warning note in the option help to help educating users. best, holger
On 13 October 2014 11:40, holger krekel <holger@merlinux.eu> wrote:
and I just noted that the very Python guide on packaging is advertising using plain --extra-index-url for private packages as well:
http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi
I can see your point here (I'm not sure I agree with it, but that's a separate issue). If you want to propose a patch for the packaging user guide, we can discuss it there.
and, besides the need for fixing the various discussions/pages, i think that PEP470 should contribute to a more careful discussion of the feature (it's fine for the actual external linking to existing pypi projects usecase, mind you).
So if I read you correctly, you're saying that the PEP 470 usage of --extra-index-url is fine. That's good. I don't think it's the place of PEP 470 to discuss *other* uses of --extra-index-url. Having an example in there seemed fine to me, but if it brings up issues unrelated to the PEP then I think it's a distraction and should be removed. And I believe that's what has happened. So again, that's good.
And i guess pip should have a warning note in the option help to help educating users.
Again, not for the PEP, but feel free to raise a PR for pip (but once again, I reserve the right to disagree with that PR when it's raised :-)). Paul
On Mon, Oct 13, 2014 at 12:00 +0100, Paul Moore wrote:
On 13 October 2014 11:40, holger krekel <holger@merlinux.eu> wrote:
and I just noted that the very Python guide on packaging is advertising using plain --extra-index-url for private packages as well:
http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi
I can see your point here (I'm not sure I agree with it, but that's a separate issue).
Sorry but what is there to agree or discuss? If recommending --extra-index-url for private packages does not come with a big fat warning that you need to publically register the name with PyPI, it exposes users to direct compromise of their machine, plain and simple. best, holger
If you want to propose a patch for the packaging user guide, we can discuss it there.
and, besides the need for fixing the various discussions/pages, i think that PEP470 should contribute to a more careful discussion of the feature (it's fine for the actual external linking to existing pypi projects usecase, mind you).
So if I read you correctly, you're saying that the PEP 470 usage of --extra-index-url is fine. That's good.
I don't think it's the place of PEP 470 to discuss *other* uses of --extra-index-url. Having an example in there seemed fine to me, but if it brings up issues unrelated to the PEP then I think it's a distraction and should be removed. And I believe that's what has happened. So again, that's good.
And i guess pip should have a warning note in the option help to help educating users.
Again, not for the PEP, but feel free to raise a PR for pip (but once again, I reserve the right to disagree with that PR when it's raised :-)).
Paul
On 13 October 2014 13:08, holger krekel <holger@merlinux.eu> wrote:
On Mon, Oct 13, 2014 at 12:00 +0100, Paul Moore wrote:
On 13 October 2014 11:40, holger krekel <holger@merlinux.eu> wrote:
and I just noted that the very Python guide on packaging is advertising using plain --extra-index-url for private packages as well:
http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi
I can see your point here (I'm not sure I agree with it, but that's a separate issue).
Sorry but what is there to agree or discuss? If recommending --extra-index-url for private packages does not come with a big fat warning that you need to publically register the name with PyPI, it exposes users to direct compromise of their machine, plain and simple.
I would view it as a matter of the trust model. If you don't trust PyPI, then you should not download direct from there. That applies whether or not you have a private index as well. if you do trust PyPI, then you presumably understand the risks. I'd be happy enough to see a note that whenever you use pip without specifying --no-index you trust PyPI. I don't mind if there's a further note that if you serve packages from your local index they will be considered as equal candidates with packages of the same name on PyPI *regardless of who uploaded them to PyPI*. But I don't accept that there's a need to over-stress the risk. After all, if I mistype an install command as "pip install devpy", I'm just as exposed to compromise of my machine. Paul.
participants (8)
-
Alex Gaynor -
Carl Meyer -
Donald Stufft -
holger krekel -
Nick Coghlan -
Paul Moore -
Stefan Krah -
Tres Seaver