On Oct 13, 2014, at 7:41 AM, holger krekel <holger@merlinux.eu> wrote:
On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote:
On 12 October 2014 09:49, Donald Stufft <donald@stufft.io> wrote:
On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links.
Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now?
Sure, we could do that too. Regardless, I don't believe we should let the conversion of the silent security failures into noisy exceptions block the other usability enhancements - if decoupling the two decisions will achieve that, then let's go down that path.
I think I suggested at one point that the PEP could potentially commit to making the backwards compatible changes on the server side, and then for the *in*compatible changes, only commit to *looking at the numbers again* in a few months time after folks have had a chance to experience the replacement model. Client applications would be officially granted permission to ignore the PEP 438 external hosting mechanism in favour of the new PEP 470 external index advertisements.
The sequence of events from a compatibility break/user experience enhancement perspective would then be:
* new model added to PyPI (no compatibility break) * old model dropped from pip (client side compatibility break, error handling improvements) * old model dropped from PyPI (conditional, based on changes in server metrics)
I like this strategy and would add that I guess it's then only a matter of when not if the old model is dropped (and the PEP could encode that).
This is more or less the sequence of events that will happen with PEP 470 as it is except that PEP 470 doesn’t make the deprecation optional. I do not believe it is a good idea to make it optional because it is a silent security risk. The other main difference is that PEP 470 doesn’t allow new projects to ever register externally hosted files in the old way.
The question is how to best do the "(almost) no compat break" for a reasonable definition of "almost". I am mostly concerned about the verified externals and "crawl urls". If they keep showing up as "rel=download" links on the simple page that should be fine for existing tools, right?
Keeping backwards compatibility for unsafe things is completely off the table as far as I’m concerned. Specifically that means the crawl links. I think keeping backwards compatibility for the ~70 projects which are doing things safely today is a waste of resources in the long run, but something that should be phased in as PEP 470 does. More people were broken when PyPI went forced TLS and then again when PyPI switched from a sha1 certificate to a sha256 certificate. The numbers of safe installs is miniscule.
But with PEP470 they might get an additional attribute so that a newer pip can recognize it and advise accordingly. How pypi/warehouse internally keeps the state during/post PEP470 is less clear to me but it might involve a one-time conversion from the existing information to the new one.
I am actually wondering if a new pip could change its UI even without any changes to pypi.python.org? For example, when "pip install pil" happens it would say:
Could not find any release file for "Pil" on https://pypi.python.org itself but the project has historically had additional external pages which linked to release files:
http://effbot.org/zon/pil-changes-115.html/ http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz http://www.pythonware.com/products/pil
Please manully check these links if they directly link to or contain release files for "Pil" and then add an appropriate one like this::
pip install --find-links URL [other options] Pil
(See <link> for more info ...)
and pip could deduce this information from the current https://pypi.python.org/simple/pil/page. (The pip message above may not be precisely right but you get the idea).
FWIW this strategy of advising to use newer/other options but keeping the old working (undocumneted or documented with a warning) is how i generally try to handle things with pytest. It allows people to use the new way and incrementally migrate their setups. And if someone complains about problems with the old ways, point them to the new ways so there is no need to maintain the old ways too much.
It’s a fine way of doing thing, and we do it in pip too, when (and this is important), it’s not a security sensitive issue.
Even then, it would probably still makes sense to advance the server side and allow explicit registration of external index pages i guess (and maybe findlinks pages so you can really just use apache/nginx directly?) and then have pip provide a more precise message.
That changes the dynamic from "the updated user experience will be better, trust us" to "if you prefer the old user experience, you can keep using the old pip, if you prefer the new user experience, upgrade".
I think it's very much fine to say "this is the future" as long as old-tool/current-server instances keep working for a longer while.
Python packaging infra is a bit like a big container ship these days -- which is both a good and a bad thing i guess -- and people build all kinds of funny containers on top.
best, holger
The eventual decision on when to drop the legacy model from PyPI can then by made based on a richer set of input data, that includes the level of adoption of the new version of pip that treats all missing link targets as potential errors.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA