On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote:
On 12 October 2014 09:49, Donald Stufft <donald@stufft.io> wrote:
On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 October 2014 04:29, Donald Stufft <donald@stufft.io> wrote:
I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes.
I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I’m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it’d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we’d not discover any files there).
If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links.
Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now?
Sure, we could do that too. Regardless, I don't believe we should let the conversion of the silent security failures into noisy exceptions block the other usability enhancements - if decoupling the two decisions will achieve that, then let's go down that path.
I think I suggested at one point that the PEP could potentially commit to making the backwards compatible changes on the server side, and then for the *in*compatible changes, only commit to *looking at the numbers again* in a few months time after folks have had a chance to experience the replacement model. Client applications would be officially granted permission to ignore the PEP 438 external hosting mechanism in favour of the new PEP 470 external index advertisements.
The sequence of events from a compatibility break/user experience enhancement perspective would then be:
* new model added to PyPI (no compatibility break) * old model dropped from pip (client side compatibility break, error handling improvements) * old model dropped from PyPI (conditional, based on changes in server metrics)
I like this strategy and would add that I guess it's then only a matter of when not if the old model is dropped (and the PEP could encode that). The question is how to best do the "(almost) no compat break" for a reasonable definition of "almost". I am mostly concerned about the verified externals and "crawl urls". If they keep showing up as "rel=download" links on the simple page that should be fine for existing tools, right? But with PEP470 they might get an additional attribute so that a newer pip can recognize it and advise accordingly. How pypi/warehouse internally keeps the state during/post PEP470 is less clear to me but it might involve a one-time conversion from the existing information to the new one. I am actually wondering if a new pip could change its UI even without any changes to pypi.python.org? For example, when "pip install pil" happens it would say: Could not find any release file for "Pil" on https://pypi.python.org itself but the project has historically had additional external pages which linked to release files: http://effbot.org/zon/pil-changes-115.html/ http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz http://www.pythonware.com/products/pil Please manully check these links if they directly link to or contain release files for "Pil" and then add an appropriate one like this:: pip install --find-links URL [other options] Pil (See <link> for more info ...) and pip could deduce this information from the current https://pypi.python.org/simple/pil/ page. (The pip message above may not be precisely right but you get the idea). FWIW this strategy of advising to use newer/other options but keeping the old working (undocumneted or documented with a warning) is how i generally try to handle things with pytest. It allows people to use the new way and incrementally migrate their setups. And if someone complains about problems with the old ways, point them to the new ways so there is no need to maintain the old ways too much. Even then, it would probably still makes sense to advance the server side and allow explicit registration of external index pages i guess (and maybe findlinks pages so you can really just use apache/nginx directly?) and then have pip provide a more precise message.
That changes the dynamic from "the updated user experience will be better, trust us" to "if you prefer the old user experience, you can keep using the old pip, if you prefer the new user experience, upgrade".
I think it's very much fine to say "this is the future" as long as old-tool/current-server instances keep working for a longer while. Python packaging infra is a bit like a big container ship these days -- which is both a good and a bad thing i guess -- and people build all kinds of funny containers on top. best, holger
The eventual decision on when to drop the legacy model from PyPI can then by made based on a richer set of input data, that includes the level of adoption of the new version of pip that treats all missing link targets as potential errors.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia