[Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))

[Noah Kantrowitz]

On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:

> From: Paul Moore
>> On 13 July 2013 10:05, Paul Moore <p.f.moore at gmail.com> wrote:
>> How robust is the process of upgrading pip using itself? Specifically on
>> Windows, where these things typically seem less reliable.
>> 
>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason
>> for the failure is simple enough to explain - the pip.exe wrapper is held open
>> by the OS while it's in use, so that the upgrade cannot replace it.
>> 
>> The result is a failed upgrade and a partially installed new version of pip. In
>> practice, the exe stubs are probably added fairly late in the install (at least
>> when installing from sdist, with a wheel that depends on the order of the files
>> in the wheel), so it's probably only a little bit broken, but "a little bit
>> broken" is still broken :-(
>> 
>> On the other hand, "python -m pip install -U pip" works fine because it avoids
>> the exe wrappers.
>> 
>> There's a lot of scope for user confusion and frustration in all this. For
>> standalone pip I've tended to recommend "don't do that" - manually uninstall and
>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective.
>> That sort of advice isn't going to be realistic for a pip bundled with CPython.
>> 
>> Does anyone have any suggestions?
> 
> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
> 
> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
> 
> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
> 
> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
> 
> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.

The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.

--Noah