[Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))

[Noah Kantrowitz]

On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote:

> 
> On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote:
> 
>> On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>>> 
>>> On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
>>> 
>>>> From: Paul Moore
>>>>> On 13 July 2013 10:05, Paul Moore <p.f.moore at gmail.com> wrote:
>>>>> How robust is the process of upgrading pip using itself? Specifically on
>>>>> Windows, where these things typically seem less reliable.
>>>>> 
>>>>> OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason
>>>>> for the failure is simple enough to explain - the pip.exe wrapper is held open
>>>>> by the OS while it's in use, so that the upgrade cannot replace it.
>>>>> 
>>>>> The result is a failed upgrade and a partially installed new version of pip. In
>>>>> practice, the exe stubs are probably added fairly late in the install (at least
>>>>> when installing from sdist, with a wheel that depends on the order of the files
>>>>> in the wheel), so it's probably only a little bit broken, but "a little bit
>>>>> broken" is still broken :-(
>>>>> 
>>>>> On the other hand, "python -m pip install -U pip" works fine because it avoids
>>>>> the exe wrappers.
>>>>> 
>>>>> There's a lot of scope for user confusion and frustration in all this. For
>>>>> standalone pip I've tended to recommend "don't do that" - manually uninstall and
>>>>> reinstall pip, or recreate your virtualenv. It's not nice, but it's effective.
>>>>> That sort of advice isn't going to be realistic for a pip bundled with CPython.
>>>>> 
>>>>> Does anyone have any suggestions?
>>>> 
>>>> Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
>>>> 
>>>> Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
>>>> 
>>>> MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
>>>> 
>>>> If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
>>>> 
>>>> So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
>>> 
>>> The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.
>> 
>> I accidentally only emailed Paul earlier, but why can't we upgrade the
>> pip module with the exe and then replace the process (using something
>> in the os.exec* family) with `python -m pip update-exe` which could
>> then succeed since the OS isn't holding onto the exe file? I could be
>> missing something entirely obvious since I haven't developed
>> (directly) on or for Windows in at least 5 years.
> 
> Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best.

Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM …

--Noah