Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote:
How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-( On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Paul. PS In better news, apart from this issue, pip upgrades of pip and setuptools seem fine.
From: Paul Moore
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it.
The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-(
On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers.
There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython.
Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. Cheers, Steve
On 14 July 2013 17:45, Steve Dower <Steve.Dower@microsoft.com> wrote:
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
Thanks. That sounds annoyingly complex (pretty much as I expected, though). My feeling is that I'd like to remove the exe wrapper altogether, and use a .py file. I need to check what issues there might be with that before recommending it, though. Paul
On Jul 14, 2013, at 1:03 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 14 July 2013 17:45, Steve Dower <Steve.Dower@microsoft.com> wrote: So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
Thanks. That sounds annoyingly complex (pretty much as I expected, though). My feeling is that I'd like to remove the exe wrapper altogether, and use a .py file. I need to check what issues there might be with that before recommending it, though.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 14 July 2013 18:06, Donald Stufft <donald@stufft.io> wrote:
Wouldn't a .py file make the command `pip.py`` and not ``pip`` ?
Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... But the various "replace the exe afterwards" hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Paul
On 15 Jul 2013 05:44, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 14 July 2013 18:06, Donald Stufft <donald@stufft.io> wrote:
Wouldn't a .py file make the command `pip.py`` and not ``pip`` ?
Not if .py is a registered extension. What I can't remember is whether it
needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so...
But the various "replace the exe afterwards" hacks sound awfully
complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure.
My strong preference here is to remove the current use of setuptools
entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best.
Question for Nick, Brett and any other core devs around: Would python-dev
be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Not sure it fits the launcher, but having something along those lines in the stdlib makes sense (especially in the context of a pip bundling PEP). Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem. Cheers, Nick.
Paul
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Jul 14, 2013, at 3:06 PM, Nick Coghlan wrote:
On 15 Jul 2013 05:44, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 14 July 2013 18:06, Donald Stufft <donald@stufft.io> wrote:
Wouldn't a .py file make the command `pip.py`` and not ``pip`` ?
Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so...
But the various "replace the exe afterwards" hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure.
My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best.
Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role?
Not sure it fits the launcher, but having something along those lines in the stdlib makes sense (especially in the context of a pip bundling PEP).
Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem.
They do the "exec a helper executable that replaces the original" approach, which works fine for non-console apps since there isn't the problem of the shell getting confused :-/ --Noah
On 14 July 2013 23:09, Noah Kantrowitz <noah@coderanger.net> wrote:
Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem.
They do the "exec a helper executable that replaces the original" approach, which works fine for non-console apps since there isn't the problem of the shell getting confused :-/
Generally, I don't think that going down the route of MSIs is a good move. They aren't a good fit for this problem. Apart from anything else, they won't support installing into a virtualenv. Paul
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
From: Paul Moore
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it.
The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-(
On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers.
There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython.
Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. --Noah
On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
From: Paul Moore
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it.
The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-(
On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers.
There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython.
Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.
I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years.
On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote:
On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
From: Paul Moore
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it.
The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-(
On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers.
There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython.
Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.
I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years.
Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. --Noah
On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote:
On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote:
On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
From: Paul Moore
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it.
The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-(
On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers.
There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython.
Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.
I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years.
Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best.
Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM … --Noah
On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote:
On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote:
On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote:
On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
From: Paul Moore
On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable.
OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it.
The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but "a little bit broken" is still broken :-(
On the other hand, "python -m pip install -U pip" works fine because it avoids the exe wrappers.
There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend "don't do that" - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython.
Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.
I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years.
Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best.
Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM …
Ignore my ignoring, with os.execl command flow does return back to the controlling terminal process (the new process continues in the background) and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. Yay for not completely misremembering, boo for this being so complicated. --Noah
On Sun, Jul 14, 2013 at 2:10 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote:
On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote:
On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote:
On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote:
From: Paul Moore > On 13 July 2013 10:05, Paul Moore <p.f.moore@gmail.com> wrote: > How robust is the process of upgrading pip using itself? Specifically on > Windows, where these things typically seem less reliable. > > OK, I just did some tests. On Windows, "pip install -U pip" FAILS. The reason > for the failure is simple enough to explain - the pip.exe wrapper is held open > by the OS while it's in use, so that the upgrade cannot replace it. > > The result is a failed upgrade and a partially installed new version of pip. In > practice, the exe stubs are probably added fairly late in the install (at least > when installing from sdist, with a wheel that depends on the order of the files > in the wheel), so it's probably only a little bit broken, but "a little bit > broken" is still broken :-( > > On the other hand, "python -m pip install -U pip" works fine because it avoids > the exe wrappers. > > There's a lot of scope for user confusion and frustration in all this. For > standalone pip I've tended to recommend "don't do that" - manually uninstall and > reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. > That sort of advice isn't going to be realistic for a pip bundled with CPython. > > Does anyone have any suggestions?
Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')?
Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output.
MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy.
If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change.
So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed.
The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done.
I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years.
Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best.
Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM …
Ignore my ignoring, with os.execl command flow does return back to the controlling terminal process (the new process continues in the background) and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. Yay for not completely misremembering, boo for this being so complicated.
I expected I was wrong, but I appreciate you looking at it to be certain.
participants (6)
-
Donald Stufft -
Ian Cordasco -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
Steve Dower