Finally fix installer to add Python to %PATH% on Windows

Hi, I'd like to You probably know that after installation on Windows system it is possible to call Python from Explorer's Run dialog (Win-R). It is because Python path is added to App Paths registry key and Windows Explorer shell checks this key when looking for executable. But Python doesn't work from cmd session and, more importantly, *Python doesn't work from .bat files*. It is because cmd shell doesn't know about App Paths and relies on system PATH to find executable. As far as I remember, there is no function in Python stdlib either, that executes processes and does lookups in App Paths. I never paid much attention to this fact, because I put several .bat files for every 25, 26, 27, 31 and 32 version of Python into PATH manually. But when bootstrap script for build environment of Native Client (NaCl) said that I have no Python available and started to install its own, I've asked myself - "How come? There are 5! possible versions of Python on my system." It appeared that the following .bat file doesn't work: ---cut mypy.bat-- python.exe ---cut mypy.bat-- C:\>mypy.bat C:\>python.exe 'python.exe' is not recognized as an internal or external command, operable program or batch file. I've seen about 7 requests to add Python into %PATH% in installer. All closed with no result, but with some fear and uncertainty. Martin feared that MSI installers are not able to remove entry from PATH and even if they can, they may kill the whole PATH instead of removing just one entry. To prove or dispel these fears, I've just installed/uninstalled Mercurial from mercurial-1.7.3-1-x86.msi and App Engine from GoogleAppEngine-1.4.1.msi several times. Both add entries to PATH and both remove them without any further problems. Should we finally add this to 3.2 installer for Python? -- anatoly t.

On Fri, Jan 28, 2011 at 10:12, anatoly techtonik <techtonik@gmail.com>wrote:
Definitely not for 3.2, but this is something I'd like to look into for 3.3. Recently I've talked to two Python trainers/educators and the major gripe their attendees see is that you can't just sit down and type "python" and have something work. For multi-Python installs, we'll have to define what that "something" is, but I think it should be possible for the installer to optionally put Python into the path, and to also remove itself on uninstall. One of said trainers is running a course inside my company right now and the training room VMs they are running on do not have the path setup. Some users were puzzled as to why "python foo.py" doesn't work, but executing "foo.py" does (via file association). One quick-and-dirty solution was to create a "Command Shell" shortcut in the start menu which would just be a batch file that adds Python to the path for that cmd session. It would be kind of similar to the "Python (command line)" shortcut, which uses pythonw.exe. I think we can do better than this, though.

On 28/01/2011 16:29, Brian Curtin wrote:
I don't think, ultimately, that there is any insurmountable technical objection. There are misgivings but they could undoubtedly be overcome or overridden. But it would require someone to patch the MSI builder so it added the functionality and -- I think -- offered it as an option which could be enabled or disabled. TJG

On 28/01/2011 16:29, Brian Curtin wrote:
I've helped quite a few "python newbies" on Windows who are also surprised / frustrated on learning that "python" on the command line doesn't work after installing python. All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Am 28.01.2011 20:29, schrieb Raymond Hettinger:
At the very least, we should add some prominent instructions for getting the command line version up and running.
/me pops out of Guido's time machine and says: "execute Tools/scripts/win_add2path.py" I'm -1 on adding Python to %PATH%. The private MSVCRT DLLs may lead to unexpected side effects and it doesn't scale at all. What about people with more than one Python installation? I suggest that we add a single user specific directory or a global directory to %PATH% for all installations. Then the Python installer or 3rd party modules can drop executables like python3.3.exe or plip-3.3.exe into this directory. A .bat file won't do good because .bat files must be called with "call python33.bat" from another .bat file or the first one gets terminated. We can even use a single and simple executable as template for all tasks: * get registry key from resource section of the executable * use the registry key to lookup the location and name of pythonXX.dll * load DLL * get optional dotted module name for resource section * either fire up interpreter as shell, with **argv or -m dotted.module.name **argv Done ;) Christian

On Fri, Jan 28, 2011 at 14:34, Christian Heimes <lists@cheimes.de> wrote:
The same "problem" exists when it comes to file associations. The last installer you've run wins the battle. Since setting file associations is optional, and only one association can exist, I don't see why we can't do the same for the PATH.

On Fri, Jan 28, 2011 at 10:34 PM, Christian Heimes <lists@cheimes.de> wrote:
Can you explain that part? There are no any MSVCRT DLLs in my Python26+ installation directories.
python33.exe, but user story about people with more than one Python installation is a different one.
Wow. I've spent so many years in Windows console and didn't know that. Thanks.
Actually, I would like to see the code that dynamically finds pythonXX.dll that is available on the system, and loads it into memory. This will be extremely useful for writing 3rd party application plugins in Python. Plugins that they only work when Python is installed and it doesn't really matter which Python version is there. But that is another story also. -- anatoly t.

Ok. Here is the patch. I used Orca to reverse installer tables of Mercurial MSI and inserted similar entry for Python. Also available for review at: http://codereview.appspot.com/4023055 -- anatoly t.

On Mon, Jan 31, 2011 at 11:49 PM, Brian Curtin <brian.curtin@gmail.com> wrote:
The master issue at http://bugs.python.org/issue3561 specifically proposes further discussion proposals on python-dev.

Am 31.01.2011 22:13, schrieb anatoly techtonik:
Ok. Here is the patch. I used Orca to reverse installer tables of Mercurial MSI and inserted similar entry for Python.
This doesn't do uninstallation correctly. Regards, Martin

On Tue, Feb 1, 2011 at 2:10 AM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I do not know where did you get this info, but I assure you that it does, and I've stressed this in the first post of this thread. You may repeat the experiment yourself. Here is the patched installer: https://docs.google.com/leaf?id=0Bwu0AJeJuth_MWJjMTgzNmQtYWIzOS00ODhlLTk3YjAtYWJiYTdmYWQwNzU5&sort=name&layout=list&num=50

On 28/01/2011 19:21, Michael Foord wrote:
Yes, I've always found it a surprising disappointment that I have to manually munge the PATH and the installer doesn't *even* offer to do it for me. But, since I don't know how to help fix the installer, I've just generally stfu'd on this issue... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On Fri, Feb 4, 2011 at 7:43 PM, Chris Withers <chris@simplistix.co.uk> wrote:
This is how to fix an installer. http://codereview.appspot.com/4023055/diff/1/Tools/msi/msi.py Right now I am waiting for Martin's decision (and probably not only me). He is responsible for MSI stuff and the only one, who can integrate the patch. I don't want to put too much pressure on him, but it would be more comfortable for all of us to know what is he up to. I'd like to see this in 3.2 release, of course. -- anatoly t.

On Sun, Feb 6, 2011 at 04:14, anatoly techtonik <techtonik@gmail.com> wrote:
We're one week from the 3.2 final release, so adding a feature such as this is definitely out of the question. Sorry to speak for Martin, but I'm certain he would agree. There are still outstanding considerations in the various issues on the tracker, so it would be best to address them before requesting integration. Example: What should happen when there is another Python installation on the path?

On 06/02/2011 15:20, Brian Curtin wrote:
Same as happens with most Windows apps: last one installed wins. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On 06/02/2011 15:25, Brian Curtin wrote:
So put the new path before the old path, or replace it? The current patch appends to the end.
I believe the last path wins in Windows land, so that would be fine. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On 6 February 2011 15:35, Nick Coghlan <ncoghlan@gmail.com> wrote:
... and "at the beginning" can be a pain due to unintended overriding of existing user commands (not likely in the case of Python, where there's only python, pythonw, w9xpopen and various bdist_wininst "RemoveXXX" commands, but still possible). "Before any existing Python directories, otherwise at the end" is the closest to what I suspect most users want (certainly it matches my preferences, and anything else would have me manually editing PATH anyway, so is of no use to me in practice). Paul.

Paul Moore writes:
Unfortunately, what is "no use to person X in practice" is a function of X. I suspect that's why this hasn't been done. Specifically, it seems to me that there are use cases for each of 1. Append (eg, if both python3 and python2 provide "python.exe", for experimental use of python3). 2. Prepend (actually, not a use case; just common, and therefore "intuitive", practice). 3. "Moore's rule" (put latest and greatest ahead of other versions but not interfere with previously installed apps). Maybe it should be user-configurable.

On 6 February 2011 17:07, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Absolutely :-) And it's also why I'm reluctant to support it - even though I agree that not having "python" just work is a PITA.
Fame at last :-) I've seen both (1) and (2) in common use. Both have disadvantages, particularly if you try to support multiple versions being installed at once (something which is nearly unheard of in Windows, and hence why no commonly used solution really does a good job of it). I've never seen (3), and in all honesty I don't expect it to be practical - too many special cases to consider. It was more of a straw man example of what "do it right" might really mean...
Maybe it should be user-configurable.
-1. Too much complexity. What I *have* seen is Oracle's "Home Selector", which is a program installed with Oracle's software which keeps track of which versions of Oracle you have installed, and gives you a GUI to move them up & down in priority, or disable versions. It then updates PATH appropriately. Ultimately, all it is in Python's terms, is a GUI means of editing PATH, so I'm not sure it's of any real use to us. (For Oracle, I think it fiddles with some other registry values, so it does have some value there...) One point - no matter what we do, we only need to consider 3.3 and later. People using 3.2 or earlier still have to manually fix up PATH how they want. So if we do add python to PATH in 3.3, we don't actually have a "what if people want to install multiple versions" issue until 3.4 comes out. I'm assuming we don't try to support multiple maintenance releases (3.3.1 and 3.3.2) being installed at once... Paul.

I've got a feeling that policy is evil and can not be applied cleanly when change falls out of scope of Python core .c sources and .py files from standard library. Right now the proposal is to add Python to %PATH% to make Python more user friendly for newbies. I can't see what can become worse than it is now. People are discussing advanced scenarios with multiple Python installations, but I propose first to make Python a normal command line user-friendly system application for Windows, and then discuss more advanced usage scenarios to make it developer-friendly in subsequent versions. Time is passing by and nothing happens. -- anatoly t.

I'd like to see this in 3.2 release, of course.
As Brian already asserted: that's not feasible. I still haven't managed to test your installer, and may not be able to for the next few weeks. It's also against the policy for release candidates to add such a change at this point. I believe the change, if implemented, needs to be optional (which I believe your change is not). Regards, Martin

Hello See http://bugs.python.org/issue3561 (rejected by Martin). Regards

On Fri, Jan 28, 2011 at 10:12, anatoly techtonik <techtonik@gmail.com>wrote:
Definitely not for 3.2, but this is something I'd like to look into for 3.3. Recently I've talked to two Python trainers/educators and the major gripe their attendees see is that you can't just sit down and type "python" and have something work. For multi-Python installs, we'll have to define what that "something" is, but I think it should be possible for the installer to optionally put Python into the path, and to also remove itself on uninstall. One of said trainers is running a course inside my company right now and the training room VMs they are running on do not have the path setup. Some users were puzzled as to why "python foo.py" doesn't work, but executing "foo.py" does (via file association). One quick-and-dirty solution was to create a "Command Shell" shortcut in the start menu which would just be a batch file that adds Python to the path for that cmd session. It would be kind of similar to the "Python (command line)" shortcut, which uses pythonw.exe. I think we can do better than this, though.

On 28/01/2011 16:29, Brian Curtin wrote:
I don't think, ultimately, that there is any insurmountable technical objection. There are misgivings but they could undoubtedly be overcome or overridden. But it would require someone to patch the MSI builder so it added the functionality and -- I think -- offered it as an option which could be enabled or disabled. TJG

On 28/01/2011 16:29, Brian Curtin wrote:
I've helped quite a few "python newbies" on Windows who are also surprised / frustrated on learning that "python" on the command line doesn't work after installing python. All the best, Michael Foord
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Am 28.01.2011 20:29, schrieb Raymond Hettinger:
At the very least, we should add some prominent instructions for getting the command line version up and running.
/me pops out of Guido's time machine and says: "execute Tools/scripts/win_add2path.py" I'm -1 on adding Python to %PATH%. The private MSVCRT DLLs may lead to unexpected side effects and it doesn't scale at all. What about people with more than one Python installation? I suggest that we add a single user specific directory or a global directory to %PATH% for all installations. Then the Python installer or 3rd party modules can drop executables like python3.3.exe or plip-3.3.exe into this directory. A .bat file won't do good because .bat files must be called with "call python33.bat" from another .bat file or the first one gets terminated. We can even use a single and simple executable as template for all tasks: * get registry key from resource section of the executable * use the registry key to lookup the location and name of pythonXX.dll * load DLL * get optional dotted module name for resource section * either fire up interpreter as shell, with **argv or -m dotted.module.name **argv Done ;) Christian

On Fri, Jan 28, 2011 at 14:34, Christian Heimes <lists@cheimes.de> wrote:
The same "problem" exists when it comes to file associations. The last installer you've run wins the battle. Since setting file associations is optional, and only one association can exist, I don't see why we can't do the same for the PATH.

On Fri, Jan 28, 2011 at 10:34 PM, Christian Heimes <lists@cheimes.de> wrote:
Can you explain that part? There are no any MSVCRT DLLs in my Python26+ installation directories.
python33.exe, but user story about people with more than one Python installation is a different one.
Wow. I've spent so many years in Windows console and didn't know that. Thanks.
Actually, I would like to see the code that dynamically finds pythonXX.dll that is available on the system, and loads it into memory. This will be extremely useful for writing 3rd party application plugins in Python. Plugins that they only work when Python is installed and it doesn't really matter which Python version is there. But that is another story also. -- anatoly t.

Ok. Here is the patch. I used Orca to reverse installer tables of Mercurial MSI and inserted similar entry for Python. Also available for review at: http://codereview.appspot.com/4023055 -- anatoly t.

On Mon, Jan 31, 2011 at 11:49 PM, Brian Curtin <brian.curtin@gmail.com> wrote:
The master issue at http://bugs.python.org/issue3561 specifically proposes further discussion proposals on python-dev.

Am 31.01.2011 22:13, schrieb anatoly techtonik:
Ok. Here is the patch. I used Orca to reverse installer tables of Mercurial MSI and inserted similar entry for Python.
This doesn't do uninstallation correctly. Regards, Martin

On Tue, Feb 1, 2011 at 2:10 AM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I do not know where did you get this info, but I assure you that it does, and I've stressed this in the first post of this thread. You may repeat the experiment yourself. Here is the patched installer: https://docs.google.com/leaf?id=0Bwu0AJeJuth_MWJjMTgzNmQtYWIzOS00ODhlLTk3YjAtYWJiYTdmYWQwNzU5&sort=name&layout=list&num=50

On 28/01/2011 19:21, Michael Foord wrote:
Yes, I've always found it a surprising disappointment that I have to manually munge the PATH and the installer doesn't *even* offer to do it for me. But, since I don't know how to help fix the installer, I've just generally stfu'd on this issue... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On Fri, Feb 4, 2011 at 7:43 PM, Chris Withers <chris@simplistix.co.uk> wrote:
This is how to fix an installer. http://codereview.appspot.com/4023055/diff/1/Tools/msi/msi.py Right now I am waiting for Martin's decision (and probably not only me). He is responsible for MSI stuff and the only one, who can integrate the patch. I don't want to put too much pressure on him, but it would be more comfortable for all of us to know what is he up to. I'd like to see this in 3.2 release, of course. -- anatoly t.

On Sun, Feb 6, 2011 at 04:14, anatoly techtonik <techtonik@gmail.com> wrote:
We're one week from the 3.2 final release, so adding a feature such as this is definitely out of the question. Sorry to speak for Martin, but I'm certain he would agree. There are still outstanding considerations in the various issues on the tracker, so it would be best to address them before requesting integration. Example: What should happen when there is another Python installation on the path?

On 06/02/2011 15:20, Brian Curtin wrote:
Same as happens with most Windows apps: last one installed wins. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On 06/02/2011 15:25, Brian Curtin wrote:
So put the new path before the old path, or replace it? The current patch appends to the end.
I believe the last path wins in Windows land, so that would be fine. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On 6 February 2011 15:35, Nick Coghlan <ncoghlan@gmail.com> wrote:
... and "at the beginning" can be a pain due to unintended overriding of existing user commands (not likely in the case of Python, where there's only python, pythonw, w9xpopen and various bdist_wininst "RemoveXXX" commands, but still possible). "Before any existing Python directories, otherwise at the end" is the closest to what I suspect most users want (certainly it matches my preferences, and anything else would have me manually editing PATH anyway, so is of no use to me in practice). Paul.

Paul Moore writes:
Unfortunately, what is "no use to person X in practice" is a function of X. I suspect that's why this hasn't been done. Specifically, it seems to me that there are use cases for each of 1. Append (eg, if both python3 and python2 provide "python.exe", for experimental use of python3). 2. Prepend (actually, not a use case; just common, and therefore "intuitive", practice). 3. "Moore's rule" (put latest and greatest ahead of other versions but not interfere with previously installed apps). Maybe it should be user-configurable.

On 6 February 2011 17:07, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Absolutely :-) And it's also why I'm reluctant to support it - even though I agree that not having "python" just work is a PITA.
Fame at last :-) I've seen both (1) and (2) in common use. Both have disadvantages, particularly if you try to support multiple versions being installed at once (something which is nearly unheard of in Windows, and hence why no commonly used solution really does a good job of it). I've never seen (3), and in all honesty I don't expect it to be practical - too many special cases to consider. It was more of a straw man example of what "do it right" might really mean...
Maybe it should be user-configurable.
-1. Too much complexity. What I *have* seen is Oracle's "Home Selector", which is a program installed with Oracle's software which keeps track of which versions of Oracle you have installed, and gives you a GUI to move them up & down in priority, or disable versions. It then updates PATH appropriately. Ultimately, all it is in Python's terms, is a GUI means of editing PATH, so I'm not sure it's of any real use to us. (For Oracle, I think it fiddles with some other registry values, so it does have some value there...) One point - no matter what we do, we only need to consider 3.3 and later. People using 3.2 or earlier still have to manually fix up PATH how they want. So if we do add python to PATH in 3.3, we don't actually have a "what if people want to install multiple versions" issue until 3.4 comes out. I'm assuming we don't try to support multiple maintenance releases (3.3.1 and 3.3.2) being installed at once... Paul.

I've got a feeling that policy is evil and can not be applied cleanly when change falls out of scope of Python core .c sources and .py files from standard library. Right now the proposal is to add Python to %PATH% to make Python more user friendly for newbies. I can't see what can become worse than it is now. People are discussing advanced scenarios with multiple Python installations, but I propose first to make Python a normal command line user-friendly system application for Windows, and then discuss more advanced usage scenarios to make it developer-friendly in subsequent versions. Time is passing by and nothing happens. -- anatoly t.

I'd like to see this in 3.2 release, of course.
As Brian already asserted: that's not feasible. I still haven't managed to test your installer, and may not be able to for the next few weeks. It's also against the policy for release candidates to add such a change at this point. I believe the change, if implemented, needs to be optional (which I believe your change is not). Regards, Martin

Hello See http://bugs.python.org/issue3561 (rejected by Martin). Regards
participants (13)
-
"Martin v. Löwis"
-
anatoly techtonik
-
Brian Curtin
-
Chris Withers
-
Christian Heimes
-
Ethan Furman
-
Michael Foord
-
Nick Coghlan
-
Paul Moore
-
Raymond Hettinger
-
Stephen J. Turnbull
-
Tim Golden
-
Éric Araujo