Re: [Python-Dev] New Windows installer for Python 3.5
Nothing is merged in yet and everything can still change, so I'm keen to hear whatever feedback people have. I've tried to make improvements fairly for first-time users through to sysadmins, but if I've missed something big I'd like to hear about it before we get too close to alpha 1.
It would be great if the new installer supported silent, portable installs. What I have in mind is a silent installation that drops all the necessary files for a working python into a folder, but does not put ANY file anywhere else and does not register anything anywhere on the system. So no PATH modification, no registering of this install as one of the available python interpreters, no uninstall entry or anything like that. By "all the necessary files" I mean that it drops things like the MSVC runtime dlls etc into that folder into which I'm installing python, but again doesn't try to install things like that system or even user wide. I'll give you my very specific use-case, but I think this might be useful for others as well: we are trying to build an installer for some other product X that internally requires a python instance to work. This python instance should NOT be visible to anything other than product X, i.e. no user should ever start it directly or anything like that, it really is an implementation detail of product X. If we had an official python installer that supported silent, portable installs, I could just call that python installer inside the setup program for X, and it would drop a fully working python installation into a sub-directory of the install directory of product X. And we would be happy :) The old MSI installer sort of had something like that with the MSI administrative install option. But it never really worked because the administrative install didn't drop the MSVC runtime dlls anywhere, as far as I could tell. At some point I hacked around that by modifying the MSI file in my setup program to also drop the MSVC runtime dlls, but that was VERY hacky... Thanks, David
On 1/9/2015 12:34 PM, David Anthoff wrote:
Nothing is merged in yet and everything can still change, so I'm keen to hear whatever feedback people have. I've tried to make improvements fairly for first-time users through to sysadmins, but if I've missed something big I'd like to hear about it before we get too close to alpha 1. It would be great if the new installer supported silent, portable installs. What I have in mind is a silent installation that drops all the necessary files for a working python into a folder, but does not put ANY file anywhere else and does not register anything anywhere on the system. So no PATH modification, no registering of this install as one of the available python interpreters, no uninstall entry or anything like that. By "all the necessary files" I mean that it drops things like the MSVC runtime dlls etc into that folder into which I'm installing python, but again doesn't try to install things like that system or even user wide.
I'll give you my very specific use-case, but I think this might be useful for others as well: we are trying to build an installer for some other product X that internally requires a python instance to work. This python instance should NOT be visible to anything other than product X, i.e. no user should ever start it directly or anything like that, it really is an implementation detail of product X. If we had an official python installer that supported silent, portable installs, I could just call that python installer inside the setup program for X, and it would drop a fully working python installation into a sub-directory of the install directory of product X. And we would be happy :)
The old MSI installer sort of had something like that with the MSI administrative install option. But it never really worked because the administrative install didn't drop the MSVC runtime dlls anywhere, as far as I could tell. At some point I hacked around that by modifying the MSI file in my setup program to also drop the MSVC runtime dlls, but that was VERY hacky...
Thanks, David Sounds like a nice feature.
David Anthoff wrote:
It would be great if the new installer supported silent, portable installs. What I have in mind is a silent installation that drops all the necessary files for a working python into a folder, but does not put ANY file anywhere else and does not register anything anywhere on the system. So no PATH modification, no registering of this install as one of the available python interpreters, no uninstall entry or anything like that. By "all the necessary files" I mean that it drops things like the MSVC runtime dlls etc into that folder into which I'm installing python, but again doesn't try to install things like that system or even user wide.
I'll give you my very specific use-case, but I think this might be useful for others as well: we are trying to build an installer for some other product X that internally requires a python instance to work. This python instance should NOT be visible to anything other than product X, i.e. no user should ever start it directly or anything like that, it really is an implementation detail of product X. If we had an official python installer that supported silent, portable installs, I could just call that python installer inside the setup program for X, and it would drop a fully working python installation into a sub-directory of the install directory of product X. And we would be happy :)
I'll look into this, but it probably isn't going to work as part of the installer. I have previously looked into being able to install arbitrary side-by-side copies of Python, but that's near impossible as well. Windows Installer doesn't really let you just copy files - it isn't part of its intended functionality. It isn't too difficult to build custom MSIs with certain parts of Python (such as the DLLs and the standard library, but no docs, headers or EXEs) in a way that won't conflict with other installs, but you're still using an MSI here which is not necessarily ideal. The easiest way is still going to be to install a copy of Python without administrative privileges, which *currently* will drop all the files you want into the main install path (wherever you customise that to be), and then copy them directly into your installer, but that may change depending on the redistribution requirements of the CRT. There are still limitations within Python itself that would be nice to fix, such as looking at the registry too soon when it could resolve directories relative to its own location, but these are independent from the installer. If you don't need python.exe, then it should be fine as long as you put pythonXY.dll alongside the executable loading it (otherwise a system version may take priority). As you can see, this is a fairly deep hole with lots of caveats :) We could release a ZIP file containing all the Python files. The only reason I hesitate on this is that it could cause significant confusion for someone who doesn't really understand the implications, while people like yourself who have thought about this are also capable of finding workarounds and don't really need the ZIP file apart from convenience. Making some of the fixes to make python.exe more portable would relieve my concerns here.
The old MSI installer sort of had something like that with the MSI administrative install option. But it never really worked because the administrative install didn't drop the MSVC runtime dlls anywhere, as far as I could tell. At some point I hacked around that by modifying the MSI file in my setup program to also drop the MSVC runtime dlls, but that was VERY hacky...
I'm a little surprised that worked at all for what you were trying to do. You'd be better off installing it once and then copying the files yourself. But overall, this is the sort of thing I do want to enable. I firmly believe that Python from python.org is for *developers*, and those who just want to run a Python application should be able to get a complete package. This is very different from the *nix approach, but it makes far more sense to Windows users. A good example is TortoiseHg, which bundles everything so that users never even know they have Python 2.7 or Mercurial installed on their machine. Making it easier for people to bundle Python into their own applications is a good thing, as far as I'm concerned. Cheers, Steve
Thanks, David
I'll look into this, but it probably isn't going to work as part of the installer. I have previously looked into being able to install arbitrary side-by-side copies of Python, but that's near impossible as well. Windows Installer doesn't really let you just copy files - it isn't part of its intended functionality. It isn't too difficult to build custom MSIs with certain parts of Python (such as the DLLs and the standard library, but no docs, headers or EXEs) in a way that won't conflict with other installs, but you're still using an MSI here which is not necessarily ideal.
Are administrative MSI installs an option, though? They don't register anything but just drop files, right? But see my comments below about a zip drop, which would be a much, much nicer option in my opinion.
We could release a ZIP file containing all the Python files.
That would be absolutely FANTASTIC and would solve all problems around this topic. In theory we could handle this ourselves on our end, but this is for a small open source project and we are really hesitant to take on another software packaging job. Doing this right just for our product would be a fair amount of work, we would have to do this with every new Python release, we might mix things up etc., and really we are more interested in our piece of software than packaging Python ;) I think it would be a much nicer model if there was a zip to download from python.org. There is another issue that keeps us from hosting our own files: we would have to figure out licensing issues, both related to python and to the msvcr*.dll and msvcp*.dll. I would feel much more comfortable if we didn't distribute python or other binaries, but just downloaded stuff from python.org as part of the setup process.
The only reason I hesitate on this is that it could cause significant confusion for someone who doesn't really understand the implications, while people like yourself who have thought about this are also capable of finding workarounds and don't really need the ZIP file apart from convenience.
I was not clear in my previous email. I have NOT found a way to work around this. I have tried various hacks, but none really works. I got pretty far, but none really worked in all cases in a robust way. So I would certainly welcome a downloadable zip file a great deal. Is there maybe a compromise for now to have such a zip on the server, but not advertise it widely, and maybe put an "experimental"/"beta" moniker into the filename? I assume you would include the MS VC runtime files msvcr100.dll and msvcp100.dll in such a zip file? Is there any chance this might even be done (as an experimental version) for Python 3.4?
Making some of the fixes to make python.exe more portable would relieve my concerns here.
I see that. For our cases things seem to work, but I agree, it would be good if python.exe would try hard to work in a xcopy mode.
The old MSI installer sort of had something like that with the MSI administrative install option. But it never really worked because the administrative install didn't drop the MSVC runtime dlls anywhere, as far as I could tell. At some point I hacked around that by modifying the MSI file in my setup program to also drop the MSVC runtime dlls, but that was VERY hacky...
I'm a little surprised that worked at all for what you were trying to do. You'd be better off installing it once and then copying the files yourself.
I got it to drop the msvcr100.dll, but haven't managed to get the msvcp100.dll out via an administrative install. The whole approach is a mess, I would MUCH prefer a zip file.
But overall, this is the sort of thing I do want to enable. I firmly believe that Python from python.org is for *developers*, and those who just want to run a Python application should be able to get a complete package. This is very different from the *nix approach, but it makes far more sense to Windows users. A good example is TortoiseHg, which bundles everything so that users never even know they have Python 2.7 or Mercurial installed on their machine. Making it easier for people to bundle Python into their own applications is a good thing, as far as I'm concerned.
Yes, those are good examples. Right now doing this in the way these guys do is too much work for our small project... Anything that makes this easier would be appreciated. Thanks! And the new installer looks great in general. Best, David
On 01/09/2015 05:01 PM, David Anthoff wrote:
On 01/09/2015 04:09 PM, Steve Dower wrote:
The only reason I hesitate on this is that it could cause significant confusion for someone who doesn't really understand the implications, while people like yourself who have thought about this are also capable of finding workarounds and don't really need the ZIP file apart from convenience.
I was not clear in my previous email. I have NOT found a way to work around this.
Is there any chance this might even be done (as an experimental version) for Python 3.4?
Couldn't this zip file be advertised in the embedded section of python.org? Or with a "for embedding in Windows apps" tag? (or whatever the proper verbiage is) -- ~Ethan~
On 10 January 2015 at 11:29, Ethan Furman <ethan@stoneleaf.us> wrote:
On 01/09/2015 05:01 PM, David Anthoff wrote:
On 01/09/2015 04:09 PM, Steve Dower wrote:
The only reason I hesitate on this is that it could cause significant confusion for someone who doesn't really understand the implications, while people like yourself who have thought about this are also capable of finding workarounds and don't really need the ZIP file apart from convenience.
I was not clear in my previous email. I have NOT found a way to work around this.
Is there any chance this might even be done (as an experimental version) for Python 3.4?
Couldn't this zip file be advertised in the embedded section of python.org? Or with a "for embedding in Windows apps" tag? (or whatever the proper verbiage is)
For the time being, things like PyInstaller, PyRun, Portable Python, etc are going to offer a better solution than anything we provide in the standard installers. As Steve says, addressing the situation properly requires eliminating various current assumptions in the interpreter startup sequence that are really only valid in a normal installation context, and if you start pulling on *that* particular thread you may find yourself writing a proposal like PEP 432 as your brain starts to hurt and you consider ways you may be able to make the pain stop :) By bypassing the normal interpreter entirely, and instead creating your own custom binary that embeds the CPython runtime, you get a lot more flexibility and control in terms of what happens during the startup sequence. It's not complete control (hence the need for PEP 432, or something like it), but it's still a lot more control than if you try to use the python.org binaries directly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jan 10, 2015 at 3:28 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For the time being, things like PyInstaller, PyRun, Portable Python, etc are going to offer a better solution than anything we provide in the standard installers.
See also Anaconda and Enthought Canopy. I think miniconda, for instance, may give you just what you need, if you don't want to go the py2exe/PyInstaller approach (though you probably do want to go that way, as far as I can tell from your description of your use-case. I'm inclined to think that this does not belong as part of the standard installer. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Sat, Jan 10, 2015 at 3:28 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For the time being, things like PyInstaller, PyRun, Portable Python, etc are going to offer a better solution than anything we provide in the standard installers.
See also Anaconda and Enthought Canopy. I think miniconda, for instance, may give you just what you need, if you don't want to go the py2exe/PyInstaller approach (though you probably do want to go that way, as far as I can tell from your description of your use-case.
Actually, both Anaconda and Canopy suffer from the initialization process issues. You can't really use virtualenv with either of them unless they set the registry PythonPath value (since they won't find the libraries they need to launch), and if you install both then they'll fight over the registry key and you'll get Canopy launching with Anaconda's path or vice-versa.
I'm inclined to think that this does not belong as part of the standard installer.
The problems are inherent to the standard python.exe, and are likely part of the standard pythonXY.dll. Nick's scared of fixing it, so I'm absolutely petrified :) Cheers, Steve
-Chris
On 13 January 2015 at 06:02, Steve Dower <Steve.Dower@microsoft.com> wrote:
On Sat, Jan 10, 2015 at 3:28 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For the time being, things like PyInstaller, PyRun, Portable Python, etc are going to offer a better solution than anything we provide in the standard installers.
See also Anaconda and Enthought Canopy. I think miniconda, for instance, may give you just what you need, if you don't want to go the py2exe/PyInstaller approach (though you probably do want to go that way, as far as I can tell from your description of your use-case.
Actually, both Anaconda and Canopy suffer from the initialization process issues. You can't really use virtualenv with either of them unless they set the registry PythonPath value (since they won't find the libraries they need to launch), and if you install both then they'll fight over the registry key and you'll get Canopy launching with Anaconda's path or vice-versa.
I'm inclined to think that this does not belong as part of the standard installer.
The problems are inherent to the standard python.exe, and are likely part of the standard pythonXY.dll. Nick's scared of fixing it, so I'm absolutely petrified :)
Technically, I'm not scared of fixing it per se, I'm just wary of trying to fix it without first refactoring it and writing some more tests :) Finding the roundtuits for PEP 432 is an eternal struggle though - so many things end up edging ahead of it on the personal priority list :( Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
David Anthoff wrote:
I'll look into this, but it probably isn't going to work as part of the installer. I have previously looked into being able to install arbitrary side-by-side copies of Python, but that's near impossible as well. Windows Installer doesn't really let you just copy files - it isn't part of its intended functionality. It isn't too difficult to build custom MSIs with certain parts of Python (such as the DLLs and the standard library, but no docs, headers or EXEs) in a way that won't conflict with other installs, but you're still using an MSI here which is not necessarily ideal.
Are administrative MSI installs an option, though? They don't register anything but just drop files, right? But see my comments below about a zip drop, which would be a much, much nicer option in my opinion.
Not to my knowledge. An administrative install puts the files in a shared location and allows users to run much faster installs that will then refer to that shared location rather than copying the files locally. As I said, MSI doesn't support plain file drops (often called "xcopy install" - you use that term later, but I'm not sure how well known it is).
We could release a ZIP file containing all the Python files.
That would be absolutely FANTASTIC and would solve all problems around this topic. In theory we could handle this ourselves on our end, but this is for a small open source project and we are really hesitant to take on another software packaging job. Doing this right just for our product would be a fair amount of work, we would have to do this with every new Python release, we might mix things up etc., and really we are more interested in our piece of software than packaging Python ;) I think it would be a much nicer model if there was a zip to download from python.org.
Apart from the bit below, this would be identical to you installing Python once on your own machine and zipping up the files yourself. For the (considerably) less than 1% of Python users who want to do this, I don't think it's a big ask.
There is another issue that keeps us from hosting our own files: we would have to figure out licensing issues, both related to python and to the msvcr*.dll and msvcp*.dll. I would feel much more comfortable if we didn't distribute python or other binaries, but just downloaded stuff from python.org as part of the setup process.
IANAL, but I've dealt with enough licensing issues in my day job that this makes me even more concerned about making a ZIP file available. If you're planning to do this, putting up a ZIP file could potentially expose Python (and presumably the PSF indirectly) to the liability that you should be taking on as a redistributor, if it's that important to your product. Without an actual legal opinion, I'm now -1 on making a ZIP download available for this purpose.
The only reason I hesitate on this is that it could cause significant confusion for someone who doesn't really understand the implications, while people like yourself who have thought about this are also capable of finding workarounds and don't really need the ZIP file apart from convenience.
I was not clear in my previous email. I have NOT found a way to work around this. I have tried various hacks, but none really works. I got pretty far, but none really worked in all cases in a robust way. So I would certainly welcome a downloadable zip file a great deal. Is there maybe a compromise for now to have such a zip on the server, but not advertise it widely, and maybe put an "experimental"/"beta" moniker into the filename?
I gave you a workaround. Install Python just for yourself and zip up the install directory.
I assume you would include the MS VC runtime files msvcr100.dll and msvcp100.dll in such a zip file?
Either that or redistribute them using the tools provided by Microsoft (there are redistributable merge modules and executable installers available). The advantage of the latter approach is that your users will automatically get the latest versions and security fixes.
Is there any chance this might even be done (as an experimental version) for Python 3.4?
You'd have to ask Martin, so probably not. But the workaround I gave you will work for 3.4.
Making some of the fixes to make python.exe more portable would relieve my concerns here.
I see that. For our cases things seem to work, but I agree, it would be good if python.exe would try hard to work in a xcopy mode.
The old MSI installer sort of had something like that with the MSI administrative install option. But it never really worked because the administrative install didn't drop the MSVC runtime dlls anywhere, as far as I could tell. At some point I hacked around that by modifying the MSI file in my setup program to also drop the MSVC runtime dlls, but that was VERY hacky...
I'm a little surprised that worked at all for what you were trying to do. You'd be better off installing it once and then copying the files yourself.
I got it to drop the msvcr100.dll, but haven't managed to get the msvcp100.dll out via an administrative install. The whole approach is a mess, I would MUCH prefer a zip file.
But overall, this is the sort of thing I do want to enable. I firmly believe that Python from python.org is for *developers*, and those who just want to run a Python application should be able to get a complete package. This is very different from the *nix approach, but it makes far more sense to Windows users. A good example is TortoiseHg, which bundles everything so that users never even know they have Python 2.7 or Mercurial installed on their machine. Making it easier for people to bundle Python into their own applications is a good thing, as far as I'm concerned.
Yes, those are good examples. Right now doing this in the way these guys do is too much work for our small project... Anything that makes this easier would be appreciated.
I don't see how. All they've done is literally copy a Python installation into their install directory. Yes, they have their own launcher executables (py2exe generated, it looks like) and have precompiled the standard library and put it in a ZIP file, but you don't even need to go that far. Without knowing anything about your project I can't give specific suggestions, but simply dropping a Python installation in is not that difficult (and until the issues that Nick referred to are fixed, will have the same problems as TortoiseHg presumably has).
Thanks! And the new installer looks great in general.
Thanks. Cheers, Steve
On Mon, 12 Jan 2015 17:26:43 +0000, Steve Dower <Steve.Dower@microsoft.com> wrote:
David Anthoff wrote:
Yes, those are good examples. Right now doing this in the way these guys do is too much work for our small project... Anything that makes this easier would be appreciated.
I don't see how. All they've done is literally copy a Python installation into their install directory. Yes, they have their own launcher executables (py2exe generated, it looks like) and have precompiled the standard library and put it in a ZIP file, but you don't even need to go that far. Without knowing anything about your project I can't give specific suggestions, but simply dropping a Python installation in is not that difficult (and until the issues that Nick referred to are fixed, will have the same problems as TortoiseHg presumably has).
That's what py2exe *does*. It does all the python-integration and launcher-building work for you. I use it for a small project myself (proprietary), which I build an installer for using Inno Setup. Works very well, supports python3. --David
participants (7)
-
Chris Barker
-
David Anthoff
-
Ethan Furman
-
Glenn Linderman
-
Nick Coghlan
-
R. David Murray
-
Steve Dower