PEP 397 (Python launcher for Windows) reference implementation

PEP 397 (Python launcher for Windows) has a reference implementation in Python. Does anyone know of a C implementation, or is planning/working on one? I realise this is the final objective, so such implementation might be premature, but perhaps someone has been experimenting ... Regards, Vinay Sajip

On 30/06/2011 3:00 AM, Vinay Sajip wrote:
Not yet - my last update of the PEP has made the existing reference implementation out-of-date, so I want to work on that before starting on the C version. However, seeing as my note about the most recent PEP update attracted zero comments, I admit I lost any sense of urgency on actually doing this... I'll make an effort to update that reference impl in the next week or so (but obviously anyone else is free to help ;) Cheers, Mark

On 30/06/2011 08:34, Tim Golden wrote:
I have that email (the update one from Mark not the silent nodding from Tim) still sitting in my inbox waiting for me to properly read through and comment on... Sorry for being useless, I'll try and move it up the priority list. I really like the PEP and think it will be a *huge* step forward for Windows users - so it's purely the details that need thrashing out (heh). In the latest update Mark also addressed my main concern, making the launcher configurable so it can also be used by alternative implementations (particularly IronPython for Windows). I've copied Jeff Hardy and Dino (IronPython maintainers) in on this, in the hope that they might take a look at the latest version of the PEP too... All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 30 June 2011 13:50, Paul Moore <p.f.moore@gmail.com> wrote:
OK, having looked through this, it looks pretty solid to me. I might try installing Vinay's implementation and seeing how it feels in use, as well... On restoring associations, I think it's entirely reasonable not to bother. It would be nice if py.exe remained installed as long as any (all-users) Python installation remained on the PC, but this may be complicated (given that older versions won't uninstall it) so maybe don't even bother with that. Actually, I'd question whether this shouldn't be packaged as a separate application, available as an independent download from python.org (I do think it's important enough to be hosted on python.org rather than PyPI, though). Future versions of python could bundle it as well, but that could be purely for convenience and to ensure that users don't miss it. If Python does bundle it, then I'd suggest that it remain a separate item in add/remove programs. That allows the user to choose whether to uninstall it or not. As py.exe and python live in separate directories (except in per-user installs) the installers don't have any nasty interactions like who deletes the directory to worry about. If you want to bother with an "active installation count" mechanism (whether SharedDLLs or a simple count/listing in the py.ini file, or something else) then when the count hits "No", just offer the user the choice to uninstall as part of the Python uninstall. I'd be inclined not to worry too much about per-user installations. Are per-user file associations possible? I've never used them, myself. As a simple approach, just install py.exe alongside python.exe in the user dir, don't worry about it being on PATH or having a separate add/remove programs item, and let the user do what he wants with it. Of course, if someone with more experience of per-user installs and the sorts of environment where they are needed wants to suggest something different, believe them rather than me! Basically, my feeling is that the core functionality is fine. Nothing in this will make life worse for anyone, so we can safely leave corner cases to be addressed later (with a standalone release, if it's urgent enough). Paul.

Paul Moore <p.f.moore <at> gmail.com> writes:
Do have a play, it would be nice to get feedback. It's only available as source, though - is that OK?
I'd be inclined not to worry too much about per-user installations. Are per-user file associations possible? I've never used them, myself.
Yes, and they can be problematic: see http://www.deadlybloodyserious.com/2007/05/no-script-arguments/ where the problem described has also bitten me. I've no idea what put that association in the registry. Regards, Vinay Sajip

Mark Hammond <skippy.hammond <at> gmail.com> writes:
Sorry, I just recently came across the PEP when Tim Golden pointed it out to me, when I asked a question about executable script support in packaging. I think I was probably not so focused on Windows at the time of the original announcement ... There's a lot to like in the PEP, and I have some questions relating to the latest version: 1. In the section on shebang line parsing, it says "If the command starts with the definition of a customized command followed by a space character, the customized command will be used." Sorry if I'm being dense, but what's the significance of the trailing space character? In fact, your vpython example in the customeised comments section doesn't show a trailing space - you've used '#! vpython' as the shebang line. 2. The section on .ini files says that "Commands specified in the [.ini file in the] "application directory" [APPDATA] will have precedence over the one next to the [launcher] executable." What's the benefit of this? If you have only one launcher executable of one type (say console, 32-bit) on a system, then there's no point in having two locations for .ini files. If you have multiple copies of the launcher and multiple .ini files next to them, then with this precedence order, you can't specialise the behaviour of anything in a specific launcher .ini that's also specified in the APPDATA .ini. Doesn't it make more sense to look for a setting in any file next to the launcher, and if not found there, look in the APPDATA? That way common things can be defined in the APPDATA .ini and overridden for special cases in the launcher-adjacent .ini. Or have I got completely the wrong end of the stick? By the way, I've already converted the existing Python reference implementation to C (I did it before I saw your response to my post). It basically works in that it does what the Python version does, but doesn't include any handling of -32 suffixes or .ini files. I can post this on BitBucket if anyone's interested, but it's a work in progress. I'm working on some simple tests. Regards, Vinay Sajip

On 30/06/2011 10:09 PM, Vinay Sajip wrote:
This is just clumsy wording on my part - what I'm trying to say is simply that 'vpython3' will not match a customized command 'vpython'.
The idea is that a user without admin permissions can customize the behaviour - so the admin can add a .ini file, but the user can override any commands in that file with their own definitions.
The idea is that some users will not have permission to edit the one next to the launcher. I'd be open to considering this yagni if others agree.
The intention is that there only be a single launcher, as only one app can be associated with .py files. OTOH though, file associations can be configured per-user IIRC, and assuming that is the case, we could avoid my multiple-ini-file usecase above by just allowing a different launcher to be registered for a specific user. This is sounding difficult from the UI perspective though (ie, does the installer then need to ask that question, will there be a post-install technique for per-user registration, etc?)
I came at this from the other angle under the assumption there will only be one launcher installed - so common things can be stored in the launcher-adjacent file and per-user special cases can be in the per-user APPDATA dir. If the expectation was multiple launchers be installed, then I'd probably then prefer to KISS and only have the launcher-adjacent file supported at all.
Sure, that would be awesome! I think that will mean your impl is fairly close to the first draft of the PEP I checked into HG, which is nice and still quite useful to use :) Thanks! Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
I don't like this, for the reasons you state. I think it would be better if the PEP was changed to say that there is intended to be just one launcher installation per machine. As the intention is for the launcher to ship with Python, and there can be multiple Pythons installed, I presume we'll have to handle this by installing the launcher in some common-to-all-Pythons location somewhere outside a Python installation location, such as "c:\Program Files\Python Launcher". This should be stated in the PEP, and we'll also need to indicate how the launcher will be cleaned up if for some strange reason someone uninstalls all Pythons from a machine. Then we can just state that there's a global .ini file (where the launcher is installed) and a local one (in the user's APPDATA). From that perspective, it makes sense to have items in the local (APPDATA) override the global (launcher installation location).
Okay, I'll do this soon - once I've added a few tests. I've updated the implementation to include help output. BTW I thought of another thing that perhaps needs handling: what if a customized command points to the launcher itself? It'd be turtles all the way down :-) Regards, Vinay Sajip

On 1/07/2011 7:20 PM, Vinay Sajip wrote:
The PEP does say "if possible, should be installed somewhere likely to already be on the system PATH (eg., the Windows System32) directory." It is silent about what to do when that isn't possible, but I'd think it OK if the launcher was installed directly in the Python directory - IOW, I'd think it OK if the PEP said "should be installed next to the PythonXX.dll being installed" - but an important point in the above working is the "already be on the system PATH" - ie, I don't really want it put in a newly created directory unless the installer also adds that directory to the PATH - and what to do on uninstall then becomes an issue. One problem with all of this is uninstallation and specifically if the user is uninstalling the most recent Python installation while leaving earlier ones. I guess there are 2 general answers to this: * The installer process could remember the previous association and restore that on uninstall. * We treat this as a "wont-fix" and document a work-around of asking the user to reinstall the previous version to restore the file association. We probably need to be mindful of adding too much extra work for the installer process as that may well end up blocking us on getting this into the next appropriate release. In particular, Martin's thoughts here would be very useful. This would force the user to reinstall that older one to re-establish the associations correctly
BTW I thought of another thing that perhaps needs handling: what if a customized command points to the launcher itself? It'd be turtles all the way down :-)
Yeah - I wonder if we can leverage the "job" api here and refuse to start if there are already 2 processes in the job? OTOH, that is tricky as it would also prevent someone using os.startfile with a .py file.... From your second mail:
I'm not too bothered to be honest - the customized commands exist purely for alternative implementations, so my initial thoughts are that additional args would be as useful for them as they are for cpython invocations. IOW, if they don't need it, then CPython invocations don't need it either, so maybe it can be dropped completely?
Sure - I think there is some policy that a Python version number will never be > 10, so that sounds fine to me. So long as the launcher doesn't blindly run off the end of such arrays I think it is fine - limitations can be addressed in later versions. It will be a few days until I can look at the implementation, but I'm very happy to see it started. Given it is now ahead of the Python reference impl, I wonder if we should just drop all wording about that reference impl and just treat the C impl as canonical? Cheers, Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
But "next to PythonXY.dll" implies multiple copies, which we want to avoid, right?
We'd need to do that in the case of the earlier Python not having come with a launcher, i.e. when its version is < 3.3.
* We treat this as a "wont-fix" and document a work-around of asking the user to reinstall the previous version to restore the file association.
This is not ideal from a user's perspective.
It sounds onerous for users to have to reinstall an older Python. I'm not that familiar with Windows Installer features, so I don't know if this is too fancy, but perhaps we could remember the last non-launcher association when we install the launcher, and either restore that when the launcher is uninstalled (if it's still pointing to an installed Python) or remove the association altogether, otherwise. If this logic is too fancy to include in the installer itself, perhaps we can provide this logic in the launcher itself or via an ancillary executable or DLL that the installer can just call into. Is there some mechanism like the SharedDLLs registry key for executables, which could be used to reference count launcher installations so that it could be uninstalled at the appropriate time? Could we perhaps used the SharedDLLs feature itself?
If there's only ever one launcher installed (which we could ensure by installing it in a Windows rather than a Python location) then perhaps we could perhaps check for the value of a customised command pointing at the one-and- only launcher, but this is circumventable. Anyway, perhaps we just need to handle a user error rather than someone deliberately trying to engineer recursion. Another approach might be - rather than limit the number of processes in the job, look to see if the launcher executable is already associated with an existing job. I'm not au fait with the job API, and hence unsure of how that would play with usages such as os.startfile, subprocess etc.
I think they would be useful, so let me check the implementation again.
Once you've taken a closer look, if you think it looks good enough, then that's fine. If you have a BitBucket account, I can add your account to the repo so you can push changes to it. Regards, Vinay Sajip

On 2/07/2011 7:08 PM, Vinay Sajip wrote:
I believe this will be most useful when people select "install for all users", which will force it into system32. If people select "just for me", then there will be multiple copies on disk, but only one will be active (ie, we will not support different ones being registered for different users. It isn't ideal, but I think it good enough, avoids some complexity and should meet the needs of the majority of users.
OTOH, we don't do that now AFAIK.
Sure, but neither is the current situation - I don't recall enough users complaining about that to make the effort worthwhile. I'm not fundamentally opposed to doing something better here - I'm just trying to avoid this kind of stuff being a requirement for acceptance. If you are more familiar with the installer than I and feel it can be done simply, then I'm happy for you to go for it!
But this only happens when they install a later version, then uninstall the later one and continue to use the old one. I'd suggest that is (a) rare and (b) possibly already broken (ie, the old association may not be restored now). If the old association currently is correctly restored, then I'd expect things to just magically work in this new model without any effort.
I'm still inclined to call YAGNI, but as above, I don't fundamentally oppose such bells-and-whistles.
We probably can't ensure this while the installer supports a non-UAC installer (ie, the "just for me" option.) OTOH, I'm not sure the "just for me" option currently works without needing UAC elevation anyway. But assuming it does (or the intention is to fix things if it does not), then we can't guarantee a writable system32.
Yeah, I agree.
Exactly - code doing os.startfile on a .py file will correctly cause the same launcher to be re-executed and preventing this would break such scripts. However, I think this would be extremely rare and the more usual case would be to use sys.executable. Indeed, any script using os.startfile for a .py file is already broken, even if the author doesn't know it yet :)
Great, thanks! Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
No, you're right - I'm just throwing ideas out. I'm not especially familiar with MSI in general, though I believe you can do the relevant things with logic in custom actions. AFAIK there are already custom actions used in the Python MSI, but I wouldn't want to propose any such changes to the MSI unless Martin were to make encouraging comments :-)
As to (a), not uncommon scenarios would be (i) later version breaks something, so go back to earlier version, or (ii) using 2.x, installing 3.x to try out, going back to 2.x. I'm not sure about (b).
I'm still inclined to call YAGNI, but as above, I don't fundamentally oppose such bells-and-whistles.
You're probably right about the cost(effort)-benefit trade-off.
Okay. From a cursory glance at msi.py, UAC elevation appears to be requested.
I had a closer look at the Job API, and it does seem to offer the required functionality, so we could tell whether the current process is in a job -> get all processes in that job -> get their executables and command lines -> see if recursion is occurring. However, would we class this as an implementation detail or does it need a mention in the PEP?
I think they would be useful, so let me check the implementation again.
I've made some updates so the following works: with the configuration containing [commands] shell=cmd /c with a shebang in doit.py of '#!shell python -v' the launcher will run 'cmd /c python -v doit.py'. Regards, Vinay Sajip

Mark Hammond <skippy.hammond <at> gmail.com> writes:
I've now checked, and it appears that we don't do the right thing now anyway. On a clean Win7-x64, I installed Python 2.7 (32-bit), for all users, with registration of extensions - and Python.File etc. were added to the registry pointing at Python 2.7. I then installed Python 3.2 the same way, and rhe registry entries then pointed to 3.2. I then uninstalled 3.2, and the registry entries were gone! No magical restoration to the earlier values :-( Okay - if users have to do this now anyway, we at least wouldn't be naking things worse :-) Regards, Vinay Sajip

Mark Hammond wrote:
On 2/07/2011 7:08 PM, Vinay Sajip wrote:
perhaps we could remember the last non-launcher association when we install the launcher,
It might be better to look in the registry for other Python installations and ask the user which one to restore if there is more than one. Trying to restore the "last" one would be prone to breakage if the user didn't uninstall versions in precisely the reverse of the order of installation. -- Greg

On 4/07/2011 3:59 PM, Greg Ewing wrote:
While that makes alot of sense, the fact we are already "broken" in exactly the same way means I hope we can treat the restoration of associations as a separate issue - a worthwhile one, but not a pre-requisite for this PEP being approved. Cheers, Mark

On Mon, Jul 4, 2011 at 2:27 AM, Mark Hammond <skippy.hammond@gmail.com> wrote:
I agree let the restoration or not of file association be an implementation detail and not a requirement. I also agree with Paul Moore that py.exe should be a separate install/uninstall It can be bundled with the python MSI but should be a standalone MSI that the python MSI executes. So uninstalling the version of python that installed it alone is not enough to remove it. This will make it seem like the file associations are just working for naive users but will still let them remove the associations. We can just put a warning in the uninstall process of py.exe that lets the user know if they still have python versions installed they will need to re-install them to get the file association again.

Mark Hammond <skippy.hammond <at> gmail.com> writes:
I agree, but there's one aspect of associations which is perhaps worth exploring: the installation of a pre-3.3 version of Python after Python 3.3 is installed with the launcher will, if the user selects "Register Extensions", hijack the laumcher's associations to that earlier Python. Then bye bye launcher - how do we deal with that? Or don't we? There'll be no warning for the user, and this problem will occur even if the launcher is packaged separately from Python. so I think we need to think about this a little more. What say? Regards, Vinay Sajip

Vinay Sajip wrote:
I don't see how anything can be done about that. It also doesn't seem like too serious a problem -- it's no worse than what currently happens if you install an older version over a newer one, i.e. associations revert to the old version. Making the launcher a separate install at least offers a way of fixing that if it happens and isn't desired, i.e. reinstall the launcher. -- Greg

On 5/07/2011 11:23 AM, Greg Ewing wrote:
Or an MSI installer may be able to offer a "repair" feature without too much pain. However, I'm a bit torn on the separate installer issue. I really think it should be installed with later Python versions even if it is made available as a separate installer for the benefit of earlier ones as it is one less step for people to get confused about - eg, in a few years when 3.3+ is the most common Python being downloaded and used, books or people helping on python-list could start referring to the launcher under the assumption it is installed, rather than avoiding mention of it simply to avoid the whole spiel about "this will only work if you have installed an optional package." IOW, it will be impossible to give unconditional advice on certain aspects of launching Python. If the launcher is such that we can unconditionally recommend its use, IMO we should just install it with Python. I'll go with the consensus though... Mark

On Tue, Jul 5, 2011 at 12:12 PM, Mark Hammond <skippy.hammond@gmail.com> wrote:
If the launcher is such that we can unconditionally recommend its use, IMO we should just install it with Python. I'll go with the consensus though...
I've installed other WIndows apps that create multiple add/remove programs entries from a single installer. I believe people are suggesting a similar thing here (i.e. have the launcher installed automatically when installing python, but create a separate add/remove entry so uninstallation leaves it behind unless removal is explicitly requested) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan <ncoghlan <at> gmail.com> writes:
Were those other Windows apps packaged as .msi, or .exe? AFAICT, although you can embed an MSI inside another one, the practice of concurrent/nested installations is strongly discouraged by Microsoft - see http://goo.gl/FJx1S (Rule 20). Also, IIUC, each entry in Add/Remove programs would correspond to a specific MSI (since you can e.g. repair that specific entry, it would imply its own installer database). So you could package Python and the launcher as separate MSIs (this would make sense so that you could restore associations to the launcher just by repairing its installation), but since nested MSIs are a no-no, that means installing via a bootstrapping .exe. This is a bigger change to our Windows packaging than some people might be comfortable with ... Regards, Vinay Sajip

On Tue, Jul 5, 2011 at 03:01, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Right, you cannot sanely embed one .msi inside another .msi; the support for "nested" or "concurrent" installations referred to in that link is indeed something to avoid.
You can certainly jump through all these hoops, but the pieces here are much more suited towards a component definition that can be shared among multiple products. If the component always installs to the same place, has the same GUID, and otherwise only changes by versions the exe, this is a completely safe correct use of one. Last I knew, msi.py allocates random GUIDs, so may or may not be suitable for generating this. If there is only rare need to update this exe, and msi.py has support for merge modules, that could be one approach. My recommendation for distributing this: each .msi which wants to include it should have a component that includes the following. Note that each .exe (py, pyw) and each architecture (x86, x64) need their own component with their own static GUID. * Defined unchanging GUID * Defined target location (perhaps SystemFolder) * msidbComponentAttributesSharedDllRefCount (cooperate with non-MSI installers), msidbComponentAttributesShared (keep highest version), and possibly msidbComponentAttributesPermanent flags set on the component * Versioned .exe (using a Windows version block) * File association information Then these components can be included in the python 3.3 installer, future releases, and even a standalone installer, and reference count correctly. Again, these can optionally be made available as merge modules for other consumers, but there's likely no need. -- Michael Urman

Michael Urman <murman <at> gmail.com> writes:
I'm currently experimenting with a merge module approach: launcher_module.msm -> launcher.msi, and x64 variants in separate .amd64.ms? files.
I'm doing that.
Thanks, that's interesting information. I'll read up about these. I'm a Windows installer tyro :-)
* Versioned .exe (using a Windows version block)
I'm doing that, too.
* File association information
Currently I'm putting the file association information in the same component as the files, since the registry values cross reference those files. Regards, Vinay Sajip

On 5 July 2011 03:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's certainly what I was meaning. I'm 100% in favour of Python 3.3 and later containing the installer as part of the core Python installer. One download, one install. (And two add/remove entries). I'd like to see a standalone installer for users of Python 2.7/3.2 and earlier. It's too useful a feature to not make it available for people who haven't installed 3.3 yet. And I'd prefer it if that standalone installer was hosted on python.org for visibility, rather than on PyPI. I'm not enough of an MSI expert to know if this can be implemented by having a standalone MSI, and then "embedding" it in the Python 3.3 MSI. That was what I'd thought of, but Vinay's later email suggests it might not be advisable:
Paul.

Mark Hammond <skippy.hammond <at> gmail.com> writes:
Or an MSI installer may be able to offer a "repair" feature without too much pain.
A few more observations to do with installation: 1. It's been mentioned that a standalone version should be available for use with earlier Python versions. This could be done with a merge module which can be used either with a standalone installer or the Python .msi. 2. For the standalone MSI, we will most likely need separate 32-bit and 64-bit MSIs, because the MSI system will look at the MSI type to determine whether registry stuff goes in the main registry or the Wow6432Node used for 32-bit applications. Regards, Vinay Sajip

One more thing about associations - we've got pyw.exe for Python.NoConFile and py.exe for Python.file, but how do we handle Python.CompiledFile? It doesn't really make sense to have the association not handled by the launcher. Unfortunately, of course, both pyw and py compile to pyo, so we don't know which launcher to use. It is, of course, easy for either py or pyw to determine which version of python is to be used to invoke a .pyc - just not which Windows variant. BTW just as a test I implemented .pyc support in the C implementation - it works fine apart from the "python.exe or pythonw.exe?" question. Regards, Vinay Sajip

On 2/07/2011 5:16 PM, I wrote:
I'm looking to update the PEP based on this discussion - does anyone object to the above? Or to put it another way, does anyone believe dropping the Python reference implementation is to the detriment of the PEP? Thanks, Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
My C implementation of the launcher is now available at https://bitbucket.org/vinay.sajip/pylauncher It's been built using VS 2008, and tested on Windows 7 (32-bit) with ActivePython 2.6.2.2 and ActivePython 3.2.0.0 installed. I've added support for .ini files [commands] section; I haven't looked at implementing [defaults] yet. There's incomplete support for -32 suffix at the moment - that can be looked at in due course. A couple of points: I've also allowed for /usr/local/bin/python as a built-in command, this can always be removed later if YAGNI. I've assumed that if a customised command is provided with arguments in the shebang line, these will be ignored - if people want to run with different options they can always define more customised commands. If you agree with this, the PEP should probably explicitly state this. In a couple of cases I've implemented using fixed size arrays - for the lists of installed Pythons and customised commands. Of course these can be made dynamic, but what's there is good enough for the moment for exploration. Do have a play, and let me know what you think. Regards, Vinay Sajip

On 30/06/2011 3:00 AM, Vinay Sajip wrote:
Not yet - my last update of the PEP has made the existing reference implementation out-of-date, so I want to work on that before starting on the C version. However, seeing as my note about the most recent PEP update attracted zero comments, I admit I lost any sense of urgency on actually doing this... I'll make an effort to update that reference impl in the next week or so (but obviously anyone else is free to help ;) Cheers, Mark

On 30/06/2011 08:34, Tim Golden wrote:
I have that email (the update one from Mark not the silent nodding from Tim) still sitting in my inbox waiting for me to properly read through and comment on... Sorry for being useless, I'll try and move it up the priority list. I really like the PEP and think it will be a *huge* step forward for Windows users - so it's purely the details that need thrashing out (heh). In the latest update Mark also addressed my main concern, making the launcher configurable so it can also be used by alternative implementations (particularly IronPython for Windows). I've copied Jeff Hardy and Dino (IronPython maintainers) in on this, in the hope that they might take a look at the latest version of the PEP too... All the best, Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 30 June 2011 13:50, Paul Moore <p.f.moore@gmail.com> wrote:
OK, having looked through this, it looks pretty solid to me. I might try installing Vinay's implementation and seeing how it feels in use, as well... On restoring associations, I think it's entirely reasonable not to bother. It would be nice if py.exe remained installed as long as any (all-users) Python installation remained on the PC, but this may be complicated (given that older versions won't uninstall it) so maybe don't even bother with that. Actually, I'd question whether this shouldn't be packaged as a separate application, available as an independent download from python.org (I do think it's important enough to be hosted on python.org rather than PyPI, though). Future versions of python could bundle it as well, but that could be purely for convenience and to ensure that users don't miss it. If Python does bundle it, then I'd suggest that it remain a separate item in add/remove programs. That allows the user to choose whether to uninstall it or not. As py.exe and python live in separate directories (except in per-user installs) the installers don't have any nasty interactions like who deletes the directory to worry about. If you want to bother with an "active installation count" mechanism (whether SharedDLLs or a simple count/listing in the py.ini file, or something else) then when the count hits "No", just offer the user the choice to uninstall as part of the Python uninstall. I'd be inclined not to worry too much about per-user installations. Are per-user file associations possible? I've never used them, myself. As a simple approach, just install py.exe alongside python.exe in the user dir, don't worry about it being on PATH or having a separate add/remove programs item, and let the user do what he wants with it. Of course, if someone with more experience of per-user installs and the sorts of environment where they are needed wants to suggest something different, believe them rather than me! Basically, my feeling is that the core functionality is fine. Nothing in this will make life worse for anyone, so we can safely leave corner cases to be addressed later (with a standalone release, if it's urgent enough). Paul.

Paul Moore <p.f.moore <at> gmail.com> writes:
Do have a play, it would be nice to get feedback. It's only available as source, though - is that OK?
I'd be inclined not to worry too much about per-user installations. Are per-user file associations possible? I've never used them, myself.
Yes, and they can be problematic: see http://www.deadlybloodyserious.com/2007/05/no-script-arguments/ where the problem described has also bitten me. I've no idea what put that association in the registry. Regards, Vinay Sajip

Mark Hammond <skippy.hammond <at> gmail.com> writes:
Sorry, I just recently came across the PEP when Tim Golden pointed it out to me, when I asked a question about executable script support in packaging. I think I was probably not so focused on Windows at the time of the original announcement ... There's a lot to like in the PEP, and I have some questions relating to the latest version: 1. In the section on shebang line parsing, it says "If the command starts with the definition of a customized command followed by a space character, the customized command will be used." Sorry if I'm being dense, but what's the significance of the trailing space character? In fact, your vpython example in the customeised comments section doesn't show a trailing space - you've used '#! vpython' as the shebang line. 2. The section on .ini files says that "Commands specified in the [.ini file in the] "application directory" [APPDATA] will have precedence over the one next to the [launcher] executable." What's the benefit of this? If you have only one launcher executable of one type (say console, 32-bit) on a system, then there's no point in having two locations for .ini files. If you have multiple copies of the launcher and multiple .ini files next to them, then with this precedence order, you can't specialise the behaviour of anything in a specific launcher .ini that's also specified in the APPDATA .ini. Doesn't it make more sense to look for a setting in any file next to the launcher, and if not found there, look in the APPDATA? That way common things can be defined in the APPDATA .ini and overridden for special cases in the launcher-adjacent .ini. Or have I got completely the wrong end of the stick? By the way, I've already converted the existing Python reference implementation to C (I did it before I saw your response to my post). It basically works in that it does what the Python version does, but doesn't include any handling of -32 suffixes or .ini files. I can post this on BitBucket if anyone's interested, but it's a work in progress. I'm working on some simple tests. Regards, Vinay Sajip

On 30/06/2011 10:09 PM, Vinay Sajip wrote:
This is just clumsy wording on my part - what I'm trying to say is simply that 'vpython3' will not match a customized command 'vpython'.
The idea is that a user without admin permissions can customize the behaviour - so the admin can add a .ini file, but the user can override any commands in that file with their own definitions.
The idea is that some users will not have permission to edit the one next to the launcher. I'd be open to considering this yagni if others agree.
The intention is that there only be a single launcher, as only one app can be associated with .py files. OTOH though, file associations can be configured per-user IIRC, and assuming that is the case, we could avoid my multiple-ini-file usecase above by just allowing a different launcher to be registered for a specific user. This is sounding difficult from the UI perspective though (ie, does the installer then need to ask that question, will there be a post-install technique for per-user registration, etc?)
I came at this from the other angle under the assumption there will only be one launcher installed - so common things can be stored in the launcher-adjacent file and per-user special cases can be in the per-user APPDATA dir. If the expectation was multiple launchers be installed, then I'd probably then prefer to KISS and only have the launcher-adjacent file supported at all.
Sure, that would be awesome! I think that will mean your impl is fairly close to the first draft of the PEP I checked into HG, which is nice and still quite useful to use :) Thanks! Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
I don't like this, for the reasons you state. I think it would be better if the PEP was changed to say that there is intended to be just one launcher installation per machine. As the intention is for the launcher to ship with Python, and there can be multiple Pythons installed, I presume we'll have to handle this by installing the launcher in some common-to-all-Pythons location somewhere outside a Python installation location, such as "c:\Program Files\Python Launcher". This should be stated in the PEP, and we'll also need to indicate how the launcher will be cleaned up if for some strange reason someone uninstalls all Pythons from a machine. Then we can just state that there's a global .ini file (where the launcher is installed) and a local one (in the user's APPDATA). From that perspective, it makes sense to have items in the local (APPDATA) override the global (launcher installation location).
Okay, I'll do this soon - once I've added a few tests. I've updated the implementation to include help output. BTW I thought of another thing that perhaps needs handling: what if a customized command points to the launcher itself? It'd be turtles all the way down :-) Regards, Vinay Sajip

On 1/07/2011 7:20 PM, Vinay Sajip wrote:
The PEP does say "if possible, should be installed somewhere likely to already be on the system PATH (eg., the Windows System32) directory." It is silent about what to do when that isn't possible, but I'd think it OK if the launcher was installed directly in the Python directory - IOW, I'd think it OK if the PEP said "should be installed next to the PythonXX.dll being installed" - but an important point in the above working is the "already be on the system PATH" - ie, I don't really want it put in a newly created directory unless the installer also adds that directory to the PATH - and what to do on uninstall then becomes an issue. One problem with all of this is uninstallation and specifically if the user is uninstalling the most recent Python installation while leaving earlier ones. I guess there are 2 general answers to this: * The installer process could remember the previous association and restore that on uninstall. * We treat this as a "wont-fix" and document a work-around of asking the user to reinstall the previous version to restore the file association. We probably need to be mindful of adding too much extra work for the installer process as that may well end up blocking us on getting this into the next appropriate release. In particular, Martin's thoughts here would be very useful. This would force the user to reinstall that older one to re-establish the associations correctly
BTW I thought of another thing that perhaps needs handling: what if a customized command points to the launcher itself? It'd be turtles all the way down :-)
Yeah - I wonder if we can leverage the "job" api here and refuse to start if there are already 2 processes in the job? OTOH, that is tricky as it would also prevent someone using os.startfile with a .py file.... From your second mail:
I'm not too bothered to be honest - the customized commands exist purely for alternative implementations, so my initial thoughts are that additional args would be as useful for them as they are for cpython invocations. IOW, if they don't need it, then CPython invocations don't need it either, so maybe it can be dropped completely?
Sure - I think there is some policy that a Python version number will never be > 10, so that sounds fine to me. So long as the launcher doesn't blindly run off the end of such arrays I think it is fine - limitations can be addressed in later versions. It will be a few days until I can look at the implementation, but I'm very happy to see it started. Given it is now ahead of the Python reference impl, I wonder if we should just drop all wording about that reference impl and just treat the C impl as canonical? Cheers, Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
But "next to PythonXY.dll" implies multiple copies, which we want to avoid, right?
We'd need to do that in the case of the earlier Python not having come with a launcher, i.e. when its version is < 3.3.
* We treat this as a "wont-fix" and document a work-around of asking the user to reinstall the previous version to restore the file association.
This is not ideal from a user's perspective.
It sounds onerous for users to have to reinstall an older Python. I'm not that familiar with Windows Installer features, so I don't know if this is too fancy, but perhaps we could remember the last non-launcher association when we install the launcher, and either restore that when the launcher is uninstalled (if it's still pointing to an installed Python) or remove the association altogether, otherwise. If this logic is too fancy to include in the installer itself, perhaps we can provide this logic in the launcher itself or via an ancillary executable or DLL that the installer can just call into. Is there some mechanism like the SharedDLLs registry key for executables, which could be used to reference count launcher installations so that it could be uninstalled at the appropriate time? Could we perhaps used the SharedDLLs feature itself?
If there's only ever one launcher installed (which we could ensure by installing it in a Windows rather than a Python location) then perhaps we could perhaps check for the value of a customised command pointing at the one-and- only launcher, but this is circumventable. Anyway, perhaps we just need to handle a user error rather than someone deliberately trying to engineer recursion. Another approach might be - rather than limit the number of processes in the job, look to see if the launcher executable is already associated with an existing job. I'm not au fait with the job API, and hence unsure of how that would play with usages such as os.startfile, subprocess etc.
I think they would be useful, so let me check the implementation again.
Once you've taken a closer look, if you think it looks good enough, then that's fine. If you have a BitBucket account, I can add your account to the repo so you can push changes to it. Regards, Vinay Sajip

On 2/07/2011 7:08 PM, Vinay Sajip wrote:
I believe this will be most useful when people select "install for all users", which will force it into system32. If people select "just for me", then there will be multiple copies on disk, but only one will be active (ie, we will not support different ones being registered for different users. It isn't ideal, but I think it good enough, avoids some complexity and should meet the needs of the majority of users.
OTOH, we don't do that now AFAIK.
Sure, but neither is the current situation - I don't recall enough users complaining about that to make the effort worthwhile. I'm not fundamentally opposed to doing something better here - I'm just trying to avoid this kind of stuff being a requirement for acceptance. If you are more familiar with the installer than I and feel it can be done simply, then I'm happy for you to go for it!
But this only happens when they install a later version, then uninstall the later one and continue to use the old one. I'd suggest that is (a) rare and (b) possibly already broken (ie, the old association may not be restored now). If the old association currently is correctly restored, then I'd expect things to just magically work in this new model without any effort.
I'm still inclined to call YAGNI, but as above, I don't fundamentally oppose such bells-and-whistles.
We probably can't ensure this while the installer supports a non-UAC installer (ie, the "just for me" option.) OTOH, I'm not sure the "just for me" option currently works without needing UAC elevation anyway. But assuming it does (or the intention is to fix things if it does not), then we can't guarantee a writable system32.
Yeah, I agree.
Exactly - code doing os.startfile on a .py file will correctly cause the same launcher to be re-executed and preventing this would break such scripts. However, I think this would be extremely rare and the more usual case would be to use sys.executable. Indeed, any script using os.startfile for a .py file is already broken, even if the author doesn't know it yet :)
Great, thanks! Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
No, you're right - I'm just throwing ideas out. I'm not especially familiar with MSI in general, though I believe you can do the relevant things with logic in custom actions. AFAIK there are already custom actions used in the Python MSI, but I wouldn't want to propose any such changes to the MSI unless Martin were to make encouraging comments :-)
As to (a), not uncommon scenarios would be (i) later version breaks something, so go back to earlier version, or (ii) using 2.x, installing 3.x to try out, going back to 2.x. I'm not sure about (b).
I'm still inclined to call YAGNI, but as above, I don't fundamentally oppose such bells-and-whistles.
You're probably right about the cost(effort)-benefit trade-off.
Okay. From a cursory glance at msi.py, UAC elevation appears to be requested.
I had a closer look at the Job API, and it does seem to offer the required functionality, so we could tell whether the current process is in a job -> get all processes in that job -> get their executables and command lines -> see if recursion is occurring. However, would we class this as an implementation detail or does it need a mention in the PEP?
I think they would be useful, so let me check the implementation again.
I've made some updates so the following works: with the configuration containing [commands] shell=cmd /c with a shebang in doit.py of '#!shell python -v' the launcher will run 'cmd /c python -v doit.py'. Regards, Vinay Sajip

Mark Hammond <skippy.hammond <at> gmail.com> writes:
I've now checked, and it appears that we don't do the right thing now anyway. On a clean Win7-x64, I installed Python 2.7 (32-bit), for all users, with registration of extensions - and Python.File etc. were added to the registry pointing at Python 2.7. I then installed Python 3.2 the same way, and rhe registry entries then pointed to 3.2. I then uninstalled 3.2, and the registry entries were gone! No magical restoration to the earlier values :-( Okay - if users have to do this now anyway, we at least wouldn't be naking things worse :-) Regards, Vinay Sajip

Mark Hammond wrote:
On 2/07/2011 7:08 PM, Vinay Sajip wrote:
perhaps we could remember the last non-launcher association when we install the launcher,
It might be better to look in the registry for other Python installations and ask the user which one to restore if there is more than one. Trying to restore the "last" one would be prone to breakage if the user didn't uninstall versions in precisely the reverse of the order of installation. -- Greg

On 4/07/2011 3:59 PM, Greg Ewing wrote:
While that makes alot of sense, the fact we are already "broken" in exactly the same way means I hope we can treat the restoration of associations as a separate issue - a worthwhile one, but not a pre-requisite for this PEP being approved. Cheers, Mark

On Mon, Jul 4, 2011 at 2:27 AM, Mark Hammond <skippy.hammond@gmail.com> wrote:
I agree let the restoration or not of file association be an implementation detail and not a requirement. I also agree with Paul Moore that py.exe should be a separate install/uninstall It can be bundled with the python MSI but should be a standalone MSI that the python MSI executes. So uninstalling the version of python that installed it alone is not enough to remove it. This will make it seem like the file associations are just working for naive users but will still let them remove the associations. We can just put a warning in the uninstall process of py.exe that lets the user know if they still have python versions installed they will need to re-install them to get the file association again.

Mark Hammond <skippy.hammond <at> gmail.com> writes:
I agree, but there's one aspect of associations which is perhaps worth exploring: the installation of a pre-3.3 version of Python after Python 3.3 is installed with the launcher will, if the user selects "Register Extensions", hijack the laumcher's associations to that earlier Python. Then bye bye launcher - how do we deal with that? Or don't we? There'll be no warning for the user, and this problem will occur even if the launcher is packaged separately from Python. so I think we need to think about this a little more. What say? Regards, Vinay Sajip

Vinay Sajip wrote:
I don't see how anything can be done about that. It also doesn't seem like too serious a problem -- it's no worse than what currently happens if you install an older version over a newer one, i.e. associations revert to the old version. Making the launcher a separate install at least offers a way of fixing that if it happens and isn't desired, i.e. reinstall the launcher. -- Greg

On 5/07/2011 11:23 AM, Greg Ewing wrote:
Or an MSI installer may be able to offer a "repair" feature without too much pain. However, I'm a bit torn on the separate installer issue. I really think it should be installed with later Python versions even if it is made available as a separate installer for the benefit of earlier ones as it is one less step for people to get confused about - eg, in a few years when 3.3+ is the most common Python being downloaded and used, books or people helping on python-list could start referring to the launcher under the assumption it is installed, rather than avoiding mention of it simply to avoid the whole spiel about "this will only work if you have installed an optional package." IOW, it will be impossible to give unconditional advice on certain aspects of launching Python. If the launcher is such that we can unconditionally recommend its use, IMO we should just install it with Python. I'll go with the consensus though... Mark

On Tue, Jul 5, 2011 at 12:12 PM, Mark Hammond <skippy.hammond@gmail.com> wrote:
If the launcher is such that we can unconditionally recommend its use, IMO we should just install it with Python. I'll go with the consensus though...
I've installed other WIndows apps that create multiple add/remove programs entries from a single installer. I believe people are suggesting a similar thing here (i.e. have the launcher installed automatically when installing python, but create a separate add/remove entry so uninstallation leaves it behind unless removal is explicitly requested) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan <ncoghlan <at> gmail.com> writes:
Were those other Windows apps packaged as .msi, or .exe? AFAICT, although you can embed an MSI inside another one, the practice of concurrent/nested installations is strongly discouraged by Microsoft - see http://goo.gl/FJx1S (Rule 20). Also, IIUC, each entry in Add/Remove programs would correspond to a specific MSI (since you can e.g. repair that specific entry, it would imply its own installer database). So you could package Python and the launcher as separate MSIs (this would make sense so that you could restore associations to the launcher just by repairing its installation), but since nested MSIs are a no-no, that means installing via a bootstrapping .exe. This is a bigger change to our Windows packaging than some people might be comfortable with ... Regards, Vinay Sajip

On Tue, Jul 5, 2011 at 03:01, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Right, you cannot sanely embed one .msi inside another .msi; the support for "nested" or "concurrent" installations referred to in that link is indeed something to avoid.
You can certainly jump through all these hoops, but the pieces here are much more suited towards a component definition that can be shared among multiple products. If the component always installs to the same place, has the same GUID, and otherwise only changes by versions the exe, this is a completely safe correct use of one. Last I knew, msi.py allocates random GUIDs, so may or may not be suitable for generating this. If there is only rare need to update this exe, and msi.py has support for merge modules, that could be one approach. My recommendation for distributing this: each .msi which wants to include it should have a component that includes the following. Note that each .exe (py, pyw) and each architecture (x86, x64) need their own component with their own static GUID. * Defined unchanging GUID * Defined target location (perhaps SystemFolder) * msidbComponentAttributesSharedDllRefCount (cooperate with non-MSI installers), msidbComponentAttributesShared (keep highest version), and possibly msidbComponentAttributesPermanent flags set on the component * Versioned .exe (using a Windows version block) * File association information Then these components can be included in the python 3.3 installer, future releases, and even a standalone installer, and reference count correctly. Again, these can optionally be made available as merge modules for other consumers, but there's likely no need. -- Michael Urman

Michael Urman <murman <at> gmail.com> writes:
I'm currently experimenting with a merge module approach: launcher_module.msm -> launcher.msi, and x64 variants in separate .amd64.ms? files.
I'm doing that.
Thanks, that's interesting information. I'll read up about these. I'm a Windows installer tyro :-)
* Versioned .exe (using a Windows version block)
I'm doing that, too.
* File association information
Currently I'm putting the file association information in the same component as the files, since the registry values cross reference those files. Regards, Vinay Sajip

On 5 July 2011 03:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's certainly what I was meaning. I'm 100% in favour of Python 3.3 and later containing the installer as part of the core Python installer. One download, one install. (And two add/remove entries). I'd like to see a standalone installer for users of Python 2.7/3.2 and earlier. It's too useful a feature to not make it available for people who haven't installed 3.3 yet. And I'd prefer it if that standalone installer was hosted on python.org for visibility, rather than on PyPI. I'm not enough of an MSI expert to know if this can be implemented by having a standalone MSI, and then "embedding" it in the Python 3.3 MSI. That was what I'd thought of, but Vinay's later email suggests it might not be advisable:
Paul.

Mark Hammond <skippy.hammond <at> gmail.com> writes:
Or an MSI installer may be able to offer a "repair" feature without too much pain.
A few more observations to do with installation: 1. It's been mentioned that a standalone version should be available for use with earlier Python versions. This could be done with a merge module which can be used either with a standalone installer or the Python .msi. 2. For the standalone MSI, we will most likely need separate 32-bit and 64-bit MSIs, because the MSI system will look at the MSI type to determine whether registry stuff goes in the main registry or the Wow6432Node used for 32-bit applications. Regards, Vinay Sajip

One more thing about associations - we've got pyw.exe for Python.NoConFile and py.exe for Python.file, but how do we handle Python.CompiledFile? It doesn't really make sense to have the association not handled by the launcher. Unfortunately, of course, both pyw and py compile to pyo, so we don't know which launcher to use. It is, of course, easy for either py or pyw to determine which version of python is to be used to invoke a .pyc - just not which Windows variant. BTW just as a test I implemented .pyc support in the C implementation - it works fine apart from the "python.exe or pythonw.exe?" question. Regards, Vinay Sajip

On 2/07/2011 5:16 PM, I wrote:
I'm looking to update the PEP based on this discussion - does anyone object to the above? Or to put it another way, does anyone believe dropping the Python reference implementation is to the detriment of the PEP? Thanks, Mark

Mark Hammond <skippy.hammond <at> gmail.com> writes:
My C implementation of the launcher is now available at https://bitbucket.org/vinay.sajip/pylauncher It's been built using VS 2008, and tested on Windows 7 (32-bit) with ActivePython 2.6.2.2 and ActivePython 3.2.0.0 installed. I've added support for .ini files [commands] section; I haven't looked at implementing [defaults] yet. There's incomplete support for -32 suffix at the moment - that can be looked at in due course. A couple of points: I've also allowed for /usr/local/bin/python as a built-in command, this can always be removed later if YAGNI. I've assumed that if a customised command is provided with arguments in the shebang line, these will be ignored - if people want to run with different options they can always define more customised commands. If you agree with this, the PEP should probably explicitly state this. In a couple of cases I've implemented using fixed size arrays - for the lists of installed Pythons and customised commands. Of course these can be made dynamic, but what's there is good enough for the moment for exploration. Do have a play, and let me know what you think. Regards, Vinay Sajip
participants (10)
-
Dj Gilcrease
-
Greg Ewing
-
Jeff Hardy
-
Mark Hammond
-
Michael Foord
-
Michael Urman
-
Nick Coghlan
-
Paul Moore
-
Tim Golden
-
Vinay Sajip