On 30/06/2011 08:34, Tim Golden wrote:

On 30/06/2011 05:23, Mark Hammond wrote:

...

On 30/06/2011 3:00 AM, Vinay Sajip wrote:

...

PEP 397 (Python launcher for Windows) has a reference implementation in Python. Does anyone know of a C implementation, or is planning/working on one? I realise this is the final objective, so such implementation might be premature, but perhaps someone has been experimenting ...

Not yet - my last update of the PEP has made the existing reference implementation out-of-date, so I want to work on that before starting on the C version. However, seeing as my note about the most recent PEP update attracted zero comments..

Sorry, didn't realise you couldn't see me silently nodding my head from the other side of the world ;) I really must work on those communication skills...

I have that email (the update one from Mark not the silent nodding from Tim) still sitting in my inbox waiting for me to properly read through and comment on... Sorry for being useless, I'll try and move it up the priority list. I really like the PEP and think it will be a *huge* step forward for Windows users - so it's purely the details that need thrashing out (heh). In the latest update Mark also addressed my main concern, making the launcher configurable so it can also be used by alternative implementations (particularly IronPython for Windows). I've copied Jeff Hardy and Dino (IronPython maintainers) in on this, in the hope that they might take a look at the latest version of the PEP too... All the best, Michael

TJG _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 30 June 2011 13:50, Paul Moore wrote:

On 30 June 2011 12:13, Michael Foord wrote:

...

I have that email (the update one from Mark not the silent nodding from Tim) still sitting in my inbox waiting for me to properly read through and comment on... Sorry for being useless, I'll try and move it up the priority list.

I really like the PEP and think it will be a *huge* step forward for Windows users - so it's purely the details that need thrashing out (heh).

That's my situation, too. I'll try to look through it properly in the next day or two.

OK, having looked through this, it looks pretty solid to me. I might try installing Vinay's implementation and seeing how it feels in use, as well... On restoring associations, I think it's entirely reasonable not to bother. It would be nice if py.exe remained installed as long as any (all-users) Python installation remained on the PC, but this may be complicated (given that older versions won't uninstall it) so maybe don't even bother with that. Actually, I'd question whether this shouldn't be packaged as a separate application, available as an independent download from python.org (I do think it's important enough to be hosted on python.org rather than PyPI, though). Future versions of python could bundle it as well, but that could be purely for convenience and to ensure that users don't miss it. If Python does bundle it, then I'd suggest that it remain a separate item in add/remove programs. That allows the user to choose whether to uninstall it or not. As py.exe and python live in separate directories (except in per-user installs) the installers don't have any nasty interactions like who deletes the directory to worry about. If you want to bother with an "active installation count" mechanism (whether SharedDLLs or a simple count/listing in the py.ini file, or something else) then when the count hits "No", just offer the user the choice to uninstall as part of the Python uninstall. I'd be inclined not to worry too much about per-user installations. Are per-user file associations possible? I've never used them, myself. As a simple approach, just install py.exe alongside python.exe in the user dir, don't worry about it being on PATH or having a separate add/remove programs item, and let the user do what he wants with it. Of course, if someone with more experience of per-user installs and the sorts of environment where they are needed wants to suggest something different, believe them rather than me! Basically, my feeling is that the core functionality is fine. Nothing in this will make life worse for anyone, so we can safely leave corner cases to be addressed later (with a standalone release, if it's urgent enough). Paul.

On 3 July 2011 19:20, Vinay Sajip wrote:

Paul Moore writes:

...

OK, having looked through this, it looks pretty solid to me. I might try installing Vinay's implementation and seeing how it feels in use, as well...

Do have a play, it would be nice to get feedback. It's only available as source, though - is that OK?

No problem - I can build it from source.

...

I'd be inclined not to worry too much about per-user installations. Are per-user file associations possible? I've never used them, myself.

Yes, and they can be problematic: see

http://www.deadlybloodyserious.com/2007/05/no-script-arguments/

where the problem described has also bitten me. I've no idea what put that association in the registry.

Ugh. That's nasty. It doesn't even look like a standard file type association. Paul.

Mark Hammond writes:

Not yet - my last update of the PEP has made the existing reference implementation out-of-date, so I want to work on that before starting on the C version. However, seeing as my note about the most recent PEP update attracted zero comments, I admit I lost any sense of urgency on actually doing this... I'll make an effort to update that reference impl in the next week or so (but obviously anyone else is free to help ;)

Sorry, I just recently came across the PEP when Tim Golden pointed it out to me, when I asked a question about executable script support in packaging. I think I was probably not so focused on Windows at the time of the original announcement ... There's a lot to like in the PEP, and I have some questions relating to the latest version: 1. In the section on shebang line parsing, it says "If the command starts with the definition of a customized command followed by a space character, the customized command will be used." Sorry if I'm being dense, but what's the significance of the trailing space character? In fact, your vpython example in the customeised comments section doesn't show a trailing space - you've used '#! vpython' as the shebang line. 2. The section on .ini files says that "Commands specified in the [.ini file in the] "application directory" [APPDATA] will have precedence over the one next to the [launcher] executable." What's the benefit of this? If you have only one launcher executable of one type (say console, 32-bit) on a system, then there's no point in having two locations for .ini files. If you have multiple copies of the launcher and multiple .ini files next to them, then with this precedence order, you can't specialise the behaviour of anything in a specific launcher .ini that's also specified in the APPDATA .ini. Doesn't it make more sense to look for a setting in any file next to the launcher, and if not found there, look in the APPDATA? That way common things can be defined in the APPDATA .ini and overridden for special cases in the launcher-adjacent .ini. Or have I got completely the wrong end of the stick? By the way, I've already converted the existing Python reference implementation to C (I did it before I saw your response to my post). It basically works in that it does what the Python version does, but doesn't include any handling of -32 suffixes or .ini files. I can post this on BitBucket if anyone's interested, but it's a work in progress. I'm working on some simple tests. Regards, Vinay Sajip

Mark Hammond writes:

The intention is that there only be a single launcher, as only one app can be associated with .py files. OTOH though, file associations can be configured per-user IIRC, and assuming that is the case, we could avoid my multiple-ini-file usecase above by just allowing a different launcher to be registered for a specific user. This is sounding difficult from the UI perspective though (ie, does the installer then need to ask that question, will there be a post-install technique for per-user registration, etc?)

I don't like this, for the reasons you state. I think it would be better if the PEP was changed to say that there is intended to be just one launcher installation per machine. As the intention is for the launcher to ship with Python, and there can be multiple Pythons installed, I presume we'll have to handle this by installing the launcher in some common-to-all-Pythons location somewhere outside a Python installation location, such as "c:\Program Files\Python Launcher". This should be stated in the PEP, and we'll also need to indicate how the launcher will be cleaned up if for some strange reason someone uninstalls all Pythons from a machine. Then we can just state that there's a global .ini file (where the launcher is installed) and a local one (in the user's APPDATA). From that perspective, it makes sense to have items in the local (APPDATA) override the global (launcher installation location).

Sure, that would be awesome! I think that will mean your impl is fairly close to the first draft of the PEP I checked into HG, which is nice and still quite useful to use :)

Okay, I'll do this soon - once I've added a few tests. I've updated the implementation to include help output. BTW I thought of another thing that perhaps needs handling: what if a customized command points to the launcher itself? It'd be turtles all the way down :-) Regards, Vinay Sajip

Mark Hammond writes:

The PEP does say "if possible, should be installed somewhere likely to already be on the system PATH (eg., the Windows System32) directory." It is silent about what to do when that isn't possible, but I'd think it OK if the launcher was installed directly in the Python directory - IOW, I'd think it OK if the PEP said "should be installed next to the PythonXX.dll being installed" - but an important point in the above working is the "already be on the system PATH" - ie, I don't really want it put in a newly created directory unless the installer also adds that directory to the PATH - and what to do on uninstall then becomes an issue.

But "next to PythonXY.dll" implies multiple copies, which we want to avoid, right?

One problem with all of this is uninstallation and specifically if the user is uninstalling the most recent Python installation while leaving earlier ones. I guess there are 2 general answers to this:

* The installer process could remember the previous association and restore that on uninstall.

We'd need to do that in the case of the earlier Python not having come with a launcher, i.e. when its version is < 3.3.

* We treat this as a "wont-fix" and document a work-around of asking the user to reinstall the previous version to restore the file association.

This is not ideal from a user's perspective.

We probably need to be mindful of adding too much extra work for the installer process as that may well end up blocking us on getting this into the next appropriate release. In particular, Martin's thoughts here would be very useful.

This would force the user to reinstall that older one to re-establish the associations correctly

It sounds onerous for users to have to reinstall an older Python. I'm not that familiar with Windows Installer features, so I don't know if this is too fancy, but perhaps we could remember the last non-launcher association when we install the launcher, and either restore that when the launcher is uninstalled (if it's still pointing to an installed Python) or remove the association altogether, otherwise. If this logic is too fancy to include in the installer itself, perhaps we can provide this logic in the launcher itself or via an ancillary executable or DLL that the installer can just call into. Is there some mechanism like the SharedDLLs registry key for executables, which could be used to reference count launcher installations so that it could be uninstalled at the appropriate time? Could we perhaps used the SharedDLLs feature itself?

Yeah - I wonder if we can leverage the "job" api here and refuse to start if there are already 2 processes in the job? OTOH, that is tricky as it would also prevent someone using os.startfile with a .py file....

If there's only ever one launcher installed (which we could ensure by installing it in a Windows rather than a Python location) then perhaps we could perhaps check for the value of a customised command pointing at the one-and- only launcher, but this is circumventable. Anyway, perhaps we just need to handle a user error rather than someone deliberately trying to engineer recursion. Another approach might be - rather than limit the number of processes in the job, look to see if the launcher executable is already associated with an existing job. I'm not au fait with the job API, and hence unsure of how that would play with usages such as os.startfile, subprocess etc.

I'm not too bothered to be honest - the customized commands exist purely for alternative implementations, so my initial thoughts are that additional args would be as useful for them as they are for cpython invocations. IOW, if they don't need it, then CPython invocations don't need it either, so maybe it can be dropped completely?

I think they would be useful, so let me check the implementation again.

It will be a few days until I can look at the implementation, but I'm very happy to see it started. Given it is now ahead of the Python reference impl, I wonder if we should just drop all wording about that reference impl and just treat the C impl as canonical?

Once you've taken a closer look, if you think it looks good enough, then that's fine. If you have a BitBucket account, I can add your account to the repo so you can push changes to it. Regards, Vinay Sajip

Mark Hammond writes:

I'm not fundamentally opposed to doing something better here - I'm just trying to avoid this kind of stuff being a requirement for acceptance. If you are more familiar with the installer than I and feel it can be done simply, then I'm happy for you to go for it!

No, you're right - I'm just throwing ideas out. I'm not especially familiar with MSI in general, though I believe you can do the relevant things with logic in custom actions. AFAIK there are already custom actions used in the Python MSI, but I wouldn't want to propose any such changes to the MSI unless Martin were to make encouraging comments :-)

But this only happens when they install a later version, then uninstall the later one and continue to use the old one. I'd suggest that is (a) rare and (b) possibly already broken (ie, the old association may not be restored now). If the old association currently is correctly restored, then I'd expect things to just magically work in this new model without any effort.

As to (a), not uncommon scenarios would be (i) later version breaks something, so go back to earlier version, or (ii) using 2.x, installing 3.x to try out, going back to 2.x. I'm not sure about (b).

I'm still inclined to call YAGNI, but as above, I don't fundamentally oppose such bells-and-whistles.

You're probably right about the cost(effort)-benefit trade-off.

We probably can't ensure this while the installer supports a non-UAC installer (ie, the "just for me" option.) OTOH, I'm not sure the "just for me" option currently works without needing UAC elevation anyway. But assuming it does (or the intention is to fix things if it does not), then we can't guarantee a writable system32.

Okay. From a cursory glance at msi.py, UAC elevation appears to be requested.

Exactly - code doing os.startfile on a .py file will correctly cause the same launcher to be re-executed and preventing this would break such scripts. However, I think this would be extremely rare and the more usual case would be to use sys.executable. Indeed, any script using os.startfile for a .py file is already broken, even if the author doesn't know it yet :)

I had a closer look at the Job API, and it does seem to offer the required functionality, so we could tell whether the current process is in a job -> get all processes in that job -> get their executables and command lines -> see if recursion is occurring. However, would we class this as an implementation detail or does it need a mention in the PEP?

...

I think they would be useful, so let me check the implementation again.

I've made some updates so the following works: with the configuration containing [commands] shell=cmd /c with a shebang in doit.py of '#!shell python -v' the launcher will run 'cmd /c python -v doit.py'. Regards, Vinay Sajip

Mark Hammond wrote:

On 2/07/2011 7:08 PM, Vinay Sajip wrote:

...

perhaps we could remember the last non-launcher association when we install the launcher,

It might be better to look in the registry for other Python installations and ask the user which one to restore if there is more than one. Trying to restore the "last" one would be prone to breakage if the user didn't uninstall versions in precisely the reverse of the order of installation. -- Greg

On Mon, Jul 4, 2011 at 2:27 AM, Mark Hammond wrote:

While that makes alot of sense, the fact we are already "broken" in exactly the same way means I hope we can treat the restoration of associations as a separate issue - a worthwhile one, but not a pre-requisite for this PEP being approved.

I agree let the restoration or not of file association be an implementation detail and not a requirement. I also agree with Paul Moore that py.exe should be a separate install/uninstall It can be bundled with the python MSI but should be a standalone MSI that the python MSI executes. So uninstalling the version of python that installed it alone is not enough to remove it. This will make it seem like the file associations are just working for naive users but will still let them remove the associations. We can just put a warning in the uninstall process of py.exe that lets the user know if they still have python versions installed they will need to re-install them to get the file association again.

Vinay Sajip wrote:

the installation of a pre-3.3 version of Python after Python 3.3 is installed with the launcher will, if the user selects "Register Extensions", hijack the laumcher's associations to that earlier Python. Then bye bye launcher

I don't see how anything can be done about that. It also doesn't seem like too serious a problem -- it's no worse than what currently happens if you install an older version over a newer one, i.e. associations revert to the old version. Making the launcher a separate install at least offers a way of fixing that if it happens and isn't desired, i.e. reinstall the launcher. -- Greg

On Tue, Jul 5, 2011 at 12:12 PM, Mark Hammond wrote:

If the launcher is such that we can unconditionally recommend its use, IMO we should just install it with Python.  I'll go with the consensus though...

I've installed other WIndows apps that create multiple add/remove programs entries from a single installer. I believe people are suggesting a similar thing here (i.e. have the launcher installed automatically when installing python, but create a separate add/remove entry so uninstallation leaves it behind unless removal is explicitly requested) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Tue, Jul 5, 2011 at 03:01, Vinay Sajip wrote:

Were those other Windows apps packaged as .msi, or .exe? AFAICT, although you can embed an MSI inside another one, the practice of concurrent/nested installations is strongly discouraged by Microsoft - see http://goo.gl/FJx1S (Rule 20).

Right, you cannot sanely embed one .msi inside another .msi; the support for "nested" or "concurrent" installations referred to in that link is indeed something to avoid.

So you could package Python and the launcher as separate MSIs (this would make sense so that you could restore associations to the launcher just by repairing its installation), but since nested MSIs are a no-no, that means installing via a bootstrapping .exe. This is a bigger change to our Windows packaging than some people might be comfortable with ...

You can certainly jump through all these hoops, but the pieces here are much more suited towards a component definition that can be shared among multiple products. If the component always installs to the same place, has the same GUID, and otherwise only changes by versions the exe, this is a completely safe correct use of one. Last I knew, msi.py allocates random GUIDs, so may or may not be suitable for generating this. If there is only rare need to update this exe, and msi.py has support for merge modules, that could be one approach. My recommendation for distributing this: each .msi which wants to include it should have a component that includes the following. Note that each .exe (py, pyw) and each architecture (x86, x64) need their own component with their own static GUID. * Defined unchanging GUID * Defined target location (perhaps SystemFolder) * msidbComponentAttributesSharedDllRefCount (cooperate with non-MSI installers), msidbComponentAttributesShared (keep highest version), and possibly msidbComponentAttributesPermanent flags set on the component * Versioned .exe (using a Windows version block) * File association information Then these components can be included in the python 3.3 installer, future releases, and even a standalone installer, and reference count correctly. Again, these can optionally be made available as merge modules for other consumers, but there's likely no need. -- Michael Urman

On 5 July 2011 03:26, Nick Coghlan wrote:

On Tue, Jul 5, 2011 at 12:12 PM, Mark Hammond wrote:

...

If the launcher is such that we can unconditionally recommend its use, IMO we should just install it with Python.  I'll go with the consensus though...

I've installed other WIndows apps that create multiple add/remove programs entries from a single installer. I believe people are suggesting a similar thing here (i.e. have the launcher installed automatically when installing python, but create a separate add/remove entry so uninstallation leaves it behind unless removal is explicitly requested)

That's certainly what I was meaning. I'm 100% in favour of Python 3.3 and later containing the installer as part of the core Python installer. One download, one install. (And two add/remove entries). I'd like to see a standalone installer for users of Python 2.7/3.2 and earlier. It's too useful a feature to not make it available for people who haven't installed 3.3 yet. And I'd prefer it if that standalone installer was hosted on python.org for visibility, rather than on PyPI. I'm not enough of an MSI expert to know if this can be implemented by having a standalone MSI, and then "embedding" it in the Python 3.3 MSI. That was what I'd thought of, but Vinay's later email suggests it might not be advisable:

AFAICT, although you can embed an MSI inside another one, the practice of concurrent/nested installations is strongly discouraged by Microsoft - see http://goo.gl/FJx1S (Rule 20). [...] So you could package Python and the launcher as separate MSIs (this would make sense so that you could restore associations to the launcher just by repairing its installation), but since nested MSIs are a no-no, that means installing via a bootstrapping .exe. This is a bigger change to our Windows packaging than some people might be comfortable with ...

Paul.

One more thing about associations - we've got pyw.exe for Python.NoConFile and py.exe for Python.file, but how do we handle Python.CompiledFile? It doesn't really make sense to have the association not handled by the launcher. Unfortunately, of course, both pyw and py compile to pyo, so we don't know which launcher to use. It is, of course, easy for either py or pyw to determine which version of python is to be used to invoke a .pyc - just not which Windows variant. BTW just as a test I implemented .pyc support in the C implementation - it works fine apart from the "python.exe or pythonw.exe?" question. Regards, Vinay Sajip

Mark Hammond writes:

Sure, that would be awesome! I think that will mean your impl is fairly close to the first draft of the PEP I checked into HG, which is nice and still quite useful to use :)

My C implementation of the launcher is now available at https://bitbucket.org/vinay.sajip/pylauncher It's been built using VS 2008, and tested on Windows 7 (32-bit) with ActivePython 2.6.2.2 and ActivePython 3.2.0.0 installed. I've added support for .ini files [commands] section; I haven't looked at implementing [defaults] yet. There's incomplete support for -32 suffix at the moment - that can be looked at in due course. A couple of points: I've also allowed for /usr/local/bin/python as a built-in command, this can always be removed later if YAGNI. I've assumed that if a customised command is provided with arguments in the shebang line, these will be ignored - if people want to run with different options they can always define more customised commands. If you agree with this, the PEP should probably explicitly state this. In a couple of cases I've implemented using fixed size arrays - for the lists of installed Pythons and customised commands. Of course these can be made dynamic, but what's there is good enough for the moment for exploration. Do have a play, and let me know what you think. Regards, Vinay Sajip