[Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Thanks for all the comments. I've responded all at once, since I don't think any of the points raised are going to immediately turn into intense discussion. Feel free to trim the rest of the post if you want to respond to just one of the points.

Donald Stufft wrote:
> Does signing the .exe’s that ship with Python effect pip at all? I’m guessing
> not since pip.exe is actually a generic .exe that we use for any project and is
> shipped inside of the .whl files but I figured i’d ask.
>
> Depending on the answer above, does it make sense to sign the generic .exe (does
> that even work if we rename it?) which will get used for anything that uses
> entry points on Windows?

Actually, I don't think the generic .exe can be signed, since embedding the script within the file will invalidate the certificate. It's better not to have the certificate at all in this case. If we were still using the separate pip-script.py file then we could sign it.

> Is there any plan to backport this to 2.7 (presumably after some experience is
> had with it in 3.5)?

No. I'm very much in the "2.7 is finished" camp. Hitting rebuild every now and then for security updates is the extent of my interest (and in large part that's because if nobody is producing builds then I'll probably end up having to produce the builds to run on Azure anyway).



Antione Pitrou wrote:
> One comment: I would find it a bit confusing if the default install path
> changes when using the customized install. OTOH, maybe you can't choose
> another default there.

It's a tough one to choose, honestly. I really don't want separate "customize" buttons for per-user and all-user installs (too many up front decisions, especially for people who really should just choose the first option). My reasoning for having "C:\Python35" as the default there is to simplify the "just give me the old style install" case, but that may turn out to be a bad reason. I'm not sure yet.



ChrisA wrote:
> You talk of installing by default into Program Files, and having a
> separate per-user installation mode. How do these two installation
> targets interact? Suppose someone installs 3.5 globally, then installs
> 3.6 for self only? Or installs 3.5.1 for self only? I would normally
> expect a per-user installation to trump a global one, but this could
> make for a lovely dep-hell on a system that's used by one person, who
> then isn't sure what was installed as admin and what wasn't.

Yeah, it gets a little messy, especially if you keep adding all the Python versions to PATH (Windows will always put per-user PATH entries at the end). The py.exe launcher handles this best, and the system version will always be found first.

As for versions, 3.5 vs 3.6 should be distinguished explicitly ("py -3.5" vs "py -3.6") and 3.5.0 vs 3.5.1 can only be installed at the same time if one is system-wide and the other is per-user. In this case, the per-user one will be used by py.exe, even if it is older than the system-wide one.

Compared to the current situation, the per-user and all-user installs do not interfere as much.

> (Also: I'm assuming that pip would require admin privs if Python is in
> Program Files, and won't require admins if it's per-user, right? Same
> as the Python installer itself?)

Correct. The pip guys are already looking into handling installs in this case since it matters on *nix, and having read-only Python installs on Windows will only make it more similar. It seems likely in this case that the user site-packages folder will be used if your user can't write to the shared site-packages. If you can write to it (either per-user install or running pip as admin) then you'll install to that folder.

I am fully expecting to find the most issues in this area once we change the install location, and I don't think we can fix all these issues without actually releasing the change. So it's a bit of a risk, but one I feel is worth it. Of course, if the community and release manager disagree, I'll happily back it out and we can focus on testing it directly for a bit before inflicting it on our users.



Glenn Linderman wrote:
> But here's another idea: automatically keep all the .msi and .cab files used for
> the first installation of Python with it in the directory from which it runs
> (naming convention... prefix them all with python-3.5.0a1.<stuff>.(msi|cab) It
> is very likely that a reinstall will use the same components (if more are needed
> on a later install, add them to the directory also). And a good naming
> convention makes it obvious what to delete when done with the installer.

Unfortunately, I'm fairly restricted on naming convention. For some reason Windows Installer requires CAB files to be 8.3 names, where some of the 8 are reserved, so the names are pretty meaningless.

However, Windows does automatically cache all the files (typically in C:\ProgramData\Package Cache) so that repairs and uninstalls don't need to redownload anything, and this cache is automatically managed to preserve disk space. If you add new features later then you won't have the files in the cache, which is about the only way that this doesn't already fit your suggestion. I think it's fairly unlikely that people will be surprised by this, though when the first complaints come it about it I'll happily reconsider.
 
> And a related idea: on the first install page, have a check box "download all
> installation components", that would do that, even if they are not used, and
> even if either of the one-click installs are chosen.

Interesting idea, but I'm worried that people would think they need to click the box simply because it is there. I've seen the same thing happen in many other places, and I think this functionality is advanced enough that the people who need it know that they need it and will look for it. I could, however, add a help link to the first page of the installer, once there's somewhere for it to go.

> And a related idea: for custom installs, record the choices made in a metadata
> file in that same directory, and after the first install, subsequent installs
> could have a 3rd single-click install: same custom install as last time. This
> would be kept in the directory with the installer, so could be applied to a
> zillion machines, and an install option /ditto would allow those choices from
> the command line. That way, the administrator could use the friendly interface
> to install the first machine, making the appropriate choices, and then just run
> "python-3.5.0a1.exe /ditto" on all the other zillion-1 machines, without needing
> to learn any other obscure command line parameters. I don't care how you spell
> /ditto, as long as there is documentation.

This is really interesting, and certainly possible. I'll keep it in mind as something to look into after alpha. (What may be easier is a message at the end showing the command line to use that will perform the same installation silently... hmm...)

> Regarding /layout, to me, that name has no mnemonic meaning of "download all
> these installation components and save them". Documentation could provide that,
> of course, but choosing a name like /download might be nicer. Saving to the same
> directory as the installer lives in seems easier than needing to specify a
> directory... the documentation can warn that users of the option should put the
> web installer in the desired, shared or private, installation directory prior to
> running the option...

It's the standard option used by Burn (the install engine I'm using) - for example, the Visual Studio web installers will also respond to /layout - and it's probably from some internal Microsoft terminology. However, it's easy enough for me to add an alias.

The installer also responds to "/?" and "/help" (IIRC, it may only be the first) to display all of its options, and I expect this will be the first thing people guess to try.


Paul Moore wrote:
> Overall, this looks good. One question - will it be possible to
> install both 32-bit and 64-bit Python on the same machine? Currently,
> you need a custom install to do this (as the default directory doesn't
> include the architecture) and IIRC there's some oddness around install
> order. It would be nice if installing both versions were a supported
> option, both for the "default" install and in custom installs.

Yes, the default install directories on the first page are different for 32-bit and 64-bit (either "Program Files [(x86)]" or "Python35[-32]"). The default value on the customize page is currently always C:\Python35, and so you'll need to change it if you're doing custom installs, but that's easy to add the "-32" by default. (I used a -32 suffix because it matches the py.exe option.)

> Also, what happens now with setting PATH? Is Python (and the scripts
> directory) added to PATH by default? If so, what happens when you
> install 2 versions of Python?

Yes, and in general the later installed version will win and system-wide installs always beat per-user installs. As I mentioned above, using py.exe with a parameter or shebang line is the most reliable way to get the version you want.

> In case it's not clear, I'm thinking of the impact on build machines,
> which often have multiple versions, in both 32- and 64-bit forms,
> installed simultaneously (but can also be used as a "normal"
> development machine, and for that purpose will want a selected Python
> version as the default one.

You should see my dev machines :) Most have 2.5, 2.6, 2.7 32-bit and 64-bit, 3.0, 3.1, 3.2, 3.3 32-bit and 64-bit, 3.4 32-bit and 64-bit, IronPython and often PyPy, Anaconda or Canopy. So I'm being fairly selfish when I try and make the multiple-versions scenario work better, but it will benefit everyone.

> Also, how does the launcher py.exe fit into the picture? Is it still
> installed into the Windows directory? What about for a user install?
> Are Python scripts associated with the launcher, and if so, how does
> it pick up the version you selected as default?

py.exe is more important than ever. It's still installed into the Windows directory for all-user installs, and into the Python directory for just-for-me. It's installed in a way that will make upgrades more reliable (so if you install 3.6 and then 3.5, you'll keep the newer launcher) and all the file associations go straight to the launcher.

The default Python for the launcher seems to be 2.7 if it's there and the latest version if not (though I could be wrong). Shebang lines are the best way to choose a specific version.

Incidentally, whoever came up with the py.exe launcher deserves a medal. It's made dealing with multiple versions amazingly easy.



Nick Coghlan wrote:
> The proposed installer changes look like a great improvement to me,
> but I think Paul's questions suggest that it will be useful to capture
> the many decisions involved in the redesign as a PEP.

I was afraid someone would suggest this :)

I certainly intend to document it thoroughly. One of my overarching goals is to make it really easy for the next person to take over producing builds, or for someone else to produce a signed release build if I'm unavailable.

My intent was just to drop a text file in Tools/msi and some of the user-facing docs (like command-line options, best practices and rationale) into the main docs. If there's added value in a PEP, then I can do that too, though I would prefer the documentation for this to be more living than most PEPs are.


Cheers,
Steve