On 16Jul2016 1254, Paul Moore wrote:
On 15 July 2016 at 23:20, Steve Dower <steve.dower@python.org> wrote:
Hi all
I'd like to get this PEP approved (status changed to Active, IIUC).
Some comments below.
Awesome, thanks! Posted a pull request at https://github.com/python/peps/pull/59 for ease of diff reading, and some commentary below (with aggressive snipping).
Motivation ==========
When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. This allows tools such as installers or IDEs to automatically detect and display a user's Python installations.
The PEP seems quite strongly focused on GUI tools ... I'd like to avoid tool developers reading this section and thinking "it only applies to GUI tools or OS integration, not to me".
Agreed. I tried to avoid any console/GUI-specific terms, but I can probably be more explicit about it being useful to both.
For example, virtualenv introspects the available Python installations - see https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L86 - to support the "-p <interpreter>" flag. To handle this well, it would be useful to allow distributions to register a "short tag", so that as well as "-p 3.5" or "-p 2", Virtualenv could support (say) "-p conda3.4" or "-p pypy2". (The short tag should be at the Company level, so "conda" or "pypy", and the version gets added to that).
Another place where this might be useful is the py.exe launcher (it's not in scope for this PEP, but having the data needed to allow the launcher to invoke any available installation could be useful for future enhancements).
virtualenv would be a great example to use. My thinking was that the Tag should be appropriate here (perhaps with the Company to disambiguate when necessary), and that is now explicit. Anaconda currently has "Anaconda_4.1.1_64-bit" as their tag, which would not be convenient, so an explicit suggestion here would help ensure this is useful.
Another key motivation for me would be to define clearly what information tools can rely on being able to get from the available registry entries describing what's installed. Whenever I've needed to scan the registry, the things I've needed to find out are where I find the Python interpreter, what Python version it is, and whether it's 32-bit or 64-bit. The first so that I can run Python, and the latter two so that I can tell if this is a version I support *without* needing to run the interpreter. For me, everything else in this PEP is about UI, but those 3 items plus the "short tag" idea are more about what capabilities I can provide.
Good points. I discussed architecture with a colleague at one point and I'm not entirely sure it's universally useful (what architecture is IronPython when built for Any CPU? what architecture is Jython?), but maybe something like the contents of importlib.machinery.IMPORT_SUFFIXES would be?
On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key that 32-bit processes transparently read and write to rather than accessing the ``Software`` key directly.
It might be worth being more explicit here that 32-bit and 64-bit processes see the registry keys slightly differently. More on this below.
I considered this and thought I had a link to the official docs about it. I don't want this PEP to be mistaken for documentation on how registry redirection works :)
Backwards Compatibility -----------------------
Also, Python 3.5 doesn't appear to include the architecture in sys.winver either.
... (Unless it adds -32 for 32-bit, and reserves the bare version for 64-bit. I've skimmed the CPython source but can't confirm that). The documentation of sys.winver makes no mention of whether it distinguishes 32- and 64-bit builds. In fact, it states "The value is normally the first three characters of version". If we're relying on sys.winver being unique by version/architecture, the docs need to say so (so that future changes don't accidentally violate that).
I'll update the docs, but your guess is correct. I changed sys.winver on 32-bit to be "3.5-32" since that matches what py.exe was already using to refer to it. I didn't want to invent yet-another-way-to-tag-architectures. (I also updated py.exe to match tags directly under PythonCore, so 3.5-32 is matched without scanning the binary type.) Also, sys.winver is defined in PCBuild/python.props, which is how we accidentally backported the suffix to 2.7.11 :(
It is not possible to detect side-by-side installations of both 64-bit and 32-bit versions of Python prior to 3.5 when they have been installed for the current user. Python 3.5 and later always uses different Tags for 64-bit and 32-bit versions.
From what I can see, this latter isn't true. I presume that 64-bit uses no suffix, but 32-bit uses a "-32" suffix? This should probably be made explicit. At a minimum, if I were writing a tool to list all installed Python versions, with only what I have available to go on (the PEP and a 64-bit Python 3.5) I wouldn't be able to write correct code, as I don't have all the information I need.
Also, if we expect to be able to distinguish 32 and 64 bit implementations in this way, that's putting a new restriction on sys.winver, that it returns a different value for 32-bit and 64-bit builds. If that's the case, I'd rather see that explicitly documented, both here and in the sys.winver documentation.
I'd actually prefer a more explicit mechanism going forward, but as this is a "backward compatibility" section I'll save that for later.
I don't want to lock in the actual scheme of the tags used by CPython. Granted, without SysArchitecture in the key you can't identify the architecture from the information you have, but Tags are supposed to be treated as opaque with the exception of those that were released prior to 3.5 (also not explicit, so I'll fix that).
The Company part of the key is intended to group related environments and to ensure that Tags are namespaced appropriately. The key name should be alphanumeric without spaces and likely to be unique. For example, a trademarked name, a UUID, or a hostname would be appropriate::
HKEY_CURRENT_USER\Software\Python\ExampleCorp HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60 HKEY_CURRENT_USER\Software\Python\www.example.com
I'd suggest adding "Human-readable Company values are preferred". UUIDs seem like a horrible idea in practice.
Maybe, but name-squatting seems equally bad (and I've seen enough collaborative registration systems like this to see the value in guaranteed uniqueness). I'll recommend trademarked names though.
It's also worth noting that "Display Name" isn't actually as useful as it sounds, in practice. A tool that relies on it would report the python.org installers as being provided by "PythonCore", which isn't particularly user friendly. Maybe we need something in the "Backward Compatibility" section going into a bit more detail as to how tools should deal with that, and maybe we need to add a "DisplayName" in 3.6+.
I'll specify defaults for PythonCore on each of them and add them to 3.6 (once the PEP is approved and we agree on the values). It certainly doesn't harm the usefulness, but we do want to make sure that tools are handling old PythonCore entries in a consistent way.
If a string value named ``DisplayName`` exists, it should be used to identify the environment to users. Otherwise, the name of the key should be used.
To an extent there's the same comment here as for DisplayName for Company - it needs to be defined with consideration for how it will be used. This is, of course, more of a "quality of implementation" matter than a standards one. But the PEP might benefit from an example of use, maybe showing the output from a hypothetical command line tool that lists all installations on the machine.
It's defined as being used to "identify ... to users". Equally, the SupportUrl "may be displayed or otherwise used to direct users to [support]". I feel that these are strong enough definitions, and that showing an hypothetical command line tool output might be seen as too prescriptive (or alternatively, swinging the pendulum too far away from GUI tools).
If a string value named ``Version`` exists, it should be used to identify the version of the environment. This is independent from the version of Python implemented by the environment.
If a string value named ``SysVersion`` exists, it must be in ``x.y`` or ``x.y.z`` format matching the version returned by ``sys.version_info`` in the interpreter. Otherwise, if the Tag matches this format it is used. If not, the Python version is unknown.
I'm not too happy with this. [...]
Note that each of these values is recommended, but optional.
SysVersion and SysArchitecture (or a Tag that works as a fallback) should be mandatory. Otherwise I'm OK with this statement.
Snipped most of the details because I agree it's unsatisfying right now, but I disagree with enough of the counterproposal that it was getting to be messy commenting on each bit. Basically, I added SysArchitecture (to match platform.architecture()[0], typically '32bit' or '64bit' but extensible without having to define all potential values in the PEP) with a note that for PythonCore it should be inferred from the registry path. SysVersion no longer allows inferring it from the Tag, except for PythonCore and only when SysVersion is missing. I'm very keen to not force any of this information to be required as it is very difficult to know how to deal with interpreters that don't include it. Does virtualenv refuse to list/use it, even if the install path is valid, just because SysArchitecture was omitted? What if the registry becomes corrupt - should Visual Studio refuse to show what information it can obtain? I already stated that all information is recommended. The change I've made now is that tools shouldn't work too hard to guess - if SysVersion is missing, they can simply say they don't know what version the interpreter is. If knowing the version is critically important, they can refuse to use it, but for many applications this is not going to be the case. And Python 3.6 will specify all of the keys. Adding it to Python 3.5 is only likely to cause issues now with people who test against 3.5.3 and not 3.5.2, which didn't have the keys.
Beneath the environment key, an ``InstallPath`` key must be created. This key is always named ``InstallPath``, and the default value must match ``sys.prefix``::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath (Default) = "C:\ExampleCorpPy36"
If a string value named ``ExecutablePath`` exists, it must be a path to the ``python.exe`` (or equivalent) executable. Otherwise, the interpreter executable is assumed to be called ``python.exe`` and exist in the directory referenced by the default value.
If a string value named ``WindowedExecutablePath`` exists, it must be a path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed interpreter executable is assumed to be called ``pythonw.exe`` and exist in the directory referenced by the default value.
These two items assume implicitly that a Python installation must provide python.exe and pythonw.exe. I'm inclined to make this explicit. Specifically, I think it's crucial that tools can read the (console or windowed) executable path as described here, and run that executable with standard Python command line arguments, and expect it to work. Otherwise there's little point in the installation registering its existence.
Again, there's a backwards compatibility argument here in that Python 3.4 and earlier did not create full paths to the executables. But that can be called out separately. When you say "assume implicitly ... python.exe and pythonw.exe", do you mean executables by those names (counterexample - IronPython includes ipy.exe and ipyw.exe, which you either know specially or would discover from these keys)? Or BOTH executables (e.g. python.exe AND pythonw.exe)? Or executables with equivalent behaviour? I'd argue that the whole PEP only applies to Python interpreters, and if you don't support standard command line arguments you aren't really a Python interpreter and shouldn't be registering as one. But I hesitate to try and define a hard rule that captures all the possible nuances here - I'd rather deal with it by having users file bugs against offending interpreters for not working correctly.
I can see an argument for a distribution providing just python.exe and omitting pythonw.exe (or even the other way around). But I can't see how I could write generic code to work with such a distribution. So let's disallow that possibility until someone comes up with a concrete use case
I think in this case, you'd either specify both keys with the same path (so tools that want the windowed executable are going to get a console window) or omit the key and make sure you don't have a "pythonw.exe" in your install directory (which seems unlikely :) ). These keys are mainly about the possibility of renaming the executables, as shown in the example. But I've added a note reminding tools developers that the executable may not exist, and attempting to launch it may fail (i.e. business as usual).
Other Keys ----------
Some other registry keys are used for defining or inferring search paths under certain conditions. A third-party installation is permitted to define these keys under their Company-Tag key, however, the interpreter must be modified and rebuilt in order to read these values. Alternatively, the interpreter may be modified to not use any registry keys for determining search paths. Making such changes is a decision for the third party; this PEP makes no recommendation either way.
I think we need to be clearer here. ...
Great suggestion. I've revised this section. (I have vague plans to make the PythonPath subkey redundant in more cases for 3.6, and I *think* we can probably drop the Modules key completely, but I'm not entirely sure it's a good idea. Still thinking about it :) ) Cheers, Steve