On 15 July 2016 at 23:20, Steve Dower <steve.dower@python.org> wrote:
Hi all
I'd like to get this PEP approved (status changed to Active, IIUC).
Some comments below.
So far (to my knowledge), Anaconda is writing out the new metadata and Visual Studio is reading it. Any changes to the schema now will require somewhat public review anyway, so I don't see any harm in approving the PEP right now.
To reiterate, this doesn't require changing anything about CPython at all and has no backwards compatibility impact on official releases (but hopefully it will stop alternative distros from overwriting our essential metadata and causing problems).
Certainly there's nothing that impacts existing releases. I've noted an issue around sys.winver below, that as an absolute minimum needs a clarification in the 3.6 docs (the documented behaviour of sys.winver isn't explicit enough to provide the uniqueness guarantees this PEP needs) and may in fact need a code change or a PEP change if sys.winver doesn't actually distinguish between 32-bit and 64-bit builds (I've not been able to confirm that either way, unfortunately). [...]
Motivation ==========
When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. This allows tools such as installers or IDEs to automatically detect and display a user's Python installations.
The PEP seems quite strongly focused on GUI tools, where the normal mode of operation would be to present the user with a list of "available installations" (with extra data where possible, not just a bare list of names) and ask for a selection. I'd like to see console tools considered as well. Basically, I'd like to avoid tool developers reading this section and thinking "it only applies to GUI tools or OS integration, not to me". For example, virtualenv introspects the available Python installations - see https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L86 - to support the "-p <interpreter>" flag. To handle this well, it would be useful to allow distributions to register a "short tag", so that as well as "-p 3.5" or "-p 2", Virtualenv could support (say) "-p conda3.4" or "-p pypy2". (The short tag should be at the Company level, so "conda" or "pypy", and the version gets added to that). Another place where this might be useful is the py.exe launcher (it's not in scope for this PEP, but having the data needed to allow the launcher to invoke any available installation could be useful for future enhancements). Another key motivation for me would be to define clearly what information tools can rely on being able to get from the available registry entries describing what's installed. Whenever I've needed to scan the registry, the things I've needed to find out are where I find the Python interpreter, what Python version it is, and whether it's 32-bit or 64-bit. The first so that I can run Python, and the latter two so that I can tell if this is a version I support *without* needing to run the interpreter. For me, everything else in this PEP is about UI, but those 3 items plus the "short tag" idea are more about what capabilities I can provide. [...]
On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key that 32-bit processes transparently read and write to rather than accessing the ``Software`` key directly.
It might be worth being more explicit here that 32-bit and 64-bit processes see the registry keys slightly differently. More on this below.
Backwards Compatibility -----------------------
Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in ``sys.winver``. As a result, it is possible to have valid side-by-side installations of both 32-bit and 64-bit interpreters.
(As Nick pointed out, "it is not possible to have valid...". I'd also add "under the rules described above"). Also, Python 3.5 doesn't appear to include the architecture in sys.winver either.
py Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.winver '3.5'
(Unless it adds -32 for 32-bit, and reserves the bare version for 64-bit. I've skimmed the CPython source but can't confirm that). The documentation of sys.winver makes no mention of whether it distinguishes 32- and 64-bit builds. In fact, it states "The value is normally the first three characters of version". If we're relying on sys.winver being unique by version/architecture, the docs need to say so (so that future changes don't accidentally violate that).
To ensure backwards compatibility, applications should treat environments listed under the following two registry keys as distinct, even when the Tag matches::
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
Environments listed under ``HKEY_CURRENT_USER`` may be treated as distinct from both of the above keys, potentially resulting in three environments discovered using the same Tag. Alternatively, a tool may determine whether the per-user environment is 64-bit or 32-bit and give it priority over the per-machine environment, resulting in a maximum of two discovered environments.
It is not possible to detect side-by-side installations of both 64-bit and 32-bit versions of Python prior to 3.5 when they have been installed for the current user. Python 3.5 and later always uses different Tags for 64-bit and 32-bit versions.
From what I can see, this latter isn't true. I presume that 64-bit uses no suffix, but 32-bit uses a "-32" suffix? This should probably be made explicit. At a minimum, if I were writing a tool to list all installed Python versions, with only what I have available to go on (the PEP and a 64-bit Python 3.5) I wouldn't be able to write correct code, as I don't have all the information I need.
Also, if we expect to be able to distinguish 32 and 64 bit implementations in this way, that's putting a new restriction on sys.winver, that it returns a different value for 32-bit and 64-bit builds. If that's the case, I'd rather see that explicitly documented, both here and in the sys.winver documentation. I'd actually prefer a more explicit mechanism going forward, but as this is a "backward compatibility" section I'll save that for later.
Environments registered under other Company names must use distinct Tags to support side-by-side installations. Tools consuming these registrations are not required to disambiguate tags other than by preferring the user's setting.
Clarification needed here? "Environments registered under other Company names have no backward compatibility requirements, and thus each distinct environment must use a distinct Tag, to support side-by-side installations."
Company -------
The Company part of the key is intended to group related environments and to ensure that Tags are namespaced appropriately. The key name should be alphanumeric without spaces and likely to be unique. For example, a trademarked name, a UUID, or a hostname would be appropriate::
HKEY_CURRENT_USER\Software\Python\ExampleCorp HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60 HKEY_CURRENT_USER\Software\Python\www.example.com
I'd suggest adding "Human-readable Company values are preferred". UUIDs seem like a horrible idea in practice.
If a string value named ``DisplayName`` exists, it should be used to identify the environment category to users. Otherwise, the name of the key should be used.
If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment.
The next few sections are talking about what data gets included in the registry. Much of this is optional, which is perfectly OK, but there are some defaulting rules here as well. I think we should clearly note those data items that tools which read the data can rely on having available. For example, the "Display Name" can always be obtained, either directly or from the Company key. But the support URL may or may not exist. This is important IMO, as it provides a guide for tool writers over what details they are entitled to assume they know about a distribution. This becomes more important later, when the technical information starts appearing. It's also worth noting that "Display Name" isn't actually as useful as it sounds, in practice. A tool that relies on it would report the python.org installers as being provided by "PythonCore", which isn't particularly user friendly. Maybe we need something in the "Backward Compatibility" section going into a bit more detail as to how tools should deal with that, and maybe we need to add a "DisplayName" in 3.6+.
The Tag part of the key is intended to uniquely identify an environment within those provided by a single company. The key name should be alphanumeric without spaces and stable across installations. For example, the Python language version, a UUID or a partial/complete hash would be appropriate; an integer counter that increases for each new environment may not::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6 HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66
Again, I'd add a recommendation that human readable Tag values be used whenever possible.
If a string value named ``DisplayName`` exists, it should be used to identify the environment to users. Otherwise, the name of the key should be used.
To an extent there's the same comment here as for DisplayName for Company - it needs to be defined with consideration for how it will be used. This is, of course, more of a "quality of implementation" matter than a standards one. But the PEP might benefit from an example of use, maybe showing the output from a hypothetical command line tool that lists all installations on the machine.
If a string value named ``Version`` exists, it should be used to identify the version of the environment. This is independent from the version of Python implemented by the environment.
If a string value named ``SysVersion`` exists, it must be in ``x.y`` or ``x.y.z`` format matching the version returned by ``sys.version_info`` in the interpreter. Otherwise, if the Tag matches this format it is used. If not, the Python version is unknown.
I'm not too happy with this. What's the benefit of allowing an installation to *not* provide the Python version? Instead, I'd prefer to say: 1. All installations must provide the Python version. They are free to use x.y or x.y.z. format (i.e., the micro version is optional - although again what's the benefit? Why not mandate x.y for consistency?). The rule given in SysVersion is fine (without the final sentence). 2. If CPython *does*, as I'm assuming, use 3.5-32, then that's an issue, because CPython doesn't follow the PEP. Maybe we should allow the Tag to be version-architecture. 3. Following on from (2) we should include a string value SysArchitecture for the architecture (32 or 64) as well. Again, this should always be available from the value or the Tag. The reason I think that the interpreter version and architecture should be mandatory is because otherwise a tool that (for example) only supports Python 3.4 or greater, or only 64-bit, has no way to exclude unsupported installations. So in summary: SysVersion = x.y SysArchitecture = 32 or 64 If SysArchitecture is missing, Tag must end in -32 or -64, and the part after the "-" is the architecture. If SysVersion is missing, Tag must be x.y or x.y-NN and the version is x.y. For backward compatibility, if Company is "PythonCore", SysArchitecture is missing, and Tag doesn't end in -NN, then SysArchitecture is 32 if the registry key is under Wow6432Node. Otherwise, it's 64 if we're a 64-bit process and 32 if we're a 32-bit process. This final heuristic could be wrong, though, and code that cannot cope with getting the wrong value (for example, it's planning on loading the Python DLL into its address space) MUST take other measures to check, or ignore any ambiguous entries. (I'm open to the above being corrected - I didn't check any references when writing it down). BTW, is there any reason why the python.org installers couldn't be modified to provide *all* the information suggested in this PEP, rather than just sticking with what we've traditionally provided? It would be a good example of how to register yourself "properly", as well as avoiding the sort of ambiguity we see above.
Note that each of these values is recommended, but optional.
SysVersion and SysArchitecture (or a Tag that works as a fallback) should be mandatory. Otherwise I'm OK with this statement.
Beneath the environment key, an ``InstallPath`` key must be created. This key is always named ``InstallPath``, and the default value must match ``sys.prefix``::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath (Default) = "C:\ExampleCorpPy36"
If a string value named ``ExecutablePath`` exists, it must be a path to the ``python.exe`` (or equivalent) executable. Otherwise, the interpreter executable is assumed to be called ``python.exe`` and exist in the directory referenced by the default value.
If a string value named ``WindowedExecutablePath`` exists, it must be a path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed interpreter executable is assumed to be called ``pythonw.exe`` and exist in the directory referenced by the default value.
These two items assume implicitly that a Python installation must provide python.exe and pythonw.exe. I'm inclined to make this explicit. Specifically, I think it's crucial that tools can read the (console or windowed) executable path as described here, and run that executable with standard Python command line arguments, and expect it to work. Otherwise there's little point in the installation registering its existence. I can see an argument for a distribution providing just python.exe and omitting pythonw.exe (or even the other way around). But I can't see how I could write generic code to work with such a distribution. So let's disallow that possibility until someone comes up with a concrete use case [...]
Other Keys ----------
Some other registry keys are used for defining or inferring search paths under certain conditions. A third-party installation is permitted to define these keys under their Company-Tag key, however, the interpreter must be modified and rebuilt in order to read these values. Alternatively, the interpreter may be modified to not use any registry keys for determining search paths. Making such changes is a decision for the third party; this PEP makes no recommendation either way.
I think we need to be clearer here. First of all, it should probably clearly state that any subkey of the <Company>\<Tag> key (and any value of that key, I guess), unless explicitly documented in this PEP, is free for any use by the vendor (Although this may make later expansion of this PEP hard - do we want to worry about that?). We should also note that PythonCore has a number of such "private" keys, and tools should not assume any particular meaning for them. Secondly, I think we should be more explicit about the search path issue. Maybe something like the following (this is based on my memory of the issue, so apologies for any inaccuracy): """ The Python core has traditionally used certain other keys under the PythonCore\<Tag> key to set interpreter paths and similar. This usage is considered historical, and is retained mainly for backward compatibility[1]. Third party installations are permitted to use a similar approach under their own <Company>\<Tag> namespace, but the interpreter must be modified and rebuilt in order to read these values. Alternatively, the interpreter may be modified to not use any registry keys (not even the PythonCore ones) for determining search paths. Making such changes is a decision for the third party; this PEP makes no recommendation either way. It should be noted, however, that without modification, the Python interpreter's behaviour will be based on the values under the PythonCore namespace, not under the vendor's namespace. """ [1] Is this sentence true? IIRC, nothing new is using that feature, and older stuff that did, such as pywin32, is removing it. But I know of no actual plans to rip it out at any point. Paul