Python environment registration in the Windows Registry

I was throwing around some ideas with colleagues about how we detect Python installations on Windows from within Visual Studio, and it came up that there are many Python distros that install into different locations but write the same registry entries. (I knew about this, of course, but this time I decided to do something.) Apart from not being detected properly by all IDEs/tools/installers, non-standard distros that register themselves in the official keys may also mess with the default sys.path values. For example, at one point (possibly still true) if you installed both Canopy and Anaconda, you would break the first one because they tried to load the other's stdlib. Other implementations have different structures or do not register themselves at all, which also makes it more complicated for tools to discover them. So here is a rough proposal to standardise the registry keys that can be set on Windows in a way that (a) lets other installers besides the official ones have equal footing, (b) provides consistent search and resolution semantics for tools, and (c) includes slightly more rich metadata (such as display names and URLs). Presented in PEP-like form here, but if feedback suggests just putting it in the docs I'm okay with that too. It is fully backwards compatible with official releases of Python (at least back to 2.5, possibly further) and does not require modifications to Python or the official installer - it is purely codifying a superset of what we already do. Any and all feedback welcomed, especially from the owners of other distros, Python implementations or tools on the list. Cheers, Steve ----- PEP: ??? Title: Python environment registration in the Windows Registry Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower@python.org> Status: Draft Type: ??? Content-Type: text/x-rst Created: 02-Feb-2016 Abstract ======== When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. Unofficial installers, such as those used by distributions, typically create identical keys for the same purpose. However, these may conflict with the official installer or other distributions. This PEP defines a schema for the Python registry key to allow unofficial installers to separately register their installation, and to allow applications to detect and correctly display all Python environments on a user's machine. No implementation changes to Python are proposed with this PEP. The schema matches the registry values that have been used by the official installer since at least Python 2.5, and the resolution behaviour matches the behaviour of the official Python releases. Specification ============= We consider there to be a single collection of Python environments on a machine, where the collection may be different for each user of the machine. There are three potential registry locations where the collection may be stored based on the installation options of each environment. These are:: HKEY_CURRENT_USER\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag> On a given machine, an environment is uniquely identified by its Company-Tag pair. Keys should be searched in the order shown, and if the same Company-Tag pair appears in more than one of the above locations, only the first occurrence is offerred. Official Python releases use ``PythonCore`` for Company, and the value of ``sys.winver`` for Tag. Other registered environments may use any values for Company and Tag. Recommendations are made in the following sections. Backwards Compatibility ----------------------- Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in ``sys.winver``. As a result, it is possible to have valid side-by-side installations of both 32-bit and 64-bit interpreters. To ensure backwards compatibility, applications should treat environments listed under the following two registry keys as distinct, even if Tag matches:: HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag> Note that this does not apply to Python 3.5 and later, which uses different Tags. Environments registered under other Company names must use distinct Tags for side-by-side installations. 1. Environments in ``HKEY_CURRENT_USER`` are always preferred 2. Environments in ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` are preferred if the interpreter is known to be 32-bit Company ------- The Company part of the key is intended to group related environments and to ensure that Tags are namespaced appropriately. The key name should be alphanumeric without spaces and likely to be unique. For example, a trademarked name, a UUID, or a hostname would be appropriate:: HKEY_CURRENT_USER\Software\Python\ExampleCorp HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60 HKEY_CURRENT_USER\Software\Python\www.example.com If a string value named ``DisplayName`` exists, it should be used to identify the environment category to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp (Default) = (value not set) DisplayName = "Example Corp" SupportUrl = "http://www.example.com" Tag --- The Tag part of the key is intended to uniquely identify an environment within those provided by a single company. The key name should be alphanumeric without spaces and stable across installations. For example, the Python language version, a UUID or a partial/complete hash would be appropriate; an integer counter that increases for each new environment may not:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6 HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 If a string value named ``DisplayName`` exists, it should be used to identify the environment to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. If a string value named ``Version`` exists, it should be used to identify the version of the environment. This is independent from the version of Python implemented by the environment. If a string value named ``SysVersion`` exists, it must be in ``x.y`` or ``x.y.z`` format matching the version returned by ``sys.version_info`` in the interpreter. Otherwise, if the Tag matches this format it is used. If not, the Python version is unknown. Note that each of these values is recommended, but optional. A complete example may look like this:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 (Default) = (value not set) DisplayName = "Distro 3" SupportUrl = "http://www.example.com/distro-3" Version = "3.0.12345.0" SysVersion = "3.6.0" InstallPath ----------- Beneath the environment key, an ``InstallPath`` key must be created. This key is always named ``InstallPath``, and the default value must match ``sys.prefix``:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath (Default) = "C:\ExampleCorpPy36" If a string value named ``ExecutablePath`` exists, it must be a path to the ``python.exe`` (or equivalent) executable. Otherwise, the interpreter executable is assumed to be called ``python.exe`` and exist in the directory referenced by the default value. If a string value named ``WindowedExecutablePath`` exists, it must be a path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed interpreter executable is assumed to be called ``pythonw.exe`` and exist in the directory referenced by the default value. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\InstallPath (Default) = "C:\ExampleDistro30" ExecutablePath = "C:\ExampleDistro30\ex_python.exe" WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe" Help ---- Beneath the environment key, a ``Help`` key may be created. This key is always named ``Help`` if present and has no default value. Each subkey of ``Help`` specifies a documentation file, tool, or URL associated with the environment. The subkey may have any name, and the default value is a string appropriate for passing to ``os.startfile`` or equivalent. If a string value named ``DisplayName`` exists, it should be used to identify the help file to users. Otherwise, the key name should be used. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help Python\ (Default) = "C:\ExampleDistro30\python36.chm" DisplayName = "Python Documentation" Extras\ (Default) = "http://www.example.com/tutorial" DisplayName = "Example Distro Online Tutorial"

...just when I thought I have solved the registry headaches I have been dealing with... I am not saying this proposal will make the registry situation worse, but it may break my solution to the headaches Python's registry use causes with some non-standard module installers (and even the standard distutils exe installers, but that is being mitigated). In the wild exist modules with their own EXE or MSI installers that check the registry for 'the system python'. No matter how hard you hit them, they will only install to *that one python*. You can imagine the sadist that builds such an installer would not be receptive to the concept of wheels. So in order to force those modules to install to the python YOU want, dammit, you have to edit the registry. I have (contributed to) a script that just sets whatever python it was last run with to the one true system python. Works with virtualenvs too.This is not a terribly obscure script either, by the way. It is in the first reply to the google search for "how to install exes in a virtualenv". So here I am in a situation where some pythons use another registry entry. I have no idea if this makes my life as a user harder. What does this kind of meddling do against tools trying to autodetect python, and finding an ever changing value? Are we trying to guarantee that the keys used by the python.org installers or only ever actually used by standard CPython? I know for PTVS manually adding a python environment to visual studio is trivial - you fill in three locations, and its done. Just today I added a python environment to my system that was not autodetected. It took under a minute and almost no effort to add it... so for that tool this adds very little benefit. I do not know about other tools. On a very personal note (like the rest of this wasn't my personal issues with possibly making my life slightly more difficult), I would much rather see python stop touching the registry all together - but I have no strong argument for that. On 2/3/2016 00:15, Steve Dower wrote:

On 2/3/2016 12:15 AM, Alexander Walters wrote:
Me too. My opinions follow, some might call them arguments, strong or weak, some might call them foolishness. I've been setting up a collection of tools and programs on Dropbox for far-flung project team members (users, not programmers) to share. It is nearly impossible to install a typical Windows program on Dropbox, because the installation information is partly in the installed directory structure, and partly in the registry. Exporting registry info from the machine that does the install to other machines is hard because other users have different paths to Dropbox. OK, for commercial software, installing to Dropbox probably violates some of the licensing, forcing each user to buy/install, but FOSS or in-house software doesn't have such restrictions: but still, most of it wants to use the registry, and Windows almost, but doesn't quite, force it to. Portable software exists, but often is 3rd party hacks of popular FOSS rather than being directly supported by the FOSS development team. Python falls into this category. Happily, I recently found WinPython Zero, which hacks it (somehow) to work portably, and I've deployed it successfully on Dropbox. I'd rather Python were portable on its own, without hacks. Portability requires not using the registry, so I agree with Alexander there. Portability, as "Windows portable software" is generally defined, is focused on moving programs and data around on a flash drive, from one computer to another, and is focused on single-user, any (Windows) machine (with sufficient specs). That doesn't quite fit the Dropbox environment. Most portable software reintroduces the idea of storing configuration information in the program folder, which is OK for "project" configuration info, done once, by one person, but not for "personal preferences". The other thing Windows GUI lost is the concept of "current working directory", which hit me hard when I first started trying to set up project working areas on Dropbox. Many Windows programs want to run only one copy of themselves, in one Window, with one set of settings, and one "Start In" directory (which generally defaults to .... the program directory, or sometimes to "My Documents". This is why I went looking for a portable Python (and other portable things), and I finally realized that I was "fighting city hall" in trying to get an environment set up that was usable for the various teams (of users, not developers). Writeup at slashdot <http://slashdot.org/firehose.pl?op=view&id=80569101> for more details on the lack of a "current working directory" concept in Windows GUI programs. The path to Dropbox folders is different for everyone, even the drive letter can be different. So here are my preferences for CPython: 1) It would be best CPython itself were fully portable. That wherever you point the installer, it lives there, and if somehow you happen to execute it from that directory, it would use its invocation path as the basis for finding the rest of the installed pieces. 2) A script could be provided that sets the association for .py and the corresponding ftype to point to the python that executed the script... and which has a parameter to clear that back out, too. This would be to allow users to set a particular python for convenient script execution, since Windows does the association thing, instead of using #! lines. 3) A script could be provided (maybe the one Alexander referred to) that sets the registry so that the "apparently one true System Python" points to the python that executed the script... and which has a parameter to clear that back out, too. This would be for compatible transition to the new registry-free Python versions for packages that have weird installers (such as Alexander alluded to). But with the registry-free Python available, those packages would hopefully see the future is registry free, and avoid requiring registry data to do installs. 4) A script could be provided to add the python that executed the script to the PATH, with an option to remove it. 5) A script could be provided to add the python that executed the script to the launcher .ini file, with an option to remove it. 6) A script could be provided to add the python that executed it to the Start menu, and/or Desktop icons, with an option to remove them. Maybe scripts 2-6 are all the same one, with different options (or different invocation shortcuts for the clicky folk). Not everyone would probably need all the scripts, but none of them would be particularly large, such that they would be burdensome for others to ignore. Such a collection of scripts would allow folks to achieve various levels of integration with Windows "conveniences", without requiring it. The portability would allow Python to be installed on Dropbox or a network share, and used from there without requiring every team member to do all the individual installs of the packages needed for a project.

On 3 February 2016 at 09:50, Glenn Linderman <v+python@g.nevcal.com> wrote:
I also would prefer that Python not use/need the registry. As far as I know, in practical terms Python works just fine without the registry entries being present, but I'd like it if that were formally supported as an option for users. Obviously it means that tools that want to see what Python versions are installed won't see such unregistered copies, I'm fine with that (and I don't have a problem with leaving it up to said tools as to whether they want to include ways to support using a Python installation that's not in the registry). Some issues with this proposal: 1. I don't like the way it states that Python distributions "must" set keys. I'd rather that it were explicitly stated that a distribution which sets no registry keys is entirely valid (with the obvious implication that tools which scan the registry looking for installed Python distributions won't be able to see such an installation). 2. It's not clear to me how alternative distributions will specify their registry keys (or the fact that they don't use them). The registry keys are built into the Python interpreter - this proposal seems to imply that distributions that simply repackage the python.org build will no longer be valid or supported, and instead anyone who wants to produce a custom Python distribution (even private in-house repackagings) will need to change the Python source and rebuild. That's a pretty major change, and if that *is* the intent, then as a minimum I'd say we need to provide compiler flags to let redistributors specify their Company/Tag values (or say that they want to omit registry use). And I'm still not happy that "repackage the python.org binaries" has been removed as an option. It's possible that the reason the above two points have been missed is because the proposal focuses purely on "informational" registry data. But Python also modifies sys.path based on the registry entries - and possibly has other behavioural changes as well. The pywin32 package, in particular, makes use of this (it's a feature of pywin32 that I disagree with and I wish it didn't do that, but it does, and it's a very widely used package on Windows). So ignoring this aspect of Python's behaviour is a big problem. (Also, what changes will pywin32 need to make to correctly support being installed into non-python.org distributions when this proposal is implemented?) Paul

On 03Feb2016 0321, Paul Moore wrote:
Good point, I never meant to imply that. If you don't want your Python install/env found then you don't have to register anything. (Of course, when users come to us IDE developers and say "your tool can't find Python XYZ", we'll all just go to Python XYZ and say "your users need you to register when you install, see this PEP" :) )
There's only one place where the registry key is used within the interpreter itself, which is PC/getpathp.c. Essentially the process is this: sys.path = [] sys.path.append('') sys.path.extend(os.getenv('PYTHONPATH').split(';')) sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**')) sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**')) home = os.getenv('PYTHONHOME') if not home: if os.path.exists(os.path.join(sys.argv[0], '..', 'Lib', 'os.py')): home = os.path.dirname(sys.argv[0]) if not home: paths = read_value(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath') if not paths: paths = read_value(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath') if paths: sys.path.extend(paths.split(';')) else: sys.path.append(r'.\Lib') # more well-known subdirs else: sys.path.append(os.path.join(home, 'Lib')) # more well-known subdirs ... So a few high-level observations: * any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath' * any environment lacking 'Lib\os.py' (e.g. venv) relies on the registry to locate enough stdlib to import site * this is too complicated, but guaranteed we will break users in production if we change it now So if repackagers follow a few rules (that I documented in https://docs.python.org/3.5/using/windows.html - I see the process above is also documented there, which I wish I remembered before writing all that out), they'll be fine. Unfortunately, following those rules means that you don't register anywhere that separate tools can find you, and so users complain and you "fix" it by doing the wrong thing. This PEP offers a right way to fix it.
I haven't looked into pywin32's use of this recently - I tend to only use Christoph Gohlke's wheels that don't register anything. But it is certainly a valid concern. Hopefully Mark Hammond is watching :) Cheers, Steve
Paul

On 3 February 2016 at 16:46, Steve Dower <steve.dower@python.org> wrote:
So a few high-level observations:
* any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath'
Yeah, that's horrid but not really something we can change without compatibility breakage. But it's essentially opt-in - if you don't need the feature you don't have to register anything under that key.
Thanks for the explanation. And for documenting it (even if I looked for the documentation, failed to find it and then whined about it not being documented - my apologies!) More specifically for the people wanting "portable" Python systems, if I read that right then if python.exe is alongside a Lib directory containing os.py, Python needs no environment variables, and no registry entries, to run perfectly. I don't see any issue with Python builds that don't register themselves not showing up in tools that look for Python installations. And as you say, if we give people who make "official" distributions a way to properly register, then that helps them and leaves the unregistered case for "homebrew" portable copies of Python. (I've just seen your other note about it being "potentially problematic". OK, let's leave it low-key for now, but when we are comfortable with it, can we publicise it then? I get a definite impression that quite a lot of people assume that "you can't have a portable build of Python").
This PEP offers a right way to fix it.
Thanks for the explanation, and I now agree that's what the PEP is doing. So +0.5 from me for this PEP (Only 0.5, because I still have some concerns that talking about registry entries in such detail gives the impression that Python is tied to them more than it actually is. If you can see a way of toning down the wording, then great, but better to document the proposal accurately than to water it down because people might get a mistaken impression).
Yeah, I've not checked if pywin32 still does this, it's a long time since I really used it. Like you, I go for wheels only these days. Paul

On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower <steve.dower@python.org> wrote:
sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**')) sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
It seems like a bug (in spirit at least) that this step isn't skipped for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).
I haven't looked into pywin32's use of this recently - I tend to only use Christoph Gohlke's wheels that don't register anything.
I install the pypiwin32 wheel using pip, which uses pypiwin32.pth: # .pth file for the PyWin32 extensions win32 win32\lib Pythonwin import os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pypiwin32_system32")) This is different from a PythonPath subkey in a couple of respects. The paths listed in .pth files are appended to sys.path instead of prepended. They also don't get added when run with -S or for a venv environment that excludes site-packages.

On 03Feb2016 0923, eryk sun wrote:
They should be skipped. If not, I'm very much in favour of fixing that immediately in all active branches.
Yeah, there are serious problems with doing these kinds of hacks in .pth files. However, this is not directly affected by the registry, so thankfully not a concern right now. Cheers, Steve

On 03Feb2016 0150, Glenn Linderman wrote:
Python has been registry independent for a while if you set the PYTHONHOME environment variable (and independent but potentially unreliable even without this). Most of the registry settings created on install are for supporting upgrades, repairs and uninstallation, none of which matter in your case. (Also many python-dev readers' cases, but there are a lot of users who get into trouble very quickly without this level of management.)
Agreed, and it already basically does this as mentioned above.
You probably want the py.exe launcher here (though that relies on Python being registered...), as it handles shebangs - even /usr/bin/env style. A script such as what you're asking for would be possible, but not something I want to be responsible for providing and maintaining.
4-6 are basically the definition of an installer, and other installers are also able to do this.
Perhaps you really want a script to run the installer and then pip? Final point I want to reiterate - Python itself is essentially registry free already in that it does not need registry settings to function. The current problems are: 1. other programs need to locate all available Pythons 2. there appears to only be one space to register your Python 3. this space is *sometimes* used by Python to locate itself and installed packages I want to fix problem 2 via documentation, and then look at the much more difficult problem 3. Cheers, Steve

On 3 February 2016 at 16:29, Steve Dower <steve.dower@python.org> wrote:
Final point I want to reiterate - Python itself is essentially registry free already in that it does not need registry settings to function.
That's something we should probably publicise better. People seem unaware of it (in much the same way that they never really noticed zip application support). Maybe we could include a section in the Python 3.6 "What's new" (even though it's not technically new - but I did a quick check of What's New back to 3.2 and couldn't see any mention)? Paul

On 03Feb2016 0839, Paul Moore wrote:
Maybe, but since it is still potentially problematic I'd rather not right now. Basically, I don't want to have to support people whose "portable" version of Python works fine on one machine, but has syntax errors in the stdlib on another machine. Adding the applocal option for 3.5 (described at https://docs.python.org/3.5/using/windows.html) helps with this, but I'm guessing always running in isolated mode is not what most people really want. Until we genuinely never rely on the registry, I don't want to claim that we don't rely on the registry. Cheers, Steve --- Rest of the email is spelling out how to create the scenario above, since I assume people won't believe it :) 1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into python34.zip and copy the binaries and zip to a "portable" folder 2. Update to Python 3.4.2 on the main machine 3. Run "-m test test_grammar" with your portable 3.4.1. 4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2 stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)

On 3 February 2016 at 17:04, Steve Dower <steve.dower@python.org> wrote:
Sigh. There's nothing so small that it isn't a compatibility break :-) But of course this process violates the rule "set PYTHONHOME or have a Lib/os.py *file* alongside python.exe". Like you say the rules are subtle enough that people will make mistakes :-( Thanks for the explanation. Paul

On 03Feb2016 0015, Alexander Walters wrote:
(I agree, but maybe if you can point them to a PEP it'll help? Right now, there's nowhere to even point them to.)
I highly doubt it will break your current solution, as everyone pre- and post-update will still look in PythonCore\X.Y for Python.
Guaranteeing that only python.org Python uses PythonCore is part of it, but the other part is officially telling everyone else that they are welcome to create their own keys.
I'm also a PTVS maintainer, so I know how much magic is going on behind those three locations :) But I don't think people should need to do that by hand at all. For example, the path to an Anaconda installation is buried deep inside AppData (as is Python 3.5+ now), and varies based on your username. Canopy does the same, and once you've found it there are (or were?) at least three copies of python.exe to choose from (we worked with Enthought to make this Just Work for PTVS users).
I also agree with that, but ultimately the registry is the global configuration store on Windows, and when we need global state it is the place to go. (My actual hope for Python 3.6 is to drop the few places where Python *reads* from the registry for configuration that shouldn't be global, at which point the Python key is solely for other programs. But those fixes are probably not PEP-worthy.) Cheers, Steve

On 2/3/2016 11:17, Steve Dower wrote: privileges to edit the registry... (It wont ask to elevate unless you install for all users, and that's where all users will install). So on that point alone, this saves nothing substantive really. (I will go off on python35 installing in insane locations some other time.)

On 3 February 2016 at 19:20, Alexander Walters <tritium-list@sdamon.com> wrote:
Per-user installs go into HKCU, which doesn't require elevation, so the proposal *does* help for per-user installs. And does the all-users install really offer no option for users to choose their own install location? Not even to switch to another drive (if, for example, C: is protected)? Paul

On 03Feb2016 1120, Alexander Walters wrote:
The install location is customisable, and users can always write to their own registry hive. The same applies to Python, so you can choose to install it as conveniently or as securely as you like. In either case, other applications need a guaranteed place to find these installations, and that place is the system registry. Cheers, Steve

On Wed, Feb 3, 2016 at 3:15 AM, Alexander Walters <tritium-list@sdamon.com> wrote:
If I remember correctly, you can use `wheel convert filename.exe` on those installers which create a wheel that you can install. I think that's what I used to do with pywin32 before pypiwin32 came along. I just tested it and it still works on the pywin32 exe.

I am well aware of this. In the SO question I referenced, being the first google hit related this this... that is the answer *I* gave. It only works, in my experience, 60% of the time, and not with two biggie packages (pywin32, for which you have to go to third parties to get the wheel, which do not include all of pywin32, and wxpython) and perhaps more. On 2/3/2016 14:32, Matthew Einhorn wrote:

I've posted an updated version of this PEP that should soon be visible at https://www.python.org/dev/peps/pep-0514. Leaving aside the fact that the current implementation of Python relies on *other* information in the registry (that is not specified in this PEP), I'm still looking for feedback or concerns from developers who are likely to create or use the keys that are described here. ---------------- PEP: 514 Title: Python registration in the Windows registry Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower@python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Feb-2016 Post-History: 02-Feb-2016 Abstract ======== This PEP defines a schema for the Python registry key to allow third-party installers to register their installation, and to allow applications to detect and correctly display all Python environments on a user's machine. No implementation changes to Python are proposed with this PEP. Python environments are not required to be registered unless they want to be automatically discoverable by external tools. The schema matches the registry values that have been used by the official installer since at least Python 2.5, and the resolution behaviour matches the behaviour of the official Python releases. Motivation ========== When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. This allows tools such as installers or IDEs to automatically detect and display a user's Python installations. Third-party installers, such as those used by distributions, typically create identical keys for the same purpose. Most tools that use the registry to detect Python installations only inspect the keys used by the official installer. As a result, third-party installations that wish to be discoverable will overwrite these values, resulting in users "losing" their Python installation. By describing a layout for registry keys that allows third-party installations to register themselves uniquely, as well as providing tool developers guidance for discovering all available Python installations, these collisions should be prevented. Definitions =========== A "registry key" is the equivalent of a file-system path into the registry. Each key may contain "subkeys" (keys nested within keys) and "values" (named and typed attributes attached to a key). ``HKEY_CURRENT_USER`` is the root of settings for the currently logged-in user, and this user can generally read and write all settings under this root. ``HKEY_LOCAL_MACHINE`` is the root of settings for all users. Generally, any user can read these settings but only administrators can modify them. It is typical for values under ``HKEY_CURRENT_USER`` to take precedence over those in ``HKEY_LOCAL_MACHINE``. On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key that 32-bit processes transparently read and write to rather than accessing the ``Software`` key directly. Structure ========= We consider there to be a single collection of Python environments on a machine, where the collection may be different for each user of the machine. There are three potential registry locations where the collection may be stored based on the installation options of each environment:: HKEY_CURRENT_USER\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag> Environments are uniquely identified by their Company-Tag pair, with two options for conflict resolution: include everything, or give priority to user preferences. Tools that include every installed environment, even where the Company-Tag pairs match, should ensure users can easily identify whether the registration was per-user or per-machine. Tools that give priority to user preferences must ignore values from ``HKEY_LOCAL_MACHINE`` when a matching Company-Tag pair exists is in ``HKEY_CURRENT_USER``. Official Python releases use ``PythonCore`` for Company, and the value of ``sys.winver`` for Tag. Other registered environments may use any values for Company and Tag. Recommendations are made in the following sections. Python environments are not required to register themselves unless they want to be automatically discoverable by external tools. Backwards Compatibility ----------------------- Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in ``sys.winver``. As a result, it is possible to have valid side-by-side installations of both 32-bit and 64-bit interpreters. To ensure backwards compatibility, applications should treat environments listed under the following two registry keys as distinct, even when the Tag matches:: HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag> Environments listed under ``HKEY_CURRENT_USER`` may be treated as distinct from both of the above keys, potentially resulting in three environments discovered using the same Tag. Alternatively, a tool may determine whether the per-user environment is 64-bit or 32-bit and give it priority over the per-machine environment, resulting in a maximum of two discovered environments. It is not possible to detect side-by-side installations of both 64-bit and 32-bit versions of Python prior to 3.5 when they have been installed for the current user. Python 3.5 and later always uses different Tags for 64-bit and 32-bit versions. Environments registered under other Company names must use distinct Tags to support side-by-side installations. There is no backwards compatibility allowance. Company ------- The Company part of the key is intended to group related environments and to ensure that Tags are namespaced appropriately. The key name should be alphanumeric without spaces and likely to be unique. For example, a trademarked name, a UUID, or a hostname would be appropriate:: HKEY_CURRENT_USER\Software\Python\ExampleCorp HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60 HKEY_CURRENT_USER\Software\Python\www.example.com The company name ``PyLauncher`` is reserved for the PEP 397 launcher (``py.exe``). It does not follow this convention and should be ignored by tools. If a string value named ``DisplayName`` exists, it should be used to identify the environment category to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp (Default) = (value not set) DisplayName = "Example Corp" SupportUrl = "http://www.example.com" Tag --- The Tag part of the key is intended to uniquely identify an environment within those provided by a single company. The key name should be alphanumeric without spaces and stable across installations. For example, the Python language version, a UUID or a partial/complete hash would be appropriate; an integer counter that increases for each new environment may not:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6 HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 If a string value named ``DisplayName`` exists, it should be used to identify the environment to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. If a string value named ``Version`` exists, it should be used to identify the version of the environment. This is independent from the version of Python implemented by the environment. If a string value named ``SysVersion`` exists, it must be in ``x.y`` or ``x.y.z`` format matching the version returned by ``sys.version_info`` in the interpreter. Otherwise, if the Tag matches this format it is used. If not, the Python version is unknown. Note that each of these values is recommended, but optional. A complete example may look like this:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 (Default) = (value not set) DisplayName = "Distro 3" SupportUrl = "http://www.example.com/distro-3" Version = "3.0.12345.0" SysVersion = "3.6.0" InstallPath ----------- Beneath the environment key, an ``InstallPath`` key must be created. This key is always named ``InstallPath``, and the default value must match ``sys.prefix``:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath (Default) = "C:\ExampleCorpPy36" If a string value named ``ExecutablePath`` exists, it must be a path to the ``python.exe`` (or equivalent) executable. Otherwise, the interpreter executable is assumed to be called ``python.exe`` and exist in the directory referenced by the default value. If a string value named ``WindowedExecutablePath`` exists, it must be a path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed interpreter executable is assumed to be called ``pythonw.exe`` and exist in the directory referenced by the default value. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\InstallPath (Default) = "C:\ExampleDistro30" ExecutablePath = "C:\ExampleDistro30\ex_python.exe" WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe" Help ---- Beneath the environment key, a ``Help`` key may be created. This key is always named ``Help`` if present and has no default value. Each subkey of ``Help`` specifies a documentation file, tool, or URL associated with the environment. The subkey may have any name, and the default value is a string appropriate for passing to ``os.startfile`` or equivalent. If a string value named ``DisplayName`` exists, it should be used to identify the help file to users. Otherwise, the key name should be used. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help Python\ (Default) = "C:\ExampleDistro30\python36.chm" DisplayName = "Python Documentation" Extras\ (Default) = "http://www.example.com/tutorial" DisplayName = "Example Distro Online Tutorial" Other Keys ---------- Some other registry keys are used for defining or inferring search paths under certain conditions. A third-party installation is permitted to define these keys under their Company-Tag key, however, the interpreter must be modified and rebuilt in order to read these values. Copyright ========= This document has been placed in the public domain.

Hi Steve, I have looked into this PEP to see what we need to do on Enthought side of things. I have a few questions: 1. Is it recommended to follow this for any python version we may provide, or just new versions (3.6 and above). Most of our customers still heavily use 2.7, and I wonder whether it would cause more trouble than it is worth backporting this to 2.7. 2. The main issue for us in practice has been `PythonPath` entry as used to build `sys.path`. I understand this is not the point of the PEP, but would it make sense to give more precise recommendations for 3rd party providers there ? IIUC, the PEP 514 would recommend for us to do the following: 1. Use HKLM for "system install" or HKCU for "user install" as the root key 2. Register under "<root>\Software\Python\Enthought" 3. We should patch our pythons to look in 2. and not in "<root>\Software\Python\PythonCore", especially for `sys.path` constructions. 4. When a python from enthought is installed, it should never register anything in the key defined in 2. Is this correct ? I am not clear about 3., especially on what should be changed. I know that for 2.7, we need to change PC\getpathp.c for sys.path, but are there any other places where the registry is used by python itself ? Thanks for working on this, David On Sat, Feb 6, 2016 at 9:01 PM, Steve Dower <python@stevedower.id.au> wrote:

On 1 March 2016 at 11:37, David Cournapeau <cournape@gmail.com> wrote:
My understanding from the earlier discussion was that you should not patch Python at all. The sys.path building via PythonPath is not covered by the PEP and you should continue as at present. The new keys are all for informational purposes - your installer should write to them, and read them if looking for your installations. But the Python interpreter itself should not know or care about your new keys. Steve can probably clarify better than I can, but that's how I recall it being intended to work. Paul

On 01Mar2016 0524, Paul Moore wrote:
Yes, the intention was to not move sys.path building out of the PythonCore key. It's solely about discovery by external tools. If you want to patch your own distribution to move the paths you are welcome to do that - there is only one string literal in getpathp.c that needs to be updated - but it's not a requirement and I deliberately avoided making a recommendation either way. (Though as discussed earlier in the thread, I'm very much in favour of deprecating and removing any use of the registry by the runtime itself in 3.6+, but still working out the implications of that.) Cheers, Steve

On Tue, Mar 1, 2016 at 5:46 PM, Steve Dower <steve.dower@python.org> wrote:
Right. For us, continuing populating sys.path from the registry "owned" by python.org official installers is more and more untenable, because every distribution writes there, and this is especially problematic when you have both 32 bits and 64 bits distributions in the same machine.
Great, I just wanted to make sure removing it ourselves do not put us in a corner or further away from where python itself is going. Would it make sense to indicate in the PEP that doing so is allowed (neither recommended or frowned upon) ? David

Hello all, 1) There are many lists to choose from - if this is the wrong one for questions about packaging - please forgive me, and point me in the right direction. 2) Normally, I have just packaged python, and then moved on. However, recently I have been asked to help with packaging an 'easier to install' python by people using cloud-init, and more recently people wanting to use salt-stack (on AIX). FYI: I have been posting about my complete failure to build 2.7.11 ( http://bugs.python.org/issue26466) - so, what I am testing is based on 2.7.10 - which built easily for me. Going through the 'base documentation' I saw a reference to both sys.argv and sys.path. atm, I am looking for an easy way to get the program name (e.g., /opt/bin/python, versus ./python). I have my reasons (basically, looking for a compiled-in library search path to help with http://bugs.python.org/issue26439) Looking on two platforms (AIX, my build, and debian for power) I am surprised that sys.argv is empty in both cases, and sys.path returns /opt/lib/python27.zip with AIX, but not with debian. root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python Python 2.7.10 (default, Nov 3 2015, 14:36:51) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information.
michael@ipv4:~$ python Python 2.7.9 (default, Mar 1 2015, 13:01:00) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
And I guess I would be interested in getting '/opt/lib/python2.7/dist-packages' in there as well (or learn a way to later add it for pre-compiled packages such as cloud-init AND that those would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init for modules added to support cloud-init - should I so choose (mainly in case of compatibility issues between say cloud-init and salt-stack that have common modules BUT may have conflicts) - Hopefully never needed for that reason, but it might also simplify packaging applications that depend on python. Many thanks for your time and pointers into the documentation, It is a bit daunting :) Michael

On Wed, 2 Mar 2016 at 09:12 Michael Felt <aixtools@gmail.com> wrote:
So in this instance you're after python-list since this is a general support question. But since I have an answer for you...
https://docs.python.org/2.7/library/sys.html#sys.executable
Did you actually build your version of Python on Debian? If not then do realize that Debian patches their version of CPython, so it wouldn't shock me if they stripped out the code that adds the zip file to sys.path. IOW don't trust the pre-installed CPython to act the same as one that is built from source.

On Wed, Mar 2, 2016 at 3:50 AM, Michael Felt <aixtools@gmail.com> wrote:
It's hard to say where this belongs best, but python-list would probably have done as well.
I think the only way to get at the compiled-in search path is to recreate it based on the compiled-in prefix, which you can get through distutils. Python purposely only uses the compiled-in path as the last resort. Instead, it searches for its home relative to the executable and adds a set of directories relative to its home (if they exist). It's not clear to me why you're focusing on these differences, as (as I describe below) they are immaterial.
When you run python interactively, sys.argv[0] will be '', yes. Since you're not launching a program, there's nothing else to set it to. 'python' (or the path to the executable) wouldn't be the right thing to set it to, because python itself isn't a Python program :) The actual python executable is sys.executable, not sys.argv[0], but you shouldn't usually care about that, either. If you want to know where to install things, distutils is the thing to use. If you want to know where Python thinks it's installed (for debugging purposes only, really), sys.prefix will tell you.
In sys.path, you're seeing the difference between a vanilla Python and Debian's patched Python. Vanilla Python adds $prefix/lib/python27.zip to sys.path unconditionally, whereas Debian removes it when it doesn't exist. Likewise, the dist-packages directory is a local modification by Debian; in vanilla Python it's called 'site-packages' instead. The subdirectories in dist-packages that you see in the Debian case are added by .pth files installed in $prefix -- third-party packages, in other words, adding their own directories to the module search path.
A vanilla Python (or non-Debian-built python, even) has no business looking in dist-packages. It should just use site-packages. Third-party packages shouldn't care whether they're installed in site-packages or dist-packages, and instead should use distutils one way or another (if not by having an actual setup.py that uses distutils or setuptools, then at least by querying distutils for the installation directory the way python-config does).
-- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!

On 2016-03-02 18:45, Thomas Wouters wrote:
Many Thanks Thomas for the extensive answer. My question is to help me understand where to look for default libraries in order to work on a patch. That has come a long way, but it is stuck now with/by something else I need to learn to debug (to find where the non-zero exit status comes from during a build). As a packager I hope to be as 'vanilla' as possible from the python perspective. Another time (another list!) I shall ask about what goes into python.zip Michael

On Tue, Feb 2, 2016 at 10:15 PM, Steve Dower <steve.dower@python.org> wrote:
Just wanted to quickly point out another use of the WIndows registry in Python: WindowsRegistryFinder [1]. This is an import "meta-path" finder that locates modules declared (*not* defined) in the registry. I'm not familiar with the Windows registry nor do I know if anyone is using this finder. That said, ISTM the finder's use of the registry does not face quite the same challenges you've described in the proposal. I expect Martin von Löwis could explain more as he was involved with adding the finder. Just wanted to throw that out there, particularly if there's a chance of the finder's registry keys conflicting in some way. -eric [1] https://hg.python.org/cpython/file/5873cfb42ebe/Lib/importlib/_bootstrap_ext...

On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
The "Modules" key (WindowsRegistryFinder in 3.3+ and previously PyWin_FindRegisteredModule) adds individual modules by subkey name, with the filepath in the default value (the filename can differ, but it can't use an arbitrary extension). The "PythonPath" and "Modules" keys both date back to Mark Hammond's Windows port in the mid 1990s.

On 03Feb2016 1851, eryk sun wrote:
Yep, essentially, I expect these keys that actually affect how Python works to remain under PythonCore, and continue not to be documented or recommended for general use. But I see no reason to deprecate or remove them. Specialised situations that use these keys should continue to set them under PythonCore. I hope that is sufficiently implied by saying nothing about them in the PEP - I really don't want to have to be more explicit about it and I definitely do not want to actually name or list them in any way. Cheers, Steve

On 3 February 2016 at 15:15, Steve Dower <steve.dower@python.org> wrote:
Presented in PEP-like form here, but if feedback suggests just putting it in the docs I'm okay with that too.
We don't really have anywhere in the docs to track platform integration topics like this, so an Informational PEP is your best bet. Cheers, Nick. P.S. While I guess you *could* try to figure out a suitable home in the docs, I don't think you'd gain anything particularly useful from the additional effort -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

...just when I thought I have solved the registry headaches I have been dealing with... I am not saying this proposal will make the registry situation worse, but it may break my solution to the headaches Python's registry use causes with some non-standard module installers (and even the standard distutils exe installers, but that is being mitigated). In the wild exist modules with their own EXE or MSI installers that check the registry for 'the system python'. No matter how hard you hit them, they will only install to *that one python*. You can imagine the sadist that builds such an installer would not be receptive to the concept of wheels. So in order to force those modules to install to the python YOU want, dammit, you have to edit the registry. I have (contributed to) a script that just sets whatever python it was last run with to the one true system python. Works with virtualenvs too.This is not a terribly obscure script either, by the way. It is in the first reply to the google search for "how to install exes in a virtualenv". So here I am in a situation where some pythons use another registry entry. I have no idea if this makes my life as a user harder. What does this kind of meddling do against tools trying to autodetect python, and finding an ever changing value? Are we trying to guarantee that the keys used by the python.org installers or only ever actually used by standard CPython? I know for PTVS manually adding a python environment to visual studio is trivial - you fill in three locations, and its done. Just today I added a python environment to my system that was not autodetected. It took under a minute and almost no effort to add it... so for that tool this adds very little benefit. I do not know about other tools. On a very personal note (like the rest of this wasn't my personal issues with possibly making my life slightly more difficult), I would much rather see python stop touching the registry all together - but I have no strong argument for that. On 2/3/2016 00:15, Steve Dower wrote:

On 2/3/2016 12:15 AM, Alexander Walters wrote:
Me too. My opinions follow, some might call them arguments, strong or weak, some might call them foolishness. I've been setting up a collection of tools and programs on Dropbox for far-flung project team members (users, not programmers) to share. It is nearly impossible to install a typical Windows program on Dropbox, because the installation information is partly in the installed directory structure, and partly in the registry. Exporting registry info from the machine that does the install to other machines is hard because other users have different paths to Dropbox. OK, for commercial software, installing to Dropbox probably violates some of the licensing, forcing each user to buy/install, but FOSS or in-house software doesn't have such restrictions: but still, most of it wants to use the registry, and Windows almost, but doesn't quite, force it to. Portable software exists, but often is 3rd party hacks of popular FOSS rather than being directly supported by the FOSS development team. Python falls into this category. Happily, I recently found WinPython Zero, which hacks it (somehow) to work portably, and I've deployed it successfully on Dropbox. I'd rather Python were portable on its own, without hacks. Portability requires not using the registry, so I agree with Alexander there. Portability, as "Windows portable software" is generally defined, is focused on moving programs and data around on a flash drive, from one computer to another, and is focused on single-user, any (Windows) machine (with sufficient specs). That doesn't quite fit the Dropbox environment. Most portable software reintroduces the idea of storing configuration information in the program folder, which is OK for "project" configuration info, done once, by one person, but not for "personal preferences". The other thing Windows GUI lost is the concept of "current working directory", which hit me hard when I first started trying to set up project working areas on Dropbox. Many Windows programs want to run only one copy of themselves, in one Window, with one set of settings, and one "Start In" directory (which generally defaults to .... the program directory, or sometimes to "My Documents". This is why I went looking for a portable Python (and other portable things), and I finally realized that I was "fighting city hall" in trying to get an environment set up that was usable for the various teams (of users, not developers). Writeup at slashdot <http://slashdot.org/firehose.pl?op=view&id=80569101> for more details on the lack of a "current working directory" concept in Windows GUI programs. The path to Dropbox folders is different for everyone, even the drive letter can be different. So here are my preferences for CPython: 1) It would be best CPython itself were fully portable. That wherever you point the installer, it lives there, and if somehow you happen to execute it from that directory, it would use its invocation path as the basis for finding the rest of the installed pieces. 2) A script could be provided that sets the association for .py and the corresponding ftype to point to the python that executed the script... and which has a parameter to clear that back out, too. This would be to allow users to set a particular python for convenient script execution, since Windows does the association thing, instead of using #! lines. 3) A script could be provided (maybe the one Alexander referred to) that sets the registry so that the "apparently one true System Python" points to the python that executed the script... and which has a parameter to clear that back out, too. This would be for compatible transition to the new registry-free Python versions for packages that have weird installers (such as Alexander alluded to). But with the registry-free Python available, those packages would hopefully see the future is registry free, and avoid requiring registry data to do installs. 4) A script could be provided to add the python that executed the script to the PATH, with an option to remove it. 5) A script could be provided to add the python that executed the script to the launcher .ini file, with an option to remove it. 6) A script could be provided to add the python that executed it to the Start menu, and/or Desktop icons, with an option to remove them. Maybe scripts 2-6 are all the same one, with different options (or different invocation shortcuts for the clicky folk). Not everyone would probably need all the scripts, but none of them would be particularly large, such that they would be burdensome for others to ignore. Such a collection of scripts would allow folks to achieve various levels of integration with Windows "conveniences", without requiring it. The portability would allow Python to be installed on Dropbox or a network share, and used from there without requiring every team member to do all the individual installs of the packages needed for a project.

On 3 February 2016 at 09:50, Glenn Linderman <v+python@g.nevcal.com> wrote:
I also would prefer that Python not use/need the registry. As far as I know, in practical terms Python works just fine without the registry entries being present, but I'd like it if that were formally supported as an option for users. Obviously it means that tools that want to see what Python versions are installed won't see such unregistered copies, I'm fine with that (and I don't have a problem with leaving it up to said tools as to whether they want to include ways to support using a Python installation that's not in the registry). Some issues with this proposal: 1. I don't like the way it states that Python distributions "must" set keys. I'd rather that it were explicitly stated that a distribution which sets no registry keys is entirely valid (with the obvious implication that tools which scan the registry looking for installed Python distributions won't be able to see such an installation). 2. It's not clear to me how alternative distributions will specify their registry keys (or the fact that they don't use them). The registry keys are built into the Python interpreter - this proposal seems to imply that distributions that simply repackage the python.org build will no longer be valid or supported, and instead anyone who wants to produce a custom Python distribution (even private in-house repackagings) will need to change the Python source and rebuild. That's a pretty major change, and if that *is* the intent, then as a minimum I'd say we need to provide compiler flags to let redistributors specify their Company/Tag values (or say that they want to omit registry use). And I'm still not happy that "repackage the python.org binaries" has been removed as an option. It's possible that the reason the above two points have been missed is because the proposal focuses purely on "informational" registry data. But Python also modifies sys.path based on the registry entries - and possibly has other behavioural changes as well. The pywin32 package, in particular, makes use of this (it's a feature of pywin32 that I disagree with and I wish it didn't do that, but it does, and it's a very widely used package on Windows). So ignoring this aspect of Python's behaviour is a big problem. (Also, what changes will pywin32 need to make to correctly support being installed into non-python.org distributions when this proposal is implemented?) Paul

On 03Feb2016 0321, Paul Moore wrote:
Good point, I never meant to imply that. If you don't want your Python install/env found then you don't have to register anything. (Of course, when users come to us IDE developers and say "your tool can't find Python XYZ", we'll all just go to Python XYZ and say "your users need you to register when you install, see this PEP" :) )
There's only one place where the registry key is used within the interpreter itself, which is PC/getpathp.c. Essentially the process is this: sys.path = [] sys.path.append('') sys.path.extend(os.getenv('PYTHONPATH').split(';')) sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**')) sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**')) home = os.getenv('PYTHONHOME') if not home: if os.path.exists(os.path.join(sys.argv[0], '..', 'Lib', 'os.py')): home = os.path.dirname(sys.argv[0]) if not home: paths = read_value(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath') if not paths: paths = read_value(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath') if paths: sys.path.extend(paths.split(';')) else: sys.path.append(r'.\Lib') # more well-known subdirs else: sys.path.append(os.path.join(home, 'Lib')) # more well-known subdirs ... So a few high-level observations: * any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath' * any environment lacking 'Lib\os.py' (e.g. venv) relies on the registry to locate enough stdlib to import site * this is too complicated, but guaranteed we will break users in production if we change it now So if repackagers follow a few rules (that I documented in https://docs.python.org/3.5/using/windows.html - I see the process above is also documented there, which I wish I remembered before writing all that out), they'll be fine. Unfortunately, following those rules means that you don't register anywhere that separate tools can find you, and so users complain and you "fix" it by doing the wrong thing. This PEP offers a right way to fix it.
I haven't looked into pywin32's use of this recently - I tend to only use Christoph Gohlke's wheels that don't register anything. But it is certainly a valid concern. Hopefully Mark Hammond is watching :) Cheers, Steve
Paul

On 3 February 2016 at 16:46, Steve Dower <steve.dower@python.org> wrote:
So a few high-level observations:
* any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath'
Yeah, that's horrid but not really something we can change without compatibility breakage. But it's essentially opt-in - if you don't need the feature you don't have to register anything under that key.
Thanks for the explanation. And for documenting it (even if I looked for the documentation, failed to find it and then whined about it not being documented - my apologies!) More specifically for the people wanting "portable" Python systems, if I read that right then if python.exe is alongside a Lib directory containing os.py, Python needs no environment variables, and no registry entries, to run perfectly. I don't see any issue with Python builds that don't register themselves not showing up in tools that look for Python installations. And as you say, if we give people who make "official" distributions a way to properly register, then that helps them and leaves the unregistered case for "homebrew" portable copies of Python. (I've just seen your other note about it being "potentially problematic". OK, let's leave it low-key for now, but when we are comfortable with it, can we publicise it then? I get a definite impression that quite a lot of people assume that "you can't have a portable build of Python").
This PEP offers a right way to fix it.
Thanks for the explanation, and I now agree that's what the PEP is doing. So +0.5 from me for this PEP (Only 0.5, because I still have some concerns that talking about registry entries in such detail gives the impression that Python is tied to them more than it actually is. If you can see a way of toning down the wording, then great, but better to document the proposal accurately than to water it down because people might get a mistaken impression).
Yeah, I've not checked if pywin32 still does this, it's a long time since I really used it. Like you, I go for wheels only these days. Paul

On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower <steve.dower@python.org> wrote:
sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**')) sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
It seems like a bug (in spirit at least) that this step isn't skipped for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).
I haven't looked into pywin32's use of this recently - I tend to only use Christoph Gohlke's wheels that don't register anything.
I install the pypiwin32 wheel using pip, which uses pypiwin32.pth: # .pth file for the PyWin32 extensions win32 win32\lib Pythonwin import os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pypiwin32_system32")) This is different from a PythonPath subkey in a couple of respects. The paths listed in .pth files are appended to sys.path instead of prepended. They also don't get added when run with -S or for a venv environment that excludes site-packages.

On 03Feb2016 0923, eryk sun wrote:
They should be skipped. If not, I'm very much in favour of fixing that immediately in all active branches.
Yeah, there are serious problems with doing these kinds of hacks in .pth files. However, this is not directly affected by the registry, so thankfully not a concern right now. Cheers, Steve

On 03Feb2016 0150, Glenn Linderman wrote:
Python has been registry independent for a while if you set the PYTHONHOME environment variable (and independent but potentially unreliable even without this). Most of the registry settings created on install are for supporting upgrades, repairs and uninstallation, none of which matter in your case. (Also many python-dev readers' cases, but there are a lot of users who get into trouble very quickly without this level of management.)
Agreed, and it already basically does this as mentioned above.
You probably want the py.exe launcher here (though that relies on Python being registered...), as it handles shebangs - even /usr/bin/env style. A script such as what you're asking for would be possible, but not something I want to be responsible for providing and maintaining.
4-6 are basically the definition of an installer, and other installers are also able to do this.
Perhaps you really want a script to run the installer and then pip? Final point I want to reiterate - Python itself is essentially registry free already in that it does not need registry settings to function. The current problems are: 1. other programs need to locate all available Pythons 2. there appears to only be one space to register your Python 3. this space is *sometimes* used by Python to locate itself and installed packages I want to fix problem 2 via documentation, and then look at the much more difficult problem 3. Cheers, Steve

On 3 February 2016 at 16:29, Steve Dower <steve.dower@python.org> wrote:
Final point I want to reiterate - Python itself is essentially registry free already in that it does not need registry settings to function.
That's something we should probably publicise better. People seem unaware of it (in much the same way that they never really noticed zip application support). Maybe we could include a section in the Python 3.6 "What's new" (even though it's not technically new - but I did a quick check of What's New back to 3.2 and couldn't see any mention)? Paul

On 03Feb2016 0839, Paul Moore wrote:
Maybe, but since it is still potentially problematic I'd rather not right now. Basically, I don't want to have to support people whose "portable" version of Python works fine on one machine, but has syntax errors in the stdlib on another machine. Adding the applocal option for 3.5 (described at https://docs.python.org/3.5/using/windows.html) helps with this, but I'm guessing always running in isolated mode is not what most people really want. Until we genuinely never rely on the registry, I don't want to claim that we don't rely on the registry. Cheers, Steve --- Rest of the email is spelling out how to create the scenario above, since I assume people won't believe it :) 1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into python34.zip and copy the binaries and zip to a "portable" folder 2. Update to Python 3.4.2 on the main machine 3. Run "-m test test_grammar" with your portable 3.4.1. 4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2 stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)

On 3 February 2016 at 17:04, Steve Dower <steve.dower@python.org> wrote:
Sigh. There's nothing so small that it isn't a compatibility break :-) But of course this process violates the rule "set PYTHONHOME or have a Lib/os.py *file* alongside python.exe". Like you say the rules are subtle enough that people will make mistakes :-( Thanks for the explanation. Paul

On 03Feb2016 0015, Alexander Walters wrote:
(I agree, but maybe if you can point them to a PEP it'll help? Right now, there's nowhere to even point them to.)
I highly doubt it will break your current solution, as everyone pre- and post-update will still look in PythonCore\X.Y for Python.
Guaranteeing that only python.org Python uses PythonCore is part of it, but the other part is officially telling everyone else that they are welcome to create their own keys.
I'm also a PTVS maintainer, so I know how much magic is going on behind those three locations :) But I don't think people should need to do that by hand at all. For example, the path to an Anaconda installation is buried deep inside AppData (as is Python 3.5+ now), and varies based on your username. Canopy does the same, and once you've found it there are (or were?) at least three copies of python.exe to choose from (we worked with Enthought to make this Just Work for PTVS users).
I also agree with that, but ultimately the registry is the global configuration store on Windows, and when we need global state it is the place to go. (My actual hope for Python 3.6 is to drop the few places where Python *reads* from the registry for configuration that shouldn't be global, at which point the Python key is solely for other programs. But those fixes are probably not PEP-worthy.) Cheers, Steve

On 2/3/2016 11:17, Steve Dower wrote: privileges to edit the registry... (It wont ask to elevate unless you install for all users, and that's where all users will install). So on that point alone, this saves nothing substantive really. (I will go off on python35 installing in insane locations some other time.)

On 3 February 2016 at 19:20, Alexander Walters <tritium-list@sdamon.com> wrote:
Per-user installs go into HKCU, which doesn't require elevation, so the proposal *does* help for per-user installs. And does the all-users install really offer no option for users to choose their own install location? Not even to switch to another drive (if, for example, C: is protected)? Paul

On 03Feb2016 1120, Alexander Walters wrote:
The install location is customisable, and users can always write to their own registry hive. The same applies to Python, so you can choose to install it as conveniently or as securely as you like. In either case, other applications need a guaranteed place to find these installations, and that place is the system registry. Cheers, Steve

On Wed, Feb 3, 2016 at 3:15 AM, Alexander Walters <tritium-list@sdamon.com> wrote:
If I remember correctly, you can use `wheel convert filename.exe` on those installers which create a wheel that you can install. I think that's what I used to do with pywin32 before pypiwin32 came along. I just tested it and it still works on the pywin32 exe.

I am well aware of this. In the SO question I referenced, being the first google hit related this this... that is the answer *I* gave. It only works, in my experience, 60% of the time, and not with two biggie packages (pywin32, for which you have to go to third parties to get the wheel, which do not include all of pywin32, and wxpython) and perhaps more. On 2/3/2016 14:32, Matthew Einhorn wrote:

I've posted an updated version of this PEP that should soon be visible at https://www.python.org/dev/peps/pep-0514. Leaving aside the fact that the current implementation of Python relies on *other* information in the registry (that is not specified in this PEP), I'm still looking for feedback or concerns from developers who are likely to create or use the keys that are described here. ---------------- PEP: 514 Title: Python registration in the Windows registry Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower@python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Feb-2016 Post-History: 02-Feb-2016 Abstract ======== This PEP defines a schema for the Python registry key to allow third-party installers to register their installation, and to allow applications to detect and correctly display all Python environments on a user's machine. No implementation changes to Python are proposed with this PEP. Python environments are not required to be registered unless they want to be automatically discoverable by external tools. The schema matches the registry values that have been used by the official installer since at least Python 2.5, and the resolution behaviour matches the behaviour of the official Python releases. Motivation ========== When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. This allows tools such as installers or IDEs to automatically detect and display a user's Python installations. Third-party installers, such as those used by distributions, typically create identical keys for the same purpose. Most tools that use the registry to detect Python installations only inspect the keys used by the official installer. As a result, third-party installations that wish to be discoverable will overwrite these values, resulting in users "losing" their Python installation. By describing a layout for registry keys that allows third-party installations to register themselves uniquely, as well as providing tool developers guidance for discovering all available Python installations, these collisions should be prevented. Definitions =========== A "registry key" is the equivalent of a file-system path into the registry. Each key may contain "subkeys" (keys nested within keys) and "values" (named and typed attributes attached to a key). ``HKEY_CURRENT_USER`` is the root of settings for the currently logged-in user, and this user can generally read and write all settings under this root. ``HKEY_LOCAL_MACHINE`` is the root of settings for all users. Generally, any user can read these settings but only administrators can modify them. It is typical for values under ``HKEY_CURRENT_USER`` to take precedence over those in ``HKEY_LOCAL_MACHINE``. On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key that 32-bit processes transparently read and write to rather than accessing the ``Software`` key directly. Structure ========= We consider there to be a single collection of Python environments on a machine, where the collection may be different for each user of the machine. There are three potential registry locations where the collection may be stored based on the installation options of each environment:: HKEY_CURRENT_USER\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag> Environments are uniquely identified by their Company-Tag pair, with two options for conflict resolution: include everything, or give priority to user preferences. Tools that include every installed environment, even where the Company-Tag pairs match, should ensure users can easily identify whether the registration was per-user or per-machine. Tools that give priority to user preferences must ignore values from ``HKEY_LOCAL_MACHINE`` when a matching Company-Tag pair exists is in ``HKEY_CURRENT_USER``. Official Python releases use ``PythonCore`` for Company, and the value of ``sys.winver`` for Tag. Other registered environments may use any values for Company and Tag. Recommendations are made in the following sections. Python environments are not required to register themselves unless they want to be automatically discoverable by external tools. Backwards Compatibility ----------------------- Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in ``sys.winver``. As a result, it is possible to have valid side-by-side installations of both 32-bit and 64-bit interpreters. To ensure backwards compatibility, applications should treat environments listed under the following two registry keys as distinct, even when the Tag matches:: HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag> Environments listed under ``HKEY_CURRENT_USER`` may be treated as distinct from both of the above keys, potentially resulting in three environments discovered using the same Tag. Alternatively, a tool may determine whether the per-user environment is 64-bit or 32-bit and give it priority over the per-machine environment, resulting in a maximum of two discovered environments. It is not possible to detect side-by-side installations of both 64-bit and 32-bit versions of Python prior to 3.5 when they have been installed for the current user. Python 3.5 and later always uses different Tags for 64-bit and 32-bit versions. Environments registered under other Company names must use distinct Tags to support side-by-side installations. There is no backwards compatibility allowance. Company ------- The Company part of the key is intended to group related environments and to ensure that Tags are namespaced appropriately. The key name should be alphanumeric without spaces and likely to be unique. For example, a trademarked name, a UUID, or a hostname would be appropriate:: HKEY_CURRENT_USER\Software\Python\ExampleCorp HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60 HKEY_CURRENT_USER\Software\Python\www.example.com The company name ``PyLauncher`` is reserved for the PEP 397 launcher (``py.exe``). It does not follow this convention and should be ignored by tools. If a string value named ``DisplayName`` exists, it should be used to identify the environment category to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp (Default) = (value not set) DisplayName = "Example Corp" SupportUrl = "http://www.example.com" Tag --- The Tag part of the key is intended to uniquely identify an environment within those provided by a single company. The key name should be alphanumeric without spaces and stable across installations. For example, the Python language version, a UUID or a partial/complete hash would be appropriate; an integer counter that increases for each new environment may not:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6 HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 If a string value named ``DisplayName`` exists, it should be used to identify the environment to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. If a string value named ``Version`` exists, it should be used to identify the version of the environment. This is independent from the version of Python implemented by the environment. If a string value named ``SysVersion`` exists, it must be in ``x.y`` or ``x.y.z`` format matching the version returned by ``sys.version_info`` in the interpreter. Otherwise, if the Tag matches this format it is used. If not, the Python version is unknown. Note that each of these values is recommended, but optional. A complete example may look like this:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 (Default) = (value not set) DisplayName = "Distro 3" SupportUrl = "http://www.example.com/distro-3" Version = "3.0.12345.0" SysVersion = "3.6.0" InstallPath ----------- Beneath the environment key, an ``InstallPath`` key must be created. This key is always named ``InstallPath``, and the default value must match ``sys.prefix``:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath (Default) = "C:\ExampleCorpPy36" If a string value named ``ExecutablePath`` exists, it must be a path to the ``python.exe`` (or equivalent) executable. Otherwise, the interpreter executable is assumed to be called ``python.exe`` and exist in the directory referenced by the default value. If a string value named ``WindowedExecutablePath`` exists, it must be a path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed interpreter executable is assumed to be called ``pythonw.exe`` and exist in the directory referenced by the default value. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\InstallPath (Default) = "C:\ExampleDistro30" ExecutablePath = "C:\ExampleDistro30\ex_python.exe" WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe" Help ---- Beneath the environment key, a ``Help`` key may be created. This key is always named ``Help`` if present and has no default value. Each subkey of ``Help`` specifies a documentation file, tool, or URL associated with the environment. The subkey may have any name, and the default value is a string appropriate for passing to ``os.startfile`` or equivalent. If a string value named ``DisplayName`` exists, it should be used to identify the help file to users. Otherwise, the key name should be used. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help Python\ (Default) = "C:\ExampleDistro30\python36.chm" DisplayName = "Python Documentation" Extras\ (Default) = "http://www.example.com/tutorial" DisplayName = "Example Distro Online Tutorial" Other Keys ---------- Some other registry keys are used for defining or inferring search paths under certain conditions. A third-party installation is permitted to define these keys under their Company-Tag key, however, the interpreter must be modified and rebuilt in order to read these values. Copyright ========= This document has been placed in the public domain.

Hi Steve, I have looked into this PEP to see what we need to do on Enthought side of things. I have a few questions: 1. Is it recommended to follow this for any python version we may provide, or just new versions (3.6 and above). Most of our customers still heavily use 2.7, and I wonder whether it would cause more trouble than it is worth backporting this to 2.7. 2. The main issue for us in practice has been `PythonPath` entry as used to build `sys.path`. I understand this is not the point of the PEP, but would it make sense to give more precise recommendations for 3rd party providers there ? IIUC, the PEP 514 would recommend for us to do the following: 1. Use HKLM for "system install" or HKCU for "user install" as the root key 2. Register under "<root>\Software\Python\Enthought" 3. We should patch our pythons to look in 2. and not in "<root>\Software\Python\PythonCore", especially for `sys.path` constructions. 4. When a python from enthought is installed, it should never register anything in the key defined in 2. Is this correct ? I am not clear about 3., especially on what should be changed. I know that for 2.7, we need to change PC\getpathp.c for sys.path, but are there any other places where the registry is used by python itself ? Thanks for working on this, David On Sat, Feb 6, 2016 at 9:01 PM, Steve Dower <python@stevedower.id.au> wrote:

On 1 March 2016 at 11:37, David Cournapeau <cournape@gmail.com> wrote:
My understanding from the earlier discussion was that you should not patch Python at all. The sys.path building via PythonPath is not covered by the PEP and you should continue as at present. The new keys are all for informational purposes - your installer should write to them, and read them if looking for your installations. But the Python interpreter itself should not know or care about your new keys. Steve can probably clarify better than I can, but that's how I recall it being intended to work. Paul

On 01Mar2016 0524, Paul Moore wrote:
Yes, the intention was to not move sys.path building out of the PythonCore key. It's solely about discovery by external tools. If you want to patch your own distribution to move the paths you are welcome to do that - there is only one string literal in getpathp.c that needs to be updated - but it's not a requirement and I deliberately avoided making a recommendation either way. (Though as discussed earlier in the thread, I'm very much in favour of deprecating and removing any use of the registry by the runtime itself in 3.6+, but still working out the implications of that.) Cheers, Steve

On Tue, Mar 1, 2016 at 5:46 PM, Steve Dower <steve.dower@python.org> wrote:
Right. For us, continuing populating sys.path from the registry "owned" by python.org official installers is more and more untenable, because every distribution writes there, and this is especially problematic when you have both 32 bits and 64 bits distributions in the same machine.
Great, I just wanted to make sure removing it ourselves do not put us in a corner or further away from where python itself is going. Would it make sense to indicate in the PEP that doing so is allowed (neither recommended or frowned upon) ? David

Hello all, 1) There are many lists to choose from - if this is the wrong one for questions about packaging - please forgive me, and point me in the right direction. 2) Normally, I have just packaged python, and then moved on. However, recently I have been asked to help with packaging an 'easier to install' python by people using cloud-init, and more recently people wanting to use salt-stack (on AIX). FYI: I have been posting about my complete failure to build 2.7.11 ( http://bugs.python.org/issue26466) - so, what I am testing is based on 2.7.10 - which built easily for me. Going through the 'base documentation' I saw a reference to both sys.argv and sys.path. atm, I am looking for an easy way to get the program name (e.g., /opt/bin/python, versus ./python). I have my reasons (basically, looking for a compiled-in library search path to help with http://bugs.python.org/issue26439) Looking on two platforms (AIX, my build, and debian for power) I am surprised that sys.argv is empty in both cases, and sys.path returns /opt/lib/python27.zip with AIX, but not with debian. root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python Python 2.7.10 (default, Nov 3 2015, 14:36:51) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information.
michael@ipv4:~$ python Python 2.7.9 (default, Mar 1 2015, 13:01:00) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
And I guess I would be interested in getting '/opt/lib/python2.7/dist-packages' in there as well (or learn a way to later add it for pre-compiled packages such as cloud-init AND that those would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init for modules added to support cloud-init - should I so choose (mainly in case of compatibility issues between say cloud-init and salt-stack that have common modules BUT may have conflicts) - Hopefully never needed for that reason, but it might also simplify packaging applications that depend on python. Many thanks for your time and pointers into the documentation, It is a bit daunting :) Michael

On Wed, 2 Mar 2016 at 09:12 Michael Felt <aixtools@gmail.com> wrote:
So in this instance you're after python-list since this is a general support question. But since I have an answer for you...
https://docs.python.org/2.7/library/sys.html#sys.executable
Did you actually build your version of Python on Debian? If not then do realize that Debian patches their version of CPython, so it wouldn't shock me if they stripped out the code that adds the zip file to sys.path. IOW don't trust the pre-installed CPython to act the same as one that is built from source.

On Wed, Mar 2, 2016 at 3:50 AM, Michael Felt <aixtools@gmail.com> wrote:
It's hard to say where this belongs best, but python-list would probably have done as well.
I think the only way to get at the compiled-in search path is to recreate it based on the compiled-in prefix, which you can get through distutils. Python purposely only uses the compiled-in path as the last resort. Instead, it searches for its home relative to the executable and adds a set of directories relative to its home (if they exist). It's not clear to me why you're focusing on these differences, as (as I describe below) they are immaterial.
When you run python interactively, sys.argv[0] will be '', yes. Since you're not launching a program, there's nothing else to set it to. 'python' (or the path to the executable) wouldn't be the right thing to set it to, because python itself isn't a Python program :) The actual python executable is sys.executable, not sys.argv[0], but you shouldn't usually care about that, either. If you want to know where to install things, distutils is the thing to use. If you want to know where Python thinks it's installed (for debugging purposes only, really), sys.prefix will tell you.
In sys.path, you're seeing the difference between a vanilla Python and Debian's patched Python. Vanilla Python adds $prefix/lib/python27.zip to sys.path unconditionally, whereas Debian removes it when it doesn't exist. Likewise, the dist-packages directory is a local modification by Debian; in vanilla Python it's called 'site-packages' instead. The subdirectories in dist-packages that you see in the Debian case are added by .pth files installed in $prefix -- third-party packages, in other words, adding their own directories to the module search path.
A vanilla Python (or non-Debian-built python, even) has no business looking in dist-packages. It should just use site-packages. Third-party packages shouldn't care whether they're installed in site-packages or dist-packages, and instead should use distutils one way or another (if not by having an actual setup.py that uses distutils or setuptools, then at least by querying distutils for the installation directory the way python-config does).
-- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!

On 2016-03-02 18:45, Thomas Wouters wrote:
Many Thanks Thomas for the extensive answer. My question is to help me understand where to look for default libraries in order to work on a patch. That has come a long way, but it is stuck now with/by something else I need to learn to debug (to find where the non-zero exit status comes from during a build). As a packager I hope to be as 'vanilla' as possible from the python perspective. Another time (another list!) I shall ask about what goes into python.zip Michael

On Tue, Feb 2, 2016 at 10:15 PM, Steve Dower <steve.dower@python.org> wrote:
Just wanted to quickly point out another use of the WIndows registry in Python: WindowsRegistryFinder [1]. This is an import "meta-path" finder that locates modules declared (*not* defined) in the registry. I'm not familiar with the Windows registry nor do I know if anyone is using this finder. That said, ISTM the finder's use of the registry does not face quite the same challenges you've described in the proposal. I expect Martin von Löwis could explain more as he was involved with adding the finder. Just wanted to throw that out there, particularly if there's a chance of the finder's registry keys conflicting in some way. -eric [1] https://hg.python.org/cpython/file/5873cfb42ebe/Lib/importlib/_bootstrap_ext...

On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
The "Modules" key (WindowsRegistryFinder in 3.3+ and previously PyWin_FindRegisteredModule) adds individual modules by subkey name, with the filepath in the default value (the filename can differ, but it can't use an arbitrary extension). The "PythonPath" and "Modules" keys both date back to Mark Hammond's Windows port in the mid 1990s.

On 03Feb2016 1851, eryk sun wrote:
Yep, essentially, I expect these keys that actually affect how Python works to remain under PythonCore, and continue not to be documented or recommended for general use. But I see no reason to deprecate or remove them. Specialised situations that use these keys should continue to set them under PythonCore. I hope that is sufficiently implied by saying nothing about them in the PEP - I really don't want to have to be more explicit about it and I definitely do not want to actually name or list them in any way. Cheers, Steve

On 3 February 2016 at 15:15, Steve Dower <steve.dower@python.org> wrote:
Presented in PEP-like form here, but if feedback suggests just putting it in the docs I'm okay with that too.
We don't really have anywhere in the docs to track platform integration topics like this, so an Informational PEP is your best bet. Cheers, Nick. P.S. While I guess you *could* try to figure out a suitable home in the docs, I don't think you'd gain anything particularly useful from the additional effort -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (13)
-
Alexander Walters
-
Brett Cannon
-
David Cournapeau
-
Eric Snow
-
eryk sun
-
Glenn Linderman
-
Matthew Einhorn
-
Michael Felt
-
Nick Coghlan
-
Paul Moore
-
Steve Dower
-
Steve Dower
-
Thomas Wouters