Use 'bin' in virtual environments on Windows

Using py in the three OS-es (unix-like, Mac and Win) has become very similar in the last years, which is great. Making portable py code has become very easy. However, there is one peculiarity in Win which I find annoying: virtual environments insists on placing executables in the `Scripts` directory, while the other platforms use `bin`. This forces portable scripts and programs on the outside of py to be OS-aware in order to interact with a venv on a Win-system vs other systems. I would like to proposing the ability to configure virtual environments to use `bin` directory for executable on Windows. Personally I think it would be smart move if it were changed to `bin` as default, making all py platform act consistently. However, I do understand that there might be existing code out there that depend on using `Scripts`. Best regards, Svein

There is a *lot* of code that depends on the Windows directory being "Scripts". There have been proposals in the past to change it (sorry, no links but it has come up a few times on the various packaging forums), but it simply breaks too much for too little gain. If you're serious about wanting this, and the gain to you is sufficient to justify the work, I'd suggest reviewing as many packaging tools as you can find (virtualenv, venv, pip, setuptools, pipenv, pipx, poetry, hatch, pew, as well as core Python in the sysconfig module and distutils would be a good starting point). Work out what changes would be needed to all of these to support using "bin" on Windows, being sure to work out how to handle backward compatibility - people will still have environments that use "Scripts" for many years to come, even if we change everything, and most of the tools I mentioned need to support Python back to at least version 3.7, if not earlier. Even after doing all of that, you'd likely *still* have a huge issue to address, which is all of the various personal and in-house utilities that never get published anywhere, but which either hard code the "if Windows, then Scripts else bin" test, or simply use whichever value is appropriate for the platform they run on (there may be no need for portability anyway). A practical approach may be to develop some form of library that "hides" the difference behind some form of API for finding the correct value, get that added to the stdlib and wait a few years until it's adopted everywhere (because it's so well-designed and convenient ;-)) Then, you can change the location. But at that point no-one will care, because they don't ever reference the actual location, they just use the API anyway :-) Paul On Thu, 21 Jul 2022 at 00:53, Svein Seldal <sveinse@seldal.com> wrote:

On 2022-07-21 09:55:21, Paul Moore wrote:
I think an option when creating the "venv" (something like "--bin-name bin", or "--use-bin") is enough. Whatever creates the environment also handles the fact that "bin" is on the "right" location now. If you don't control the venv creation, you just use the default, or accept a similar option. I would vote against having a environment variable or some kind of global setting, that will break backwards compatibility.

On Thu, 21 Jul 2022 at 11:07, Simão Afonso <simao.afonso@powertools-tech.com> wrote:
How would that work? Would the value of bin-name be stored somewhere and then all tools would need to refer to that rather than just selecting based on platform like now? You'd still need to change all the tools, or your choice of directory simply wouldn't make any difference... Paul

Is this the canonical location of this information? https://github.com/python/cpython/blob/3.10/Lib/sysconfig.py#L56

On Thu, 21 Jul 2022 at 11:33, Thomas Grainger <tagrain@gmail.com> wrote:
In theory, yes. In practice, if that worked, we wouldn't get people asking about changing this in the first place... Non-Python programs can get the script location from sysconfig using py -c "import sysconfig; print(sysconfig.get_path('scripts'))" But yes, it's likely that sysconfig *is* that API - it's just that for whatever reason, people haven't adopted it well enough that we can afford to change the location without breaking things. Which makes this more of a social problem than a technical one (get people to use sysconfig and the problem goes away). Paul

On 2022-07-21 13:05, Paul Moore wrote:
I ran a grep Scripts in a freshly created venv and I do get quite many hits. Even pip is hard coded: https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils... https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py... It also creates files which seems to come from Py 3 standard lib. venv/Lib/site-packages/pip/_vendor/distlib/_backport/sysconfig.cfg And this is just considering pip and setuptools... I see now that this proposal is futile, and I'm backing down. I'd wish it thou, but we're too committed to the existing design choices and that ship has sailed. Best regards, Svein

On 2022-07-21 17:04:16, Svein Seldal wrote:
https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils...
This actually works great! If the "Scripts" folder does not exist, it uses "bin".
https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py...
This is a just a small patch anyway.

I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight. However, I’m no Windows expert, but I *think* the modern Windows file system(s?) support something like symlinks. It’s an under-the-hood feature, but maybe it’s possible to add a symlink for bin. Maybe I’m wrong, and/or it’s not possible on all file systems Python needs to support, in which case *nix systems do support linking, so we could support “Scripts” on all systems. Just a thought. -CHB On Thu, Jul 21, 2022 at 8:26 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, 21 Jul 2022 at 16:45, Christopher Barker <pythonchb@gmail.com> wrote:
I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight.
Indeed. I've no idea why we had the difference in the first place, but that's water under the bridge at this point.
Maybe. But symlinks are not available on all Windows systems (they need certain settings enabled) so we need to be able to work without them (the same is true on Unix, of course, as symlink support requires the filesystem to support it, and not all filesystems do, - but that's not relevant here). The long and short of it is that for a significant amount of time, the *best* we can hope for is something that allows tools to support *both* "Scripts" and "bin". Only after something like that has been in place for an extended period (years) will we have any chance of desupporting environments that use "Scripts".
I'd strongly encourage anyone who is still interested in pursuing this to at least work on a proof of concept set of changes, rather than just discussing how it would be better than what we're doing at the moment. This has come up enough times now that anyone involved in packaging is pretty much resigned to us not being able to change this, and we're not likely to do anything just because someone says "this is worth doing". However, it's possible we're so resigned to the current situation that we haven't thought of some approach that will work - and if so, the best way to demonstrate that will certainly be to come with working (or at least partly working) code. If people want to speculate on how a change might be implemented, then please go ahead (this is python-ideas, after all). But without someone willing to write code, that's all it's likely to be, speculation. Sorry. Paul

On 7/24/22, Barry Scott <barry@barrys-emacs.org> wrote:
Creating symlinks requires the filesystem to support NT reparse points. That's guaranteed for the system volume, which must be NTFS, but it's unreliable when development is spread across various filesystems. This is the main obstacle to relying on a "Scripts" -> "bin" link. It's not technically correct to state that creating symlinks requires administrator access. It requires SeCreateSymbolicLinkPrivilege, or no privilege at all if developer mode is enabled for the system in Windows 10+. By default this privilege is granted to just the administrators group. However, an administrator can grant it to any user or group. I prefer to grant it to the "Authenticated Users" group. If creating a directory symlink isn't allowed, and the filesystem supports reparse points, then a junction mount point can be created instead. In Unix terms, this is like using a bind mount instead of a symlink. In Windows, creating a mount point doesn't require any privilege or special access. (Registering it with the mount-point manager requires administrator access, but that's only done for volume mount points, as created by SetVolumeMountPointW.)

I'd recommend writing a virtualenv plugin that configures https://virtualenv.pypa.io/en/latest/extend.html#virtualenv.discovery.discov... Overriding the interpreter install path here: https://github.com/pypa/virtualenv/blob/6f8fb11aed7c05a535c30040ce314ae5eec1... And see what you can make work and what doesn't work

There is a *lot* of code that depends on the Windows directory being "Scripts". There have been proposals in the past to change it (sorry, no links but it has come up a few times on the various packaging forums), but it simply breaks too much for too little gain. If you're serious about wanting this, and the gain to you is sufficient to justify the work, I'd suggest reviewing as many packaging tools as you can find (virtualenv, venv, pip, setuptools, pipenv, pipx, poetry, hatch, pew, as well as core Python in the sysconfig module and distutils would be a good starting point). Work out what changes would be needed to all of these to support using "bin" on Windows, being sure to work out how to handle backward compatibility - people will still have environments that use "Scripts" for many years to come, even if we change everything, and most of the tools I mentioned need to support Python back to at least version 3.7, if not earlier. Even after doing all of that, you'd likely *still* have a huge issue to address, which is all of the various personal and in-house utilities that never get published anywhere, but which either hard code the "if Windows, then Scripts else bin" test, or simply use whichever value is appropriate for the platform they run on (there may be no need for portability anyway). A practical approach may be to develop some form of library that "hides" the difference behind some form of API for finding the correct value, get that added to the stdlib and wait a few years until it's adopted everywhere (because it's so well-designed and convenient ;-)) Then, you can change the location. But at that point no-one will care, because they don't ever reference the actual location, they just use the API anyway :-) Paul On Thu, 21 Jul 2022 at 00:53, Svein Seldal <sveinse@seldal.com> wrote:

On 2022-07-21 09:55:21, Paul Moore wrote:
I think an option when creating the "venv" (something like "--bin-name bin", or "--use-bin") is enough. Whatever creates the environment also handles the fact that "bin" is on the "right" location now. If you don't control the venv creation, you just use the default, or accept a similar option. I would vote against having a environment variable or some kind of global setting, that will break backwards compatibility.

On Thu, 21 Jul 2022 at 11:07, Simão Afonso <simao.afonso@powertools-tech.com> wrote:
How would that work? Would the value of bin-name be stored somewhere and then all tools would need to refer to that rather than just selecting based on platform like now? You'd still need to change all the tools, or your choice of directory simply wouldn't make any difference... Paul

Is this the canonical location of this information? https://github.com/python/cpython/blob/3.10/Lib/sysconfig.py#L56

On Thu, 21 Jul 2022 at 11:33, Thomas Grainger <tagrain@gmail.com> wrote:
In theory, yes. In practice, if that worked, we wouldn't get people asking about changing this in the first place... Non-Python programs can get the script location from sysconfig using py -c "import sysconfig; print(sysconfig.get_path('scripts'))" But yes, it's likely that sysconfig *is* that API - it's just that for whatever reason, people haven't adopted it well enough that we can afford to change the location without breaking things. Which makes this more of a social problem than a technical one (get people to use sysconfig and the problem goes away). Paul

On 2022-07-21 13:05, Paul Moore wrote:
I ran a grep Scripts in a freshly created venv and I do get quite many hits. Even pip is hard coded: https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils... https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py... It also creates files which seems to come from Py 3 standard lib. venv/Lib/site-packages/pip/_vendor/distlib/_backport/sysconfig.cfg And this is just considering pip and setuptools... I see now that this proposal is futile, and I'm backing down. I'd wish it thou, but we're too committed to the existing design choices and that ship has sailed. Best regards, Svein

On 2022-07-21 17:04:16, Svein Seldal wrote:
https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils...
This actually works great! If the "Scripts" folder does not exist, it uses "bin".
https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py...
This is a just a small patch anyway.

I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight. However, I’m no Windows expert, but I *think* the modern Windows file system(s?) support something like symlinks. It’s an under-the-hood feature, but maybe it’s possible to add a symlink for bin. Maybe I’m wrong, and/or it’s not possible on all file systems Python needs to support, in which case *nix systems do support linking, so we could support “Scripts” on all systems. Just a thought. -CHB On Thu, Jul 21, 2022 at 8:26 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, 21 Jul 2022 at 16:45, Christopher Barker <pythonchb@gmail.com> wrote:
I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight.
Indeed. I've no idea why we had the difference in the first place, but that's water under the bridge at this point.
Maybe. But symlinks are not available on all Windows systems (they need certain settings enabled) so we need to be able to work without them (the same is true on Unix, of course, as symlink support requires the filesystem to support it, and not all filesystems do, - but that's not relevant here). The long and short of it is that for a significant amount of time, the *best* we can hope for is something that allows tools to support *both* "Scripts" and "bin". Only after something like that has been in place for an extended period (years) will we have any chance of desupporting environments that use "Scripts".
I'd strongly encourage anyone who is still interested in pursuing this to at least work on a proof of concept set of changes, rather than just discussing how it would be better than what we're doing at the moment. This has come up enough times now that anyone involved in packaging is pretty much resigned to us not being able to change this, and we're not likely to do anything just because someone says "this is worth doing". However, it's possible we're so resigned to the current situation that we haven't thought of some approach that will work - and if so, the best way to demonstrate that will certainly be to come with working (or at least partly working) code. If people want to speculate on how a change might be implemented, then please go ahead (this is python-ideas, after all). But without someone willing to write code, that's all it's likely to be, speculation. Sorry. Paul

On 7/24/22, Barry Scott <barry@barrys-emacs.org> wrote:
Creating symlinks requires the filesystem to support NT reparse points. That's guaranteed for the system volume, which must be NTFS, but it's unreliable when development is spread across various filesystems. This is the main obstacle to relying on a "Scripts" -> "bin" link. It's not technically correct to state that creating symlinks requires administrator access. It requires SeCreateSymbolicLinkPrivilege, or no privilege at all if developer mode is enabled for the system in Windows 10+. By default this privilege is granted to just the administrators group. However, an administrator can grant it to any user or group. I prefer to grant it to the "Authenticated Users" group. If creating a directory symlink isn't allowed, and the filesystem supports reparse points, then a junction mount point can be created instead. In Unix terms, this is like using a bind mount instead of a symlink. In Windows, creating a mount point doesn't require any privilege or special access. (Registering it with the mount-point manager requires administrator access, but that's only done for volume mount points, as created by SetVolumeMountPointW.)

I'd recommend writing a virtualenv plugin that configures https://virtualenv.pypa.io/en/latest/extend.html#virtualenv.discovery.discov... Overriding the interpreter install path here: https://github.com/pypa/virtualenv/blob/6f8fb11aed7c05a535c30040ce314ae5eec1... And see what you can make work and what doesn't work
participants (7)
-
Barry Scott
-
Christopher Barker
-
Eryk Sun
-
Paul Moore
-
Simão Afonso
-
Svein Seldal
-
Thomas Grainger