First of all, an apology. I know this is probably going to be of limited interest for many on python-dev, but I'm honestly not sure where there's a better audience for the question. As it's related to how people should use the new "embeddable" distribution of Python 3.5, I'm hoping it's sufficiently relevant - but if not, feel free to direct me to a better forum. I've been trying to use the new embeddable distribution to build a Windows installation of vim that includes a Python interpreter, rather than relying on the user having a system install of Python on PATH. This strikes me as precisely the sort of usage that the embeddable distribution would be great for. And indeed, it works fine for that. But when it comes to distribution, I have an issue. The problem is that for vim to find the Python DLL, it needs to be alongside vim.exe (there's a small complication here I'll come to later, but let's ignore that for now). So I unzip the embeddable distribution into the directory containing vim.exe. However, I want that directory on PATH (so I can run vim from the command line). Now, as the embeddable distribution includes python.exe, this pollutes my PATH with an extra copy of python, which may or may not be picked up in preference to my "real" python, depending on how I install python. That's bad - I can delete python.exe and pythonw.exe from my vim installation, but I still have a load of Python DLLs on PATH that could potentially mess up other Python installations on my machine (I honestly don't know how likely that is, particularly if I distribute my vim build to other users with different setups). What I'd like to do is to put the Python stuff in a subdirectory of the Vim installation, so it's safely isolated. In the case of vim, I can do this, as vim has an option to dynamically load the Python DLL via LoadLibrary, and I can patch vim to look in <vim dir>\python before searching PATH. But the LoadLibrary stuff is horribly fragile, and I doubt anyone else embedding Pthon in their application is likely to do that - they will probably just link to python35.dll at load-time. I'm not aware of any way to adjust the loader's search path for DLLs to include a subdirectory that's not on PATH, so it seems to me that there's no good way to avoid having the embeddable distribution visible via the user's PATH. For vim, I'll probably go for the dynamic loading approach with a patch to look in a subdirectory. But have I missed any other approaches that make an embedded Python more isolated from the user's system? Paul
A good overview on this topic is given on https://github.com/numpy/numpy/wiki/windows-dll-notes. As a side note the PATH is usually the latest place in the search order of DLLs. Pre-loading python35.dll into the process space from within vim could be a possible solution. 2015-09-09 16:30 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
First of all, an apology. I know this is probably going to be of limited interest for many on python-dev, but I'm honestly not sure where there's a better audience for the question. As it's related to how people should use the new "embeddable" distribution of Python 3.5, I'm hoping it's sufficiently relevant - but if not, feel free to direct me to a better forum.
I've been trying to use the new embeddable distribution to build a Windows installation of vim that includes a Python interpreter, rather than relying on the user having a system install of Python on PATH. This strikes me as precisely the sort of usage that the embeddable distribution would be great for. And indeed, it works fine for that. But when it comes to distribution, I have an issue.
The problem is that for vim to find the Python DLL, it needs to be alongside vim.exe (there's a small complication here I'll come to later, but let's ignore that for now). So I unzip the embeddable distribution into the directory containing vim.exe.
However, I want that directory on PATH (so I can run vim from the command line). Now, as the embeddable distribution includes python.exe, this pollutes my PATH with an extra copy of python, which may or may not be picked up in preference to my "real" python, depending on how I install python. That's bad - I can delete python.exe and pythonw.exe from my vim installation, but I still have a load of Python DLLs on PATH that could potentially mess up other Python installations on my machine (I honestly don't know how likely that is, particularly if I distribute my vim build to other users with different setups).
What I'd like to do is to put the Python stuff in a subdirectory of the Vim installation, so it's safely isolated. In the case of vim, I can do this, as vim has an option to dynamically load the Python DLL via LoadLibrary, and I can patch vim to look in <vim dir>\python before searching PATH. But the LoadLibrary stuff is horribly fragile, and I doubt anyone else embedding Pthon in their application is likely to do that - they will probably just link to python35.dll at load-time.
I'm not aware of any way to adjust the loader's search path for DLLs to include a subdirectory that's not on PATH, so it seems to me that there's no good way to avoid having the embeddable distribution visible via the user's PATH.
For vim, I'll probably go for the dynamic loading approach with a patch to look in a subdirectory. But have I missed any other approaches that make an embedded Python more isolated from the user's system?
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
On 9 September 2015 at 16:11, Carl Kleffner <cmkleffner@gmail.com> wrote:
A good overview on this topic is given on https://github.com/numpy/numpy/wiki/windows-dll-notes. As a side note the PATH is usually the latest place in the search order of DLLs. Pre-loading python35.dll into the process space from within vim could be a possible solution.
Thank you for this. From a very quick read, it looks like I could put the embedded Python files in a directory ...\gvim.exe.local. I'll give that a try (there's a downside, in that I'd need a duplicate copy in ...\vim.exe.local if I wanted the console version to work too). It looks like SxS assemblies might help too, but that'll need a bit more reading before I understand it :-) Paul
On 09Sep2015 0819, Paul Moore wrote:
On 9 September 2015 at 16:11, Carl Kleffner <cmkleffner@gmail.com> wrote:
A good overview on this topic is given on https://github.com/numpy/numpy/wiki/windows-dll-notes. As a side note the PATH is usually the latest place in the search order of DLLs. Pre-loading python35.dll into the process space from within vim could be a possible solution.
Thank you for this. From a very quick read, it looks like I could put the embedded Python files in a directory ...\gvim.exe.local. I'll give that a try (there's a downside, in that I'd need a duplicate copy in ...\vim.exe.local if I wanted the console version to work too). It looks like SxS assemblies might help too, but that'll need a bit more reading before I understand it :-)
Don't bother reading into SxS assemblies. It may work, but it will destroy more brain cells than are worth wasting on it. :) Certainly putting the distro into a subdirectory is what I had expected would be common. In general, putting application directories in PATH is considered poor form, but unfortunately we don't have a better approach (there are App Paths, but those only work via the shell). Another approach you could use is making a separate directory to put on PATH that contains stub executables (or symlinks?) to launch the actual ones from a separate directory. That way you can control exactly what is available on PATH while still launching from a directory that is not on PATH. Cheers, Steve
Paul
On 9 September 2015 at 17:16, Steve Dower <steve.dower@python.org> wrote:
Don't bother reading into SxS assemblies. It may work, but it will destroy more brain cells than are worth wasting on it. :)
:-) Yeah, I looked at SxS once before and sprained my brain. But the summary on the numpy wiki looked like a digestible summary, so I may just take another look for curiosity. (That's probably what the guys who summoned Cthulhu thought... ;-))
Certainly putting the distro into a subdirectory is what I had expected would be common. In general, putting application directories in PATH is considered poor form, but unfortunately we don't have a better approach (there are App Paths, but those only work via the shell).
Yep, that's sort of the norm, and I don't *really* object to it. But I'd delete python(w).exe so they didn't hide a full distribution.
Another approach you could use is making a separate directory to put on PATH that contains stub executables (or symlinks?) to launch the actual ones from a separate directory. That way you can control exactly what is available on PATH while still launching from a directory that is not on PATH.
That's nice. The only problems with that are that I've never been comfortable with symlinks on Windows (because they need admin privs, mainly) and I prefer to avoid stubs because they mean there's an extra process - that latter's a bit of a silly objection (it works fine for py.exe and pip's executable wrappers), though. Thanks for the ideas. As I thought, it's more of a general Windows development question in the end, rather than a Python one. So sorry for the off-topic question (I blame the Python community for being so helpful ;-)) Paul
participants (3)
-
Carl Kleffner -
Paul Moore -
Steve Dower