Is anyone knowledgeable about dll deployment on windows ?
Hi,
My head hurt trying to understand dll management with windows. I wanted to find a sane way to use dll for numscons, but I can't see how to do this, so I was wondering if anyone on this ML had any deep knowledge on how to install dll, and reuse them with python extensions ? The problem is the following: - you have some dll (say MKL) in a path (say C:\program files\MKL). - how do you tell another dll (more exactly a python extension, a .pyd) to look for mkl dll into the MKL path ?
It seems that there is no rpath-like capability on windows, and the only way to load dll is to put them into some windows directory, or to put the dll path into PATH. Is it really the only way ?
cheers,
David
David Cournapeau wrote:
Hi,
My head hurt trying to understand dll management with windows. Iwanted to find a sane way to use dll for numscons, but I can't see how to do this, so I was wondering if anyone on this ML had any deep knowledge on how to install dll, and reuse them with python extensions ? The problem is the following: - you have some dll (say MKL) in a path (say C:\program files\MKL). - how do you tell another dll (more exactly a python extension, a .pyd) to look for mkl dll into the MKL path ?
Hi,
To be honest I'd be nervous about spreading your depedencies all over the disk, the potential interactions go as n^2. According to msdn there is a function which might work:
"SetDllDirectory: Adds a directory to the search path used to locate DLLs for the application." http://msdn2.microsoft.com/en-us/library/ms686203.aspx
Perhaps this is only for vista/xp? Otherwise you might have to copy the mkl dll into the "directory from which the application loaded". This would seem like a better choice, but I do wonder where that is for python.
Hope this helps. I'm certainly not an expert, but keen to try to support your work on help the masses access optimised libraries.
Best,
Jon
It seems that there is no rpath-like capability on windows, and the only way to load dll is to put them into some windows directory, or to put the dll path into PATH. Is it really the only way ?
cheers,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Jon Wright wrote:
Hi,
To be honest I'd be nervous about spreading your depedencies all over the disk, the potential interactions go as n^2.
Well, I am more than nervous, this is totally insane, and I tried different things (registry: per application path, etc...). But it just looks like windows is even more f***-up than I thought. I am tired of wasting my time for this broken platform: I am really near giving up supporting MS compilers at all.
According to msdn there is a function which might work:
"SetDllDirectory: Adds a directory to the search path used to locate DLLs for the application." http://msdn2.microsoft.com/en-us/library/ms686203.aspx
Well, this does not sound right to me either; anyway, this is a C function which does not seem to be available from python.
Otherwise you might have to copy the mkl dll into the "directory from which the application loaded". This would seem like a better choice, but I do wonder where that is for python.
This directory depends on how you start python, I think, so this is not better. Having looked at how installers do it, there does not seem to be much choice: either use PATH (actually, that's what the MKL is doing), or put everything in hardcoded directories (system32 and co). The other solution I see is to add the paths to the registry for the python application (it seems that windows has a registry subtree App Paths to handle per application dll path, but touching the registry makes me quite nervous too).
Hope this helps. I'm certainly not an expert, but keen to try to support your work on help the masses access optimised libraries.
Thanks,
David
When Visual Studio 2008 will be used, there might be a way of using the manifest files (that were created for a similar purpose). For the moment, All I know is that you must put the dll in the Windows/system32 folder or somewhere in the PATH.
Matthieu
2008/2/15, David Cournapeau david@ar.media.kyoto-u.ac.jp:
Hi,
My head hurt trying to understand dll management with windows. Iwanted to find a sane way to use dll for numscons, but I can't see how to do this, so I was wondering if anyone on this ML had any deep knowledge on how to install dll, and reuse them with python extensions ? The problem is the following: - you have some dll (say MKL) in a path (say C:\program files\MKL). - how do you tell another dll (more exactly a python extension, a .pyd) to look for mkl dll into the MKL path ?
It seems that there is no rpath-like capability on windows, and the only way to load dll is to put them into some windows directory, or to put the dll path into PATH. Is it really the only way ?
cheers,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Matthieu Brucher wrote:
When Visual Studio 2008 will be used, there might be a way of using the manifest files (that were created for a similar purpose). For the moment, All I know is that you must put the dll in the Windows/system32 folder or somewhere in the PATH.
Do you know where to find some informations about that ? I tried to find something readable, but I could only find some msdn pages with almost no concrete information on how to use those manifest files.
cheers,
David
Matthieu Brucher wrote:
When Visual Studio 2008 will be used, there might be a way of using the manifest files (that were created for a similar purpose). For the moment, All I know is that you must put the dll in the Windows/system32 folder or somewhere in the PATH.
That's not enough for some DLLs. .NET assemblies as well as Side-by-Side dlls (SxS) must be registered properly. You can install a SxS dll in PATH but simple copying the DLL isn't enough. It also depends on the OS.
Have fun! The SxS issue has bitten us in the a.. when we delivered the second beta of Python 3.0.
Christian
Christian Heimes wrote:
That's not enough for some DLLs. .NET assemblies as well as Side-by-Side dlls (SxS) must be registered properly. You can install a SxS dll in PATH but simple copying the DLL isn't enough. It also depends on the OS.
Ah, that reminds me of something I tried, that is using the global cache something, with gacutil, which is supposed to work with 'bare' dll. Of course, it did not work as said on MSDN, and I got some obscure error message (I am starting to wonder if anything coming from MS works at all).
Have fun! The SxS issue has bitten us in the a.. when we delivered the second beta of Python 3.0.
Do you have a link to the related python ML discussion by any chance ?
cheers,
David
David Cournapeau wrote:
Do you have a link to the related python ML discussion by any chance ?
No, I'm sorry. It was a private chat between between Guido, Martin and me during the release phase of Python 3.0a2.
The MSDN website has some articles about SxS DLLs though. I had to read about ten articles to get the big picture. The information is scattered all over the place. :/
Christian
On Sat, Feb 16, 2008 at 12:43 AM, Christian Heimes lists@cheimes.de wrote:
The MSDN website has some articles about SxS DLLs though. I had to read about ten articles to get the big picture. The information is scattered all over the place. :/
10 pages, xml files and not even compatible across OS versions: it it what is meant by MS when they talk about rich API ? :) I won't read those pages to understand something which is available on unix for 20 years and understandable in 5 minutes. I wasted one half day, I won't waste more than that. Well, this is it, I give up, I won't support linking dll with MS compilers. I don't have easy access to windows, and I don't even use this crap.
Thanks for your help, though
David
David,
I know this discussion first took place months ago, but I'd like to know whether or not you find a solution to the SxS assemblies issues using MSVC9.0.
In case you haven't (the binaries package for windows is built using mingw), I'd like to know if this would be possible to relocate all numpy *.pyd from their original location (i.e. site-packages/numpy/core/, etc.) to the main executable one (i.e. python.exe). AFAIK, this would eventually solve the redistributing issue we are facing with python extensions built with SxS policy. Indeed, both Microsoft.VC90.CRT.manifest and msvcr90.dll files are located next to the python.exe executable in the standard distribution (http://www.python.org/ftp/python/2.6.4/python-2.6.4.msi). This way, all numpy *.pyd extension would be able to use these files (considering that python and numpy are built using the same revision of the crt library). IIRC, the SxS look-up sequence starts with the dirname of the executable... (i.e. ./python.exe/.)
Regards, Eloi
On Fri, Nov 27, 2009 at 11:01 PM, Eloi Gaudry eg@fft.be wrote:
David,
I know this discussion first took place months ago, but I'd like to know whether or not you find a solution to the SxS assemblies issues using MSVC9.0.
Which issue exactly are you talking about ? When python is installed for one user (when installed for all users, the C runtime is installed in the SxS) ?
In case you haven't (the binaries package for windows is built using mingw), I'd like to know if this would be possible to relocate all numpy *.pyd from their original location (i.e. site-packages/numpy/core/, etc.) to the main executable one (i.e. python.exe).
This sounds like a very bad idea: if packages start doing that, there will quickly be clashes between extensions.
You would need to describe the exact scenario which is failing: how python is installed, how you build numpy, what is not working with which message, etc...
David
On 27/11/2009 18:21, David Cournapeau wrote:
On Fri, Nov 27, 2009 at 11:01 PM, Eloi Gaudryeg@fft.be wrote:
David,
I know this discussion first took place months ago, but I'd like to know whether or not you find a solution to the SxS assemblies issues using MSVC9.0.
Which issue exactly are you talking about ? When python is installed for one user (when installed for all users, the C runtime is installed in the SxS) ?
Sorry, I was thinking of the "single-user" installation one, where no C runtime can be installed to the SxS folder but rather made available next to python.exe executable.
In case you haven't (the binaries package for windows is built using mingw), I'd like to know if this would be possible to relocate all numpy *.pyd from their original location (i.e. site-packages/numpy/core/, etc.) to the main executable one (i.e. python.exe).
This sounds like a very bad idea: if packages start doing that, there will quickly be clashes between extensions.
well, the DLLs directory of the nt python install is somehow using the very same concept. Moreover this could offer a proper solution to the single-user installation C runtime "issue". By making sure every binaries are located in the same directory where the C runtime are, one doesn't need to install the C runtime libraries in the SxS folder, thus no administrative rights are needed at set-up.
You would need to describe the exact scenario which is failing: how python is installed, how you build numpy, what is not working with which message, etc...
I'm distributing a software using python as its main interpreter and taking advantage of the several python extensions available (such as numpy, matplotlib, reportlab, vtk bindings, etc.). Python and its extension binaries (*.pyd, *.dll and *.exe) are dynamically built using msvc 2008 and thus rely on the SxS assemblies at runtime. As this software need to be installed without administrative privileges, I cannot install the mandatory runtimes libraries in the shared SxS assembly directory (if missing). I then need to directly provide the runtime libraries in a private directory of the software I'm distributing. Actually, this is why I provide the C runtime libraries in the 'bin' directory, where all binaries are stored (python.exe, *.dll, vtk*.pyd, etc.).
Eloi
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Eloi: take a look at http://bugs.python.org/issue4120, which contains a discussion on, and a patch for, not embedding manifest files in *.pyd when compiling with MSVC.
You could recompile the PYD/DLL files that depend on the VC90.CRT in winSXS with the patch applied, or manually remove the manifests from those PYD/DLL files using a HEX editor or script. Also make sure that all DLLs that your PYD files depend on are in the Windows search PATH.
Christoph
On Mon, Nov 30, 2009 at 4:25 AM, Christoph Gohlke cgohlke@uci.edu wrote:
Eloi: take a look at http://bugs.python.org/issue4120, which contains a discussion on, and a patch for, not embedding manifest files in *.pyd when compiling with MSVC.
Note that the official binaries are built with mingw, and thus do not embed any manifest in the .pyd.
David
David, I think that what Christoph pointed here applies to the msvc built libraries only (I need to build the whole python/extensions using msvc).
May I ask you what re the "clashes between extensions" you were referring to in your previous answer ? Have you already encountered such issues ? I'm wondering how it could break things asp roviding libraries in an unique location is a known and common practice.
Eloi
David Cournapeau wrote:
On Mon, Nov 30, 2009 at 4:25 AM, Christoph Gohlke cgohlke@uci.edu wrote:
Eloi: take a look at http://bugs.python.org/issue4120, which contains a discussion on, and a patch for, not embedding manifest files in *.pyd when compiling with MSVC.
Note that the official binaries are built with mingw, and thus do not embed any manifest in the .pyd.
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Eloi Gaudry wrote:
David, I think that what Christoph pointed here applies to the msvc built libraries only (I need to build the whole python/extensions using msvc).
Ok, I did not understand that point. Do you know if the patch discussed in 4120 has been applied to 2.6.4 ?
May I ask you what re the "clashes between extensions" you were referring to in your previous answer ? Have you already encountered such issues ?
If every python package starts to put its extensions (*.pyd) into a directory, what happens when two different packages have an extension with the same name (e.g. package foo has a package multiarray.pyd) ? I would also be really annoyed if a 3rd party extension starts polluting my C:\Python26.
I'm wondering how it could break things asp roviding libraries in an unique location is a known and common practice.
The problem is that numpy does not "own" C:\Python*, and it is not expected that projects start dumping their stuff there. It usually "works" on windows because each applications is independent of each other. But here, numpy extends python, so we cannot just do as we please - especially to deal with this whole manifest stupidity. Given that even MS has implicitly recognized how broken manifests were (they are giving it up for VS 2010), I don't think it worths spending time on this issue which may well be unsolvable in a reliable way anyway. AFAIK, the related issue of broken "install for me" python installers on vista and windows 7 has not been solved yet.
cheers,
David
David Cournapeau wrote:
Eloi Gaudry wrote:
David, I think that what Christoph pointed here applies to the msvc built libraries only (I need to build the whole python/extensions using msvc).
Ok, I did not understand that point. Do you know if the patch discussed in 4120 has been applied to 2.6.4 ?
I don't think so... I'm using 2.6.4 and still wondering how to circumvent the runtime redistribution issue.
May I ask you what re the "clashes between extensions" you were referring to in your previous answer ? Have you already encountered such issues ?
If every python package starts to put its extensions (*.pyd) into a directory, what happens when two different packages have an extension with the same name (e.g. package foo has a package multiarray.pyd) ? I would also be really annoyed if a 3rd party extension starts polluting my C:\Python26.
You made your point.
I'm wondering how it could break things asp roviding libraries in an unique location is a known and common practice.
The problem is that numpy does not "own" C:\Python*, and it is not expected that projects start dumping their stuff there. It usually "works" on windows because each applications is independent of each other. But here, numpy extends python, so we cannot just do as we please
- especially to deal with this whole manifest stupidity. Given that even
MS has implicitly recognized how broken manifests were (they are giving it up for VS 2010), I don't think it worths spending time on this issue which may well be unsolvable in a reliable way anyway. AFAIK, the related issue of broken "install for me" python installers on vista and windows 7 has not been solved yet.
Well, I wasn't aware of Microsoft willing to giving up the whole SxS/manifest thing. Is there any MSDN information available? Thanks for your input David, Eloi
cheers,
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Mon, Nov 30, 2009 at 8:52 PM, Eloi Gaudry eg@fft.be wrote:
Well, I wasn't aware of Microsoft willing to giving up the whole SxS/manifest thing. Is there any MSDN information available?
I have seen this mentioned for the first time on the python-dev ML:
http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3764855
The mention of including the version in the dll file, if true, is tragically comic. Maybe in 20 years windows will be able to have a system which exists for more than a decade on conventional unix... The link given by M.A Lemburg has changed since, though, as the description is nowhere to be found in the link. I think I have read that VS 2010 will never install the runtime in the SxS configuration, but I of course cannot find this information anymore. Maybe it is not true anymore, VS 2010 has not yet been released.
You can also find useful manifest troubleshooting information there:
http://blogs.msdn.com/junfeng/archive/2006/04/14/576314.aspx
cheers,
David
Thanks for these references (that's a pity we currently can't find anything related to runtime libraries versioning on the msdn database).
Eloi
David Cournapeau wrote:
On Mon, Nov 30, 2009 at 8:52 PM, Eloi Gaudry eg@fft.be wrote:
Well, I wasn't aware of Microsoft willing to giving up the whole SxS/manifest thing. Is there any MSDN information available?
I have seen this mentioned for the first time on the python-dev ML:
http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3764855
The mention of including the version in the dll file, if true, is tragically comic. Maybe in 20 years windows will be able to have a system which exists for more than a decade on conventional unix... The link given by M.A Lemburg has changed since, though, as the description is nowhere to be found in the link. I think I have read that VS 2010 will never install the runtime in the SxS configuration, but I of course cannot find this information anymore. Maybe it is not true anymore, VS 2010 has not yet been released.
You can also find useful manifest troubleshooting information there:
http://blogs.msdn.com/junfeng/archive/2006/04/14/576314.aspx
cheers,
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
David Cournapeau skrev:
If every python package starts to put its extensions (*.pyd) into a directory, what happens when two different packages have an extension with the same name (e.g. package foo has a package multiarray.pyd) ? I would also be really annoyed if a 3rd party extension starts polluting my C:\Python26.
In disutils, data_files can install DLLs where Python wants them. Just pass 'DLLs' as path, and the rest it up to distutils. If anyone pollutes your c:\Python26 it is distutils, not a third party extension. This is not any different from installing other data files.
Sturla
On Mon, Nov 30, 2009 at 10:48 PM, Sturla Molden sturla@molden.no wrote:
David Cournapeau skrev:
If every python package starts to put its extensions (*.pyd) into a directory, what happens when two different packages have an extension with the same name (e.g. package foo has a package multiarray.pyd) ? I would also be really annoyed if a 3rd party extension starts polluting my C:\Python26.
In disutils, data_files can install DLLs where Python wants them. Just pass 'DLLs' as path, and the rest it up to distutils. If anyone pollutes your c:\Python26 it is distutils, not a third party extension. This is not any different from installing other data files.
We are talking about the numpy extensions here, which are not installed through the install_data command. The problem is about how windows looks for dll with the manifest mechanism, and how to build/install extensions when the C runtime (or any other "system" dll) is not installed in the SxS cache.
David
David Cournapeau skrev:
We are talking about the numpy extensions here, which are not installed through the install_data command. The problem is about how windows looks for dll with the manifest mechanism, and how to build/install extensions when the C runtime (or any other "system" dll) is not installed in the SxS cache.
Do we need to support Windows 2000?
S.M.
David Cournapeau skrev:
We are talking about the numpy extensions here, which are not installed through the install_data command. The problem is about how windows looks for dll with the manifest mechanism, and how to build/install extensions when the C runtime (or any other "system" dll) is not installed in the SxS cache.
Is COM (aka ActiveX) out of the question?
Christoph, thanks for pointing this discussion. That's a perfect match.
If the workaround provided offers a solution to the current redistribution issue, I'm wondering if it will still be the case when an update to the assembly check function will be activated/implemented (within Windows). The manifest edition (removing the "assemblyIdentity" tag) doesn't seem to be a popular/official/supported way of dealing with the whole runtime libraries issue. Don't you think ?
Christoph Gohlke wrote:
Eloi: take a look at http://bugs.python.org/issue4120, which contains a discussion on, and a patch for, not embedding manifest files in *.pyd when compiling with MSVC.
You could recompile the PYD/DLL files that depend on the VC90.CRT in winSXS with the patch applied, or manually remove the manifests from those PYD/DLL files using a HEX editor or script. Also make sure that all DLLs that your PYD files depend on are in the Windows search PATH.
Christoph _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
The most popular/simple way to deal with the VC90.CRT dependency issue is to have the user install the runtime redistributable on their system. If you don't want to put that burden on the user, which I understand, you have to make adjustments to the assembly manifests. This is not unofficial or unsupported. It is a bug in Python that it embeds the assemblyIdentity for VC90.CRT in all extensions build with distutils/msvc9compiler.py. In fact, the *.pyd distributed with Python 2.6.3+ don't have that problem. Maybe you can raise your concerns about future compatibility at http://bugs.python.org/issue4120.
Christoph
On 11/30/2009 1:11 AM, Eloi Gaudry wrote:
Christoph, thanks for pointing this discussion. That's a perfect match.
If the workaround provided offers a solution to the current redistribution issue, I'm wondering if it will still be the case when an update to the assembly check function will be activated/implemented (within Windows). The manifest edition (removing the "assemblyIdentity" tag) doesn't seem to be a popular/official/supported way of dealing with the whole runtime libraries issue. Don't you think ?
I've done so, thanks for pointing the discussion.
In the meantime, I've just patched distutils/msvc9compiler.py so that it neither embed nor create a manifest assembly. This way, I'll be sure that the assembly information would be fetched from the main python (or python-based) binaries (i.e. pythonX.dll). That may be a very strong prerequisites in some cases, but never in my very particular case.
Eloi
Christoph Gohlke wrote:
The most popular/simple way to deal with the VC90.CRT dependency issue is to have the user install the runtime redistributable on their system. If you don't want to put that burden on the user, which I understand, you have to make adjustments to the assembly manifests. This is not unofficial or unsupported. It is a bug in Python that it embeds the assemblyIdentity for VC90.CRT in all extensions build with distutils/msvc9compiler.py. In fact, the *.pyd distributed with Python 2.6.3+ don't have that problem. Maybe you can raise your concerns about future compatibility at http://bugs.python.org/issue4120.
Christoph
On 11/30/2009 1:11 AM, Eloi Gaudry wrote:
Christoph, thanks for pointing this discussion. That's a perfect match.
If the workaround provided offers a solution to the current redistribution issue, I'm wondering if it will still be the case when an update to the assembly check function will be activated/implemented (within Windows). The manifest edition (removing the "assemblyIdentity" tag) doesn't seem to be a popular/official/supported way of dealing with the whole runtime libraries issue. Don't you think ?
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (8)
-
Christian Heimes -
Christoph Gohlke -
David Cournapeau -
David Cournapeau -
Eloi Gaudry -
Jon Wright -
Matthieu Brucher -
Sturla Molden