Binary installer on Windows can't install extension
Hello, I have a problem with an install script in Python 2.7.5 on Windows 7 64-Bit. I have written an install script to register a COM server for a Windows-specific package. I specify the script with the --install-script option to the bdist_wininst command. When I run the installer, the install script fails trying to install an extension: Traceback (most recent call last): ... File "C:\Program Files\Python\lib\site-packages\mx\DateTime\DateTime.py", line 9, in <module> from mxDateTime import * File "C:\Program Files\Python\lib\site-packages\mx\DateTime\mxDateTime\__init__.py", line 13, in <module> raise ImportError, why ImportError: DLL load failed: %1 is not a valid Win32 application. close failed in file object destructor: sys.excepthook is missing lost sys.stderr The last three lines of output are probably caused by a known, unrelated issue (http://bugs.python.org/issue15321). I can run the install script by hand without problems. Importing mx.DateTime in Python works fine as well. I know very litte about Windows internals, but have a guess nevertheless: I checked mx\DateTime\mxDateTime\mxDateTime.pyd with Dependency Walker (depends.exe). It seems, the extension does not find the proper version of MSVCR90.DLL. Instead, it is trying to use a 32-bit version of the DLL that was installed as part of a 32-bit application. The application's directory is included in the user search path (PATH). Obviously, a 64-bit version of the DLL exists and is used by Python itself. What is causing my problem? How can I solve it? Thanks, Malte
You will need a 64-bit version of mxDateTime.pyd when used with 64-bit Python. The DLL load fails because a 64-bit process can only load 64-bit DLLs. If you can't get a 64-bit version, it should not be a problem to install and use a 32-bit version of Python on 64-bit Windows (and I generally recommend it - the only thing you gain with 64-bit Python is a higher memory limit). Cheers, Steve -----Original Message----- From: Distutils-SIG [mailto:distutils-sig-bounces+steve.dower=microsoft.com@python.org] On Behalf Of Malte Forkel Sent: Wednesday, June 12, 2013 1010 To: Distutils-Sig@Python.Org Subject: [Distutils] Binary installer on Windows can't install extension Hello, I have a problem with an install script in Python 2.7.5 on Windows 7 64-Bit. I have written an install script to register a COM server for a Windows-specific package. I specify the script with the --install-script option to the bdist_wininst command. When I run the installer, the install script fails trying to install an extension: Traceback (most recent call last): ... File "C:\Program Files\Python\lib\site-packages\mx\DateTime\DateTime.py", line 9, in <module> from mxDateTime import * File "C:\Program Files\Python\lib\site-packages\mx\DateTime\mxDateTime\__init__.py", line 13, in <module> raise ImportError, why ImportError: DLL load failed: %1 is not a valid Win32 application. close failed in file object destructor: sys.excepthook is missing lost sys.stderr The last three lines of output are probably caused by a known, unrelated issue (http://bugs.python.org/issue15321). I can run the install script by hand without problems. Importing mx.DateTime in Python works fine as well. I know very litte about Windows internals, but have a guess nevertheless: I checked mx\DateTime\mxDateTime\mxDateTime.pyd with Dependency Walker (depends.exe). It seems, the extension does not find the proper version of MSVCR90.DLL. Instead, it is trying to use a 32-bit version of the DLL that was installed as part of a 32-bit application. The application's directory is included in the user search path (PATH). Obviously, a 64-bit version of the DLL exists and is used by Python itself. What is causing my problem? How can I solve it? Thanks, Malte _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Am 14.06.2013 00:05, schrieb Steve Dower:
You will need a 64-bit version of mxDateTime.pyd when used with 64-bit Python. The DLL load fails because a 64-bit process can only load 64-bit DLLs.
If you can't get a 64-bit version, it should not be a problem to install and use a 32-bit version of Python on 64-bit Windows (and I generally recommend it - > the only thing you gain with 64-bit Python is a higher memory limit).
I'm afraid its not that easy. Everything I installed is 64-bit, both Python itself and the eGenix mx Base Distribution which includes mx.DateTime. According to Dependency Walker, MSVCR90.DLL is the only 32-bit DLL referenced by mxDateTime.pyd. All others DLLs (inluding the version of MSVCR90.DLL referenced by PYTHON27.DLL) are 64-bit. When imported from a regular Python process, mxDateTime loads fine. The problem only occurred when mxDateTime was imported in the context of the installer's (post)install script. Note the past tense, though. When I ran my tests today, there was no error anymore. I guess if I knew what change has caused that, I knew what caused the initial problem. I started my tests with Python 2.7.3 and mx 2.3.1, then upgraded to 2.7.5 and 2.3.6 respectively. The problem still occurred. Since, I have rebooted the computer and installed another Python package of my own. The paths (user, system) have not changed.
Malte Forkel wrote:
Am 14.06.2013 00:05, schrieb Steve Dower:
You will need a 64-bit version of mxDateTime.pyd when used with 64-bit Python. The DLL load fails because a 64-bit process can only load 64-bit DLLs.
If you can't get a 64-bit version, it should not be a problem to install and use a 32-bit version of Python on 64-bit Windows (and I generally recommend it - > the only thing you gain with 64-bit Python is a higher memory limit).
I'm afraid its not that easy. Everything I installed is 64-bit, both Python itself and the eGenix mx Base Distribution which includes mx.DateTime.
According to Dependency Walker, MSVCR90.DLL is the only 32-bit DLL referenced by mxDateTime.pyd. All others DLLs (inluding the version of MSVCR90.DLL referenced by PYTHON27.DLL) are 64-bit.
Are you using our prebuilt or egg distribution of eGenix mx Base, or compiling it from source ?
When imported from a regular Python process, mxDateTime loads fine. The problem only occurred when mxDateTime was imported in the context of the installer's (post)install script.
Note the past tense, though. When I ran my tests today, there was no error anymore. I guess if I knew what change has caused that, I knew what caused the initial problem. I started my tests with Python 2.7.3 and mx 2.3.1, then upgraded to 2.7.5 and 2.3.6 respectively. The problem still occurred. Since, I have rebooted the computer and installed another Python package of my own. The paths (user, system) have not changed.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Am 19.06.2013 00:19, schrieb M.-A. Lemburg:
Are you using our prebuilt or egg distribution of eGenix mx Base, or compiling it from source ?
I'm using the prebuilt distribution (egenix-mx-base-3.2.6.win-amd64-py2.7.msi). Thanks to the kind advice of Steve Dower, I have since updated OpenVPN, the application had installed the 32-bit version of MSVCR90.DLL in its own directory, to a newer 64-bit version which does not install local CRTs. According to Dependency Walker, it had been OpenVPN's copy of MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the OpenVPN's entry in the user's search path. Now, according to Dependency Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved. But there are no problems when I run the install script or import mx.DateTime myself.
On 19.06.2013 10:01, Malte Forkel wrote:
Am 19.06.2013 00:19, schrieb M.-A. Lemburg:
Are you using our prebuilt or egg distribution of eGenix mx Base, or compiling it from source ?
I'm using the prebuilt distribution (egenix-mx-base-3.2.6.win-amd64-py2.7.msi).
Thanks to the kind advice of Steve Dower, I have since updated OpenVPN, the application had installed the 32-bit version of MSVCR90.DLL in its own directory, to a newer 64-bit version which does not install local CRTs.
According to Dependency Walker, it had been OpenVPN's copy of MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the OpenVPN's entry in the user's search path.
Interesting. I just tried that on my system and indeed, see the reference to OpenVPN as well. The same happens with all other .pyd files in the Python installation.
Now, according to Dependency Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved. But there are no problems when I run the install script or import mx.DateTime myself.
We'll investigate that some more. It may have something to do with the way distutils handles manifests for Python extensions at compile/link time. The extensions should get all the MSVCR90.DLL symbols from the DLL loaded into the PYTHON27.DLL process. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 19 2013)
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2013-06-18: Released mxODBC Django DE 1.2.0 ... http://egenix.com/go47 2013-07-01: EuroPython 2013, Florence, Italy ... 12 days to go 2013-07-16: Python Meeting Duesseldorf ... 27 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 19.06.2013 12:51, M.-A. Lemburg wrote:
On 19.06.2013 10:01, Malte Forkel wrote:
Am 19.06.2013 00:19, schrieb M.-A. Lemburg:
Are you using our prebuilt or egg distribution of eGenix mx Base, or compiling it from source ?
I'm using the prebuilt distribution (egenix-mx-base-3.2.6.win-amd64-py2.7.msi).
Thanks to the kind advice of Steve Dower, I have since updated OpenVPN, the application had installed the 32-bit version of MSVCR90.DLL in its own directory, to a newer 64-bit version which does not install local CRTs.
According to Dependency Walker, it had been OpenVPN's copy of MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the OpenVPN's entry in the user's search path.
Interesting. I just tried that on my system and indeed, see the reference to OpenVPN as well. The same happens with all other .pyd files in the Python installation.
Now, according to Dependency Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved. But there are no problems when I run the install script or import mx.DateTime myself.
We'll investigate that some more. It may have something to do with the way distutils handles manifests for Python extensions at compile/link time. The extensions should get all the MSVCR90.DLL symbols from the DLL loaded into the PYTHON27.DLL process.
The unresolved reference to MSVCR90.DLL is shown for all distutils compiled extensions, so this is most likely the result of distutils removing the manifest from compiled C extensions to force the linker to use the Python DLL's CRT libs - this in return avoids problem with having to place the CRT manifests into each extension directory. I have no idea why the Windows linker attempts to load a x86 DLL into a x64 process. The manifest included with OpenVPN clearly shows that the DLL is for a different platform, even though it uses the same version as the one listed in the python.exe manifest. -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/
Am 19.06.2013 13:57, schrieb M.-A. Lemburg:
The unresolved reference to MSVCR90.DLL is shown for all distutils compiled extensions, so this is most likely the result of distutils removing the manifest from compiled C extensions to force the linker to use the Python DLL's CRT libs - this in return avoids problem with having to place the CRT manifests into each extension directory.
After my initial problems, this now to works for me. I still don't understand what went wrong initially, though.
I have no idea why the Windows linker attempts to load a x86 DLL into a x64 process. The manifest included with OpenVPN clearly shows that the DLL is for a different platform, even though it uses the same version as the one listed in the python.exe manifest.
I can a confirm this for the manifest included with OpenVPN 2.2.1 which I had installed before: <assemblyIdentity type="win32" name="Microsoft.VC90.CRT" version="9.0.21022.8" processorArchitecture="x86" publicKeyToken="1fc8b3b9a1e18e3b" /> When Dependency Walker showed a reference from mxDateTime.pyd to the 32-bit version of MSVCR90.DLL, it always also showed a reference from PYTHON27.DLL to the 64-bit version of MSVCR90.DLL.
I have no idea why the Windows linker attempts to load a x86 DLL into a x64 process. The manifest included with OpenVPN clearly shows that the DLL is for a different platform, even though it uses the same version as the one listed in the python.exe manifest.
[snip]
When Dependency Walker showed a reference from mxDateTime.pyd to the 32-bit version of MSVCR90.DLL, it always also showed a reference from PYTHON27.DLL to the 64-bit version of MSVCR90.DLL.
My suspicion is that this is due to OpenVPN adding itself to the PATH variable. The default Python install will put PYTHON27.DLL into System32 (for a 64-bit install), and so it will try and resolve dependencies in the same directory first. This is probably why it finds the correct MSVCR90.DLL (it may also be that the manifest is correct). Since mxDateTime.pyd is not in System32, it will go through the usual resolution process. System32 is supposed to be checked first, so I'm not sure why the correct version is being bypassed, but that may be because of the lack of manifest. I'd guess that it works for most people because by the time Windows reaches the end of its search, the correct DLL is the only/best/most recent option found. That OpenVPN version provides an alternative, and Windows is choosing it and then failing to load because of the platform mismatch. There's not much I can offer by way of solution. Adding a manifest to the pyd 'should' be the 'correct' solution, but I can't guarantee that it won't cause conflicts elsewhere. Adding a copy of MSVCR90.DLL to the pyd directory would also work, but this is probably riskier when it comes to version mismatches. As I understand, OpenVPN has corrected their side of things already, so this may just be an issue to be aware of for when people hit it in the future. Cheers, Steve
participants (4)
-
M.-A. Lemburg -
M.-A. Lemburg -
Malte Forkel -
Steve Dower