Compiling Cython on Windows runs into link problems because of directories containing spaces
Hi,
I have a problem with compiling Cython code on Windows. I think this is a bug in distutils which is easy to solve. I have reproduced this on Python 2.6 and Python 3.2 (32 bit).
The problem occurs with the native msvc compiler. Using gcc works fine. And I prefer using gcc, but sometimes you just need msvc :/
The problem is that the command to link the libraries does not put double quotes around paths that have spaces in them. Unfortunately, the path where many users have Python installed have spaces in it (e.g. c:/program files/python26). Small example: */LIBPATH:C:\Program Files (x86)\python32\libs*. Oh, and the include_dirs DO have double quotes around them.
The problem is easily solved (I confirmed this) by changing msvc9compiler.py and msvccompiler.py:
def library_dir_option(self, dir): # OLD VERSION
return "/LIBPATH:" + dir
def library_dir_option(self, dir): # FIXED VERSION
if ' ' in dir and not dir.startswith('"'):
dir = '"%s"' % dir
return "/LIBPATH:" + dir
===== Below follows a minimal example =====
===== test_.pyx def foo():
print('hello')
===== setup.py
import os, sys
from Cython.Distutils import build_ext
from distutils.core import setup
from distutils.extension import Extension
from numpy.distutils.misc_util import get_numpy_include_dirs
# Ugly hack so I can run setup.py in my IDE
sys.argv = ['setup.py', 'build_ext', '--inplace']
# Init include dirs
include_dirs = ['.']
include_dirs.extend(get_numpy_include_dirs())
# Creat Extensions
ext_modules = [
Extension('test_', ['test_.pyx'],
include_dirs=include_dirs,
),
]
# Compile
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
)
print('Successfully compiled cython file: test_')
===== output when running setup.py
running build_ext
No module named msvccompiler in numpy.distutils; trying from distutils
cythoning test_.pyx to test_.c
building 'test_' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c /Fobuild\temp.win32-3.2\Release\test_.obj
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO */LIBPATH:C:\Program Files (x86)\python32\libs /LIBPATH:C:\Program Files (x86)\python32\PCbuild* /EXPORT:PyInit_test_ build\temp.win32-3.2\Release\test_.obj /OUT:C:\almar\projects\py\cmu1394\test_.pyd /IMPLIB:build\temp.win32-3.2\Release\test_.lib /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe
LINK : fatal error LNK1181: cannot open input file 'Files.obj'
Since there've been no responses yet, let me make my question more clear :)
Is this is a bug, or is this a known "feature" that I should solve in another manner. If this is a bug, can someone help me fix it?
Further, I'm a bit surprised to find this bug. I can't be the only one who wants to compile stuff with the MS compiler and who has Python installed in his "program files". Why is that?
Regards, Almar
On 5 January 2012 11:49, Almar Klein almar.klein@gmail.com wrote:
Hi,
I have a problem with compiling Cython code on Windows. I think this is a bug in distutils which is easy to solve. I have reproduced this on Python 2.6 and Python 3.2 (32 bit).
The problem occurs with the native msvc compiler. Using gcc works fine. And I prefer using gcc, but sometimes you just need msvc :/
The problem is that the command to link the libraries does not put double quotes around paths that have spaces in them. Unfortunately, the path where many users have Python installed have spaces in it (e.g. c:/program files/python26). Small example: */LIBPATH:C:\Program Files (x86)\python32\libs*. Oh, and the include_dirs DO have double quotes around them.
The problem is easily solved (I confirmed this) by changing msvc9compiler.py and msvccompiler.py:
def library_dir_option(self, dir): # OLD VERSION
return "/LIBPATH:" + dirdef library_dir_option(self, dir): # FIXED VERSION
if ' ' in dir and not dir.startswith('"'): dir = '"%s"' % dir return "/LIBPATH:" + dir===== Below follows a minimal example =====
===== test_.pyx def foo():
print('hello')
===== setup.py
import os, sys
from Cython.Distutils import build_ext
from distutils.core import setup
from distutils.extension import Extension
from numpy.distutils.misc_util import get_numpy_include_dirs
# Ugly hack so I can run setup.py in my IDE
sys.argv = ['setup.py', 'build_ext', '--inplace']
# Init include dirs
include_dirs = ['.']
include_dirs.extend(get_numpy_include_dirs())
# Creat Extensions
ext_modules = [
Extension('test_', ['test_.pyx'], include_dirs=include_dirs, ), ]# Compile
setup(
cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules, )print('Successfully compiled cython file: test_')
===== output when running setup.py
running build_ext
No module named msvccompiler in numpy.distutils; trying from distutils
cythoning test_.pyx to test_.c
building 'test_' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c /Fobuild\temp.win32-3.2\Release\test_.obj
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO */LIBPATH:C:\Program Files (x86)\python32\libs /LIBPATH:C:\Program Files (x86)\python32\PCbuild* /EXPORT:PyInit_test_ build\temp.win32-3.2\Release\test_.obj /OUT:C:\almar\projects\py\cmu1394\test_.pyd /IMPLIB:build\temp.win32-3.2\Release\test_.lib /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe
LINK : fatal error LNK1181: cannot open input file 'Files.obj'
On 9/01/2012 9:10 AM, Almar Klein wrote:
Since there've been no responses yet, let me make my question more clear :)
Is this is a bug, or is this a known "feature" that I should solve in another manner. If this is a bug, can someone help me fix it?
It probably can be considered a bug - the Python issue tracker is the place where it should be reported (but be sure to check it hasn't already been reported). From your earlier message, it sounds like you already have a handle on how to fix it (although I suspect it might be better for the caller of library_dir_option to add the quotes).
Further, I'm a bit surprised to find this bug. I can't be the only one who wants to compile stuff with the MS compiler and who has Python installed in his "program files".
You may well be. "program files" isn't the default location and even if it was, I suspect most developers would override it. So I suspect that most people either (a) use the default location or (b) override it to something without spaces in the path.
Mark
Why is that?
Regards, Almar
On 5 January 2012 11:49, Almar Klein <almar.klein@gmail.com mailto:almar.klein@gmail.com> wrote:
Hi, I have a problem with compiling Cython code on Windows. I think this is a bug in distutils which is easy to solve. I have reproduced this on Python 2.6 and Python 3.2 (32 bit). The problem occurs with the native msvc compiler. Using gcc works fine. And I prefer using gcc, but sometimes you just need msvc :/ The problem is that the command to link the libraries does not put double quotes around paths that have spaces in them. Unfortunately, the path where many users have Python installed have spaces in it (e.g. c:/program files/python26). Small example: _/LIBPATH:C:\Program Files (x86)\python32\libs_. Oh, and the include_dirs DO have double quotes around them. The problem is easily solved (I confirmed this) by changing msvc9compiler.py and msvccompiler.py: def library_dir_option(self, dir): # OLD VERSION return "/LIBPATH:" + dir def library_dir_option(self, dir): # FIXED VERSION if ' ' in dir and not dir.startswith('"'): dir = '"%s"' % dir return "/LIBPATH:" + dir ===== Below follows a minimal example ===== ===== test_.pyx def foo(): print('hello') ===== setup.py import os, sys from Cython.Distutils import build_ext from distutils.core import setup from distutils.extension import Extension from numpy.distutils.misc_util import get_numpy_include_dirs # Ugly hack so I can run setup.py in my IDE sys.argv = ['setup.py', 'build_ext', '--inplace'] # Init include dirs include_dirs = ['.'] include_dirs.extend(get_numpy_include_dirs()) # Creat Extensions ext_modules = [ Extension('test_', ['test_.pyx'], include_dirs=include_dirs, ), ] # Compile setup( cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules, ) print('Successfully compiled cython file: test_') ===== output when running setup.py running build_ext No module named msvccompiler in numpy.distutils; trying from distutils cythoning test_.pyx to test_.c building 'test_' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c /Fobuild\temp.win32-3.2\Release\test_.obj Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO _/LIBPATH:C:\Program Files (x86)\python32\libs /LIBPATH:C:\Program Files (x86)\python32\PCbuild_ /EXPORT:PyInit_test_ build\temp.win32-3.2\Release\test_.obj /OUT:C:\almar\projects\py\cmu1394\test_.pyd /IMPLIB:build\temp.win32-3.2\Release\test_.lib /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe LINK : fatal error LNK1181: cannot open input file 'Files.obj'
Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Dear Mark,
Thanks for the response.
Is this is a bug, or is this a known "feature" that I should solve in
another manner. If this is a bug, can someone help me fix it?
It probably can be considered a bug - the Python issue tracker is the place where it should be reported (but be sure to check it hasn't already been reported). From your earlier message, it sounds like you already have a handle on how to fix it (although I suspect it might be better for the caller of library_dir_option to add the quotes).
Ok, I will report the bug there. I wasn't sure whether distutils had its own issue tracker (I couldn't find it anyway).
Further, I'm a bit surprised to find this bug. I can't be the only one
who wants to compile stuff with the MS compiler and who has Python installed in his "program files".
You may well be. "program files" isn't the default location and even if it was, I suspect most developers would override it. So I suspect that most people either (a) use the default location or (b) override it to something without spaces in the path.
Well I didn't :) And I can hardly be the only one. I can't imagine I'm the first to run into this bug. Maybe because the produced link error is not helpful at all, people did not know what to do with it?
Regards, Almar
Ok, I will report the bug there. I wasn't sure whether distutils had its own issue tracker (I couldn't find it anyway).
I reported the bug and supplied a patch: http://bugs.python.org/issue13765
Almar
participants (2)
-
Almar Klein -
Mark Hammond