On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers wrote:

On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock wrote:

...

Dear all,

Since the NumPy 1.7.0b2 release didn't include a Windows (32 bit) installer for Python 3.3, I am considering compiling it myself for local testing. What compiler is recommended?

Either MSVC or MinGW 3.4.5. For the latter see https://github.com/certik/numpy-vendor

Thanks Ralf, I was trying with MSVC 9.0 installed, but got this cryptic error: C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build ... error: Unable to find vcvarsall.bat After sprinkling distutils with debug statements, I found it was looking for MSVC v10 (not numpy's fault but the error is most unhelpful). Presumably "Microsoft Visual C++ 2010 Express Edition" is the appropriate thing to download? http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express Thanks, Peter

I meant to click on "save" not "send", anyway: On Sat, Nov 10, 2012 at 11:13 PM, Peter Cock wrote:

...

Either MSVC or MinGW 3.4.5. For the latter see https://github.com/certik/numpy-vendor

Ralf

I was trying with mingw32 via cygwin with gcc 2.4.4,

Typo, gcc 3.4.4

which also failed with a cryptic error:

... File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.p y", line 94, in __init__ msvcr_success = build_msvcr_library() File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.p y", line 336, in build_msvcr_library if int(msvcr_name.lstrip('msvcr')) < 80: AttributeError: 'NoneType' object has no attribute 'lstrip'

I think part of the problem could be in numpy/distutils/misc_util.py where there is no code to detect MSCV 10, def msvc_runtime_library(): "Return name of MSVC runtime library if Python was built with MSVC >= 7" msc_pos = sys.version.find('MSC v.') if msc_pos != -1: msc_ver = sys.version[msc_pos+6:msc_pos+10] lib = {'1300' : 'msvcr70', # MSVC 7.0 '1310' : 'msvcr71', # MSVC 7.1 '1400' : 'msvcr80', # MSVC 8 '1500' : 'msvcr90', # MSVC 9 (VS 2008) }.get(msc_ver, None) else: lib = None return lib https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L353 Under Python 3.3, we have: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

...

...

import sys sys.version '3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)]'

i.e. It looks to me like that dictionary needs another entry for key '1600'. Peter

On Sun, Nov 11, 2012 at 12:43 AM, Peter Cock wrote:

On Sat, Nov 10, 2012 at 11:24 PM, Peter Cock wrote:

...

I think part of the problem could be in numpy/distutils/misc_util.py where there is no code to detect MSCV 10,

def msvc_runtime_library(): "Return name of MSVC runtime library if Python was built with MSVC = 7" msc_pos = sys.version.find('MSC v.') if msc_pos != -1: msc_ver = sys.version[msc_pos+6:msc_pos+10] lib = {'1300' : 'msvcr70', # MSVC 7.0 '1310' : 'msvcr71', # MSVC 7.1 '1400' : 'msvcr80', # MSVC 8 '1500' : 'msvcr90', # MSVC 9 (VS 2008) }.get(msc_ver, None) else: lib = None return lib

https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L353

...

Under Python 3.3, we have:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import sys sys.version '3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit

(Intel)]'

...

i.e. It looks to me like that dictionary needs another entry for key

'1600'.

...

Adding this line seems to help,

'1600' : 'msvcr100', # MSVC 10 (aka 2010)

Now my compile gets further, but runs into another issue:

File "c:\python33\lib\distutils\command\config.py", line 246, in try_link libraries, library_dirs, lang) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py", line 146, in _link generate_manifest(self) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 562, in generate_manifest check_embedded_msvcr_match_linked(msver) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 541, in check_embedded_msvcr_match_linked "(%d)" % (int(msver), maj)) ValueError: Discrepancy between linked msvcr (10) and the one about to be embedded (1)

My hunch was version something about the new three digit version number is breaking things... which appears to be breaking here:

def check_embedded_msvcr_match_linked(msver): """msver is the ms runtime version used for the MANIFEST.""" # check msvcr major version are the same for linking and # embedding msvcv = msvc_runtime_library() if msvcv: maj = int(msvcv[5:6]) if not maj == int(msver): raise ValueError( "Discrepancy between linked msvcr " \ "(%d) and the one about to be embedded " \ "(%d)" % (int(msver), maj))

https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler....

As you can see, to get the major version number from the string it looks at the first digit. When the string was something like "81" or "90" that was fine, but now it is "100". Instead it should look at all the digits up to the final one, i.e. use:

maj = int(msvcv[5:-1])

Now (finally), I get an understandable (but hopefully wrong) error message from trying to build NumPy 1.7.0b2 under Python 3.3 on Windows XP,

File "numpy\core\setup.py", line 646, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "c:\python33\lib\distutils\command\config.py", line 246, in try_link libraries, library_dirs, lang) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py", line 146, in _link generate_manifest(self) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 568, in generate_manifest manxml = msvc_manifest_xml(ma, mi) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py", line 484, in msvc_manifest_xml % (maj, min)) ValueError: Version 10,0 of MSVCRT not supported yet

Presumably those two changes I have described are worth committing to the trunk anyway? I can prepare a patch or pull request, but currently I've been working on this Windows box remotely and I'd prefer to wait until next week when I can do it directly on the machine concerned.

Those changes look correct, a PR would be great. Fixing the next error also seems straightforward; around line 465 of mingw32ccompiler a check is needed for 10.0 (not sure which one) and then a line like _MSVCRVER_TO_FULLVER['10'] = "10.0.xxxxx.x" needs to be added. Ralf

Files affected: numpy/distutils/misc_util.py function msvc_runtime_library numpy/distutils/mingw32ccompiler.py function check_embedded_msvcr_match_linked

Peter _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Sun, Nov 11, 2012 at 11:20 PM, Peter Cock wrote:

On Sun, Nov 11, 2012 at 9:05 PM, Ralf Gommers wrote:

...

Those changes look correct, a PR would be great.

I'll do that later this week - but feel free to do it yourself immediately if more convenient.

Hi again Ralf, OK, new branch from the master here: https://github.com/peterjc/numpy/tree/msvc10 The first two commits are the changes already discussed. https://github.com/peterjc/numpy/commit/d8ead4c83364f9c7d4543690eb12e4543c60... https://github.com/peterjc/numpy/commit/1d498f54668a9a87286fa31f2779acbb048e...

...

Fixing the next error also seems straightforward; around line 465 of mingw32ccompiler a check is needed for 10.0 (not sure which one) and then a line like _MSVCRVER_TO_FULLVER['10'] = "10.0.xxxxx.x" needs to be added.

I'd got that far last night before calling turning in - my question was where do I get the last two parts of the version number. Presumably I can just use the values from the msvcr100.dll on my machine?

The third commit provides a fall back value (based on the DLL on my machine) as done for MSVC v8 and v9, and updates the live lookup via msvcrt.CRT_ASSEMBLY_VERSION to actually check what version that is (the current code wrongly assumes it is going to be for MSCV v9): https://github.com/peterjc/numpy/commit/24523565b5dbb23d6de0591ef2a4c1d01472... Do you want a pull request for that work so far? Or can you suggest what I should investigate next since there is at least one more hurdle to clear before I can build it (build output below). Regards, Peter ------ I now get further trying to build with mingw32 (from this branch from master, not the numpy-1.7.0b2 files this time), an excerpt from which this line may be note-worthy: Cannot build msvcr library: "msvcr100d.dll" not found This omits the failure to detect ALTAS etc which seems irrelevant. I would guess the MismatchCAPIWarning is an unrelated issue from working from the master branch this time. C:\repositories\numpy>c:\python33\python setup.py build --compiler=mingw32 Converting to Python3 via 2to3... <snip> numpy\core\setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 8, with checksum f4362353e2d72f889fda0128aa015037, but recorded checksum for C API version 8 in codegen_dir/cversions.txt is 17321775fc884de0b1eda478cd61c74b. If functions were added in the C API, you have to update C_API_VERSION in numpy\core\setup_common.py. MismatchCAPIWarning) <snip> running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources building library "npymath" sources Cannot build msvcr library: "msvcr100d.dll" not found Unable to find productdir in registry Checking environ VS100COMNTOOLS customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelVisualFCompiler Could not locate executable ifort Could not locate executable ifl customize AbsoftFCompiler Could not locate executable f90 customize CompaqVisualFCompiler Found executable C:\cygwin\bin\DF.exe Found executable C:\cygwin\bin\DF.exe customize IntelItaniumVisualFCompiler Could not locate executable efl customize Gnu95FCompiler Could not locate executable gfortran Could not locate executable f95 customize G95FCompiler Could not locate executable g95 customize IntelEM64VisualFCompiler customize IntelEM64TFCompiler Could not locate executable efort Could not locate executable efc don't know how to compile Fortran code on platform 'nt' C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes compile options: '-DNPY_MINGW_USE_CUSTOM_MSVCR -D__MSVCRT_VERSION__=0x1000 -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include -Ic:\python33\include -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -DNPY_MINGW_USE_CUSTOM_MSVCR -D__MSVCRT_VERSION__=0x1000 -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include -Ic:\python33\include -c _configtest.c -o _configtest.o Found executable C:\cygwin\usr\bin\gcc.exe g++ -mno-cygwin _configtest.o -lmsvcr100 -o _configtest.exe Could not locate executable g++ Executable g++ does not exist failure. removing: _configtest.exe.manifest _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "C:\repositories\numpy\build\py3k\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "c:\python33\lib\distutils\core.py", line 148, in setup dist.run_commands() File "c:\python33\lib\distutils\dist.py", line 917, in run_commands self.run_command(cmd) File "c:\python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "c:\python33\lib\distutils\command\build.py", line 126, in run self.run_command(cmd_name) File "c:\python33\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "c:\python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 648, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program Peter

On Mon, Nov 12, 2012 at 9:20 PM, Ralf Gommers wrote:

...

... Found executable C:\cygwin\usr\bin\gcc.exe g++ -mno-cygwin _configtest.o -lmsvcr100 -o _configtest.exe Could not locate executable g++ Executable g++ does not exist

A C++ compiler shouldn't be needed for numpy, so it shouldn't try to find one. On Linux it doesn't try this. It looks like on Windows (with MinGW at least) it does. I looked at a build log from a compile with MinGW and g++ available (http://projects.scipy.org/numpy/attachment/ticket/1909/dc6b601-builld.log), and g++ is used exactly once: for the linking stage of numpy.core._dummy.pyd.

Looking at core/setup.py that is related to __STDC_FORMAT_MACROS. That led me to https://github.com/numpy/numpy/commit/64423fdb18e8. So the easy way forward is to install g++. The hard way is figuring out why that define was necessary and find a way around it.

Ralf

Very strange (wanting a C++ compiler). I had some trouble updating my cygwin installation (probably in part due to issues stemming from a change in Windows username) and gave up and reinstalled cygwin. This presented a new issue that gcc and g++ wouldn't work from the Windows cmd prompt (due to being symlinks), so I deleted the symlinks /usr/bin/gcc[.exe] and /usr/bin/g++[.exe] and just replaced them with copies of the binaries /usr/bin/gcc-3.exe and /usr/bin/g++-3.exe - which seems to work. [I don't recall having to do that before, perhaps a relatively recent regression in cygwin?] Now, back to numpy, ... C:\repositories\numpy>c:\python33\python setup.py build --compiler=mingw32 <snip> gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -DNPY_NEEDS_MINGW_TIME_WORKAROUND -DNPY_MINGW_USE_CUSTOM_MSVCR -D__MSVCRT_VERSION__=0x1000 -Inumpy\core\include -Ibuild\src.win32-3.3\numpy\core\include/numpy -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include -Ic:\python33\include -Ibuild\src.win32-3.3\numpy\core\src\multiarray -Ibuild\src.win32-3.3\numpy\core\src\umath -c numpy\random\mtrand\distributions.c -o build\temp.win32-3.3\Release\numpy\random\mtrand\distributions.o g++ -mno-cygwin -shared build\temp.win32-3.3\Release\numpy\random\mtrand\mtrand.o build\temp.win32-3.3\Release\numpy\random\mtrand\randomkit.o build\temp.win32-3.3\Release\numpy\random\mtrand\initarray.o build\temp.win32-3.3\Release\numpy\random\mtrand\distributions.o -Lc:\python33\libs -Lc:\python33\PCbuild -Lbuild\temp.win32-3.3 -lpython33 -lmsvcr100 -o build\lib.win32-3.3\numpy\random\mtrand.pyd running scons running build_scripts creating build\scripts.win32-3.3 Creating build\scripts.win32-3.3\f2py.py adding 'build\scripts.win32-3.3\f2py.py' to scripts C:\repositories\numpy> Horray! It appears to work. I guess I need nose for the tests now... However, what is the correct way to now install this build? I have a workaround (given below), but the obvious routes failed: ------------------------------------------------------------- C:\repositories\numpy>c:\python33\python setup.py install --compiler=mingw32 Converting to Python3 via 2to3... Running from numpy source directory. usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: option --compiler not recognized ------------------------------------------------------------- Yet the only other likely choice of leaving out the compiler option also fails: C:\repositories\numpy>c:\python33\python setup.py install Converting to Python3 via 2to3... Running from numpy source directory. non-existing path in 'numpy\\distutils': 'site.cfg' F2PY Version 2 numpy\core\setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 8, with checks um f4362353e2d72f889fda0128aa015037, but recorded checksum for C API version 8 i n codegen_dir/cversions.txt is 17321775fc884de0b1eda478cd61c74b. If functions we re added in the C API, you have to update C_API_VERSION in numpy\core\setup_com mon.py. MismatchCAPIWarning) blas_opt_info: <snip> running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources building library "npymath" sources No module named 'numpy.distutils.msvccompiler' in numpy.distutils; trying from distutils Unable to find productdir in registry Checking environ VS100COMNTOOLS error: Unable to find vcvarsall.bat C:\repositories\numpy> ------------------------------------------------------------- I went with plan (B), creating C:\Python33\Lib\distutils\distutils.cfg containing: [build] compiler=mingw32 Now I don't have to give a compiler switch and distutils will default to using mingw32 with Python 3.3, and build and install seem to work. I've not yet run the numpy tests yet, but I think this means my github branches are worth merging: https://github.com/peterjc/numpy/commits/msvc10 Regards, Peter P.S. I am really looking forward to official Windows installers for NumPy on Python 3.3 on Windows...