Ralf Gommers wrote:

At http://github.com/rgommers/NumPy-release-guide you can find a summary of how to set up your system to build numpy binaries on OS X. I still have to add info on scipy (that's turning out to be fairly painful) but for numpy it is pretty complete.

Any feedback is appreciated!

We really don't want to supply binaries for python 2.5 anymore? (maybe this will get me to finally switch! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Wed, Mar 24, 2010 at 1:40 AM, Christopher Barker wrote:

Ralf Gommers wrote:

...

At http://github.com/rgommers/NumPy-release-guide you can find a summary of how to set up your system to build numpy binaries on OS X. I still have to add info on scipy (that's turning out to be fairly painful) but for numpy it is pretty complete.

Any feedback is appreciated!

We really don't want to supply binaries for python 2.5 anymore?

(maybe this will get me to finally switch!

That's fast, I was hoping to get away with that one! I discussed a bit with David, and he suggested to not make 2.5 binaries for scipy 0.7.2 and then see if there was still any demand. It's a lot of work with perhaps not so much return. Of course a maintenance release of scipy with no new features may be a different story from a new release of numpy/scipy. So by no means is this decided or even publicly discussed. I do want to point out that we have not provided 2.4 binaries for a while without much demand. So now the question: who still wants and uses 2.5 binaries? Cheers, Ralf

A Wednesday 24 March 2010 01:49:50 Ralf Gommers escrigué:

On Wed, Mar 24, 2010 at 1:40 AM, Christopher Barker

wrote:

...

Ralf Gommers wrote:

...

At http://github.com/rgommers/NumPy-release-guide you can find a summary of how to set up your system to build numpy binaries on OS X. I still have to add info on scipy (that's turning out to be fairly painful) but for numpy it is pretty complete.

Any feedback is appreciated!

We really don't want to supply binaries for python 2.5 anymore?

(maybe this will get me to finally switch!

That's fast, I was hoping to get away with that one!

I discussed a bit with David, and he suggested to not make 2.5 binaries for scipy 0.7.2 and then see if there was still any demand. It's a lot of work with perhaps not so much return. Of course a maintenance release of scipy with no new features may be a different story from a new release of numpy/scipy. So by no means is this decided or even publicly discussed. I do want to point out that we have not provided 2.4 binaries for a while without much demand.

So now the question: who still wants and uses 2.5 binaries?

I do. I think at least 2 binary versions is a recommendable practice for allowing people to change versions more comfortably. Also, I have read the draft and I cannot see references to 64-bit binary packages. With the advent of Windows 7 and Mac OSX Snow Leopard, 64-bit are way more spread than before, so they would be a great thing to deliver, IMO. Thanks, -- Francesc Alted

On Wed, Mar 24, 2010 at 6:50 PM, Francesc Alted wrote:

Also, I have read the draft and I cannot see references to 64-bit binary packages.  With the advent of Windows 7 and Mac OSX Snow Leopard, 64-bit are way more spread than before, so they would be a great thing to deliver, IMO.

For Mac OS X, we should wait for official 64 bits binaries from python.org - EPD offers 64 bits binaries if necessary. On windows, the situation is more complicated, because there is no way to build numpy and scipy correctly with free compilers. For various reasons, I think we should avoid distributing binaries based on non-free compilers (not available to everyone, problem of compatibility and redistribution of non-free code, especially for fortran) - and for people who don't mind/care, there are unofficial win64 binaries out there (we may put a link somewhere). Unfortunately, I don't think I will have much time to spend on making scipy and gfortran work together on win64 in the near future, cheers, David

On Wed, Mar 24, 2010 at 11:25 PM, Francesc Alted wrote:

Ok.  I've been having a try at mingw-w64 project:

http://mingw-w64.sourceforge.net/

with no success so far with build numpy:

$ python setup.py build --compiler=mingw32

Oh, it is not that easy :) First, for some reason, the mingw-w64 project does not provide 64 hosted compilers, and since pushing for mingw cross compilation support in distutils would redefine the meaning of insanity, I build my gcc. Since building gcc on windows is not a fun ride either, you have to build it on unix (I have the scripts on my github account: github.com/cournape/). Then, you can hope to build numpy with mingw 64 on windows 64. But then, it randomly crashes, and last time I looked at it (~ 8 months ago), there was no gdb support and the mingw debug symbols cannot be understood by MS tools, so I was stuck at trying to understand what went wrong without a meaningful backtrace. I also looked into making gfortran works with MS compiler, but this means porting libgfortran to something MS compilers can understand. This is not as crazy as it sounds, because we don't use many fortran runtime functions in scipy. Of course, atlas on windows 64 is nowhere near buildable. cheers, David

A Wednesday 24 March 2010 15:38:58 David Cournapeau escrigué:

Oh, it is not that easy :)

First, for some reason, the mingw-w64 project does not provide 64 hosted compilers, and since pushing for mingw cross compilation support in distutils would redefine the meaning of insanity, I build my gcc. Since building gcc on windows is not a fun ride either, you have to build it on unix (I have the scripts on my github account: github.com/cournape/).

Mmh, not sure about what you mean by hosted compiler, but there is certainly a native compiler package for Win64: mingw-w64-bin_x86_64-mingw_20100322_sezero.zip It comes with gcc, g++ and gfortran 4.4.4. gdb support is also there. So it seems like a pretty complete toolset for windows amd64. With it, and with some fixes in numpy sources (very few), I achieved to pass the build phase. I can provide the patch in case someone is interested. The generated extensions are: 24/03/2010 19:34 1.492.313 multiarray.pyd 24/03/2010 19:34 124.866 multiarray_tests.pyd 24/03/2010 19:34 453.377 scalarmath.pyd 24/03/2010 19:34 1.079.827 umath.pyd 24/03/2010 19:34 121.651 umath_tests.pyd 24/03/2010 19:34 304.014 _sort.pyd which looks good to my eyes. Now, when I try to generate the installable package I'm in trouble again: $ python setup.py bdist [...] File "C: \Users\francesc\Desktop\NumPy\1.4.x\numpy\distutils\command\config.py" , line 56, in _check_compiler self.compiler.initialize() File "C:\Python26_64\lib\distutils\msvc9compiler.py", line 359, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python26_64\lib\distutils\msvc9compiler.py", line 275, in query_vcvar sall raise ValueError(str(list(result.keys()))) ValueError: [u'path'] [...] So, it looks like either numpy or python cannot determine that the used compiler is mingw instead of msvc9. However, when I try to specify mingw explicitly, I get the next error: $ python setup.py bdist --compiler=mingw32 Running from numpy source directory. Forcing DISTUTILS_USE_SDK=1 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: option --compiler not recognized Someone could tell me why distutils can be told to use mingw32 compiler for the build stage but not for bdist? What is more, why the need for a compiler for bdist if numpy is already built? I feel that I'm almost there, but some piece still resists... -- Francesc Alted

What happens if you try to build a windows installer "python setup.py bdist_wininst" Also, have you attempted to specify the compiler in "$PYTHON_ROOT\Lib\distutils\distutils.cfg" ? I've got a script I use to manually change the "build" section of this file between [build] msvc and [build] mingw32 to switch between default compilers when I've had issues with setup.py. On Wed, Mar 24, 2010 at 2:31 PM, Francesc Alted wrote:

A Wednesday 24 March 2010 15:38:58 David Cournapeau escrigué:

...

Oh, it is not that easy :)

First, for some reason, the mingw-w64 project does not provide 64 hosted compilers, and since pushing for mingw cross compilation support in distutils would redefine the meaning of insanity, I build my gcc. Since building gcc on windows is not a fun ride either, you have to build it on unix (I have the scripts on my github account: github.com/cournape/).

Mmh, not sure about what you mean by hosted compiler, but there is certainly a native compiler package for Win64:

mingw-w64-bin_x86_64-mingw_20100322_sezero.zip

It comes with gcc, g++ and gfortran 4.4.4. gdb support is also there. So it seems like a pretty complete toolset for windows amd64.

With it, and with some fixes in numpy sources (very few), I achieved to pass the build phase. I can provide the patch in case someone is interested. The generated extensions are:

24/03/2010 19:34 1.492.313 multiarray.pyd 24/03/2010 19:34 124.866 multiarray_tests.pyd 24/03/2010 19:34 453.377 scalarmath.pyd 24/03/2010 19:34 1.079.827 umath.pyd 24/03/2010 19:34 121.651 umath_tests.pyd 24/03/2010 19:34 304.014 _sort.pyd

which looks good to my eyes.

Now, when I try to generate the installable package I'm in trouble again:

$ python setup.py bdist [...] File "C: \Users\francesc\Desktop\NumPy\1.4.x\numpy\distutils\command\config.py" , line 56, in _check_compiler self.compiler.initialize() File "C:\Python26_64\lib\distutils\msvc9compiler.py", line 359, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python26_64\lib\distutils\msvc9compiler.py", line 275, in query_vcvar sall raise ValueError(str(list(result.keys()))) ValueError: [u'path'] [...]

So, it looks like either numpy or python cannot determine that the used compiler is mingw instead of msvc9. However, when I try to specify mingw explicitly, I get the next error:

$ python setup.py bdist --compiler=mingw32 Running from numpy source directory. Forcing DISTUTILS_USE_SDK=1 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help

error: option --compiler not recognized

Someone could tell me why distutils can be told to use mingw32 compiler for the build stage but not for bdist? What is more, why the need for a compiler for bdist if numpy is already built? I feel that I'm almost there, but some piece still resists...

-- Francesc Alted _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com

A Thursday 25 March 2010 02:00:36 David Cournapeau escrigué:

Hosted compiler refers to the platform the compiler itself runs on (so here I mean a native 64 bits compiler, instead of a 32 bits compiler which targets 64 bits). It is nice that mingw-w64 gives a 64 bits hosted, that's recent.

Yeah, but unfortunately it is not enough for numpy to work. After creating the binary package (thanks for the "build -c mingw32 bdist_wininst" trick), and installing nose, I get a big crash when running the test units. When trying to use gdb to see the backtrace, I got: C:\Users\francesc\Desktop\NumPy>gdb python GNU gdb (GDB) 7.1.50.20100318-cvs Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-w64-mingw32". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from \Python26_64/python.exe...(no debugging symbols found)...done. (gdb) r Starting program: \Python26_64/python.exe [New Thread 2880.0x410] Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

...

...

import numpy C:\Python26_64\lib\site-packages\numpy\core\__init__.py:5: Warning: Numpy built with MINGW-W64 on Windows 64 bits is experimental, and only available for testing. You are advised not to use it for production.

CRASHES ARE TO BE EXPECTED - PLEASE REPORT THEM TO NUMPY DEVELOPERS import multiarray

...

...

numpy.test() Running unit tests for numpy NumPy version 1.4.0 NumPy is installed in C:\Python26_64\lib\site-packages\numpy Python version 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD 64)] nose version 0.11.3 Forcing DISTUTILS_USE_SDK=1 .....warning: HEAP[python.exe]: warning: Invalid address specified to RtlFreeHeap( 0000000002240000, 0000000000C 25110 )

Program received signal SIGTRAP, Trace/breakpoint trap. 0x0000000076fa6061 in ntdll!DbgBreakPoint () from C:\Windows\system32\ntdll.dll (gdb) bt #0 0x0000000076fa6061 in ntdll!DbgBreakPoint () from C:\Windows\system32\ntdll.dll #1 0x0000000076ffe17a in ntdll!EtwEventProviderEnabled () from C:\Windows\system32\ntdll.dll #2 0x00000000022af0d8 in ?? () #3 0x000000005104095c in ?? () #4 0x0000000000219a08 in ?? () #5 0x000000000e040001 in ?? () #6 0x0000000002240000 in ?? () #7 0x0000000076fe27a1 in ntdll!MD4Final () from C:\Windows\system32\ntdll.dll #8 0x0000000076fb9630 in ntdll!LdrGetProcedureAddress () from C:\Windows\system32\ntdll.dll #9 0x0000000076fb9500 in ntdll!LdrGetProcedureAddress () from C:\Windows\system32\ntdll.dll #10 0x0000000002240000 in ?? () #11 0x0000000000c25110 in ?? () #12 0x0000000002240000 in ?? () #13 0x000000000623f197 in ?? () #14 0x0000000002240000 in ?? () #15 0x00000000770151a9 in ntdll!RtlTraceDatabaseCreate () from C:\Windows\system32\ntdll.dll #16 0x0000000000000000 in ?? () (gdb) So, it is still exactly as you said: it seems like gdb cannot introspect MS debug symbols. Unfortunately I don't think solving this would be easy at all :-/ Perhaps reporting this to mingw-w64 would help...

...

With it, and with some fixes in numpy sources (very few), I achieved to pass the build phase. I can provide the patch in case someone is interested.

Building should work out of the box, actually - what are the issues ?

I'm attaching the patch. Caveat emptor: the patch is *very* crude, and it is just meant to allow a clean compile with mingw-w64. A more elaborated patch should be implemented in case we want it to go into numpy.

But the main problem is not to build numpy (the 1.3.0 64 bits msi was built with mingw-w64 already) - it is to find out what causes the various crashes people encountered. Some were due to bugs in mingw which have been fixed since then.

Exactly. OK, I think I'm done with this try for the moment. It is a pity because mingw-w64 does an excellent job with my preliminary tests with Blosc compressor (it achieves 4x better performance than mingw-w32 and 2x better than MSVC 2008 32-bit). But before going MSVC 64-bit route, I'd like to test Intel compiler 64-bit. Anyone knows if it can cope with numpy? A quick look at setup.py seems to say that only 32-bit Intel compiler is supported. Thanks, -- Francesc Alted

A Thursday 25 March 2010 10:53:34 David Cournapeau escrigué:

Believe it or not, but this is already much better than what I had last time I looked at it (the stack was corrupted after two items, and gdb often crashed). I had to build custom mingw runtimes to get there last year :)

Well, I've reported the problem just in case: https://sourceforge.net/projects/mingw-w64/forums/forum/723797/topic/3638920

What works well is MS compiler + Intel Fortran compiler (at least with numscons). Surprisingly, when I tried using the Intel C compiler + Intel Fortran compiler, I also had a lot of issues, not unlike the ones with mingw. What I am afraid is that the C runtimes issues are unsolvable on win64 because of python, and that we have to use the MS compiler (for C and C++).

Ok. So it seems the MS compiler venue for 64-bit is unavoidable (at this moment, at least). One question though: is a fortran compiler really necessary for compiling just numpy? If so, why? -- Francesc Alted