On Sun, Feb 3, 2013 at 12:28 AM, wrote:

On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett wrote:

...

Hi,

I see there is no Windows 64 bit installer for the 1.7 rc1.

related: Is there any chance to get newer mingw or mingw-w64 support "soonish"?

The problem has no solution until we can restrict support to windows 7 and above. Otherwise, any acceptable solution would require user to be an admin. David

Hi, On Mon, Feb 4, 2013 at 12:27 PM, Ondřej Čertík wrote:

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau wrote:

...

On Sun, Feb 3, 2013 at 12:28 AM, wrote:

...

On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett wrote:

...

Hi,

I see there is no Windows 64 bit installer for the 1.7 rc1.

related: Is there any chance to get newer mingw or mingw-w64 support "soonish"?

The problem has no solution until we can restrict support to windows 7 and above. Otherwise, any acceptable solution would require user to be an admin.

The installer is built with this VM/scripts:

https://github.com/certik/numpy-vendor

currently the VM itself is 32 bit. I think that might be upgraded to 64bit, and maybe it's possible to use 64 bit Wine:

http://wiki.winehq.org/Wine64

but then we would need to figure out how to use Mingw with 64 bits.

I would be very happy to accept patches to the above repository.

Alternatively, if the actual Windows 64bit machine would have to be used, is there any way to automate the process? Would you compile it from command line (cmd.exe), just like I do in Wine? I would much prefer if we can figure out how to do this in Wine, so that the process can be automated and other people can easily reproduce it.

I wonder whether getting ming64 to work on 64 bit Wine is too hard to get working before the release? I often can't get 32-bit Wine working, and we've apparently got problems with mingw64 on native windows. As a short term fix, how about an Amazon image with the Windows 64 bit compilers on it? Or can Christophe Gohlke help us out here? It seems a shame not to provide these builds. Cheers, Matthew

On Mon, Feb 4, 2013 at 9:39 PM, Nathaniel Smith wrote:

...

Hi,

On Mon, Feb 4, 2013 at 12:27 PM, Ondřej Čertík wrote:

...

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau wrote:

...

On Sun, Feb 3, 2013 at 12:28 AM, wrote:

...

On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett < matthew.brett@gmail.com> wrote:

...

Hi,

I see there is no Windows 64 bit installer for the 1.7 rc1.

related: Is there any chance to get newer mingw or mingw-w64 support "soonish"?

The problem has no solution until we can restrict support to windows 7 and above. Otherwise, any acceptable solution would require user to be an admin.

The installer is built with this VM/scripts:

https://github.com/certik/numpy-vendor

currently the VM itself is 32 bit. I think that might be upgraded to 64bit, and maybe it's possible to use 64 bit Wine:

http://wiki.winehq.org/Wine64

but then we would need to figure out how to use Mingw with 64 bits.

I would be very happy to accept patches to the above repository.

Alternatively, if the actual Windows 64bit machine would have to be used, is there any way to automate the process? Would you compile it from command line (cmd.exe), just like I do in Wine? I would much prefer if we can figure out how to do this in Wine, so that the process can be automated and other

On Mon, Feb 4, 2013 at 12:36 PM, Matthew Brett wrote: people

...

...

can easily reproduce it.

I wonder whether getting ming64 to work on 64 bit Wine is too hard to get working before the release? I often can't get 32-bit Wine working, and we've apparently got problems with mingw64 on native windows.

As a short term fix, how about an Amazon image with the Windows 64 bit compilers on it?

-1 on providing an "official" solution which will require admin rights for the produced installer and not work for scipy.

...

Or can Christophe Gohlke help us out here?

As a temporary measure it might make sense to just deem the cgolke builds official and upload them to the usual places -- or at least, it seems like it might make sense to me, but it depends on what Christophe thinks :-).

I'm +0 on that. MSVC + MKL is currently the only real option for 64-bit binaries, so providing Christoph's installers as "official" could make sense. On the other hand, Christoph has a matching set of other packages on his site, so users will be better off being redirected there than just grabbing only a numpy installer from SF and not finding a scipy one there. Ralf

On Mon, Feb 4, 2013 at 8:27 PM, Ondřej Čertík wrote:

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau wrote:

...

On Sun, Feb 3, 2013 at 12:28 AM, wrote:

...

On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett wrote:

...

Hi,

I see there is no Windows 64 bit installer for the 1.7 rc1.

related: Is there any chance to get newer mingw or mingw-w64 support "soonish"?

The problem has no solution until we can restrict support to windows 7 and above. Otherwise, any acceptable solution would require user to be an admin.

The installer is built with this VM/scripts:

I am not sure whether you're replying to my observation or just giving a status update: with mingw-w64 (or recent mingw), the built installer will depend on several .dll (libgcc_s_sjil.dll) that we can't easily distribute. The only place we can realistically put them is in C:\Python$VERSION (or wherever python happens to be installed), and I think it is a very bad idea to install dll from NumPy there. In Windows 2008 and above, one can refer in .pyd where to look for dlls in another directory which is private to numpy. David

On Mon, Feb 4, 2013 at 10:06 PM, Matthew Brett wrote:

Hi,

On Mon, Feb 4, 2013 at 12:55 PM, Ralf Gommers wrote:

...

On Mon, Feb 4, 2013 at 9:39 PM, Nathaniel Smith wrote:

...

On Mon, Feb 4, 2013 at 12:36 PM, Matthew Brett

...

wrote:

...

Hi,

On Mon, Feb 4, 2013 at 12:27 PM, Ondřej Čertík <

...

...

...

wrote:

...

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau

...

...

...

wrote:

...

On Sun, Feb 3, 2013 at 12:28 AM, wrote: > On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett > wrote: >> Hi, >> >> I see there is no Windows 64 bit installer for the 1.7 rc1. > > related: > Is there any chance to get newer mingw or mingw-w64 support > "soonish"?

The problem has no solution until we can restrict support to windows 7 and above. Otherwise, any acceptable solution would require user to be an admin.

The installer is built with this VM/scripts:

https://github.com/certik/numpy-vendor

currently the VM itself is 32 bit. I think that might be upgraded to 64bit, and maybe it's possible to use 64 bit Wine:

http://wiki.winehq.org/Wine64

but then we would need to figure out how to use Mingw with 64 bits.

I would be very happy to accept patches to the above repository.

Alternatively, if the actual Windows 64bit machine would have to be used, is there any way to automate the process? Would you compile it from command line (cmd.exe), just like I do in Wine? I would much prefer if we can

ondrej.certik@gmail.com> figure

...

...

...

...

out how to do this in Wine, so that the process can be automated and other people can easily reproduce it.

I wonder whether getting ming64 to work on 64 bit Wine is too hard to get working before the release? I often can't get 32-bit Wine working, and we've apparently got problems with mingw64 on native windows.

As a short term fix, how about an Amazon image with the Windows 64 bit compilers on it?

-1 on providing an "official" solution which will require admin rights for the produced installer and not work for scipy.

Sorry if I am being slow, but I don't follow.

Am I right in thinking that we can currently build numpy 64 bit installers with the Microsoft tools, and that these would be distributable without admin rights for windows >=XP ?

MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you provide an Amazon image for those?

Why would this solution not work for scipy?

It would. gfortran doesn't. Looking at your mail, I still read it as providing an image with mingw64 + gfortran. Ralf

Hi, On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers wrote:

On Mon, Feb 4, 2013 at 10:06 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 12:55 PM, Ralf Gommers wrote:

...

On Mon, Feb 4, 2013 at 9:39 PM, Nathaniel Smith wrote:

...

On Mon, Feb 4, 2013 at 12:36 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 12:27 PM, Ondřej Čertík wrote:

...

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau wrote: > On Sun, Feb 3, 2013 at 12:28 AM, wrote: >> On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett >> wrote: >>> Hi, >>> >>> I see there is no Windows 64 bit installer for the 1.7 rc1. >> >> related: >> Is there any chance to get newer mingw or mingw-w64 support >> "soonish"? > > The problem has no solution until we can restrict support to > windows 7 > and above. Otherwise, any acceptable solution would require user to > be > an admin.

The installer is built with this VM/scripts:

https://github.com/certik/numpy-vendor

currently the VM itself is 32 bit. I think that might be upgraded to 64bit, and maybe it's possible to use 64 bit Wine:

http://wiki.winehq.org/Wine64

but then we would need to figure out how to use Mingw with 64 bits.

I would be very happy to accept patches to the above repository.

Alternatively, if the actual Windows 64bit machine would have to be used, is there any way to automate the process? Would you compile it from command line (cmd.exe), just like I do in Wine? I would much prefer if we can figure out how to do this in Wine, so that the process can be automated and other people can easily reproduce it.

I wonder whether getting ming64 to work on 64 bit Wine is too hard to get working before the release? I often can't get 32-bit Wine working, and we've apparently got problems with mingw64 on native windows.

As a short term fix, how about an Amazon image with the Windows 64 bit compilers on it?

-1 on providing an "official" solution which will require admin rights for the produced installer and not work for scipy.

Sorry if I am being slow, but I don't follow.

Am I right in thinking that we can currently build numpy 64 bit installers with the Microsoft tools, and that these would be distributable without admin rights for windows >=XP ?

MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you provide an Amazon image for those?

You can make an image that is not public, I guess. I suppose anyone who uses the image would have to have their own licenses for the Intel stuff? Does anyone have experience of this? Does ATLAS / openBLAS not build for windows? Sorry for my ignorance. Cheers, Matthew

On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern wrote:

On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers wrote:

...

...

MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you provide an Amazon image for those?

You can make an image that is not public, I guess. I suppose anyone who uses the image would have to have their own licenses for the Intel stuff? Does anyone have experience of this?

You need to purchase one license per developer:

http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing...

I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It would be a bit much to get it implemented in the next week or two. Chuck

On Mon, Feb 4, 2013 at 3:49 PM, Christoph Gohlke wrote:

On 2/4/2013 12:59 PM, David Cournapeau wrote:

...

On Mon, Feb 4, 2013 at 8:27 PM, Ondřej Čertík wrote:

...

On Sun, Feb 3, 2013 at 2:57 AM, David Cournapeau wrote:

...

On Sun, Feb 3, 2013 at 12:28 AM, wrote:

...

On Sat, Feb 2, 2013 at 6:14 PM, Matthew Brett wrote:

...

Hi,

I see there is no Windows 64 bit installer for the 1.7 rc1.

related: Is there any chance to get newer mingw or mingw-w64 support "soonish"?

The problem has no solution until we can restrict support to windows 7 and above. Otherwise, any acceptable solution would require user to be an admin.

The installer is built with this VM/scripts:

I am not sure whether you're replying to my observation or just giving a status update: with mingw-w64 (or recent mingw), the built installer

I was just giving a general status, sorry about not being clear.

...

will depend on several .dll (libgcc_s_sjil.dll) that we can't easily distribute. The only place we can realistically put them is in C:\Python$VERSION (or wherever python happens to be installed), and I think it is a very bad idea to install dll from NumPy there. In Windows 2008 and above, one can refer in .pyd where to look for dlls in another directory which is private to numpy.

Yes.

...

David

If I understand correctly the problem is distributing dependency/runtime DLLs with a package and ensuring the DLLs are found by Windows when the pyd extensions are imported? For numpy-MKL and other packages I include/install the extra DLLs in the package directories and, if necessary, (i) append the package directory to os.environ['PATH'] or (ii) "pre-load" the DLLs into the process using Ctypes, both early in the package's main __init__.py. No admin rights are required.

So that seems to be the only option. Is there any other solution? Ondrej

Hi, On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris wrote:

On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern wrote:

...

On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers wrote:

...

...

MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you provide an Amazon image for those?

You can make an image that is not public, I guess. I suppose anyone who uses the image would have to have their own licenses for the Intel stuff? Does anyone have experience of this?

You need to purchase one license per developer:

http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing...

I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It would be a bit much to get it implemented in the next week or two.

The problem with not providing these binaries is that they are at the bottom of everyone's stack, so a delay in numpy holds everyone back. I can't find completely convincing stats, but it looks as though 64 bit windows 7 is now the most common version of Windows, at least for Gamers [1] around now, and it was getting that way for everyone in 2010 [2]. It don't think it reflects well on on us that we don't appear to support 64 bits out of the box; just for example, R already has a 32 bit / 64 bit installer. If I understand correctly, the options for doing this right now are: 1) Minimal cost in time : ask Christophe nicely whether we can distribute his binaries via the Numpy page 2) Small cost in time / money : pay for licenses for Ondrej or me or someone to install the dependencies on my Berkeley machine / an Amazon image. Ralf : I suppose we qualify for the free licenses you referred to earlier ? [3] . I guess that covers us for the Numpy build? Then it's only a question of paying for ifort licenses when it comes to do the Scipy build? So, if the cost of option 2 is too high, how about option 1? Cheers, Matthew [1] http://store.steampowered.com/hwsurvey [2] http://blogs.windows.com/windows/b/bloggingwindows/archive/2010/07/08/64-bit... [3] http://numpy-discussion.10968.n7.nabble.com/MKL-licenses-for-core-scientific...

On 02/04/2013 06:09 PM, Matthew Brett wrote:

The problem with not providing these binaries is that they are at the bottom of everyone's stack, so a delay in numpy holds everyone back.

OTOH, so far it's been an *excellent* excuse for those of us further up the stack not to make a 64-bit binary. It sounds like we're completely out of luck on 64-bit Windows with free tools? So the rest of the community is going to face shelling out for compiler licenses as well? -- Jonathan Niehof ISR-3 Space Data Systems Los Alamos National Laboratory MS-D466 Los Alamos, NM 87545 Phone: 505-667-9595 email: jniehof@lanl.gov Correspondence / Technical data or Software Publicly Available

Hi, On Mon, Feb 4, 2013 at 5:09 PM, Matthew Brett wrote:

Hi,

On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris wrote:

...

On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern wrote:

...

On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers wrote:

...

...

MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you provide an Amazon image for those?

You can make an image that is not public, I guess. I suppose anyone who uses the image would have to have their own licenses for the Intel stuff? Does anyone have experience of this?

You need to purchase one license per developer:

http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing...

I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It would be a bit much to get it implemented in the next week or two.

The problem with not providing these binaries is that they are at the bottom of everyone's stack, so a delay in numpy holds everyone back.

I can't find completely convincing stats, but it looks as though 64 bit windows 7 is now the most common version of Windows, at least for Gamers [1] around now, and it was getting that way for everyone in 2010 [2].

It don't think it reflects well on on us that we don't appear to support 64 bits out of the box; just for example, R already has a 32 bit / 64 bit installer.

If I understand correctly, the options for doing this right now are:

1) Minimal cost in time : ask Christophe nicely whether we can distribute his binaries via the Numpy page 2) Small cost in time / money : pay for licenses for Ondrej or me or someone to install the dependencies on my Berkeley machine / an Amazon image.

In order not to leave this discussion without a resolution: Christophe - would you allow us to distribute your numpy binaries for 1.7 from the numpy sourceforge page? Cheers, Matthew

Hi, On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke wrote:

On 2/5/2013 10:51 AM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 5:09 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris wrote:

...

On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern wrote:

...

On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers wrote:

...

> MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you > provide an Amazon image for those?

You can make an image that is not public, I guess. I suppose anyone who uses the image would have to have their own licenses for the Intel stuff? Does anyone have experience of this?

You need to purchase one license per developer:

http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing...

I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It would be a bit much to get it implemented in the next week or two.

The problem with not providing these binaries is that they are at the bottom of everyone's stack, so a delay in numpy holds everyone back.

I can't find completely convincing stats, but it looks as though 64 bit windows 7 is now the most common version of Windows, at least for Gamers [1] around now, and it was getting that way for everyone in 2010 [2].

It don't think it reflects well on on us that we don't appear to support 64 bits out of the box; just for example, R already has a 32 bit / 64 bit installer.

If I understand correctly, the options for doing this right now are:

1) Minimal cost in time : ask Christophe nicely whether we can distribute his binaries via the Numpy page 2) Small cost in time / money : pay for licenses for Ondrej or me or someone to install the dependencies on my Berkeley machine / an Amazon image.

In order not to leave this discussion without a resolution:

Christophe - would you allow us to distribute your numpy binaries for 1.7 from the numpy sourceforge page?

Cheers,

Matthew

I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy compiled with MSVC compilers and linked to Intel's MKL) for official numpy releases.

Thank you - that is great.

However:

1) There seems to be no real consensus and urge for doing this.

I certainly feel the urge and feel it strongly. As a packager for two or three projects myself, it's a royal pain having to tell someone to go to two different places for binaries depending on the number of bits of their Windows. I think Chuck was worried about the time it would take to do it, and I think you've already solved this problem. Ralf was worried about Scipy - see below.

Using a free toolchain capable of building the whole scipy-stack would be much preferred.

That's true, but there seems general agreement this is not practical in the very near future.

Several 64 bit Python distributions containing numpy-MKL are already available, some for free.

You mean EPD and AnacondaCE? I don't think we should withhold easily available vanilla builds because they are also available in company-sponsored projects. Python.org provides windows builds even though ActiveState is free-as-in-beer.

2) Releasing 64 bit numpy without matching scipy binaries would make little sense to me.

Would you consider also releasing your scipy binaries?

3) Please do not just redistribute the binaries from my website and declare them official. They might contain unreleased fixes from git master and pull requests that are needed for my work and other packages.

Right - would you consider then being the release provider for numpy / scipy binaries on windows, much as it appears that Martin v Lowis supplies Windows builds for Python?

4) Numpy-MKL requires the Intel runtime DLLs (MKL is linked statically btw). I ship those with the installers and append the directory containing the DLLs to os.environ['PATH'] in numpy/__init__.py. This is a big no-no according to numpy developers. I don't agree. Anyway, those changes are not in the numpy source repositories.

5) My numpy-MKL installers are Python distutils bdist_wininst installers. That means if Python was installed for all users, installing numpy-MKL on Windows >6.0 will prompt for UAC elevation. Another no-no?

I defer to others on these ones, Thanks a lot, Matthew

Hi, On Tue, Feb 5, 2013 at 4:55 PM, Chris Barker - NOAA Federal wrote:

On Tue, Feb 5, 2013 at 4:32 PM, Matthew Brett wrote:

...

...

4) Numpy-MKL requires the Intel runtime DLLs (MKL is linked statically btw). I ship those with the installers and append the directory containing the DLLs to os.environ['PATH'] in numpy/__init__.py. This is a big no-no according to numpy developers. I don't agree. Anyway, those changes are not in the numpy source repositories.

I think you pointed out that another option is to load the dlls with ctypes -- is it much work to make that change?

...

...

5) My numpy-MKL installers are Python distutils bdist_wininst installers. That means if Python was installed for all users, installing numpy-MKL on Windows >6.0 will prompt for UAC elevation. Another no-no?

not sure about the UAC elevation -- but:

1) most folks use bdist_wininst for Windows binaries -- including the current numpy builds, and python.org python -- yes?

2) UAC aside, It would be great to have binaries that could be used with virtualenv -- binary eggs?

easy_install can install into virtualenvs from bdist_wininst installers, at least the ones I have built... See you, Matthew

On Wed, Feb 6, 2013 at 1:32 AM, Matthew Brett wrote:

Hi,

On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke wrote:

...

On 2/5/2013 10:51 AM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 5:09 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris wrote:

...

On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern

wrote:

...

...

On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett <

matthew.brett@gmail.com>

...

wrote: > Hi, > > On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers < ralf.gommers@gmail.com> > wrote:

>> MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can you >> provide an Amazon image for those? > > You can make an image that is not public, I guess. I suppose anyone > who uses the image would have to have their own licenses for the Intel > stuff? Does anyone have experience of this?

You need to purchase one license per developer:

http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing...

...

I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It would be a bit much to get it implemented in the next week or two.

The problem with not providing these binaries is that they are at the bottom of everyone's stack, so a delay in numpy holds everyone back.

I can't find completely convincing stats, but it looks as though 64 bit windows 7 is now the most common version of Windows, at least for Gamers [1] around now, and it was getting that way for everyone in 2010 [2].

It don't think it reflects well on on us that we don't appear to support 64 bits out of the box; just for example, R already has a 32 bit / 64 bit installer.

If I understand correctly, the options for doing this right now are:

1) Minimal cost in time : ask Christophe nicely whether we can distribute his binaries via the Numpy page 2) Small cost in time / money : pay for licenses for Ondrej or me or someone to install the dependencies on my Berkeley machine / an Amazon image.

In order not to leave this discussion without a resolution:

Christophe - would you allow us to distribute your numpy binaries for 1.7 from the numpy sourceforge page?

Cheers,

Matthew

I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy compiled with MSVC compilers and linked to Intel's MKL) for official numpy releases.

Thank you - that is great.

...

However:

1) There seems to be no real consensus and urge for doing this.

I certainly feel the urge and feel it strongly. As a packager for two or three projects myself, it's a royal pain having to tell someone to go to two different places for binaries depending on the number of bits of their Windows.

If you're relying on .exe installers on SF, then you have to send your users to more places than that probably. Really the separate installers are a poor alternative to the available scientific distributions. And the more packages you need as a user, the more annoying these separate installers are.

I think Chuck was worried about the time it would take to do it, and I think you've already solved this problem. Ralf was worried about Scipy - see below.

Not just Scipy - that would be my worry number one, but the same holds for all packages based on Numpy. You double the number of Windows installers that every single project needs to provide.

...

Using a free toolchain capable of building the whole scipy-stack would be much preferred.

That's true, but there seems general agreement this is not practical in the very near future.

...

Several 64 bit Python distributions containing numpy-MKL are already available, some for free.

You mean EPD and AnacondaCE? I don't think we should withhold easily available vanilla builds because they are also available in company-sponsored projects. Python.org provides windows builds even though ActiveState is free-as-in-beer.

If the company-sponsored bit bothers you, there's also a 64-bit Python(x,y) now. Ralf

...

2) Releasing 64 bit numpy without matching scipy binaries would make little sense to me.

Would you consider also releasing your scipy binaries?

...

3) Please do not just redistribute the binaries from my website and declare them official. They might contain unreleased fixes from git master and pull requests that are needed for my work and other packages.

Right - would you consider then being the release provider for numpy / scipy binaries on windows, much as it appears that Martin v Lowis supplies Windows builds for Python?

...

4) Numpy-MKL requires the Intel runtime DLLs (MKL is linked statically btw). I ship those with the installers and append the directory containing the DLLs to os.environ['PATH'] in numpy/__init__.py. This is a big no-no according to numpy developers. I don't agree. Anyway, those changes are not in the numpy source repositories.

5) My numpy-MKL installers are Python distutils bdist_wininst installers. That means if Python was installed for all users, installing numpy-MKL on Windows >6.0 will prompt for UAC elevation. Another no-no?

I defer to others on these ones,

Thanks a lot,

Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi, On Wed, Feb 6, 2013 at 10:48 AM, Ralf Gommers wrote:

On Wed, Feb 6, 2013 at 1:32 AM, Matthew Brett wrote:

...

Hi,

On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke wrote:

...

On 2/5/2013 10:51 AM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 5:09 PM, Matthew Brett wrote:

...

Hi,

On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris wrote:

...

On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern wrote: > > On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett > > wrote: >> Hi, >> >> On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers >> >> wrote: > >>> MSVC + Intel Fortran + MKL, yes. But those aren't free. So how can >>> you >>> provide an Amazon image for those? >> >> You can make an image that is not public, I guess. I suppose >> anyone >> who uses the image would have to have their own licenses for the >> Intel >> stuff? Does anyone have experience of this? > > You need to purchase one license per developer: > > > > http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing... >

I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It would be a bit much to get it implemented in the next week or two.

The problem with not providing these binaries is that they are at the bottom of everyone's stack, so a delay in numpy holds everyone back.

I can't find completely convincing stats, but it looks as though 64 bit windows 7 is now the most common version of Windows, at least for Gamers [1] around now, and it was getting that way for everyone in 2010 [2].

It don't think it reflects well on on us that we don't appear to support 64 bits out of the box; just for example, R already has a 32 bit / 64 bit installer.

If I understand correctly, the options for doing this right now are:

1) Minimal cost in time : ask Christophe nicely whether we can distribute his binaries via the Numpy page 2) Small cost in time / money : pay for licenses for Ondrej or me or someone to install the dependencies on my Berkeley machine / an Amazon image.

In order not to leave this discussion without a resolution:

Christophe - would you allow us to distribute your numpy binaries for 1.7 from the numpy sourceforge page?

Cheers,

Matthew

I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy compiled with MSVC compilers and linked to Intel's MKL) for official numpy releases.

Thank you - that is great.

...

However:

1) There seems to be no real consensus and urge for doing this.

I certainly feel the urge and feel it strongly. As a packager for two or three projects myself, it's a royal pain having to tell someone to go to two different places for binaries depending on the number of bits of their Windows.

If you're relying on .exe installers on SF, then you have to send your users to more places than that probably. Really the separate installers are a poor alternative to the available scientific distributions. And the more packages you need as a user, the more annoying these separate installers are.

...

I think Chuck was worried about the time it would take to do it, and I think you've already solved this problem. Ralf was worried about Scipy - see below.

Not just Scipy - that would be my worry number one, but the same holds for all packages based on Numpy. You double the number of Windows installers that every single project needs to provide.

...

...

Using a free toolchain capable of building the whole scipy-stack would be much preferred.

That's true, but there seems general agreement this is not practical in the very near future.

...

Several 64 bit Python distributions containing numpy-MKL are already available, some for free.

You mean EPD and AnacondaCE? I don't think we should withhold easily available vanilla builds because they are also available in company-sponsored projects. Python.org provides windows builds even though ActiveState is free-as-in-beer.

If the company-sponsored bit bothers you, there's also a 64-bit Python(x,y) now.

I'm sure you've seen that the question 'where are the 64-bit installers' comes up often for Numpy. It seems to me that we'd have to have a good reason not provide them. There will always be some number of people like me who like to install the various parts by hand, and don't like using large packages, free or or open or not. For example, I don't use macports. At the moment, the reasons I see you are giving are: 1) Then everyone would have to provide 64-bit binaries 2) You should use a super-package instead On those arguments, we should withdraw all binary installers. Or withdraw the 32-bit ones, on the basis that 64-bit is likely the more common now. Of course we shouldn't do that, because it will put off some measurable number of users, and convey the impression that the numpy stack is a bit shaky, because it cannot be easily installed without a monolithic build framework. I think that's a real shame, and would harm numpy, to no benefit. I guess I'm saying, please, please, pretty please, oh please oh please oh please can we have a Windows 64-bit installer? Cheers, Matthew

I'm trying to weed out the issues here: 1) what should the binary installer for Windows look like: * bdist_wininst is the obvious choice but apparently has issues with the newer Windows security stuff -- the real solution is for distutils to be fixed/use something else/ etc -- but is there anyone here that has expertise, interest or time to get in and add-to or fix distutils? Do binary eggs solve any of this ? maybe, but pip doesn't support that last I saw, and setuptools easy-install is kind-of sort-of broken. But coming up with the best binary installer tool is orthogonal to the other questions: 2) Should we distribute binaries at all? - No: most people need a whole stack anyway, so they should get all that from the same place: - Enthought - Python(x.y) - Chris Gohlke's repository... - Yes: Some folks just want numpy damn it! Those big ol' distributions are a mess that don't work for everyone. (I'm in that camp.....) But if we distribute binaries, we really should: - distribute them for 32 & 64 bit - Clearly define what the "official" binaries are, so that third-party packagers can build against that. Historically, this has been a huge mess on the Mac, as there are many options for Python itself: Apple's builds, Python,org builds, macports, fink, homebrew, ...... Over the years, we in teh MacPython community have tried hard to establish that the python.org builds are the "official" builds that folks should support with binaries -- this has kind-of, sort-of worked, but I still don't see a better solution. (macports et al aren't really the issue, they aren't based on binaries anyway) On Windows, this has been pretty much a non-issue: MS doesn't provide a build, and the pyton.org builds are well accepted as the default binaries. AFAIU, third party distributions are compatible as well (Active State, Enthought) The parallel here is that we can establish in the scientific python community (that is, folks building packages based on numpy) that the binaries distributed by numpy are the "official" ones that should be supported by third part binaries. If we succeed in that, then folks can get the rest of the stack from any number of sources. the python.org python builds are built with the MS compilers, is there any reason we HAVE to build with a open-source stack? Can you build third party packages against an MS-built binary numpy with open-source compilers? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Wed, Feb 6, 2013 at 3:16 PM, Matthew Brett wrote:

Can you clarify the people you think will get stuck? I think I'm right in saying that anyone with a C extension should be able to build them against numpy, by installing the free (as-in-beer) MS tools?

yup -- and that's the recommended (easiest) way to do it against the the python.org python. and you can use mingw to compile extensions that will run with the python.org python (built with MSVC), can you not use mingw to build extensions that will work with a MSVC-build numpy?

So do you just mean people needing a Fortran compiler? That's a small constituency, I think.

particularly small overlap between needing fortran (and knowing how to deal with building it) and needing binaries of numpy.... if someone has a fortran-extension-building solution -- is it likeley not to work with a MSVC-built numpy?

And, if we start providing installers, I suspect we'll find that people start solving the remaining issues much more quickly than would happen if we don't provide them.

+1 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Hi, On Wed, Feb 6, 2013 at 9:20 PM, Dag Sverre Seljebotn wrote:

On 02/07/2013 12:16 AM, Matthew Brett wrote:

...

Hi,

On Wed, Feb 6, 2013 at 1:36 PM, Ralf Gommers wrote:

...

On Wed, Feb 6, 2013 at 8:46 PM, Matthew Brett wrote:

...

Hi,

On Wed, Feb 6, 2013 at 10:48 AM, Ralf Gommers wrote:

...

On Wed, Feb 6, 2013 at 1:32 AM, Matthew Brett wrote:

...

Hi,

On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke wrote: > On 2/5/2013 10:51 AM, Matthew Brett wrote: >> Hi, >> >> On Mon, Feb 4, 2013 at 5:09 PM, Matthew Brett >> >> wrote: >>> Hi, >>> >>> On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris >>> wrote: >>>> >>>> >>>> On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern >>>> >>>> wrote: >>>>> >>>>> On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett >>>>> >>>>> wrote: >>>>>> Hi, >>>>>> >>>>>> On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers >>>>>> >>>>>> wrote: >>>>> >>>>>>> MSVC + Intel Fortran + MKL, yes. But those aren't free. So how >>>>>>> can >>>>>>> you >>>>>>> provide an Amazon image for those? >>>>>> >>>>>> You can make an image that is not public, I guess. I suppose >>>>>> anyone >>>>>> who uses the image would have to have their own licenses for the >>>>>> Intel >>>>>> stuff? Does anyone have experience of this? >>>>> >>>>> You need to purchase one license per developer: >>>>> >>>>> >>>>> >>>>> >>>>> http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing... >>>>> >>>> >>>> I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It >>>> would be a >>>> bit much to get it implemented in the next week or two. >>> >>> The problem with not providing these binaries is that they are at >>> the >>> bottom of everyone's stack, so a delay in numpy holds everyone >>> back. >>> >>> I can't find completely convincing stats, but it looks as though 64 >>> bit windows 7 is now the most common version of Windows, at least >>> for >>> Gamers [1] around now, and it was getting that way for everyone in >>> 2010 [2]. >>> >>> It don't think it reflects well on on us that we don't appear to >>> support 64 bits out of the box; just for example, R already has a >>> 32 >>> bit / 64 bit installer. >>> >>> If I understand correctly, the options for doing this right now >>> are: >>> >>> 1) Minimal cost in time : ask Christophe nicely whether we can >>> distribute his binaries via the Numpy page >>> 2) Small cost in time / money : pay for licenses for Ondrej or me >>> or >>> someone to install the dependencies on my Berkeley machine / an >>> Amazon >>> image. >> >> In order not to leave this discussion without a resolution: >> >> Christophe - would you allow us to distribute your numpy binaries >> for >> 1.7 from the numpy sourceforge page? >> >> Cheers, >> >> Matthew > > > I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy > compiled with MSVC compilers and linked to Intel's MKL) for official > numpy releases.

Thank you - that is great.

> However: > > 1) There seems to be no real consensus and urge for doing this.

I certainly feel the urge and feel it strongly. As a packager for two or three projects myself, it's a royal pain having to tell someone to go to two different places for binaries depending on the number of bits of their Windows.

If you're relying on .exe installers on SF, then you have to send your users to more places than that probably. Really the separate installers are a poor alternative to the available scientific distributions. And the more packages you need as a user, the more annoying these separate installers are.

...

I think Chuck was worried about the time it would take to do it, and I think you've already solved this problem. Ralf was worried about Scipy - see below.

Not just Scipy - that would be my worry number one, but the same holds for all packages based on Numpy. You double the number of Windows installers that every single project needs to provide.

...

> Using a > free toolchain capable of building the whole scipy-stack would be > much > preferred.

That's true, but there seems general agreement this is not practical in the very near future.

> Several 64 bit Python distributions containing numpy-MKL are > already available, some for free.

You mean EPD and AnacondaCE? I don't think we should withhold easily available vanilla builds because they are also available in company-sponsored projects. Python.org provides windows builds even though ActiveState is free-as-in-beer.

If the company-sponsored bit bothers you, there's also a 64-bit Python(x,y) now.

I'm sure you've seen that the question 'where are the 64-bit installers' comes up often for Numpy.

It seems to me that we'd have to have a good reason not provide them. There will always be some number of people like me who like to install the various parts by hand, and don't like using large packages, free or or open or not. For example, I don't use macports. At the moment, the reasons I see you are giving are:

1) Then everyone would have to provide 64-bit binaries

Indeed. And since many packages won't do that because there's no free compilers, users just get stuck a bit later in the "installing the stack" process. I'm sure providing the binaries will help some people, but I expect it to cause as many problems as it solves.

Can you clarify the people you think will get stuck? I think I'm right in saying that anyone with a C extension should be able to build them against numpy, by installing the free (as-in-beer) MS tools? So do you just mean people needing a Fortran compiler? That's a small constituency, I think.

Off the top of my head there's SciPy and pymc...

We covered Scipy a while back in this thread. The proposal was that Christophe also supply these. Perhaps we can at least agree that the large majority of numpy-based projects that require compilation do not need FORTRAN. And if they do, then, as Chris said, they are likely to have sophisticated developers who are not likely to be confused by a new Windows build for numpy.

Anyway, I'm butting in because I wish this discussion could separate between the user perspective and the developer perspective.

FWIW,

1) From a user's perspective, I don't understand this either. If you are already using a closed source, not-free-as-in-beer operating system, why would you not use (or buy!) a closed source, not-free-as-in-beer Fortran compiler?

I think you might be asking whether Windows users care about having free (beer and or freedom) software? I think the answer to this is yes. As indeed OSX users care about having free software.

2) BUT, the argument I've seen that I can at least understand is that the release manager should be able to do a release using only open source tools (even using Wine instead of Windows) and not rely on a limited number of licenses. And that the release manager must be able to perform all the official builds directly.

I haven't heard that argument made. I've heard the general wish that we could build numpy with open-source tools. I haven't heard anyone assert that we won't release anything that can't be build with open-source tools. Python itself is not built with open-source tools, so I can't personally see the reason for this restriction. Perhaps you can explain it? I haven't heard the argument that the release manager should be able to make all the builds. That would be nice, obviously, but as an absolute rule it seems unnecessary. For example, is it your understanding that all the Python builds are made by the same person? Why would we want to be more restrictive than that? I don't know if anyone has estimated the number of people running Numpy via windows, but I notice from the download stats on Sourceforge, that there are around 800 downloads of OSX binary installers and over 4000 downloads of windows binaries. To be clear, what we seem to be heading for at the moment is this: * We have a experienced builder in Christophe building Windows 64 binaries for numpy and scipy * He has offered us at least Numpy and maybe Scipy * We are saying no to this zero-cost-to-us build because of one of the following reasons 1) Anyone using Windows 64 should be using EPD or AnacondaCE [1]. To enforce this we should not provide our own build 2) If we provide a Windows 64 build then other people will have to provide them too 3) We insist for some reason on only using open-source build tools 4) We insist for some reason that the release manager personally builds all the builds. This means that there is no way that I has a packager of - say - nipy - can give you a binary installer for Windows 64 bits without making you install one of (Christophe's builds, EPD, AnacondaCE). Which one should I build and installer for? All three? You have to give up your Python.org python to use my package? We are, because of this, leaving more than half of what appears to be our largest user base without a standard binary installer. I suppose, in 6 months time, someone will look for a Windows 64 bit installer, and they will not see one, and they think 'I wonder why?'. Then they will look at this thread. I wonder if they'll think we made the right decision or not. Cheers, Matthew [1] I can't find a 64-bit Python X Y but maybe one exists.

Christoph, On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke wrote: [...]

...

In order not to leave this discussion without a resolution:

Christophe - would you allow us to distribute your numpy binaries for 1.7 from the numpy sourceforge page?

Cheers,

Matthew

I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy compiled with MSVC compilers and linked to Intel's MKL) for official numpy releases.

However:

1) There seems to be no real consensus and urge for doing this. Using a free toolchain capable of building the whole scipy-stack would be much preferred. Several 64 bit Python distributions containing numpy-MKL are already available, some for free.

2) Releasing 64 bit numpy without matching scipy binaries would make little sense to me.

3) Please do not just redistribute the binaries from my website and declare them official. They might contain unreleased fixes from git master and pull requests that are needed for my work and other packages.

4) Numpy-MKL requires the Intel runtime DLLs (MKL is linked statically btw). I ship those with the installers and append the directory containing the DLLs to os.environ['PATH'] in numpy/__init__.py. This is a big no-no according to numpy developers. I don't agree. Anyway, those changes are not in the numpy source repositories.

5) My numpy-MKL installers are Python distutils bdist_wininst installers. That means if Python was installed for all users, installing numpy-MKL on Windows >6.0 will prompt for UAC elevation. Another no-no?

I think that all these things should be possible to fix so that the binary is acceptable for the official NumPy binary. How exactly do you build the binaries? I wasn't able to find the info at: http://www.lfd.uci.edu/~gohlke/pythonlibs/ Do you have some scripts to do that? Do you use PowerShell? Or you do it by hand by mouse and clicks in Visual Studio somehow? If I can figure out how to do these builds, I'll be happy to figure out how to automate it and then we can try to figure out a solution that works for NumPy. Ondrej

On Wed, Feb 6, 2013 at 10:41 PM, Matthew Brett wrote:

Hi,

On Wed, Feb 6, 2013 at 10:21 PM, Ondřej Čertík wrote:

...

On Wed, Feb 6, 2013 at 9:20 PM, Dag Sverre Seljebotn wrote:

...

On 02/07/2013 12:16 AM, Matthew Brett wrote: [...]

...

Can you clarify the people you think will get stuck? I think I'm right in saying that anyone with a C extension should be able to build them against numpy, by installing the free (as-in-beer) MS tools? So do you just mean people needing a Fortran compiler? That's a small constituency, I think.

Off the top of my head there's SciPy and pymc...

Anyway, I'm butting in because I wish this discussion could separate between the user perspective and the developer perspective.

FWIW,

1) From a user's perspective, I don't understand this either. If you are already using a closed source, not-free-as-in-beer operating system, why would you not use (or buy!) a closed source, not-free-as-in-beer Fortran compiler?

Indeed. Though I really have no clue on the Windows use cases. Maybe most Windows users don't want to compile anything, just use numpy and scipy from Python?

Well - yes - as a packager I really want to be able to provide a binary so my binary consumers don't have to have a C compiler installed. I imagine it's the same for all of us packagers out there.

...

...

2) BUT, the argument I've seen that I can at least understand is that the release manager should be able to do a release using only open source tools (even using Wine instead of Windows) and not rely on a limited number of licenses. And that the release manager must be able to perform all the official builds directly.

As the release manager, I really only have two requirements:

* I want to ssh in there from my Ubuntu * I want to automate the whole process

For Mac, linux and Wine I can do that. So I have just spend few hours browsing the net and it looks like that the combination of Windows PowerShell 2.0:

http://en.wikipedia.org/wiki/Windows_PowerShell

and some SSH server, there are quite a few, one commercial but free for one user one connection (perfect for me!):

http://www.powershellinside.com/powershell/ssh/

So if I understand the pages correctly, I can login there from linux, and then I use the PowerShell commands to script anything. It looks like I can even use my Fabric fabfiles with powershell:

https://gist.github.com/diyan/2850866

I can also use git with PowerShell:

http://windows.github.com/ http://haacked.com/archive/2011/12/13/better-git-with-powershell.aspx

So the final problem is how to execute MSVC and Fortran from Power Shell on Windows. These links might help for MSVC:

http://stackoverflow.com/questions/4398136/use-powershell-for-visual-studio-... http://geekswithblogs.net/dwdii/archive/2011/05/20/automating-a-visual-studi...

Finally, for Intel Fortran + powershell:

http://software.intel.com/en-us/forums/topic/284425

So I think it is all possible. If somebody can provide a machine with Windows, MSVC, PowerShell2.0, SSH server and some Fortran compiler, it should be possible for me to automate everything from Ubuntu using my Fabric files (https://github.com/certik/numpy-vendor).

Many many thanks for trying to solve this. I had really started to give up hope.

I think you will need a developer's license for MKL for Numpy. Ralf - any ETA for those?

I think I'm right in thinking you'll need a Fortran compiler for Scipy but not Numpy? Can we defer the Scipy build until after the Numpy build?

I will try to get you set up with ssh on my Windows 7 machine in case you can use it. It has the MS tools.

That would be amazing! If you can set me up with the Power Shell and some ssh server, I'll start playing with this right away. Ondrej

On 2/6/2013 10:35 PM, Ondřej Čertík wrote:

Christoph,

On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke wrote: [...]

...

...

In order not to leave this discussion without a resolution:

Christophe - would you allow us to distribute your numpy binaries for 1.7 from the numpy sourceforge page?

Cheers,

Matthew

I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy compiled with MSVC compilers and linked to Intel's MKL) for official numpy releases.

However:

1) There seems to be no real consensus and urge for doing this. Using a free toolchain capable of building the whole scipy-stack would be much preferred. Several 64 bit Python distributions containing numpy-MKL are already available, some for free.

2) Releasing 64 bit numpy without matching scipy binaries would make little sense to me.

3) Please do not just redistribute the binaries from my website and declare them official. They might contain unreleased fixes from git master and pull requests that are needed for my work and other packages.

4) Numpy-MKL requires the Intel runtime DLLs (MKL is linked statically btw). I ship those with the installers and append the directory containing the DLLs to os.environ['PATH'] in numpy/__init__.py. This is a big no-no according to numpy developers. I don't agree. Anyway, those changes are not in the numpy source repositories.

5) My numpy-MKL installers are Python distutils bdist_wininst installers. That means if Python was installed for all users, installing numpy-MKL on Windows >6.0 will prompt for UAC elevation. Another no-no?

I think that all these things should be possible to fix so that the binary is acceptable for the official NumPy binary.

How exactly do you build the binaries? I wasn't able to find the info at:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Do you have some scripts to do that? Do you use PowerShell? Or you do it by hand by mouse and clicks in Visual Studio somehow? If I can figure out how to do these builds, I'll be happy to figure out how to automate it and then we can try to figure out a solution that works for NumPy.

Ondrej

My development/build environment is listed at http://www.lfd.uci.edu/~gohlke/pythonlibs/#buildenv. Not that it helps much... Assuming that Windows 7|8 Pro 64 bit, Visual Studio 2008 Pro SP1 (with 64 bit compiler option), Visual Studio 2010 Pro, Intel Composer XE 2013, 64 bit CPython 2.6, 2.7, 3.2 and 3.3 are installed, the following batch script (no need for PowerShell or an IDE) should build 64 bit numpy-MKL installers when run from within the numpy source directory. I do not really use this script but the "secrets" are there. It can be extended for building eggs and MSIs, 32 bit, and automated testing. Probably not all the libraries listed in site.cfg are needed but this works for me also with scipy and other packages. @echo off setlocal set ICDIR=C:/Program Files (x86)/Intel/Composer XE rem Work around a bug in numpy distutils. Requires admin privileges fsutil hardlink create "%ICDIR%/mkl/lib/intel64/libiomp5md.lib" "%ICDIR%/compiler/lib/intel64/libiomp5md.lib" fsutil hardlink create "%ICDIR%/mkl/lib/intel64/libifportmd.lib" "%ICDIR%/compiler/lib/intel64/libifportmd.lib" rem Create site.cfg for static linking to 64 bit MKL echo [mkl] > site.cfg echo include_dirs = %ICDIR%/mkl/include >> site.cfg echo library_dirs = %ICDIR%/mkl/lib/intel64;%ICDIR%/compiler/lib/intel64

...

site.cfg echo mkl_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md,libifportmd site.cfg echo lapack_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md,libifportmd site.cfg rem Build installers using distutils rd /q /s build call C:\Python26\python.exe setup.py bdist_wininst --user-access-control=auto rd /q /s build call C:\Python27\python.exe setup.py bdist_wininst --user-access-control=auto rd /q /s build call C:\Python32\python.exe setup.py bdist_wininst --user-access-control=auto copy /Y build\py3k\dist\*.exe dist\ rd /q /s build call C:\Python33\python.exe setup.py bdist_wininst --user-access-control=auto copy /Y build\py3k\dist\*.exe dist\ rd /q /s build endlocal

-- Christoph

Hi, On Thu, Feb 7, 2013 at 10:47 AM, Ralf Gommers wrote:

On Thu, Feb 7, 2013 at 7:41 AM, Matthew Brett wrote:

...

I think you will need a developer's license for MKL for Numpy. Ralf - any ETA for those?

No, I'll have to ask again.

...

I think I'm right in thinking you'll need a Fortran compiler for Scipy but not Numpy?

Correct.

...

Can we defer the Scipy build until after the Numpy build?

That doesn't sound like a good idea to me.

...

I will try to get you set up with ssh on my Windows 7 machine in case you can use it. It has the MS tools.

As pointed out before by Robert, sharing the machine between you requires multiple licenses. Intel has promised us some licenses, but those are given to specific people.

Right. Ondrej has his own account on my machine, and it will be Ondrej's license, when it is available. Cheers, Matthew

On Thu, Feb 7, 2013 at 10:59 AM, Matthew Brett wrote:

Hi,

On Thu, Feb 7, 2013 at 10:47 AM, Ralf Gommers wrote:

...

On Thu, Feb 7, 2013 at 7:41 AM, Matthew Brett wrote:

...

I think you will need a developer's license for MKL for Numpy. Ralf - any ETA for those?

No, I'll have to ask again.

...

I think I'm right in thinking you'll need a Fortran compiler for Scipy but not Numpy?

Correct.

...

Can we defer the Scipy build until after the Numpy build?

That doesn't sound like a good idea to me.

I must say I'm a little confused as to how we're going to make the decisions here.

I'm sure you agree that there's an opposite argument to be made, and I would make it if I thought it would make a difference, but I'm losing faith in my ability to keep the discussion on track, and I don't know what to do about that.

Matthew I don't see any problem here. I agree with Ralf that we need to figure out how to build scipy with Fortran pretty much at the same time, to see if the solution is a viable solution. With Christoph help and experience, I am sure it can get done in a satisfactory way. Ondrej

On Thu, Feb 7, 2013 at 11:05 AM, Ralf Gommers wrote:

On Thu, Feb 7, 2013 at 7:59 PM, Matthew Brett wrote:

...

...

...

Can we defer the Scipy build until after the Numpy build?

That doesn't sound like a good idea to me.

I must say I'm a little confused as to how we're going to make the decisions here.

How about: attempt to reach consensus? David's concern on DLLs hasn't been addressed yet, nor has mine on packages being unavailable. I was actually still answering another of your emails, but I can't seem to reply fast enough.

Yep, we will need to address those.

...

I'm sure you agree that there's an opposite argument to be made, and I would make it if I thought it would make a difference, but I'm losing faith in my ability to keep the discussion on track, and I don't know what to do about that.

I don't see the problem. Before you offered to put in work. Ondrej is willing to help, so is Christoph. So why is it impossible to do Scipy builds?

I can see us getting to a solution here, but offering Numpy installers without Scipy ones is not a solution in my book.

Exactly. There is no problem here. Fortran needs to be working as a first class citizen. I personally use modern Fortran a lot. I've setup this page: http://fortran90.org/ with a relevant FAQ about binary compatibility: http://fortran90.org/src/faq.html#are-fortran-compilers-abi-compatible and based on how things work on Windows, I'll be happy to extend the information there. Ondrej

On Thu, Feb 7, 2013 at 8:38 PM, Matthew Brett wrote:

Hi,

On Thu, Feb 7, 2013 at 11:05 AM, Ralf Gommers wrote:

...

On Thu, Feb 7, 2013 at 7:59 PM, Matthew Brett wrote:

...

...

...

Can we defer the Scipy build until after the Numpy build?

That doesn't sound like a good idea to me.

I must say I'm a little confused as to how we're going to make the decisions here.

How about: attempt to reach consensus? David's concern on DLLs hasn't

been

...

addressed yet, nor has mine on packages being unavailable. I was actually still answering another of your emails, but I can't seem to reply fast enough.

Right - consensus is good - but at the moment I keep getting lost because the arguments seem to shift and get lost, and sometimes they are not made.

So, here is the summary as I understand it, please correct if I am wrong

I think we agree that:

1) Having a binary installer for numpy Windows 64 bit is desirable 2) It is desirable to have a matching binary installer for Scipy as soon as possible 3) It is preferable to build with free tools 4) It is acceptable to use non-free tools 5) The build will need to do some run-time linking to MKL and / or mingw 6) It is preferable that the build should be fully automated 7) It is preferable that one person can build all numpy / scipy builds

The points of potential disagreement are:

a) If we cannot build Scipy now, it may or may not be acceptable to release numpy now. I think it is, you (Ralf) think it isn't, we haven't discussed that. It may not come up. b) It may or may not be acceptable for someone other than Ondrej to be responsible for the Windows 64-bit builds. I think it should be, if necessary, we haven't really discussed that, it may not come up. c) It may or may not be acceptable for the build to be only partially automated. Ditto. d) It may or may not be acceptable to add the DLL directory to the PATH on numpy import. David says not, Christophe disagrees, we haven't really discussed that.

Is that right?

Good summary, looks complete to me. Ralf

Hi, On Thu, Feb 7, 2013 at 12:29 PM, Chris Barker - NOAA Federal wrote:

On Thu, Feb 7, 2013 at 11:38 AM, Matthew Brett wrote:

...

a) If we cannot build Scipy now, it may or may not be acceptable to release numpy now. I think it is, you (Ralf) think it isn't, we haven't discussed that. It may not come up.

Is anyone suggesting we hold the whole release for this? I fnot, then why would you say we shouldnt' put out a 64 bit Windows numpy binary because we dont' ahve a matching Scipy one? folks that need Scipy will have to find a way to built it themselves anyway, and folks that use numpy without scipy would have a nice binary to use -- That seems like value-added to me.

Right - me too - but we could hold off that question until Ondrej has had a chance to build both I guess?

...

d) It may or may not be acceptable to add the DLL directory to the PATH on numpy import. David says not, Christophe disagrees, we haven't really discussed that.

I like the "load them with ctypes" approach better, but I don't know how much effort it would be to implement that.

I'm still not totally clear if there is an issue with buildings other extensions tht rely on numpy/scipy -- can they be built with mingGW if numpy/scipy were built with MSVC (Or any other compiler for that matter).

I had the impression that the problem with Mingw64 was the need to load Mingw DLLs at run time - if that's the same for MSVC / MKL, maybe there's no advantage to using the MS tools? But I don't understand the issues very well. Cheers, Matthew

On Thu, Feb 7, 2013 at 12:29 PM, Chris Barker - NOAA Federal wrote:

On Thu, Feb 7, 2013 at 11:38 AM, Matthew Brett wrote:

...

a) If we cannot build Scipy now, it may or may not be acceptable to release numpy now. I think it is, you (Ralf) think it isn't, we haven't discussed that. It may not come up.

Is anyone suggesting we hold the whole release for this? I fnot, then

Just to make it clear, I do not plan to hold the whole release because of this. Previous releases also didn't have this official 64bit Windows binary, so there is no regression. Once we figure out how to create 64bit binaries, then we'll start uploading them. Ondrej

Hi Matthew, On Tue, Mar 26, 2013 at 5:48 PM, Matthew Brett wrote:

Hi Ondrej,

On Thu, Feb 7, 2013 at 3:18 PM, Ondřej Čertík wrote:

...

On Thu, Feb 7, 2013 at 12:29 PM, Chris Barker - NOAA Federal wrote:

...

On Thu, Feb 7, 2013 at 11:38 AM, Matthew Brett wrote:

...

a) If we cannot build Scipy now, it may or may not be acceptable to release numpy now. I think it is, you (Ralf) think it isn't, we haven't discussed that. It may not come up.

Is anyone suggesting we hold the whole release for this? I fnot, then

Just to make it clear, I do not plan to hold the whole release because of this. Previous releases also didn't have this official 64bit Windows binary, so there is no regression.

Once we figure out how to create 64bit binaries, then we'll start uploading them.

Did you make any progress with this? Worth making some notes? Anything we can do to help?

Unfortunately I've been too busy the last month to push this through, so right now I am just concentrating on getting 1.7.1 out of the door, as that is higher priority. I am starting a new job on Monday, so once things settle down, I should be able to get back to this. I will post notes once I get to this again. Ondrej