[Numpy-discussion] Any plans for windows 64-bit installer for 1.7?

Thu Feb 7 01:10:10 EST 2013

Hi,

On Wed, Feb 6, 2013 at 9:20 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 02/07/2013 12:16 AM, Matthew Brett wrote:
>> Hi,
>>
>> On Wed, Feb 6, 2013 at 1:36 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>
>>>
>>>
>>> On Wed, Feb 6, 2013 at 8:46 PM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Wed, Feb 6, 2013 at 10:48 AM, Ralf Gommers <ralf.gommers at gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 6, 2013 at 1:32 AM, Matthew Brett <matthew.brett at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Tue, Feb 5, 2013 at 3:04 PM, Christoph Gohlke <cgohlke at uci.edu>
>>>>>> wrote:
>>>>>>> On 2/5/2013 10:51 AM, Matthew Brett wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Mon, Feb 4, 2013 at 5:09 PM, Matthew Brett
>>>>>>>> <matthew.brett at gmail.com>
>>>>>>>> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Mon, Feb 4, 2013 at 3:46 PM, Charles R Harris
>>>>>>>>> <charlesr.harris at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 4, 2013 at 4:04 PM, Robert Kern
>>>>>>>>>> <robert.kern at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 4, 2013 at 10:38 PM, Matthew Brett
>>>>>>>>>>> <matthew.brett at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 4, 2013 at 1:15 PM, Ralf Gommers
>>>>>>>>>>>> <ralf.gommers at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> MSVC + Intel Fortran + MKL, yes. But those aren't free. So how
>>>>>>>>>>>>> can
>>>>>>>>>>>>> you
>>>>>>>>>>>>> provide an Amazon image for those?
>>>>>>>>>>>>
>>>>>>>>>>>> You can make an image that is not public, I guess.   I suppose
>>>>>>>>>>>> anyone
>>>>>>>>>>>> who uses the image would have to have their own licenses for the
>>>>>>>>>>>> Intel
>>>>>>>>>>>> stuff?   Does anyone have experience of this?
>>>>>>>>>>>
>>>>>>>>>>> You need to purchase one license per developer:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://software.intel.com/en-us/articles/intel-math-kernel-library-licensing-faq#eula1
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think 64 bits on windows is best pushed off to 1.7.1 or 1.8. It
>>>>>>>>>> would be a
>>>>>>>>>> bit much to get it implemented in the next week or two.
>>>>>>>>>
>>>>>>>>> The problem with not providing these binaries is that they are at
>>>>>>>>> the
>>>>>>>>> bottom of everyone's stack, so a delay in numpy holds everyone
>>>>>>>>> back.
>>>>>>>>>
>>>>>>>>> I can't find completely convincing stats, but it looks as though 64
>>>>>>>>> bit windows 7 is now the most common version of Windows, at least
>>>>>>>>> for
>>>>>>>>> Gamers [1] around now, and it was getting that way for everyone in
>>>>>>>>> 2010 [2].
>>>>>>>>>
>>>>>>>>> It don't think it reflects well on on us that we don't appear to
>>>>>>>>> support 64 bits out of the box; just for example, R already has a
>>>>>>>>> 32
>>>>>>>>> bit / 64 bit installer.
>>>>>>>>>
>>>>>>>>> If I understand correctly, the options for doing this right now
>>>>>>>>> are:
>>>>>>>>>
>>>>>>>>> 1) Minimal cost in time : ask Christophe nicely whether we can
>>>>>>>>> distribute his binaries via the Numpy page
>>>>>>>>> 2) Small cost in time / money : pay for licenses for Ondrej or me
>>>>>>>>> or
>>>>>>>>> someone to install the dependencies on my Berkeley machine / an
>>>>>>>>> Amazon
>>>>>>>>> image.
>>>>>>>>
>>>>>>>> In order not to leave this discussion without a resolution:
>>>>>>>>
>>>>>>>> Christophe - would you allow us to distribute your numpy binaries
>>>>>>>> for
>>>>>>>> 1.7 from the numpy sourceforge page?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Matthew
>>>>>>>
>>>>>>>
>>>>>>> I am OK with providing 64 bit "numpy-MKL" binaries (that is numpy
>>>>>>> compiled with MSVC compilers and linked to Intel's MKL) for official
>>>>>>> numpy releases.
>>>>>>
>>>>>> Thank you - that is great.
>>>>>>
>>>>>>> However:
>>>>>>>
>>>>>>> 1) There seems to be no real consensus and urge for doing this.
>>>>>>
>>>>>> I certainly feel the urge and feel it strongly. As a packager for two
>>>>>> or three projects myself, it's a royal pain having to tell someone to
>>>>>> go to two different places for binaries depending on the number of
>>>>>> bits of their Windows.
>>>>>
>>>>>
>>>>> If you're relying on .exe installers on SF, then you have to send your
>>>>> users
>>>>> to more places than that probably. Really the separate installers are a
>>>>> poor
>>>>> alternative to the available scientific distributions. And the more
>>>>> packages
>>>>> you need as a user, the more annoying these separate installers are.
>>>>>
>>>>>>
>>>>>>   I think Chuck was worried about the time it
>>>>>> would take to do it, and I think you've already solved this problem.
>>>>>> Ralf was worried about Scipy - see below.
>>>>>
>>>>>
>>>>> Not just Scipy - that would be my worry number one, but the same holds
>>>>> for
>>>>> all packages based on Numpy. You double the number of Windows installers
>>>>> that every single project needs to provide.
>>>>>
>>>>>>
>>>>>>
>>>>>>> Using a
>>>>>>> free toolchain capable of building the whole scipy-stack would be
>>>>>>> much
>>>>>>> preferred.
>>>>>>
>>>>>> That's true, but there seems general agreement this is not practical
>>>>>> in the very near future.
>>>>>>
>>>>>>> Several 64 bit Python distributions containing numpy-MKL are
>>>>>>> already available, some for free.
>>>>>>
>>>>>> You mean EPD and AnacondaCE?   I don't think we should withhold easily
>>>>>> available vanilla builds because they are also available in
>>>>>> company-sponsored projects.  Python.org provides windows builds even
>>>>>> though ActiveState is free-as-in-beer.
>>>>>
>>>>>
>>>>> If the company-sponsored bit bothers you, there's also a 64-bit
>>>>> Python(x,y)
>>>>> now.
>>>>
>>>> I'm sure you've seen that the question 'where are the 64-bit
>>>> installers' comes up often for Numpy.
>>>>
>>>> It seems to me that we'd have to have a good reason not provide them.
>>>> There will always be some number of people like me who like to install
>>>> the various parts by hand, and don't like using large packages, free
>>>> or or open or not.  For example, I don't use macports.  At the moment,
>>>> the reasons I see you are giving are:
>>>>
>>>> 1) Then everyone would have to provide 64-bit binaries
>>>
>>>
>>> Indeed. And since many packages won't do that because there's no free
>>> compilers, users just get stuck a bit later in the "installing the stack"
>>> process. I'm sure providing the binaries will help some people, but I expect
>>> it to cause as many problems as it solves.
>>
>> Can you clarify the people you think will get stuck?  I think I'm
>> right in saying that anyone with a C extension should be able to build
>> them against numpy, by installing the free (as-in-beer) MS tools?  So
>> do you just mean people needing a Fortran compiler?  That's a small
>> constituency, I think.
>
> Off the top of my head there's SciPy and pymc...

We covered Scipy a while back in this thread.  The proposal was that
Christophe also supply these.

Perhaps we can at least agree that the large majority of numpy-based
projects that require compilation do not need FORTRAN.  And if they
do, then, as Chris said, they are likely to have sophisticated
developers who are not likely to be confused by a new Windows build
for numpy.

> Anyway, I'm butting in because I wish this discussion could separate
> between the user perspective and the developer perspective.
>
> FWIW,
>
> 1) From a user's perspective, I don't understand this either. If you are
> already using a closed source, not-free-as-in-beer operating system, why
> would you not use (or buy!) a closed source, not-free-as-in-beer Fortran
> compiler?

I think you might be asking whether Windows users care about having
free (beer and or freedom) software?   I think the answer to this is
yes.  As indeed OSX users care about having free software.

> 2) BUT, the argument I've seen that I can at least understand is that
> the release manager should be able to do a release using only open
> source tools (even using Wine instead of Windows) and not rely on a
> limited number of licenses. And that the release manager must be able to
> perform all the official builds directly.

I haven't heard that argument made.  I've heard the general wish that
we could build numpy with open-source tools.  I haven't heard anyone
assert that we won't release anything that can't be build with
open-source tools.  Python itself is not built with open-source tools,
so I can't personally see the reason for this restriction.   Perhaps
you can explain it?

I haven't heard the argument that the release manager should be able
to make all the builds.  That would be nice, obviously, but as an
absolute rule it seems unnecessary.   For example, is it your
understanding that all the Python builds are made by the same person?
Why would we want to be more restrictive than that?

I don't know if anyone has estimated the number of people running
Numpy via windows, but I notice from the download stats on
Sourceforge, that there are around 800 downloads of OSX binary
installers and over 4000 downloads of windows binaries.

To be clear, what we seem to be heading for at the moment is this:

* We have a experienced builder in Christophe building Windows 64
binaries for numpy and scipy
* He has offered us at least Numpy and maybe Scipy
* We are saying no to this zero-cost-to-us build because of one of the
following reasons

1) Anyone using Windows 64 should be using EPD or AnacondaCE [1].  To
enforce this we should not provide our own build
2) If we provide a Windows 64 build then other people will have to
provide them too
3) We insist for some reason on only using open-source build tools
4) We insist for some reason that the release manager personally
builds all the builds.

This means that there is no way that I has a packager of - say - nipy
- can give you a binary installer for Windows 64 bits without making
you install one of (Christophe's builds, EPD, AnacondaCE).   Which one
should I build and installer for?  All three?  You have to give up
your Python.org python to use my package?

We are, because of this, leaving more than half of what appears to be
our largest user base without a standard binary installer.

I suppose, in 6 months time, someone will look for a Windows 64 bit
installer, and they will not see one, and they think 'I wonder why?'.
Then they will look at this thread.  I wonder if they'll think we made
the right decision or not.

Cheers,

Matthew

[1] I can't find a 64-bit Python X Y but maybe one exists.