Proposal: stop providing official win32 downloads (for now)
Hi all, I'm wondering what people think of the idea of us (= numpy) stopping providing our "official" win32 builds (the "superpack installers" distributed on sourceforge) starting with the next release. These builds are: - low quality: they're linked to an old & untuned build of ATLAS, so linear algebra will be dramatically slower than builds using MKL or OpenBLAS. They're win32 only and will never support win64. They're using an ancient version of gcc. They will never support python 3.5 or later. - a dead end: there's a lot of work going on to solve the windows build problem, and hopefully we'll have something better in the short-to-medium-term future; but, any solution will involve throwing out the current system entirely and switching to a new toolchain, wheel-based distribution, etc. - a drain on our resources: producing these builds is time-consuming and finicky; I'm told that these builds alone are responsible for a large proportion of the energy spent preparing each release, and take away from other things that our release managers could be doing (e.g. QA and backporting fixes). So the idea would be that for 1.11, we create a 1.11 directory on sourceforge and upload one final file: a README explaining the situation, a pointer to the source releases on pypi, and some links to places where users can find better-supported windows builds (Gohlke's page, Anaconda, etc.). I think this would serve our users better than the current system, while also freeing up a drain on our resources. Thoughts? -n -- Nathaniel J. Smith -- http://vorpus.org
On Fri, Dec 18, 2015 at 9:12 AM, Nathaniel Smith <njs@pobox.com> wrote:
Hi Nathaniel, Speaking as a downstream library (Biopython) using the NumPy C API, we have to ensure binary compatibility with your releases. We've continued to produce our own Windows 32 bit installers - originally the .exe kind (from python setup.py bdist_wininst) but now also .msi (from python setup.py bdist_msi). However, in the absence of an official 64bit Windows NumPy installer we've simply pointed people at Chris Gohlke's stack http://www.lfd.uci.edu/~gohlke/pythonlibs/ and will likely also start to recommend using Anaconda. This means we don't have any comparable download metrics to gauge 32 bit vs 64 bit Windows usage, but personally I'm quite happy for NumPy to phase out their 32 bit Windows installers (and then we can do the same). I hope we can follow NumPy's lead with wheel distribution etc. Thanks, Peter
On Fri, Dec 18, 2015 at 2:12 AM, Nathaniel Smith <njs@pobox.com> wrote:
Once numpy-vendor is set up, preparing and running the builds take about fifteen minutes on my machine. That assumes familiarity with the process, a first time user will spend significantly more time. Most of the work in a release is keeping track of reported bugs and fixing them. Tracking deprecations and such also takes time.
What about beta releases? I have nothing against offloading part of the release process, but if we do, we need to determine how to coordinate it among the different parties, which might be something of a time sink in itself. Chuck
On Fri, Dec 18, 2015 at 5:55 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
+1 from me. Despite the number of downloads still being high, I don't think there's too much value in these binaries anymore. We have been recommending Anaconda/Canopy for a couple of years now, and that's almost always a much better option for users.
Well, it builds but the current setup is just broken. Try building a binary and running the tests - you should find that there's a segfault in the np.fromfile tests (see https://github.com/scipy/scipy/issues/5540). And that kind of thing is incredibly painful to debug and fix.
We need to ensure that the MSVC builds work. But that's not new, that was always necessary for a release. Christophe has always tested beta/rc releases which is super helpful, but we need to get Appveyor CI to work soon. Ralf
On Fri, Dec 18, 2015 at 2:51 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
An appveyor setup is a great idea. An appveyor build matrix with the various supported MSVC versions would do a lot more to prevent compatibility issues than periodically building installers with old versions of MinGW. The effort toward a MinGW-based build is valuable, but having a CI system test for MSVC compatibility will be valuable regardless of where things go with that. Best, -Ian
I believe that a lot can be learned from matplotlib's recent foray into appveyor. Don't hesitate to ask questions on our dev mailing list (I wasn't personally involved, so I don't know what was learned). Cheers! Ben Root On Fri, Dec 18, 2015 at 5:27 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Dec 18, 2015 at 3:27 PM Nathaniel Smith <njs@pobox.com> wrote:
I'll take a look at setting that up. On the other hand, getting everything working with the various MSVC versions isn't likely to be a smooth sailing process, so I can't guarantee anything. Best, -Ian
On Sat, Dec 19, 2015 at 12:07 AM, Ian Henriksen < insertinterestingnamehere@gmail.com> wrote:
This may also be helpful (also contains a link to a previous attempt to get numpy working on Appveyor): https://github.com/numpy/numpy-vendor/issues/6#issuecomment-147004444 Ralf
On Fri, Dec 18, 2015 at 1:51 PM, Ralf Gommers
+1 from me. Despite the number of downloads still being high, I don't think there's too much value in these binaries anymore.
If there are a lot of downloads, then there is value. At least until we have binary wheels on PyPi. What's up with that, by the way? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Dec 21, 2015 at 11:11 PM, Chris Barker <chris.barker@noaa.gov> wrote:
There's a good chance that many downloads are from unsuspecting users with a 64-bit Python, and they then just get an unhelpful "cannot find Python" error from the installer. At least until we have binary wheels on PyPi.
What's up with that, by the way?
I expect those to appear in 2016, built with MinGW-w64 and OpenBLAS. Ralf
On Mon, Dec 21, 2015 at 10:05 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
could be -- hard to know.
nice. Anyway, I do think it's important to have a "official" easy way to get numpy for pyton.org pythons. numpy does/can/should see a lot of use outside the "scientific computing" community. And people are wary of dependencies. people should be able to use numpy in their projects, without requiring that their users start all over with Anaconda or ??? The ideal is for "pip install" to "just work" -- sound sike we're getting there. BTW, we've been wary of putting a 32 bit wheel up 'cause of the whole "what processor features to require" issue, but if we think it's OK to drop the binary installer altogether, I can't see the harm in putting a 32 bit SSE2 wheel up. Any way to know how many people are running 32 bit Python on Windows these days?? -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Dec 22, 2015 at 7:11 PM, Chris Barker <chris.barker@noaa.gov> wrote:
I don't claim we are representative of the whole community, but as far as canopy is concerned, it is still a significant platform. That's the only 32 bit platform we still support (both linux and osx 32 bits were < 1 % of our downloads) David
On Tue, Dec 22, 2015 at 11:18 AM, David Cournapeau <cournape@gmail.com> wrote:
thanks -- I think that's a good data point -- presumably, people can grab 64 bit Canopy just as easily as 64 bit -- and get all the extra packages. And they still want, or think they do :-), 32 bit. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Dec 22, 2015 at 7:22 PM, Chris Barker <chris.barker@noaa.gov> wrote:
I recently updated the data on 32 and 64 bit windows from the Steam hardware survey here: https://github.com/numpy/numpy/wiki/Windows-versions The take-home is that about 12 percent of gamers have 32 bit Windows. It's easy to believe business users will use 32-bit more. Also, I believe that the default Windows Python.org installers are 32-bit, at least, that's what the filenames suggest when I try it now. Cheers, Matthew
On Tue, Dec 22, 2015 at 11:32 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
The take-home is that about 12 percent of gamers have 32 bit Windows. It's easy to believe business users will use 32-bit more.
yup -- I tend to think gamers are on the cutting edge.... Though I work on gov't which is very slow moving, and we're on 64 bit finally... Also, I believe that the default Windows Python.org installers are
32-bit, at least, that's what the filenames suggest when I try it now.
what is "default" -- when I go to pyton.org to downloads, I get: https://www.python.org/downloads/release/python-2711/ where they are both there with equal footing. though I'm running a Mac -- so it starts out suggesting a Mac download -- maybe if I was running Windows, I'd get a default. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Dec 22, 2015 at 7:38 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On a virtual Windows machine I just span up, the Python.org site gave me default buttons to download Python 3.5 or 2.7, and the linked installers look like they are 32-bit. I can also go to the full list where there is no preference for one over the other, Cheers, Matthew
On Tue, Dec 22, 2015 at 11:43 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
It's probably time for python.org to change that -- but this does mean that there will be people using 32 bit pytohn on windows purely by happenstance. so I think it's important to continue to support those folks. Again, 32 bit binary wheels on PyPi is probably the way to do it these days. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hi, On 12/22/2015 08:11 PM, Chris Barker wrote:
Any way to know how many people are running 32 bit Python on Windows these days??
Approximately 25% of total Winpython downloads are 32bit. Exact numbers depend on the release and python version. Python 2.7 support has been dropped already, last release with 2.7 was in October. More details on download rates (but unfortunately without absolute numbers) here: http://sourceforge.net/projects/winpython/files/ Sebastian -- python programming - mail server - photo - video - https://sebix.at To verify my cryptographic signature or send me encrypted mails, get my key at https://sebix.at/DC9B463B.asc and on public keyservers.
On Tue, Dec 22, 2015 at 11:11 AM, Chris Barker <chris.barker@noaa.gov> wrote:
FWIW, most ArcGIS user are probably using 32-bit Windows unless they view python as more than "just that thing with the chevron the nerd keeps telling me to learn."
On Tue, Dec 22, 2015 at 1:22 PM, Paul Hobson <pmhobson@gmail.com> wrote:
Sorry, I meant 32-bit python *on Windows*, since that's what Esri ships by default. I know how to make ArcWhatever use 64-bit python, but I'd say most don't. (And I do update the numpy that comes with Esri's python).
On Fri, Dec 18, 2015 at 9:12 AM, Nathaniel Smith <njs@pobox.com> wrote:
Hi Nathaniel, Speaking as a downstream library (Biopython) using the NumPy C API, we have to ensure binary compatibility with your releases. We've continued to produce our own Windows 32 bit installers - originally the .exe kind (from python setup.py bdist_wininst) but now also .msi (from python setup.py bdist_msi). However, in the absence of an official 64bit Windows NumPy installer we've simply pointed people at Chris Gohlke's stack http://www.lfd.uci.edu/~gohlke/pythonlibs/ and will likely also start to recommend using Anaconda. This means we don't have any comparable download metrics to gauge 32 bit vs 64 bit Windows usage, but personally I'm quite happy for NumPy to phase out their 32 bit Windows installers (and then we can do the same). I hope we can follow NumPy's lead with wheel distribution etc. Thanks, Peter
On Fri, Dec 18, 2015 at 2:12 AM, Nathaniel Smith <njs@pobox.com> wrote:
Once numpy-vendor is set up, preparing and running the builds take about fifteen minutes on my machine. That assumes familiarity with the process, a first time user will spend significantly more time. Most of the work in a release is keeping track of reported bugs and fixing them. Tracking deprecations and such also takes time.
What about beta releases? I have nothing against offloading part of the release process, but if we do, we need to determine how to coordinate it among the different parties, which might be something of a time sink in itself. Chuck
On Fri, Dec 18, 2015 at 5:55 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
+1 from me. Despite the number of downloads still being high, I don't think there's too much value in these binaries anymore. We have been recommending Anaconda/Canopy for a couple of years now, and that's almost always a much better option for users.
Well, it builds but the current setup is just broken. Try building a binary and running the tests - you should find that there's a segfault in the np.fromfile tests (see https://github.com/scipy/scipy/issues/5540). And that kind of thing is incredibly painful to debug and fix.
We need to ensure that the MSVC builds work. But that's not new, that was always necessary for a release. Christophe has always tested beta/rc releases which is super helpful, but we need to get Appveyor CI to work soon. Ralf
On Fri, Dec 18, 2015 at 2:51 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
An appveyor setup is a great idea. An appveyor build matrix with the various supported MSVC versions would do a lot more to prevent compatibility issues than periodically building installers with old versions of MinGW. The effort toward a MinGW-based build is valuable, but having a CI system test for MSVC compatibility will be valuable regardless of where things go with that. Best, -Ian
I believe that a lot can be learned from matplotlib's recent foray into appveyor. Don't hesitate to ask questions on our dev mailing list (I wasn't personally involved, so I don't know what was learned). Cheers! Ben Root On Fri, Dec 18, 2015 at 5:27 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Dec 18, 2015 at 3:27 PM Nathaniel Smith <njs@pobox.com> wrote:
I'll take a look at setting that up. On the other hand, getting everything working with the various MSVC versions isn't likely to be a smooth sailing process, so I can't guarantee anything. Best, -Ian
On Sat, Dec 19, 2015 at 12:07 AM, Ian Henriksen < insertinterestingnamehere@gmail.com> wrote:
This may also be helpful (also contains a link to a previous attempt to get numpy working on Appveyor): https://github.com/numpy/numpy-vendor/issues/6#issuecomment-147004444 Ralf
On Fri, Dec 18, 2015 at 1:51 PM, Ralf Gommers
+1 from me. Despite the number of downloads still being high, I don't think there's too much value in these binaries anymore.
If there are a lot of downloads, then there is value. At least until we have binary wheels on PyPi. What's up with that, by the way? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Dec 21, 2015 at 11:11 PM, Chris Barker <chris.barker@noaa.gov> wrote:
There's a good chance that many downloads are from unsuspecting users with a 64-bit Python, and they then just get an unhelpful "cannot find Python" error from the installer. At least until we have binary wheels on PyPi.
What's up with that, by the way?
I expect those to appear in 2016, built with MinGW-w64 and OpenBLAS. Ralf
On Mon, Dec 21, 2015 at 10:05 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
could be -- hard to know.
nice. Anyway, I do think it's important to have a "official" easy way to get numpy for pyton.org pythons. numpy does/can/should see a lot of use outside the "scientific computing" community. And people are wary of dependencies. people should be able to use numpy in their projects, without requiring that their users start all over with Anaconda or ??? The ideal is for "pip install" to "just work" -- sound sike we're getting there. BTW, we've been wary of putting a 32 bit wheel up 'cause of the whole "what processor features to require" issue, but if we think it's OK to drop the binary installer altogether, I can't see the harm in putting a 32 bit SSE2 wheel up. Any way to know how many people are running 32 bit Python on Windows these days?? -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Dec 22, 2015 at 7:11 PM, Chris Barker <chris.barker@noaa.gov> wrote:
I don't claim we are representative of the whole community, but as far as canopy is concerned, it is still a significant platform. That's the only 32 bit platform we still support (both linux and osx 32 bits were < 1 % of our downloads) David
On Tue, Dec 22, 2015 at 11:18 AM, David Cournapeau <cournape@gmail.com> wrote:
thanks -- I think that's a good data point -- presumably, people can grab 64 bit Canopy just as easily as 64 bit -- and get all the extra packages. And they still want, or think they do :-), 32 bit. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Dec 22, 2015 at 7:22 PM, Chris Barker <chris.barker@noaa.gov> wrote:
I recently updated the data on 32 and 64 bit windows from the Steam hardware survey here: https://github.com/numpy/numpy/wiki/Windows-versions The take-home is that about 12 percent of gamers have 32 bit Windows. It's easy to believe business users will use 32-bit more. Also, I believe that the default Windows Python.org installers are 32-bit, at least, that's what the filenames suggest when I try it now. Cheers, Matthew
On Tue, Dec 22, 2015 at 11:32 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
The take-home is that about 12 percent of gamers have 32 bit Windows. It's easy to believe business users will use 32-bit more.
yup -- I tend to think gamers are on the cutting edge.... Though I work on gov't which is very slow moving, and we're on 64 bit finally... Also, I believe that the default Windows Python.org installers are
32-bit, at least, that's what the filenames suggest when I try it now.
what is "default" -- when I go to pyton.org to downloads, I get: https://www.python.org/downloads/release/python-2711/ where they are both there with equal footing. though I'm running a Mac -- so it starts out suggesting a Mac download -- maybe if I was running Windows, I'd get a default. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Dec 22, 2015 at 7:38 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On a virtual Windows machine I just span up, the Python.org site gave me default buttons to download Python 3.5 or 2.7, and the linked installers look like they are 32-bit. I can also go to the full list where there is no preference for one over the other, Cheers, Matthew
On Tue, Dec 22, 2015 at 11:43 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
It's probably time for python.org to change that -- but this does mean that there will be people using 32 bit pytohn on windows purely by happenstance. so I think it's important to continue to support those folks. Again, 32 bit binary wheels on PyPi is probably the way to do it these days. -CHB
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hi, On 12/22/2015 08:11 PM, Chris Barker wrote:
Any way to know how many people are running 32 bit Python on Windows these days??
Approximately 25% of total Winpython downloads are 32bit. Exact numbers depend on the release and python version. Python 2.7 support has been dropped already, last release with 2.7 was in October. More details on download rates (but unfortunately without absolute numbers) here: http://sourceforge.net/projects/winpython/files/ Sebastian -- python programming - mail server - photo - video - https://sebix.at To verify my cryptographic signature or send me encrypted mails, get my key at https://sebix.at/DC9B463B.asc and on public keyservers.
On Tue, Dec 22, 2015 at 11:11 AM, Chris Barker <chris.barker@noaa.gov> wrote:
FWIW, most ArcGIS user are probably using 32-bit Windows unless they view python as more than "just that thing with the chevron the nerd keeps telling me to learn."
On Tue, Dec 22, 2015 at 1:22 PM, Paul Hobson <pmhobson@gmail.com> wrote:
Sorry, I meant 32-bit python *on Windows*, since that's what Esri ships by default. I know how to make ArcWhatever use 64-bit python, but I'd say most don't. (And I do update the numpy that comes with Esri's python).
participants (11)
-
Benjamin Root -
Charles R Harris -
Chris Barker -
David Cournapeau -
Ian Henriksen -
Matthew Brett -
Nathaniel Smith -
Paul Hobson -
Peter Cock -
Ralf Gommers -
Sebastian