Hey all, I work for edX (http://code.edx.org), and we include NumPy and SciPy in our software platform. We have a lot of tests for our code, and running them requires making a virtual environment and installing all of our dependencies each time — but NumPy and SciPy take a long time to compile. We’ve actually built Python wheel (http://pythonwheels.com) files for all of our dependencies, which we host on an AWS server: http://edx-wheels.s3-website-us-east-1.amazonaws.com It would be super convenient if we could get wheel files for NumPy and SciPy on PyPI (https://pypi.python.org/pypi), so that its easier for us to install them in our test environment — and I imagine it would help a whole lot of other people, as well! Who has the authorization to upload new packages to the NumPy and SciPy projects on PyPI, and how can I contribute our wheel files back to the community? Alternatively, how can I help the SciPy project build and distribute wheel files automatically, as part of the standard release process? Thanks for the info! -David Baumgold
On Mon, Apr 28, 2014 at 4:55 PM, David Baumgold <db@edx.org> wrote:
Hey all, I work for edX, and we include NumPy and SciPy in our software platform. We have a lot of tests for our code, and running them requires making a virtual environment and installing all of our dependencies each time — but NumPy and SciPy take a long time to compile. We’ve actually built Python wheel files for all of our dependencies, which we host on an AWS server: http://edx-wheels.s3-website-us-east-1.amazonaws.com It would be super convenient if we could get wheel files for NumPy and SciPy on PyPI, so that its easier for us to install them in our test environment — and I imagine it would help a whole lot of other people, as well! Who has the authorization to upload new packages to the NumPy and SciPy projects on PyPI, and how can I contribute our wheel files back to the community? Alternatively, how can I help the SciPy project build and distribute wheel files automatically, as part of the standard release process? Thanks for the info!
It looks like you only have Linux and OS X wheels for numpy and scipy. The general consensus, Python-community-wide, is that only noarch and Windows wheels should be put up on PyPI. The PyPI server even enforces this, at this present time. Linux and even OS X environments are a little too variable. The stumbling block for getting official Windows wheels up has been finding working, not-embarassingly-out-of-date accelerated BLAS and LAPACK libraries that work. If you look on the recent numpy-discussion threads on this subject, it appears that this will be resolved soonish. Or at least, there are people actively working on it, and you can expect regular Windows wheels to be uploaded to PyPI as soon as it is possible to do so. -- Robert Kern
On Mon, Apr 28, 2014 at 6:08 PM, Robert Kern <robert.kern@gmail.com> wrote:
Hey all, I work for edX, and we include NumPy and SciPy in our software platform. We have a lot of tests for our code, and running them requires making a virtual environment and installing all of our dependencies each time — but NumPy and SciPy take a long time to compile. We’ve actually built Python wheel files for all of our dependencies, which we host on an AWS server: http://edx-wheels.s3-website-us-east-1.amazonaws.com It would be super convenient if we could get wheel files for NumPy and SciPy on PyPI, so that its easier for us to install them in our test environment — and I imagine it would help a whole lot of other people, as well! Who has the authorization to upload new packages to the NumPy and SciPy projects on PyPI, and how can I contribute our wheel files back to the community? Alternatively, how can I help the SciPy project build and distribute wheel files automatically, as part of the standard release process? Thanks for
On Mon, Apr 28, 2014 at 4:55 PM, David Baumgold <db@edx.org> wrote: the
info!
It looks like you only have Linux and OS X wheels for numpy and scipy. The general consensus, Python-community-wide, is that only noarch and Windows wheels should be put up on PyPI. The PyPI server even enforces this, at this present time. Linux and even OS X environments are a little too variable.
For Linux that's correct, but for OS X we are going to put those on PyPi. Actually, for Numpy those are already there thanks to Matthew's efforts: https://pypi.python.org/pypi/numpy. For Scipy we plan to do the same for 0.14.0. Specifically for testing on TravisCI, AstroPy hosts a wheelhouse with Linux wheels that may be of use: https://www.mpia-hd.mpg.de/~robitaille/wheelhouse/ https://github.com/scipy/scipy/blob/master/.travis.yml Ralf
The stumbling block for getting official Windows wheels up has been finding working, not-embarassingly-out-of-date accelerated BLAS and LAPACK libraries that work. If you look on the recent numpy-discussion threads on this subject, it appears that this will be resolved soonish. Or at least, there are people actively working on it, and you can expect regular Windows wheels to be uploaded to PyPI as soon as it is possible to do so.
-- Robert Kern _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Mon, Apr 28, 2014 at 6:04 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Mon, Apr 28, 2014 at 6:08 PM, Robert Kern <robert.kern@gmail.com> wrote:
It looks like you only have Linux and OS X wheels for numpy and scipy. The general consensus, Python-community-wide, is that only noarch and Windows wheels should be put up on PyPI. The PyPI server even enforces this, at this present time. Linux and even OS X environments are a little too variable.
For Linux that's correct, but for OS X we are going to put those on PyPi. Actually, for Numpy those are already there thanks to Matthew's efforts: https://pypi.python.org/pypi/numpy. For Scipy we plan to do the same for 0.14.0.
Great! I didn't realize PyPI had expanded their policy to allow OS X uploads too. -- Robert Kern
Hi, On Mon, Apr 28, 2014 at 12:23 PM, Robert Kern <robert.kern@gmail.com> wrote:
On Mon, Apr 28, 2014 at 6:04 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Mon, Apr 28, 2014 at 6:08 PM, Robert Kern <robert.kern@gmail.com> wrote:
It looks like you only have Linux and OS X wheels for numpy and scipy. The general consensus, Python-community-wide, is that only noarch and Windows wheels should be put up on PyPI. The PyPI server even enforces this, at this present time. Linux and even OS X environments are a little too variable.
For Linux that's correct, but for OS X we are going to put those on PyPi. Actually, for Numpy those are already there thanks to Matthew's efforts: https://pypi.python.org/pypi/numpy. For Scipy we plan to do the same for 0.14.0.
Great! I didn't realize PyPI had expanded their policy to allow OS X uploads too.
Yes, that happened in January. There's a summary of current OSX wheel status here: https://github.com/MacPython/wiki/wiki/Spinning-wheels Cheers, Matthew
I've brought this up before, but it is probably worth repeating. conda provides a completely free and accessible method to manage binaries including NumPy and SciPy. We are working with the PyPA to make as many tools work with conda packages as possible, and get official recognition as a PyPA-approved approach to managing the integration problem with Python binaries. I had many discussions with Nick Coghlan from the PyPA at PyCon to help people understand the benefits / trade-offs of conda and other approaches. We built this system, and made it completely free to solve the exact problem you are having. It is another mechanism that works completely today and anyone can use it to manage your binary installations of NumPy and SciPy. You are free to grab the Anaconda-built NumPy or SciPy conda packages and host them on a server of your choosing -- or rely on our free server at repo.continuum.io. If you want to build your own NumPy and SciPy packages (and avoid the attribution requirement of Anaconda binaries), then building a conda package is very straight-forward and quite easy. We are working on improved documentation, but if you can build a "tree of binaries," you can package it in a conda .tar.gz file. In fact, the "conda pipbuild" command will take anything installable via pip and build a conda package from it. You can also use the freely available binstar.org service to host your binaries. As long as the binaries are freely available to everyone, then there will never be a charge for this service. Currently binstar.org is still in beta, but request an invite and you will get access pretty quickly. You can also host .whl packages at binstar.org (look for pypi.binstar.org). This system works very well, and it works for Windows, Mac, and Linux. If someone else would like to host a conda repository, the conda index command and a webserver is all you need. All of these tools are BSD licensed and developed on Github. It really is worth your while to check it out. It would be easy for edX to have their own distro that they define which could either be a channel on binstar.org or your own distribution hosted from your site. Either way. I don't have as much time as I'd like to participate on this mailing list, but I am still very interested in making sure *everyone* can get the numpy-stack installed as easily as possible and take advantage of all the work we have done in building binaries and building frameworks that solve the exact problems this community has. Best, -Travis On Mon, Apr 28, 2014 at 10:55 AM, David Baumgold <db@edx.org> wrote:
Hey all, I work for edX <http://code.edx.org>, and we include NumPy and SciPy in our software platform. We have a lot of tests for our code, and running them requires making a virtual environment and installing all of our dependencies each time -- but NumPy and SciPy take a long time to compile. We've actually built Python wheel <http://pythonwheels.com> files for all of our dependencies, which we host on an AWS server: http://edx-wheels.s3-website-us-east-1.amazonaws.com It would be super convenient if we could get wheel files for NumPy and SciPy on PyPI<https://pypi.python.org/pypi>, so that its easier for us to install them in our test environment -- and I imagine it would help a whole lot of other people, as well! Who has the authorization to upload new packages to the NumPy and SciPy projects on PyPI, and how can I contribute our wheel files back to the community? Alternatively, how can I help the SciPy project build and distribute wheel files automatically, as part of the standard release process? Thanks for the info! -David Baumgold
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
-- Travis Oliphant CEO Continuum Analytics, Inc. http://www.continuum.io
Hi, On Mon, Apr 28, 2014 at 8:55 AM, David Baumgold <db@edx.org> wrote:
Hey all, I work for edX, and we include NumPy and SciPy in our software platform. We have a lot of tests for our code, and running them requires making a virtual environment and installing all of our dependencies each time — but NumPy and SciPy take a long time to compile. We’ve actually built Python wheel files for all of our dependencies, which we host on an AWS server: http://edx-wheels.s3-website-us-east-1.amazonaws.com It would be super convenient if we could get wheel files for NumPy and SciPy on PyPI, so that its easier for us to install them in our test environment — and I imagine it would help a whole lot of other people, as well! Who has the authorization to upload new packages to the NumPy and SciPy projects on PyPI, and how can I contribute our wheel files back to the community? Alternatively, how can I help the SciPy project build and distribute wheel files automatically, as part of the standard release process? Thanks for the info!
As promised some time ago, I've put up scipy wheels for 0.13.3 on pypi. Assuming you've got Python.org Python, you should now be able to do: pip install numpy scipy matplotlib from the OSX command line (after upgrading pip to latest). Cheers, Matthew
Hi, On Mon, Apr 28, 2014 at 7:34 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Mon, Apr 28, 2014 at 8:55 AM, David Baumgold <db@edx.org> wrote:
Hey all, I work for edX, and we include NumPy and SciPy in our software platform. We have a lot of tests for our code, and running them requires making a virtual environment and installing all of our dependencies each time — but NumPy and SciPy take a long time to compile. We’ve actually built Python wheel files for all of our dependencies, which we host on an AWS server: http://edx-wheels.s3-website-us-east-1.amazonaws.com It would be super convenient if we could get wheel files for NumPy and SciPy on PyPI, so that its easier for us to install them in our test environment — and I imagine it would help a whole lot of other people, as well! Who has the authorization to upload new packages to the NumPy and SciPy projects on PyPI, and how can I contribute our wheel files back to the community? Alternatively, how can I help the SciPy project build and distribute wheel files automatically, as part of the standard release process? Thanks for the info!
As promised some time ago, I've put up scipy wheels for 0.13.3 on pypi.
Assuming you've got Python.org Python, you should now be able to do:
pip install numpy scipy matplotlib
from the OSX command line (after upgrading pip to latest).
Automated tests on a clean 10.6 machine showing tests passing with pypi wheels: http://nipy.bic.berkeley.edu/builders/scipy-2.7.6-wheel/builds/6 http://nipy.bic.berkeley.edu/builders/scipy-3.3.5-wheel/builds/0 http://nipy.bic.berkeley.edu/builders/scipy-3.4.0-wheel/builds/0 The tests start with nothing but the system python, install python.org python, install wheels into a virtualenv and run the scipy tests. The test machine has no compilers installed. Enjoy, Matthew
participants (5)
-
David Baumgold -
Matthew Brett -
Ralf Gommers -
Robert Kern -
Travis Oliphant