Thanks for the very helpful update, Oliver! I can take care of updating the following two repositories: https://github.com/MacPython/pywavelets-wheels <https://github.com/MacPython/pandas-wheels> https://github.com/pyFFTW/pyFFTW-wheels <https://github.com/MacPython/pandas-wheels> Neither of those are doing nightly builds, but both have been using wheels.scipy.org for release staging. My anaconda.org account is under: grlee77 switching to https://anaconda.org/multibuild-wheels-staging/ should be fine for our purposes On Wed, Feb 12, 2020 at 12:45 PM Tom Augspurger <tom.w.augspurger@gmail.com> wrote:
Thanks for working on this Olivier.
My Anaconda.org account is tomaugspurger.
I'll take care of updating https://github.com/MacPython/pandas-wheels over the next couple days.
On Wed, Feb 12, 2020 at 11:36 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Status update:
- I got a reply from Rackspace and and they agree to continue the discount through April, 2020. This should give us enough time to move operations. I also deleted many useless costly resources on that account so that the running cost for the coming month should not exceed $100/month even if we go beyond end of April.
- I already upgraded the https://github.com/MacPython/scikit-learn-wheels infrastructure to upload nightly builds and temporary release artifacts to anaconda.org, details below.
- OpenBLAS builds used for numpy and scipy will be pushed to github releases: https://github.com/MacPython/openblas-libs/issues/14
- We still need to update the MacPython/*-wheels for the rest of the projects but I can help prepare some pull requests.
Based on past observed CDN traffic on the Rackspace account (around 800 GB/month), it should be fine to use the free anaconda.org package hosting service to host the nightly builds for all the community projects that currently rely on http://wheels.scipy.org.
For scikit-learn, I decided to keep the release staging artifacts separated from the nightly build artifacts:
- https://anaconda.org/scipy-wheels-nightly/ as a shared host for all nightly builds - https://anaconda.org/scikit-learn-wheels-staging/ as a temporary store for scikit-learn wheels prior to final upload to pypi.org when making a release.
Here is the configuration in scikit-learn CI that does the wheel upload to anaconda.org:
https://github.com/MacPython/scikit-learn-wheels/blob/c0f96681684657dc94a7c0...
## Shared nightly build hosting
The scipy-wheels-nightly organization is meant to make it easier for numpy, scipy, pandas, scikit-learn and other projects with long build time to publish binaries for their development branch on a daily basis so that continuous integration can efficiently run their tests against the dev branch of their dependencies, so as to spot regressions, ASAP before making a release.
For instance, once numpy and scipy have been configured to upload nightly builds there it will be possible to reconfigure the scheduled tests of a project that depends on scikit-learn to run against the dev branch of the full stack using:
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn <https://pypi.anaconda.org/scipy-wheels-nightly/simplescikit-learn> pip install --no-build-isolation -e . pytest
If you are a maintainer of one of the projects that publishes nightly-builds on http://wheels.scipy.org, **please create an account on https://anaconda.org and reply to this email with your identifier** so that I can grant you permissions on this shared organization.
From there you will be able to generate tokens to use as secret variable in your CI servers. To use those access tokens in a CI setup will need to enable permissions for the token:
- Write access to API - Upload pypi packages
## Staging hosting for release artifacts
Each project is free to use its own staging area. Staging areas are just useful as a synchronization step in a multi-CI release automation setup. There is really no need to publish those wheels to end-users. The end users will grab the wheels of the released versions from the main https://pypi.org index once the release is done.
That being said, I plan to also create a default staging area for convenience, e.g.:
- https://anaconda.org/multibuild-wheels-staging/
for projects who do not really care and want to use the https://github.com/matthew-brett/multibuild scripts with minimal admin efforts.
Thanks for your attention, let me know if you have any comment on this plan.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev