Re: [SciPy-Dev] Discontinued Rackspace Open Compute Discount
Several CI services are free for open source projects although there are throughput limits https://about.gitlab.com https://github.com/features/actions https://travis-ci.org https://www.appveyor.com -----Original Message-----
From: Olivier Grisel <olivier.grisel@ensta.org> Sent: Feb 8, 2020 11:30 AM To: SciPy Developers List <scipy-dev@python.org> Subject: [SciPy-Dev] Discontinued Rackspace Open Compute Discount
Hi all,
I received an email end of January that I just discovered today telling me that the Rackspace Open Compute Discount grant we use to host http://wheels.scipy.org for nightly builds and release management of our community has been discontinued on December 31st 2019.
I would like to take down the Rackspace hosted cloudfiles as soon as possible but we need to find a replacement as this will break all the CIs that rely on nightly wheels and also all the wheel release infrastructure for several project in the ecosystem.
Apparently the fees for January and the beginning of February already sum to $936.51. This is a large cost because there were a bunch of unused left-over block volumes that I already deleted.
To host wheels for nightly builds and release automation, a possible alternatives include using the github release pages but they are not really designed for this kind of workflow.
Alternatively there is a PyPI repo hosting feature in Azure Pipelines in public preview but I am not sure how it works, if it's free for open source projects and if we could use it as a replacement for the Rackspace cloud storage we currently use to host http://wheels.scipy.org .
https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pypi?view=azure-devops&tabs=yaml
Important: if you have uploaded any important files to one of the cloud file storage / blob containers on Rackspace, please download a copy as soon as possible.
But please do not download things you do not really want to keep to avoid adding any unnecessary bandwidth costs. If you have file write permission on one of those containers, please delete the files you own and do not need anymore.
Feel free to join this room to discuss the matter interactively: https://gitter.im/scikit-learn/nightly-build-hosting
Feel free to reply to this email on the scipy-dev mailing list if you have suggestions for setting up a free replacement.
-- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
The problem is to find a common PyPI-style host for nightly builds that we would agree on. The scikit-learn CI publishes a wheels for the master branch on a nightly basis and uses the numpy and scipy wheels to tests itself against their master branch. For this we all use the http://wheels.scipy.org via the https://github.com/ogrisel/wheelhouse-uploader tool that uploads wheels to a shared rackspace file container, delete older dev wheels and update the index page if necessary. We do not want to publish nightly wheels to the main PyPI to avoid bloating it with files that are only useful for continuous integration and testing. In particular the main PyPI host would not allow for the automatic deletion of older nightly builds as far as I know. -- Olivier
Here is a tweet by Van Lindberg from the PSF who used to work at Rackspace: https://twitter.com/VanL/status/1225928953350774785 -- Olivier
FWIW, they gave NumFOCUS 2 more years of discount. On Sat, Feb 8, 2020 at 11:58 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Here is a tweet by Van Lindberg from the PSF who used to work at Rackspace:
https://twitter.com/VanL/status/1225928953350774785
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Hi, Can we use the NumFOCUS account for scientific Python wheels? Cheers, Matthew On Sat, Feb 8, 2020 at 7:52 PM Andy Ray Terrel <andy.terrel@gmail.com> wrote:
FWIW, they gave NumFOCUS 2 more years of discount.
On Sat, Feb 8, 2020 at 11:58 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Here is a tweet by Van Lindberg from the PSF who used to work at Rackspace:
https://twitter.com/VanL/status/1225928953350774785
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Well, this sounds like a mess! I'll try to follow here/gitter convo to see what exactly you want us to do from SciPy release side. On Sat, 8 Feb 2020 at 13:03, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Can we use the NumFOCUS account for scientific Python wheels?
Cheers,
Matthew
On Sat, Feb 8, 2020 at 7:52 PM Andy Ray Terrel <andy.terrel@gmail.com> wrote:
FWIW, they gave NumFOCUS 2 more years of discount.
On Sat, Feb 8, 2020 at 11:58 AM Olivier Grisel <olivier.grisel@ensta.org>
wrote:
Here is a tweet by Van Lindberg from the PSF who used to work at
Rackspace:
https://twitter.com/VanL/status/1225928953350774785
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
I was also looking for possibilities via GitHub actions. Here are a few pages helped me to cover some initial distance. https://help.github.com/en/actions/automating-your-workflow-with-github-acti... https://github.com/marketplace/actions/python-wheels-manylinux-build https://github.com/polm/fugashi/pull/5/files On Sun, Feb 9, 2020 at 5:07 AM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
Well, this sounds like a mess! I'll try to follow here/gitter convo to see what exactly you want us to do from SciPy release side.
On Sat, 8 Feb 2020 at 13:03, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Can we use the NumFOCUS account for scientific Python wheels?
Cheers,
Matthew
On Sat, Feb 8, 2020 at 7:52 PM Andy Ray Terrel <andy.terrel@gmail.com> wrote:
FWIW, they gave NumFOCUS 2 more years of discount.
On Sat, Feb 8, 2020 at 11:58 AM Olivier Grisel <
olivier.grisel@ensta.org> wrote:
Here is a tweet by Van Lindberg from the PSF who used to work at
Rackspace:
https://twitter.com/VanL/status/1225928953350774785
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Making wheel artefacts from a CI run is reasonably straightforward. The bigger issue is finding a wheelhouse to store the builds.
On Sun, Feb 9, 2020 at 2:02 AM Andrew Nelson <andyfaff@gmail.com> wrote:
Making wheel artefacts from a CI run is reasonably straightforward. The bigger issue is finding a wheelhouse to store the builds.
Yeah. And unfortunately Github Actions don't have any usable way to store artifacts from builds. (There is an artifact system that lets you pass files between steps within a single run, but there's no programmatic way to fetch those files afterwards.) And Github Releases do let you attach arbitrary files, but I don't know of any way to point pip at a Github Release. You might ping Ernest Durbin @ the PSF to see if they can help out. The PSF already has substantial cloud infrastructure and relationships with the big vendors (e.g. I think PyPI gets free storage in S3), so they might be able to set up another storage bucket pretty easily. -n -- Nathaniel J. Smith -- https://vorpus.org
Using Azure Artifacts feed seems to work: https://github.com/MacPython/scikit-learn-wheels/pull/23#issuecomment-583804... But apparently it will require project maintainers to create a new Azure account (it's possible to do so using your existing github credentials), create public project there and a feed whose content that can be publicly accessed via pip by anyone. The CI script. It's possible to configure a retention policy to automatically clean-up old nightly builds and release artifacts to stay below the limit of 2GB (which would be more than enough for release automation and nightly builds publishing). For scikit-learn we already use Azure Pipelines for most automated testing (pytest in PR) and we wanted to migrated our https://github.com/MacPython/scikit-learn-wheels release automation (based on https://github.com/matthew-brett/multibuild/ and backed by on travis and appveyor) to use it Azure Pipelines instead. Appveyor is too slow and Azure Pipelines is much faster for us. So migrating to Azure Artifacts to also host the nightly build wheelhouse would be quite natural in our case. The upload credentials is automatically handled via the TwineAuthenticate@1 task setup in the Azure Pipelines yaml configuration. However for projects that do not want to use Azure Pipelines for building, I cannot find how to generate a secret token to "twine upload" their wheels from a different CI to an Azure Artifact feed. Another alternative would be to use anaconda.org that can also host wheelhouse. The clean-up is not automatic but can be scripted with a cron task on the CI. anaconda uploads might be more flexible w.r.t. credentials for uploading from various CIs. -- Olivier
And unfortunately Github Actions don't have any usable way to store artifacts from builds.
This is a planned feature but it does not support PyPI-style, pip compatible wheelhouse just yet: https://github.com/features/packages -- Olivier
You might ping Ernest Durbin @ the PSF to see if they can help out. The PSF already has substantial cloud infrastructure and relationships with the big vendors (e.g. I think PyPI gets free storage in S3), so they might be able to set up another storage bucket pretty easily.
The problem is that we would need to automate the clean-up of old build artifacts. On Rackspace, the https://github.com/ogrisel/wheelhouse-uploader tool I developed would do that automatically (for nightly-builds) but honestly. I used Apache Libcloud so it might work almost out of the box with AWS S3 blob storage but I have never tested and honestly I am not interested in maintaining that tool if more standard alternatives based on twine can do handle retention logic automatically without custom development. Also, managing AWS upload credentials for a bunch of loosely coupled open source projects will be a mess (it was for the rackspace based solution, Matthew Brett was a SPOF with only myself as a high latency backup spare). I would rather have each project be autonomous in their ability to share and revoke upload credentials among maintainers. -- Olivier
Left field, I wonder if we could use something like test.pypi.org?
Left field, I wonder if we could use something like test.pypi.org?
Good point. Last time I checked it wasn't possible to programmatically delete old packages to avoid wasting resources for nightly builds. I cannot find a deletion action in the API docs: https://warehouse.readthedocs.io/api-reference/ But maybe test.pypi.org has some specific built-in retention policy that would work for us? I tried to google a bit and it's not how long it keeps artifacts around. Also I do not know if the test instance is meant to be reliable enough to share nightly-build artifacts between the continuous integration servers for numpy, scipy and their dependencies. If it's not available enough it might trigger annoying random CI failures. -- Olivier
Yes, but I recommend ya’ll only using it for the short term, as Rackspace sprung this so suddenly. I asked them to extend it so we didn’t owe them money, but my intention is to set it up so we can move servers off Rackspace. Additionally I have a bunch of AWS credits that we are still figuring out. So if you want to use AWS, we can set up an org and y’all will have full control. Finally if you do want to use Azure pipeline, Microsoft offered us some discounts but most teams I worked with had enough on the free their. — Andy On Sat, Feb 8, 2020 at 2:03 PM Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Can we use the NumFOCUS account for scientific Python wheels?
Cheers,
Matthew
On Sat, Feb 8, 2020 at 7:52 PM Andy Ray Terrel <andy.terrel@gmail.com> wrote:
FWIW, they gave NumFOCUS 2 more years of discount.
On Sat, Feb 8, 2020 at 11:58 AM Olivier Grisel <olivier.grisel@ensta.org>
wrote:
Here is a tweet by Van Lindberg from the PSF who used to work at
Rackspace:
https://twitter.com/VanL/status/1225928953350774785
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
I asked the question about using test.pypi.org for nightly builds here: https://discuss.python.org/t/publishing-nightly-builds-on-https-test-pypi-or... Note that the tensorflow project is abusing much more than we would: they upload 2.2GB worth of binaries every day just for the CPU version, on the main pypi.org instance (under the tf-nightly project name). If there was an automated way to setup a retention policy for dev wheels on the test.pypi.org warehouse that would be ideal from an operational standpoint point of view. -- Olivier
Paul Moore answered on discord to recommend us to use our own PEP 503 index (instead of test.pypi.org) to publish nightly builds. I think setting up a new shared organization on https://anaconda.org/ to shared the wheels of nightly builds for cython, numpy, scipy, pandas, scikit-learn, gensim,... would be the best way to go forward. It makes it possible to preserve the convenience of the shared nightly builds currently uploaded to http://wheels.scipy.org . anaconda.org does not have a built-in retention policy but the anaconda-client package provides a command line "anaconda remove" to delete individual files. So it should be possible to write a CRON job somewhere to clean-up old nightly builds (which is currently not possible to do on test.pypi.org). I have also investigated further if we could do the same on Azure Artifacts but it is more cumbersome: to share upload credentials to a shared feed to several projects' maintainer teams requires generating "Personal Access Tokens". Each token has a maximum validity limit of 1 year. We would have to reconfigure all the CIs at least once a year which is a bummer. For the name of the shared anaconda cloud organization, would "scipy-nightly" be fine? Or maybe "scipy-wheels-nightly" to be extra explicit? The intent would be to share this space with the aforementioned projects of the scipy ecosystem. For the second use case of a staging wheelhouse for (tagged) release automation, one could use test.scipy.org as it matches the intended use case of that service. -- Olivier
Status update: - I got a reply from Rackspace and and they agree to continue the discount through April, 2020. This should give us enough time to move operations. I also deleted many useless costly resources on that account so that the running cost for the coming month should not exceed $100/month even if we go beyond end of April. - I already upgraded the https://github.com/MacPython/scikit-learn-wheels infrastructure to upload nightly builds and temporary release artifacts to anaconda.org, details below. - OpenBLAS builds used for numpy and scipy will be pushed to github releases: https://github.com/MacPython/openblas-libs/issues/14 - We still need to update the MacPython/*-wheels for the rest of the projects but I can help prepare some pull requests. Based on past observed CDN traffic on the Rackspace account (around 800 GB/month), it should be fine to use the free anaconda.org package hosting service to host the nightly builds for all the community projects that currently rely on http://wheels.scipy.org. For scikit-learn, I decided to keep the release staging artifacts separated from the nightly build artifacts: - https://anaconda.org/scipy-wheels-nightly/ as a shared host for all nightly builds - https://anaconda.org/scikit-learn-wheels-staging/ as a temporary store for scikit-learn wheels prior to final upload to pypi.org when making a release. Here is the configuration in scikit-learn CI that does the wheel upload to anaconda.org: https://github.com/MacPython/scikit-learn-wheels/blob/c0f96681684657dc94a7c0... ## Shared nightly build hosting The scipy-wheels-nightly organization is meant to make it easier for numpy, scipy, pandas, scikit-learn and other projects with long build time to publish binaries for their development branch on a daily basis so that continuous integration can efficiently run their tests against the dev branch of their dependencies, so as to spot regressions, ASAP before making a release. For instance, once numpy and scipy have been configured to upload nightly builds there it will be possible to reconfigure the scheduled tests of a project that depends on scikit-learn to run against the dev branch of the full stack using: pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn pip install --no-build-isolation -e . pytest If you are a maintainer of one of the projects that publishes nightly-builds on http://wheels.scipy.org, **please create an account on https://anaconda.org and reply to this email with your identifier** so that I can grant you permissions on this shared organization.
From there you will be able to generate tokens to use as secret variable in your CI servers. To use those access tokens in a CI setup will need to enable permissions for the token:
- Write access to API - Upload pypi packages ## Staging hosting for release artifacts Each project is free to use its own staging area. Staging areas are just useful as a synchronization step in a multi-CI release automation setup. There is really no need to publish those wheels to end-users. The end users will grab the wheels of the released versions from the main https://pypi.org index once the release is done. That being said, I plan to also create a default staging area for convenience, e.g.: - https://anaconda.org/multibuild-wheels-staging/ for projects who do not really care and want to use the https://github.com/matthew-brett/multibuild scripts with minimal admin efforts. Thanks for your attention, let me know if you have any comment on this plan. -- Olivier
Thanks for working on this Olivier. My Anaconda.org account is tomaugspurger. I'll take care of updating https://github.com/MacPython/pandas-wheels over the next couple days. On Wed, Feb 12, 2020 at 11:36 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Status update:
- I got a reply from Rackspace and and they agree to continue the discount through April, 2020. This should give us enough time to move operations. I also deleted many useless costly resources on that account so that the running cost for the coming month should not exceed $100/month even if we go beyond end of April.
- I already upgraded the https://github.com/MacPython/scikit-learn-wheels infrastructure to upload nightly builds and temporary release artifacts to anaconda.org, details below.
- OpenBLAS builds used for numpy and scipy will be pushed to github releases: https://github.com/MacPython/openblas-libs/issues/14
- We still need to update the MacPython/*-wheels for the rest of the projects but I can help prepare some pull requests.
Based on past observed CDN traffic on the Rackspace account (around 800 GB/month), it should be fine to use the free anaconda.org package hosting service to host the nightly builds for all the community projects that currently rely on http://wheels.scipy.org.
For scikit-learn, I decided to keep the release staging artifacts separated from the nightly build artifacts:
- https://anaconda.org/scipy-wheels-nightly/ as a shared host for all nightly builds - https://anaconda.org/scikit-learn-wheels-staging/ as a temporary store for scikit-learn wheels prior to final upload to pypi.org when making a release.
Here is the configuration in scikit-learn CI that does the wheel upload to anaconda.org:
https://github.com/MacPython/scikit-learn-wheels/blob/c0f96681684657dc94a7c0...
## Shared nightly build hosting
The scipy-wheels-nightly organization is meant to make it easier for numpy, scipy, pandas, scikit-learn and other projects with long build time to publish binaries for their development branch on a daily basis so that continuous integration can efficiently run their tests against the dev branch of their dependencies, so as to spot regressions, ASAP before making a release.
For instance, once numpy and scipy have been configured to upload nightly builds there it will be possible to reconfigure the scheduled tests of a project that depends on scikit-learn to run against the dev branch of the full stack using:
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn <https://pypi.anaconda.org/scipy-wheels-nightly/simplescikit-learn> pip install --no-build-isolation -e . pytest
If you are a maintainer of one of the projects that publishes nightly-builds on http://wheels.scipy.org, **please create an account on https://anaconda.org and reply to this email with your identifier** so that I can grant you permissions on this shared organization.
From there you will be able to generate tokens to use as secret variable in your CI servers. To use those access tokens in a CI setup will need to enable permissions for the token:
- Write access to API - Upload pypi packages
## Staging hosting for release artifacts
Each project is free to use its own staging area. Staging areas are just useful as a synchronization step in a multi-CI release automation setup. There is really no need to publish those wheels to end-users. The end users will grab the wheels of the released versions from the main https://pypi.org index once the release is done.
That being said, I plan to also create a default staging area for convenience, e.g.:
- https://anaconda.org/multibuild-wheels-staging/
for projects who do not really care and want to use the https://github.com/matthew-brett/multibuild scripts with minimal admin efforts.
Thanks for your attention, let me know if you have any comment on this plan.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Hi Olivier, My Anaconda.org account is treddy On Wed, 12 Feb 2020 at 10:46, Tom Augspurger <tom.w.augspurger@gmail.com> wrote:
Thanks for working on this Olivier.
My Anaconda.org account is tomaugspurger.
I'll take care of updating https://github.com/MacPython/pandas-wheels over the next couple days.
On Wed, Feb 12, 2020 at 11:36 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Status update:
- I got a reply from Rackspace and and they agree to continue the discount through April, 2020. This should give us enough time to move operations. I also deleted many useless costly resources on that account so that the running cost for the coming month should not exceed $100/month even if we go beyond end of April.
- I already upgraded the https://github.com/MacPython/scikit-learn-wheels infrastructure to upload nightly builds and temporary release artifacts to anaconda.org, details below.
- OpenBLAS builds used for numpy and scipy will be pushed to github releases: https://github.com/MacPython/openblas-libs/issues/14
- We still need to update the MacPython/*-wheels for the rest of the projects but I can help prepare some pull requests.
Based on past observed CDN traffic on the Rackspace account (around 800 GB/month), it should be fine to use the free anaconda.org package hosting service to host the nightly builds for all the community projects that currently rely on http://wheels.scipy.org.
For scikit-learn, I decided to keep the release staging artifacts separated from the nightly build artifacts:
- https://anaconda.org/scipy-wheels-nightly/ as a shared host for all nightly builds - https://anaconda.org/scikit-learn-wheels-staging/ as a temporary store for scikit-learn wheels prior to final upload to pypi.org when making a release.
Here is the configuration in scikit-learn CI that does the wheel upload to anaconda.org:
https://github.com/MacPython/scikit-learn-wheels/blob/c0f96681684657dc94a7c0...
## Shared nightly build hosting
The scipy-wheels-nightly organization is meant to make it easier for numpy, scipy, pandas, scikit-learn and other projects with long build time to publish binaries for their development branch on a daily basis so that continuous integration can efficiently run their tests against the dev branch of their dependencies, so as to spot regressions, ASAP before making a release.
For instance, once numpy and scipy have been configured to upload nightly builds there it will be possible to reconfigure the scheduled tests of a project that depends on scikit-learn to run against the dev branch of the full stack using:
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn <https://pypi.anaconda.org/scipy-wheels-nightly/simplescikit-learn> pip install --no-build-isolation -e . pytest
If you are a maintainer of one of the projects that publishes nightly-builds on http://wheels.scipy.org, **please create an account on https://anaconda.org and reply to this email with your identifier** so that I can grant you permissions on this shared organization.
From there you will be able to generate tokens to use as secret variable in your CI servers. To use those access tokens in a CI setup will need to enable permissions for the token:
- Write access to API - Upload pypi packages
## Staging hosting for release artifacts
Each project is free to use its own staging area. Staging areas are just useful as a synchronization step in a multi-CI release automation setup. There is really no need to publish those wheels to end-users. The end users will grab the wheels of the released versions from the main https://pypi.org index once the release is done.
That being said, I plan to also create a default staging area for convenience, e.g.:
- https://anaconda.org/multibuild-wheels-staging/
for projects who do not really care and want to use the https://github.com/matthew-brett/multibuild scripts with minimal admin efforts.
Thanks for your attention, let me know if you have any comment on this plan.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Thanks for the very helpful update, Oliver! I can take care of updating the following two repositories: https://github.com/MacPython/pywavelets-wheels <https://github.com/MacPython/pandas-wheels> https://github.com/pyFFTW/pyFFTW-wheels <https://github.com/MacPython/pandas-wheels> Neither of those are doing nightly builds, but both have been using wheels.scipy.org for release staging. My anaconda.org account is under: grlee77 switching to https://anaconda.org/multibuild-wheels-staging/ should be fine for our purposes On Wed, Feb 12, 2020 at 12:45 PM Tom Augspurger <tom.w.augspurger@gmail.com> wrote:
Thanks for working on this Olivier.
My Anaconda.org account is tomaugspurger.
I'll take care of updating https://github.com/MacPython/pandas-wheels over the next couple days.
On Wed, Feb 12, 2020 at 11:36 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Status update:
- I got a reply from Rackspace and and they agree to continue the discount through April, 2020. This should give us enough time to move operations. I also deleted many useless costly resources on that account so that the running cost for the coming month should not exceed $100/month even if we go beyond end of April.
- I already upgraded the https://github.com/MacPython/scikit-learn-wheels infrastructure to upload nightly builds and temporary release artifacts to anaconda.org, details below.
- OpenBLAS builds used for numpy and scipy will be pushed to github releases: https://github.com/MacPython/openblas-libs/issues/14
- We still need to update the MacPython/*-wheels for the rest of the projects but I can help prepare some pull requests.
Based on past observed CDN traffic on the Rackspace account (around 800 GB/month), it should be fine to use the free anaconda.org package hosting service to host the nightly builds for all the community projects that currently rely on http://wheels.scipy.org.
For scikit-learn, I decided to keep the release staging artifacts separated from the nightly build artifacts:
- https://anaconda.org/scipy-wheels-nightly/ as a shared host for all nightly builds - https://anaconda.org/scikit-learn-wheels-staging/ as a temporary store for scikit-learn wheels prior to final upload to pypi.org when making a release.
Here is the configuration in scikit-learn CI that does the wheel upload to anaconda.org:
https://github.com/MacPython/scikit-learn-wheels/blob/c0f96681684657dc94a7c0...
## Shared nightly build hosting
The scipy-wheels-nightly organization is meant to make it easier for numpy, scipy, pandas, scikit-learn and other projects with long build time to publish binaries for their development branch on a daily basis so that continuous integration can efficiently run their tests against the dev branch of their dependencies, so as to spot regressions, ASAP before making a release.
For instance, once numpy and scipy have been configured to upload nightly builds there it will be possible to reconfigure the scheduled tests of a project that depends on scikit-learn to run against the dev branch of the full stack using:
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn <https://pypi.anaconda.org/scipy-wheels-nightly/simplescikit-learn> pip install --no-build-isolation -e . pytest
If you are a maintainer of one of the projects that publishes nightly-builds on http://wheels.scipy.org, **please create an account on https://anaconda.org and reply to this email with your identifier** so that I can grant you permissions on this shared organization.
From there you will be able to generate tokens to use as secret variable in your CI servers. To use those access tokens in a CI setup will need to enable permissions for the token:
- Write access to API - Upload pypi packages
## Staging hosting for release artifacts
Each project is free to use its own staging area. Staging areas are just useful as a synchronization step in a multi-CI release automation setup. There is really no need to publish those wheels to end-users. The end users will grab the wheels of the released versions from the main https://pypi.org index once the release is done.
That being said, I plan to also create a default staging area for convenience, e.g.:
- https://anaconda.org/multibuild-wheels-staging/
for projects who do not really care and want to use the https://github.com/matthew-brett/multibuild scripts with minimal admin efforts.
Thanks for your attention, let me know if you have any comment on this plan.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Tyler, Tom: I added you as co-owners of the scipy-wheels-nightly organization. Gregory, Tyler, Tom: I added you as co-owners of the multibuild-wheels-staging organization. -- Olivier
Following Olivier's lead I've observed our first automatic upload to anaconda.org from scipy-wheels from t his testing PR: https://github.com/MacPython/scipy-wheels/pull/69 I've restored the full build matrix & the cron requirement after confirming the upload worked to the target location using a secret key: https://anaconda.org/scipy-wheels-nightly/scipy/files I'll probably address the staging area for SciPy release wheels in a separate PR from this cron job for weekly uploads. Since our master branch wheels builds typically have some failures in the matrix for unrelated reasons it is complicated enough as it is. I did notice a 3.0 GB "free limit" I think---hopefully that doesn't become an issue. On Sat, 15 Feb 2020 at 03:11, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Tyler, Tom: I added you as co-owners of the scipy-wheels-nightly organization. Gregory, Tyler, Tom: I added you as co-owners of the multibuild-wheels-staging organization.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Quick followup on this: I spoke to the site manager for Anaconda.org and they raised the storage limit for the scipy-wheels-nightly org to 50GB, so I think you are set for a while. On Sun, Feb 16, 2020 at 11:03 PM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
Following Olivier's lead I've observed our first automatic upload to anaconda.org from scipy-wheels from t his testing PR: https://github.com/MacPython/scipy-wheels/pull/69
I've restored the full build matrix & the cron requirement after confirming the upload worked to the target location using a secret key: https://anaconda.org/scipy-wheels-nightly/scipy/files
I'll probably address the staging area for SciPy release wheels in a separate PR from this cron job for weekly uploads. Since our master branch wheels builds typically have some failures in the matrix for unrelated reasons it is complicated enough as it is.
I did notice a 3.0 GB "free limit" I think---hopefully that doesn't become an issue.
On Sat, 15 Feb 2020 at 03:11, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Tyler, Tom: I added you as co-owners of the scipy-wheels-nightly organization. Gregory, Tyler, Tom: I added you as co-owners of the multibuild-wheels-staging organization.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Mon, Feb 17, 2020 at 10:20 AM Stanley Seibert <sseibert@anaconda.com> wrote:
Quick followup on this: I spoke to the site manager for Anaconda.org and they raised the storage limit for the scipy-wheels-nightly org to 50GB, so I think you are set for a while.
Thanks Stan! And thanks Olivier, Tyler and everyone else who is helping to resolve this! Ralf
On Sun, Feb 16, 2020 at 11:03 PM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
Following Olivier's lead I've observed our first automatic upload to anaconda.org from scipy-wheels from t his testing PR: https://github.com/MacPython/scipy-wheels/pull/69
I've restored the full build matrix & the cron requirement after confirming the upload worked to the target location using a secret key: https://anaconda.org/scipy-wheels-nightly/scipy/files
I'll probably address the staging area for SciPy release wheels in a separate PR from this cron job for weekly uploads. Since our master branch wheels builds typically have some failures in the matrix for unrelated reasons it is complicated enough as it is.
I did notice a 3.0 GB "free limit" I think---hopefully that doesn't become an issue.
On Sat, 15 Feb 2020 at 03:11, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Tyler, Tom: I added you as co-owners of the scipy-wheels-nightly organization. Gregory, Tyler, Tom: I added you as co-owners of the multibuild-wheels-staging organization.
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Sat, Feb 15, 2020 at 5:11 AM Olivier Grisel <olivier.grisel@ensta.org> wrote:
Tyler, Tom: I added you as co-owners of the scipy-wheels-nightly organization. Gregory, Tyler, Tom: I added you as co-owners of the multibuild-wheels-staging organization.
Hi Olivier, I was previously added as a co-owner for multibuild-wheels-staging. Can you add me to scipy-wheels-nightly as well (username: grlee77)? (this is for the scikit-image team).
-- Olivier _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (12)
-
Andrew Nelson -
Andy Ray Terrel -
Gregory Lee -
Ilhan Polat -
Matthew Brett -
Nathaniel Smith -
Olivier Grisel -
Ralf Gommers -
Robert Lucente -
Stanley Seibert -
Tom Augspurger -
Tyler Reddy