Recent changes on the CI jobs run on pull requests
Hi,
Changes on CIs run on GitHub pull requests:
- Travis CI became mandatory
- Azure Pipelines is no longer mandatory
- Please watch Windows x64 job: I would like to make it mandatory in 2 weeks
--
I asked to make Azure Pipelines CI not mandatory on Python PRs since there were more and more random failures and no one was available to investigate. Also, sadly, currently Azure Pipelines has no way to only re-trigger a single failed job, but requires to close/re-open a PR to schedule all jobs. Sadly, some Python tests fail randomly and so sometimes another CI fails. Also, for backport PRs, closing a PR triggers miss-inlington which removes the branch which prevents to re-open the PR. A new backport PR must be created which is not convenient.
Until Azure Pipelines issues are fixed, I asked Mariatta to make Travis CI mandatory. In fact, I expected Travis CI to be mandatory, I never noticed that it was optional. I didn't know that only Azure Pipelines was mandatory!
--
I also asked Mariatta to make the GitHub Action ("GHA") Windows x64 job mandatory. GHA workflow jobs are skipped for "documentation only" changes, to not waste CI resources. GHA configuration used "paths-ignored". Sadly (again!?), if a job using "paths-ignored" is mandatory, a PR cannot be merged because the mandatory job is skipped and will never be run... Sadly, it even seems to be a known GitHub issue which is not going to be fixed soon!
I removed "paths-ignored" to always run jobs, to be able to make a job mandatory. Then Filipe Laíns found a way to add a new quick (around 15 seconds!) job just for check if a change is documentation only: the job result is then used to decide if build jobs must be skipped.
In short, the behavior is the same than previously: GHA build jobs are skipped on documentation only changes. The main difference is that it became possible to make a GHA job mandatory.
I backported the GHA workflow configuration to Python 3.8 and 3.7. Mariatta suggested to watch the Windows x64 job for 2 weeks to check if it's stable before making it mandatory.
Hopefully, in my experience, the Windows x64 job is (very) reliable.
--
By the way, the GHA macOS job fails randomly, I have no idea why. For example, I saw the job being cancelled after 5 minutes for an unknown reason. I read that a job can be cancelled if "fail fast" is used and another job fails. It doesn't seem like we use "fail fast" and others jobs passed successfully.
--
Links to CI changes.
Make Azure Pipelines optional on GitHub PRs: https://bugs.python.org/issue39837
Always run GitHub action jobs, even on documentation-only jobs: https://bugs.python.org/issue40548
Make one Windows CI job mandatory: https://discuss.python.org/t/make-one-windows-ci-job-mandatory/4047
Make Travis CI (and Windows x64 ?) mandatory: https://github.com/python/core-workflow/issues/365
Make Windows (x64) GitHub action mandatory on master branch: https://github.com/python/core-workflow/issues/368
--
It's so hard to get a bunch of reliable CIs on many platforms (Linux, Windows, macOS) and minimize the failure rate :-(
Victor
Night gathers, and now my watch begins. It shall not end until my death.
Hi,
FYI the Windows x64 CI is now mandatory on Python pull requests (on the master branch).
Victor
Le ven. 15 mai 2020 à 17:09, Victor Stinner <vstinner@python.org> a écrit :
Hi,
Changes on CIs run on GitHub pull requests:
- Travis CI became mandatory
- Azure Pipelines is no longer mandatory
- Please watch Windows x64 job: I would like to make it mandatory in 2 weeks
--
I asked to make Azure Pipelines CI not mandatory on Python PRs since there were more and more random failures and no one was available to investigate. Also, sadly, currently Azure Pipelines has no way to only re-trigger a single failed job, but requires to close/re-open a PR to schedule all jobs. Sadly, some Python tests fail randomly and so sometimes another CI fails. Also, for backport PRs, closing a PR triggers miss-inlington which removes the branch which prevents to re-open the PR. A new backport PR must be created which is not convenient.
Until Azure Pipelines issues are fixed, I asked Mariatta to make Travis CI mandatory. In fact, I expected Travis CI to be mandatory, I never noticed that it was optional. I didn't know that only Azure Pipelines was mandatory!
--
I also asked Mariatta to make the GitHub Action ("GHA") Windows x64 job mandatory. GHA workflow jobs are skipped for "documentation only" changes, to not waste CI resources. GHA configuration used "paths-ignored". Sadly (again!?), if a job using "paths-ignored" is mandatory, a PR cannot be merged because the mandatory job is skipped and will never be run... Sadly, it even seems to be a known GitHub issue which is not going to be fixed soon!
I removed "paths-ignored" to always run jobs, to be able to make a job mandatory. Then Filipe Laíns found a way to add a new quick (around 15 seconds!) job just for check if a change is documentation only: the job result is then used to decide if build jobs must be skipped.
In short, the behavior is the same than previously: GHA build jobs are skipped on documentation only changes. The main difference is that it became possible to make a GHA job mandatory.
I backported the GHA workflow configuration to Python 3.8 and 3.7. Mariatta suggested to watch the Windows x64 job for 2 weeks to check if it's stable before making it mandatory.
Hopefully, in my experience, the Windows x64 job is (very) reliable.
--
By the way, the GHA macOS job fails randomly, I have no idea why. For example, I saw the job being cancelled after 5 minutes for an unknown reason. I read that a job can be cancelled if "fail fast" is used and another job fails. It doesn't seem like we use "fail fast" and others jobs passed successfully.
--
Links to CI changes.
Make Azure Pipelines optional on GitHub PRs: https://bugs.python.org/issue39837
Always run GitHub action jobs, even on documentation-only jobs: https://bugs.python.org/issue40548
Make one Windows CI job mandatory: https://discuss.python.org/t/make-one-windows-ci-job-mandatory/4047
Make Travis CI (and Windows x64 ?) mandatory: https://github.com/python/core-workflow/issues/365
Make Windows (x64) GitHub action mandatory on master branch: https://github.com/python/core-workflow/issues/368
--
It's so hard to get a bunch of reliable CIs on many platforms (Linux, Windows, macOS) and minimize the failure rate :-(
Victor
Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
participants (1)
-
Victor Stinner