[Twisted-Python] Problem with osx10.10-py2.7 builder
Hi, Can someone check this builder: https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7 It is failing randomly in weird ways during builds. For example, in some cases it fails to install tox. In other cases, the tests terminate in some random spot. This builder is marked as "Required", so it is a bit annoying. -- Craig
Hi, On 25 March 2018 at 05:14, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
Can someone check this builder:
https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7
It is failing randomly in weird ways during builds. For example, in some cases it fails to install tox. In other cases, the tests terminate in some random spot.
I will try and take a look at it. If someone knows more about the slave running this build, please add the info on wiki at: https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves
This builder is marked as "Required", so it is a bit annoying.
My understanding from the previous discussion about the supported platforms is that any build is required. The osx is there as I reminder to run the buildbot tests and not to merge a PR with only Travis and Appveyor. I have reconfigured GitHub to require ubuntu16.04-py2.7-coverage as this is backed by 3 slaves. ----- We have the circle-ci macOS and for the next 2 days, we can use that :) We are already used 55% of the macOS time and Circle-CI was enalbled in trunk in the last 2 days...so I don't think that we can use Circle-CI for macOS for the long term. I will write them to see if we can get an exception or a discount. Otherwise, I plan to rent a macOS so that it can act as a second slave for osx10.10-py2.7 So far, the support from macincloud was not helpful or friendly, but I guess that we can just try their service for 1 month and see how it goes. -- Adi Roiban
On Mar 25, 2018, at 5:01 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi,
On 25 March 2018 at 05:14, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
Can someone check this builder:
https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7
It is failing randomly in weird ways during builds. For example, in some cases it fails to install tox. In other cases, the tests terminate in some random spot.
I will try and take a look at it.
Adi, myself, and Mark looked at it by turns and eventually I think we got it into a mostly working state. Builds are now passing. However, if you attempt to view the logs of a build in progress, it appears to fail with a purple exception. This actually looks like it's a bug in the build master which I'm a little confused about why it isn't affecting other builders.
If someone knows more about the slave running this build, please add the info on wiki at:
https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves <https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves>
Looks like you beat me to the punch: https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves/d3 <https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves/d3> Is there other information you need?
This builder is marked as "Required", so it is a bit annoying.
My understanding from the previous discussion about the supported platforms is that any build is required. The osx is there as I reminder to run the buildbot tests and not to merge a PR with only Travis and Appveyor.
I have reconfigured GitHub to require ubuntu16.04-py2.7-coverage as this is backed by 3 slaves.
Primarily it's required because it's the only reasonable way we have to verify that things work on macOS, which is by far our most popular developer operating system. I've reverted this configuration change, since it's more reasonable to give the reviewers some discretion about the fact that Travis is representative of Linux than that a random assortment of non-macOS buildbots are representative of macOS.
We have the circle-ci macOS and for the next 2 days, we can use that :) We are already used 55% of the macOS time and Circle-CI was enalbled in trunk in the last 2 days...so I don't think that we can use Circle-CI for macOS for the long term. I will write them to see if we can get an exception or a discount.
I would definitely like it if we could move to something hosted :).
Otherwise, I plan to rent a macOS so that it can act as a second slave for osx10.10-py2.7 So far, the support from macincloud was not helpful or friendly, but I guess that we can just try their service for 1 month and see how it goes.
Bummer. This seems like the sort of thing that we should have the Conservancy pay for, rather than you personally. Better yet, of course, to get a commercially interested macOS user to supply some hardware or mac cloud boxes or something. Alternately... The reason we're doing this at all is that Travis's macOS situation was pretty dismal when we last tried it. But then, the graphs on https://www.traviscistatus.com <https://www.traviscistatus.com/> at the time matched that experience: routinely there were build queues in the hundreds. Right now the "Backlog macOS Builds for Open Source Projects" shows a completely flat line, zero for the last 24 hours. It might be worth trying out Travis again. -glyph
On Mon, Mar 26, 2018 at 2:58 AM, Glyph <glyph@twistedmatrix.com> wrote:
Alternately...
The reason we're doing this at all is that Travis's macOS situation was pretty dismal when we last tried it. But then, the graphs on https://www.traviscistatus.com at the time matched that experience: routinely there were build queues in the hundreds. Right now the "Backlog macOS Builds for Open Source Projects" shows a completely flat line, zero for the last 24 hours.
It might be worth trying out Travis again.
FWIW, I moved another project's macOS CI from Travis to CircleCI about a month ago because the Travis macOS job queue was so long. Counting queue time, CircleCI finishes those jobs around 6x faster than TravisCI. Maybe things have improved on Travis but they're still pretty bad. Jean-Paul
On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Mon, Mar 26, 2018 at 2:58 AM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote: Alternately...
The reason we're doing this at all is that Travis's macOS situation was pretty dismal when we last tried it. But then, the graphs on https://www.traviscistatus.com <https://www.traviscistatus.com/> at the time matched that experience: routinely there were build queues in the hundreds. Right now the "Backlog macOS Builds for Open Source Projects" shows a completely flat line, zero for the last 24 hours.
It might be worth trying out Travis again.
FWIW, I moved another project's macOS CI from Travis to CircleCI about a month ago because the Travis macOS job queue was so long. Counting queue time, CircleCI finishes those jobs around 6x faster than TravisCI.
Maybe things have improved on Travis but they're still pretty bad.
That's a bummer, but thanks for providing the data point. -g
On Mon, Mar 26, 2018, at 10:39 PM, Glyph wrote:
On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:>> On Mon, Mar 26, 2018 at 2:58 AM, Glyph <glyph@twistedmatrix.com> wrote:>>> Alternately...
The reason we're doing this at all is that Travis's macOS situation was pretty dismal when we last tried it. But then, the graphs on https://www.traviscistatus.com[1] at the time matched that experience: routinely there were build queues in the hundreds. Right now the "Backlog macOS Builds for Open Source Projects" shows a completely flat line, zero for the last 24 hours.>>> It might be worth trying out Travis again.
FWIW, I moved another project's macOS CI from Travis to CircleCI about a month ago because the Travis macOS job queue was so long. Counting queue time, CircleCI finishes those jobs around 6x faster than TravisCI.>> Maybe things have improved on Travis but they're still pretty bad.
That's a bummer, but thanks for providing the data point.
-g
I guess I just needed to look during the PST workday... Links: 1. https://www.traviscistatus.com/
On 27 March 2018 at 23:54, Glyph <glyph@twistedmatrix.com> wrote:
On Mon, Mar 26, 2018, at 10:39 PM, Glyph wrote:
On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Mon, Mar 26, 2018 at 2:58 AM, Glyph <glyph@twistedmatrix.com> wrote:
Alternately...
The reason we're doing this at all is that Travis's macOS situation was pretty dismal when we last tried it. But then, the graphs on https://www.traviscistatus.com at the time matched that experience: routinely there were build queues in the hundreds. Right now the "Backlog macOS Builds for Open Source Projects" shows a completely flat line, zero for the last 24 hours.
It might be worth trying out Travis again.
FWIW, I moved another project's macOS CI from Travis to CircleCI about a month ago because the Travis macOS job queue was so long. Counting queue time, CircleCI finishes those jobs around 6x faster than TravisCI.
Maybe things have improved on Travis but they're still pretty bad.
That's a bummer, but thanks for providing the data point.
-g
I guess I just needed to look during the PST workday...
FYI buildbot/osx10.10-py2.7 is broken again --------- Circle-CI is pretty fast, but we are also burning the time pretty fast. We have used 90% of the time allocated between 19 March of 19 April. Regards, -- Adi Roiban
On Apr 2, 2018, at 7:13 AM, Adi Roiban <adi@roiban.ro> wrote:
FYI buildbot/osx10.10-py2.7 is broken again
FWIW, `pip` on OSX (perhaps others?) swallows the 403 error about TLS brownouts on pypi for the 1.0/1.1 ending next week. No error is raised, and you just get an empty match for installable versions of anything. (it needs to be run as `-vvvv` to see/know there is an actual error). If that is the issue, the 9.0.3 version of pip is compatible with the new TLS and upgrading to it will fix things. This broke my CI system a few weeks ago, and has done the same for everyone i know with a CI running on macs.
On Mon, Apr 2, 2018, at 4:13 AM, Adi Roiban wrote:
FYI buildbot/osx10.10-py2.7 is broken again
The Build master's configuration was updated outside of twisted-infra/braid. I've checked it in and brought the repo more in line with reality: https://github.com/twisted-infra/braid/commit/55402aaa0a8da3b8d6ee7377642878... I also have PR that should fix the intermittent failures: https://github.com/twisted-infra/braid/pull/250 If you're interested, please review that PR.
---------
Circle-CI is pretty fast, but we are also burning the time pretty fast. We have used 90% of the time allocated between 19 March of 19 April.
Regards, -- Adi Roiban _________________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Mark Williams mrw@enotuniq.org
Adi Roiban graciously reviewed the PR and I deployed the new master.cfg to buildbot.twistedmatrix.com. It looks like the macOS builder is now working (more) reliably. Please note that the required macOS build is now known as "buildbot/osx10.11-py2.7-coverage", where before it was "buildbot/osx10.10-py2.7-coverage". This name change reflects the updated operating system. On Wed, Apr 4, 2018, at 5:28 PM, Mark Williams wrote:
On Mon, Apr 2, 2018, at 4:13 AM, Adi Roiban wrote:
FYI buildbot/osx10.10-py2.7 is broken again
The Build master's configuration was updated outside of twisted-infra/ braid. I've checked it in and brought the repo more in line with reality:
https://github.com/twisted-infra/braid/commit/55402aaa0a8da3b8d6ee7377642878...
I also have PR that should fix the intermittent failures:
https://github.com/twisted-infra/braid/pull/250
If you're interested, please review that PR.
---------
Circle-CI is pretty fast, but we are also burning the time pretty fast. We have used 90% of the time allocated between 19 March of 19 April.
Regards, -- Adi Roiban _________________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Mark Williams mrw@enotuniq.org
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Mark Williams mrw@enotuniq.org
On Sun, Mar 25, 2018 at 5:01 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi,
On 25 March 2018 at 05:14, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
Can someone check this builder:
https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7
It is failing randomly in weird ways during builds. For example, in some cases it fails to install tox. In other cases, the tests terminate in some random spot.
I will try and take a look at it.
If someone knows more about the slave running this build, please add the info on wiki at:
https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves
Where is this MacOS builder? Is it in Rackspace, some cloud provider, or other? -- Craig
participants (6)
-
Adi Roiban -
Craig Rodrigues -
Glyph -
Jean-Paul Calderone -
jonathan vanasco -
Mark Williams