[Twisted-Python] a farewell to buildbot
Thanks to our long-suffering contributors Adi Roiban and Kyle Altendorf, we now have macOS builds running on circleCI! Supposedly we have sufficient resources to actually run all our builds, too, without running out of CI juice before the end of each month :-). As such, I've changed our repository configuration to drop the old, somewhat outdated macOS buildbot, and replace it with the Circle CI infrastructure which should cover that platform. With this change, none of the buildbot builders define our gating-to-trunk continuous integration. This is a great thing for the project, as it means external contributors will be able to get a "this is acceptable to merge" green checkmark without ./admin/pr_as_branch or any other similar repo:write-person-requiring shenanigans. However, it also means that we are now spending a not-insignificant amount of contributor time maintaining a farm of machines that do tons of continuous integration work, which may not really be telling us anything interesting about Twisted's quality or correctness. I think it might be worth considering decommissioning buildbot.twistedmatrix.com <http://buildbot.twistedmatrix.com/> entirely, unless some of the vendors of the platforms and kernels covered there would like to step up to do some maintenance themselves. It's been a decade or so since Twisted was spotting regular regressions in Linux, FreeBSD or Darwin, so I think this style of build infrastructure may be a relic of a bygone era. For my part, I probably will start doing any contributions on my own fork, since that will mean I don't have to constantly kick random spurious RHEL7 buildbot failures to avoid getting a red "X" on my PRs. Furthermore, if we decom'd buildbot as software infrastructure, we'd still have a significant amount of cloud / hardware resources we could potentially throw at *other* problems facing the project which cloud CI doesn't cover as well, like SpeedCenter. So, do folks have any strong feelings, or would anyone like to volunteer to help with some aspect of this? As always: we don't have enough folks to keep up with the operational demands of twistedmatrix.com <http://twistedmatrix.com/>, so if you want to dev some ops or ops some infra, please speak up :). -g
I have been switching matrix.org to Buildkite's open source offering (where you bring your own builders). It might be an option for things like codespeed, which we still would want to be consistent. Other than that, I was taking a look at Azure Pipelines last year. I think we could get a decent amount of hosted, easy to maintain build infrastructure through Microsoft, which includes the potential to also ditch running our own Windows for testing. - Amber On Sat., 2 Mar. 2019, 22:33 Glyph, <glyph@twistedmatrix.com> wrote:
Thanks to our long-suffering contributors Adi Roiban and Kyle Altendorf, we now have macOS builds running on circleCI! Supposedly we have sufficient resources to actually run all our builds, too, without running out of CI juice before the end of each month :-).
As such, I've changed our repository configuration to drop the old, somewhat outdated macOS buildbot, and replace it with the Circle CI infrastructure which should cover that platform.
With this change, none of the buildbot builders define our gating-to-trunk continuous integration. This is a great thing for the project, as it means external contributors will be able to get a "this is acceptable to merge" green checkmark without ./admin/pr_as_branch or any other similar repo:write-person-requiring shenanigans.
However, it also means that we are now spending a not-insignificant amount of contributor time maintaining a farm of machines that do tons of continuous integration work, which may not really be telling us anything interesting about Twisted's quality or correctness. I think it might be worth considering decommissioning buildbot.twistedmatrix.com entirely, unless some of the vendors of the platforms and kernels covered there would like to step up to do some maintenance themselves. It's been a decade or so since Twisted was spotting regular regressions in Linux, FreeBSD or Darwin, so I think this style of build infrastructure may be a relic of a bygone era.
For my part, I probably will start doing any contributions on my own fork, since that will mean I don't have to constantly kick random spurious RHEL7 buildbot failures to avoid getting a red "X" on my PRs.
Furthermore, if we decom'd buildbot as software infrastructure, we'd still have a significant amount of cloud / hardware resources we could potentially throw at *other* problems facing the project which cloud CI doesn't cover as well, like SpeedCenter.
So, do folks have any strong feelings, or would anyone like to volunteer to help with some aspect of this? As always: we don't have enough folks to keep up with the operational demands of twistedmatrix.com, so if you want to dev some ops or ops some infra, please speak up :).
-g
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 2019-03-03 09:07, Chris Withers wrote:
What do you make of CircleCI?
I found it mostly pleasant but I'm expecting to be using Azure personally. It's the only one that I've worked with that covers what I think ought to be the basics. Linux/macOS/Windows, CPython 2,7/3.4+ (ok, maybe 3.5+...), and PyPy 2/3 (I had to download it but it wasn't too painful to get it working), and artifact storage. CircleCI was nice but it lacks Windows entirely and only has CPython 2/3 on macOS iirc. They do provide some nice templating features. AppVeyor lacks macOS (I haven't tried their Linux) and just one worker at a time. Travis now has the OSes covered and their matrixing is nice (until you do anything 'special' and it can't be used afaik) but only really supports Python on Linux and doesn't have artifact storage. All that said, I just started with Azure Pipelines this week and have only put one pure Python project on it so who knows what will turn up as I do more. https://dev.azure.com/altendky/exttr/_build/results?buildId=120 Cheers, -kyle
Hawkowl: I didn't realize you were involved in matrix. I love matrix! On Sun, Mar 3, 2019 at 9:40 AM Kyle Altendorf <sda@fstab.net> wrote:
On 2019-03-03 09:07, Chris Withers wrote:
What do you make of CircleCI?
I found it mostly pleasant but I'm expecting to be using Azure personally. It's the only one that I've worked with that covers what I think ought to be the basics. Linux/macOS/Windows, CPython 2,7/3.4+ (ok, maybe 3.5+...), and PyPy 2/3 (I had to download it but it wasn't too painful to get it working), and artifact storage.
CircleCI was nice but it lacks Windows entirely and only has CPython 2/3 on macOS iirc. They do provide some nice templating features. AppVeyor lacks macOS (I haven't tried their Linux) and just one worker at a time. Travis now has the OSes covered and their matrixing is nice (until you do anything 'special' and it can't be used afaik) but only really supports Python on Linux and doesn't have artifact storage.
All that said, I just started with Azure Pipelines this week and have only put one pure Python project on it so who knows what will turn up as I do more.
https://dev.azure.com/altendky/exttr/_build/results?buildId=120
Cheers, -kyle
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Justin Myles Holmes justinholmes.com thisisthebus.com github.com/jMyles/
Am 03/03/2019 um 7:32 schrieb Glyph:
So, do folks have any strong feelings, or would anyone like to volunteer to help with some aspect of this? As always: we don't have enough folks to keep up with the operational demands of twistedmatrix.com <http://twistedmatrix.com>, so if you want to dev some ops or ops some infra, please speak up :).
This together with the 1password email raises following question: Is there anything particular that needs more support? Amber's point about Buildkite's bring-your-own-builders offering sounds interesting, and if lack of Mensch-power is the only issue, maybe a call to arms in a somewhat more specific fashion would be in order? I mention because that would help with some ML readers who may think "no way I can help with that" and actually could have the expertise on specific bits that are needed. Also, listing needs could be a good exercise overall and it could be listed here: https://twistedmatrix.com/trac/wiki/Infrastructure PS: Since Amber is part of Matrix' team, I just realised the silliest thing: following Twisted's tradition of awesome/awful naming puns, Matrix' Homeserver reference implementation, synapse, is written on Twisted, and this project's website is... twistedmatrix.com (!) -- Evilham
On Mar 3, 2019, at 9:46 AM, Evilham <contact@evilham.com> wrote:
Am 03/03/2019 um 7:32 schrieb Glyph:
So, do folks have any strong feelings, or would anyone like to volunteer to help with some aspect of this? As always: we don't have enough folks to keep up with the operational demands of twistedmatrix.com <http://twistedmatrix.com>, so if you want to dev some ops or ops some infra, please speak up :).
This together with the 1password email raises following question: Is there anything particular that needs more support?
Thanks for asking!
Amber's point about Buildkite's bring-your-own-builders offering sounds interesting, and if lack of Mensch-power is the only issue, maybe a call to arms in a somewhat more specific fashion would be in order?
The bystander effect might indeed be a big reason few folks have stepped forward. Broadly speaking there are three areas that would be good to get help with, in rough order of importance (although they're all super important): reducing the amount of regular work it takes to keep things running setting up systems to get more concrete and immediate benefits out of our dogfooding Twisted and related technology keeping an eye on the running systems and reacting to alerts Projects that fall into bucket number 1 include: dockerizing everything so that instead of a long, elaborate fabric script (https://github.com/twisted-infra/braid/ <https://github.com/twisted-infra/braid/>) we can just `docker-compose up` to get something resembling our current infrastructure. these services include: the website the DNS service the mailman instance serving this list codespeed buildbot, if it keeps being A Thing the IRC bot (kenaan) that reminds us of the work that needs doing decomissioning Trac and replacing its workflows with a Github bot (note that we would want to preserve stuff like the regular dev-stats email, the IRC bot, https://twisted.reviews/ <https://twisted.reviews/> etc) adding an API to our DNS service so that we could use i.e. DNS-01 certificate provisioning to get TLS certificates for other services In the second bucket would come things like setting up Sentry to capture errors from all those services (web, dns, trac, etc) all of which have bits of Twisted running in them. In the third would of course be getting admin access to these machines and restarting things when all else has failed, so that hawkowl, mark, or myself doesn't need to do it.
I mention because that would help with some ML readers who may think "no way I can help with that" and actually could have the expertise on specific bits that are needed.
The main thing that is required here is the enthusiasm, energy, and spare time required to be responsible for some of this stuff, and to regularly dedicate a little energy to any or all of the above things. Right now, the operations work is drudgery because nobody has the energy to take it to the next level, and it's just about trying not to break things too badly. But if someone really put some time into it, we could potentially build some really cool event-driven demos with Twisted that could get people (and core developers!) really excited to develop more interesting features, and we have plenty of hardware to run such things.
Also, listing needs could be a good exercise overall and it could be listed here: https://twistedmatrix.com/trac/wiki/Infrastructure <https://twistedmatrix.com/trac/wiki/Infrastructure>
A good initial project for an interested party would be perhaps to start editing that page by reading Braid and this email.
PS: Since Amber is part of Matrix' team, I just realised the silliest thing: following Twisted's tradition of awesome/awful naming puns, Matrix' Homeserver reference implementation, synapse, is written on Twisted, and this project's website is... twistedmatrix.com (!)
I'm really hoping to bring back more of this. For example, the "twisted matrix" was originally so named because the sub-projects were twisted dot something, and these were termed "dot products". ;-). I hope this message was helpful! -g
participants (6)
-
Amber Brown -
Chris Withers -
Evilham -
Glyph -
Justin Myles Holmes -
Kyle Altendorf