[Twisted-Python] operations: baby steps. buildbot move?
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
In the spirit of having more of our discussions of what's going on with the operations of twistedmatrix.com <http://twistedmatrix.com/> happening out in the open so that folks can jump in and help, we are going to start discussing this stuff on the public mailing list instead of in private side discussions or the administrative list. Credentials and other operationally sensitive things will of course still be sent via private GPG-encrypted messages but general "what's going on" stuff should be here. Right now we have a creaky old machine (dornkirk) running in a datacenter operated by Tummy. Although they have been generous with discounts, we do have actually free hardware elsewhere, including some sponsored by my employer (Rackspace). I would like to get everything off of Tummy's hardware for several reasons: we actually have to pay for tummy's hosting, which is a drain (albeit a minor one) on our very taxed resources from the Conservancy. (hey slackers: get out your credit cards and head to the sponsor form twistedmatrix.com <http://twistedmatrix.com/> or the site is literally going to go down one day). by co-locating so many services we can't upgrade the host operating system as often as we'd like without fear of breaking something else buildbot's TLS cert is expired and I'd like to use Let's Encrypt to get a new one, but the way the web server is configured makes this a bit tricky; it would be easier if it lived on its own host. the buildmaster is slow and it would be nice if it were on faster hardware given that many of our builders themselves are running in the cloud, we'd have lower latency if the buildmaster actually ran there So at some point in the coming week I'd like to just move exactly the version of the Buildmaster we're running right now to a new cloud host on Rackspace; probably the smallest one that comes with an SSD. Hawkowl is in the process of upgrading us to Buildbot 9 which will be wonderful when it happens, but I want to be clear that this effort is wholly independent from that and neither depends on the other :-). It looks to me like the thing I need to do is 'fab config.production buildbot.stop; fab config.production buildbot.dump', then make the relevant DNS change. At this point I think I need to add a new environment to ENVIRONMENTS and do 'fab config.buildbot buildbot.dump'. It's not obvious to me how to have two hosts present in 'config.production' but if somebody knows this please let me know. If you're interested in getting involved in operational stuff to help us out with this, I'll be available for pair-opsing (what do you call pair programming when it's ops?) so just get in touch. -glyph
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
On 1 Feb 2016, at 08:46, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
So at some point in the coming week I'd like to just move exactly the version of the Buildmaster we're running right now to a new cloud host on Rackspace; probably the smallest one that comes with an SSD.
Sounds good. The 2GB General Purpose that I use for the buildbots is plenty fast, and has enough RAM that we don't have to care (although I think RAX is smart enough that we can start at 1GB and move up to 2GB if needed?).
Hawkowl is in the process of upgrading us to Buildbot 9 which will be wonderful when it happens, but I want to be clear that this effort is wholly independent from that and neither depends on the other :-).
Yep, Nine is separate.
It looks to me like the thing I need to do is 'fab config.production buildbot.stop; fab config.production buildbot.dump', then make the relevant DNS change. At this point I think I need to add a new environment to ENVIRONMENTS and do 'fab config.buildbot buildbot.dump'. It's not obvious to me how to have two hosts present in 'config.production' but if somebody knows this please let me know.
I think "config.production" is a bit of a eh name to have after, we should change it if we have more machines. Eg. config.prod-dornkirk, or whatever. Then we just need to write what goes where. - Amber
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
I think "config.production" is a bit of a eh name to have after, we should change it if we have more machines. Eg. config.prod-dornkirk, or whatever. Then we just need to write what goes where.
I have create a PR to look at the actual implementation and to make this happen. https://github.com/twisted-infra/braid/pull/183 Feel fee to push changes to that branch. -------- On my installation with about 30 slaves the buildmaster process on py2.7 - 64bit uses 100MB for RES and 1.5G for VIRT CPU usage is very, very low. Regards -- Adi Roiban
![](https://secure.gravatar.com/avatar/e81edff3af564b86f4c9d780a8023299.jpg?s=120&d=mm&r=g)
Amber "Hawkie" Brown <hawkowl@atleastfornow.net> writes:
It looks to me like the thing I need to do is 'fab config.production buildbot.stop; fab config.production buildbot.dump', then make the relevant DNS change. At this point I think I need to add a new environment to ENVIRONMENTS and do 'fab config.buildbot buildbot.dump'. It's not obvious to me how to have two hosts present in 'config.production' but if somebody knows this please let me know.
I think "config.production" is a bit of a eh name to have after, we should change it if we have more machines. Eg. config.prod-dornkirk, or whatever. Then we just need to write what goes where.
I actually originally included some code to support having different machines for different services in the same environment. `t-names' is configured this way, although it runs on the same host. https://github.com/twisted-infra/braid/compare/buildbot-migration-poc is an untested change to make buildbot use a potentially different host. Tom
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Jan 31, 2016, at 4:56 PM, Amber Hawkie Brown <hawkowl@atleastfornow.net <mailto:hawkowl@atleastfornow.net>> wrote:
I think "config.production" is a bit of a eh name to have after, we should change it if we have more machines. Eg. config.prod-dornkirk, or whatever. Then we just need to write what goes where.
As Tom pointed out on https://github.com/twisted-infra/braid/pull/183 <https://github.com/twisted-infra/braid/pull/183> we do have a way to have multiple hosts participate in "production". I think it's much better to have one "production" environment so we just tell ops to work against that env, rather than having them have to memorize names like "dilandau", "dornkirk", "wolfwood", etc. I don't want to make watching a lot of 90s animé a hard requirement of being a Twisted systems admin ;-). -glyph
participants (4)
-
Adi Roiban
-
Amber "Hawkie" Brown
-
Glyph Lefkowitz
-
Tom Prince