Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

On Aug 1, 2021, at 2:49 AM, Adi Roiban mailto:adiroiban@gmail.com> wrote:

Hi Glyph

Thanks for sending the message.

As a wise computer once sung, "We do what we must, because we can".

This is bad, but I am confident that we can do the migration in 60 days, as long as we don't expect to have the exact same Twisted infrastructure.

It's certainly annoying (and I'm very sad to see tummy go, they were great) but I think we may be happy with the opportunity to reduce the amount of stuff we operate.

We have a monthly credit of 150 USD from MS to spend on Azure services.

This is great news! Presumably I … already have access to this, somehow? Is it in the 1password team vault? How do I see it? Do we have a commitment from MS to actually maintain this credit?

A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and migrate the existing server to it.

If we need to provision a new server anyway, could we provision it on the most recent ubuntu? Presumably most of the stuff we require will still be on there? If it's a hassle let's fall back to 18.04 but let's not set ourselves up to be outdated as long as we're doing this work.

Do we have other servers hosted with tummy.com http://tummy.com/, other than dornkirk.twistedmatrix.com http://dornkirk.twistedmatrix.com/ ?

Nope. We used to have a VPS called Wolfwood, but in an act of nominative determinism, it had to die for the greater good. (It was our source-control server, back when that was a thing you had to run yourself. We shut it down when we migrated to github.)

I will not have too much time to do the migration, but I can ask a sysadmin from my work to help with that and my company will cover that costs.

Thanks Adi, this is very generous. Let me know if I'm needed to give this sysadmin any requisite privileges.

I think that for now, we should see who is interested with helping with the migration and after 1 week start to work with that team.

Looks like a team is coming together :).

Maybe we will need to schedule an online meeting, but I hope we can do async work via GitHub Issues (braid) and IRC/Gitter.

Yeah, async seems to be working fine.

I saw that you have already considered using containers for Twisted infra https://github.com/twisted-infra/braid/issues/265 https://github.com/twisted-infra/braid/issues/265 Yes. I'd love to do this, and if such a migration reduces effort, then let's do it by all means - I just don't want to get caught up in trying to do this and ending up with things broken when the clock runs out.

In parallel with migrating dornkirk.twistedmatrix.com http://dornkirk.twistedmatrix.com/, we can look at outsourcing some of our services. For example

* Move mailing list to the main Python mailing list server

Looks like this is full steam ahead!

* Move DNS to a cloud provider (Azure DNS)

I've already pulled the trigger to migrate our DNS to Gandi, since they're our registrar and I already know I can provision certificates against their LiveDNS API :). We do still have to get a few other domains off of this infrastructure though, I'll do the ones we control (Divunal) and contact folks at the others. Longer-term, I would like to host some DNS, since I want to make sure that twisted.names does not bit-rot. However, perhaps a prerequisite to setting this up again will be (A) containerizing it and (B) providing something like Sentry for us to track exceptions, so that the dog-fooding here is interesting and not just a tedious obligation that produces gigabytes full of tracback-laden log files that nobody ever looks at or fixes.

* Move Trac Wiki to GitHub Wiki

The trac wiki's main purpose is not really to serve as a wiki-style editable document repository, but rather to host the website. I think migrating it to something like Pelican, or some other static-site management thing might make more sense.

* Move Trac Tickets to GitHub Issues

Sure, sure, as soon as we figure out a way to preserve all the links :).

* Move static file hosting to GitHub pages

I wonder if it would make sense for www.twistedmatrix.com http://www.twistedmatrix.com/ itself to be hosted by

* Move highscore to a container and update it to pull info from GitHub hooks

+1. This thing might be interesting <https://github.com/leereilly/github-high-scores https://github.com/leereilly/github-high-scores> in that it might be a better starting point than our own code (which is not even open source because, among other reasons, it is such trash) this version gets the point of the scoring wrong, which is to weight

* IRC bot to some sort of integration to Gitter and then IRC bridge

Given that this is already broken right now, I'd say "don't bother migrating the IRC bot right away". This also means we don't need amptrac, and we can rebuild the IRC bot in a more sustainable way, inside a container, later

* Retire speed tests and buildbot

Buildbot yes, speed tests no. I agree that speedcenter is not super well maintained, but it does at least give us some sense of performance, and right now it's all we've got. (If you can find some cloud CI thing like Github Actions which does anything with perf we should destroy speedcenter immediately, but given that PyPy still uses it, I have to imagine it's the best thing going. We should absolutely put it in a container though. We should expect all of our relative perf data to become garbage when we move to new hosting hardware, of course.

From past experience, I would say that most active Twisted contributors are software developers and not sysadmin. So we don't have the resources to keep custom Twisted infrastructure. And with the limited resources that we have, I would like to see Twisted contributors work at doing code review, fixing bugs rather than doing infrastructure work.

Absolutely.

So I am +1 for outsourcing as many Twisted services as possible.

My only caveat to this is that I think it would be interesting to be able to host fun stuff that uses Twisted. Some kind of goofy webchat thing or minimal real-time game to show off event-driven communication. (Also maybe we should have first-party websockets support, but I digress.) Maybe some public IMAP server that you can connect a mail client to that does something weird that isn't quite email. Maybe a logbot that can talk to the IRC channel. These things should all be containerized, and should not be permanently required for the project's workflows to work (writing such a toy should not be the same as committing to maintain it forever; if it breaks, it can break and we let it go), but we shouldn't aspire to pare down to nothing running in the cloud. That said, to have the energy to write, set up, and maintain such things, we do have to discard the perpetual albatross of propping up a years-old non-containerized system with tons of mystery miserable mutable state in it. (I really think relieving ourselves of Mailman will get rid of most of the stuff that makes this difficult, though, so I'm quite excited to get that migration underway.)

I think that we have already outsourced a few services:

* Automated testing to Azure Pipelines and GitHub Actions * Git Hosting to GitHub * Some Twisted sub-projects use GitHub Issues * Python package downloads to PyPi * Documentation hosting to Read The Docs

Don't forget moving our mail hosting & forwarding to Mailgun (which mailgun donates for free, thank you mailgunners)!