On dt., ag. 03 2021, Adi Roiban wrote:

On Sun, 1 Aug 2021 at 10:29, Evilham <contact@evilham.com> wrote:

...

\o

On dg., ag. 01 2021, Glyph wrote:

...

Hello Twistors,

Our venerable hosting provider, tummy.com <http://tummy.com/>, will be terminating their business operations on September 30, 2021.

...

...

...

datetime.date(2021, 9, 30) - datetime.date.today() datetime.timedelta(days=60)

In addition to being a bit personally sad - Tummy has been an avid supporter of the Twisted community since the very beginning - this means we have just about 60 days to fully migrate our website, mailing list, and DNS's hosting from Dornkirk, the machine where it's been running continuously since 2013, to ... something else.

it is sad indeed when friendly businesses terminate operations, but Twisted will go on :-).

...

There are 2 problems here:

We need a sponsor to help us find a new hosting or cloud provider where things can be hosted. We do pay tummy currently, and we can probably ask the SFC to pay a different hosting provider a similar price for equivalent services, but ideally we'd find someone willing to donate something so as not to put a drain on those resources.

Let's define this so I and others can ask around with various infra providers:

- What is the service currently? (e.g. VPS/Physical? Which resources?)

We only have a single server. It is called "dornkirk" I don't know if it's a VM or bare-metal... but it shouldn't matter.

- Are the resources allocated adequate? Can/should they be

...

increased/reduced?

I don't understand this question. Sorry. Can you please rephrase ? Thanks.

This was very poorly worded, my bad. I meant how big of a machine (storage excluding ML archives, RAM, CPU, traffic) is currently used and whether or not it is over/under-provisioned. Without knowing this concretely, it is hard to try to get an entity to commit to donating resources. It is good to know that VM vs physical and location are not important factors.

...

- Is currently something on the "wishlist" for this service? (e.g. IPv6 support!)

Many things are on the wishlist. The priority is to have the services up and running, even after dornkirk is dead.

...

- Even if the goal is and should be for that expense to disappear or go down significantly: roughly what is the compensation being offered for this service? (it might be easier to convince someone to offer it at cost/partial compensation than it is to offer it for free)

I don't know what is the current cost to cover the expenses for dornkirk.

...

- Is location important? E.g. if I asked around and it were fruitful, it'd be EU or Switzerland-based and not US-based as it is right now.

No. Location is not important.

...

...

I will need significant help with the operational aspects of the migration. I typically do a bunch of behind-the-scenes system administration for twistedmatrix.com <http://twistedmatrix.com/> to keep the whole thing clunking along, but after my brain has been fried by a year and a half's worth of pandemic stress, I simply don't have the time, energy, or mental capacity to do this. I know I've asked for help before and rarely received any, but if I have to do this entirely by myself, the most likely outcome is that I'll migrate DNS to some free cloud thing so at least my email address doesn't break (I migrated SMTP, and thereby everything associated with personal email addresses to Mailgun some years ago, so Tummy is not in that loop for that), and the website (and this mailing list) will simply stop working sometime between now and October. So if you'd like to keep www.twistedmatrix.com <http://www.twistedmatrix.com/>, speed.twistedmatrix.com <http://speed.twistedmatrix.com/>, our IRC bot (which is offline anyway right now due to Freenode shutting down), https://twistedmatrix.com/highscores/ <https://twistedmatrix.com/highscores/>, this mailing list, or our dogfooding instance for Twisted Names DNS, please volunteer so we can start to coordinate.

Thank you for both doing this so long and so far and for being straightforward about not being able to do it as much now.

I'd like to help and would be particularly (aka more intensively) able to do so during August, even if that's just helping document what needs to be done and how. On a day-to-day basis I do some other sysadmin work and if necessary would be able to accommodate taking care of Twisted-related things into that.

To coordinate migration and also day-to-day Twisted project things we use the IRC Libera channel #twisted-dev

If you don't like IRC ... we are also experiencing with Gitter/Matrix https://gitter.im/twisted/twisted

I start this to-do list https://github.com/twisted-infra/braid/issues/277

IM is particularly taxing these days. The TODO list looks quite comprehensive, thank you.

...

This is the documentation that I am aware of: https://twistedmatrix.com/trac/wiki/Infrastructure

There is not much documentation... we need to rely on memories :) So don't bother to search for documentation, and just ask :)

...

Particularly braid: https://github.com/twisted-infra/braid

Yes. braid is our deployment as code thing... is the best we have in terms of documenting the infrastructure.

"braid" is (was?) the Twisted customization on top of fabric for doing primitive automated deployment.

...

Is there anything else to look at? Something known to not be supported in braid / only partially supported?

The plan is to stop using braid and do all the deployment via images and containers.

...

Would you rather schedule some online voice session where I poke your brain at what needs to be done (IIRC something similar was done about 5 years ago) and take notes that can be used (also by someone else) while you let your mind flow happily?

We are split between Europe time zone (UTC -> UTC+2) and US ( something like UTC-6.. but also UTC-7) For now, we have done only ad-hoc meetings.

...

...

If you've been wanting to just use Github Issues and don't care about preserving any of the data in Trac, congratulations, you're about to win that argument by default ;-).

In the worst case scenario, I will download a backup image before things get turned off in case someone wants to deal with this

I personally hope this doesn't happen; particularly the mailing list is a great low-bandwidth way of staying mostly up-to-date with the project.

I think that the plan is to migrate the mailing list to the Python Org mailing list hosting.

We will need some help with setting up archive redirections.

---------------

I think the current problem is finding or deciding for the next hosting / VM / container provider.

We have 150 USD per month from MS to spend on Azure, but this is renewed any year without any guarantee that next year we will receive the free credit. The subscription was requested by Python Software Foundation and was granted by MS, but it was associated to the Azure AD of my organization.

I have no idea how to create an Azure AD for Twisted organization, so that I can transfer the subscription.

So...if anyone has any experience with Azure AD, it would help to get this thing sorted.

For now, using Azure is plan B... so if we don't manage to migrate the services somewhere else, we (I) can just create an Ubuntu VM in Azure and migrate the existing dornkirk services there.

------------

The DNS was already migrated by Glyph. Thanks Glyph :)

The mailing lists will be migrated soon to Python Org... we only have to see how to do it in a seamless way.

Mailing list was certainly the most concerning point, it's great that it's already being solved \o/ handling mailing lists is really not fun. Using the registrar's DNS makes sense short-term, mid-term as Glyph mentioned twisted.names should be used by the project so there is starker incentive to keep it working and somewhat up to date. Having this functionality allows for interesting use-cases: https://github.com/evilham/weneeda.name IIRC there was a way to setup Gandi to mirror a zone periodically, so that'd be good redundancy. Anyway, that's for somewhen else.

--------

So Evilham, if you want to help, try to see if you can hang out over IRC or Gitter and we can see how you can help. Thanks!

After catching up on some other things, I'll take a more serious look at what I can help with from the TODO.

On Aug 1, 2021, at 2:49 AM, Adi Roiban <adiroiban@gmail.com <mailto:adiroiban@gmail.com>> wrote:

Hi Glyph

Thanks for sending the message.

As a wise computer once sung, "We do what we must, because we can".

This is bad, but I am confident that we can do the migration in 60 days, as long as we don't expect to have the exact same Twisted infrastructure.

It's certainly annoying (and I'm very sad to see tummy go, they were great) but I think we may be happy with the opportunity to reduce the amount of stuff we operate.

We have a monthly credit of 150 USD from MS to spend on Azure services.

This is great news! Presumably I … already have access to this, somehow? Is it in the 1password team vault? How do I see it? Do we have a commitment from MS to actually maintain this credit?

A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and migrate the existing server to it.

If we need to provision a new server anyway, could we provision it on the most recent ubuntu? Presumably most of the stuff we require will still be on there? If it's a hassle let's fall back to 18.04 but let's not set ourselves up to be outdated as long as we're doing this work.

Do we have other servers hosted with tummy.com <http://tummy.com/>, other than dornkirk.twistedmatrix.com <http://dornkirk.twistedmatrix.com/> ?

Nope. We used to have a VPS called Wolfwood, but in an act of nominative determinism, it had to die for the greater good. (It was our source-control server, back when that was a thing you had to run yourself. We shut it down when we migrated to github.)

I will not have too much time to do the migration, but I can ask a sysadmin from my work to help with that and my company will cover that costs.

Thanks Adi, this is very generous. Let me know if I'm needed to give this sysadmin any requisite privileges.

I think that for now, we should see who is interested with helping with the migration and after 1 week start to work with that team.

Looks like a team is coming together :).

Maybe we will need to schedule an online meeting, but I hope we can do async work via GitHub Issues (braid) and IRC/Gitter.

Yeah, async seems to be working fine.

I saw that you have already considered using containers for Twisted infra https://github.com/twisted-infra/braid/issues/265 <https://github.com/twisted-infra/braid/issues/265> Yes. I'd love to do this, and if such a migration reduces effort, then let's do it by all means - I just don't want to get caught up in trying to do this and ending up with things broken when the clock runs out.

In parallel with migrating dornkirk.twistedmatrix.com <http://dornkirk.twistedmatrix.com/>, we can look at outsourcing some of our services. For example

* Move mailing list to the main Python mailing list server

Looks like this is full steam ahead!

* Move DNS to a cloud provider (Azure DNS)

I've already pulled the trigger to migrate our DNS to Gandi, since they're our registrar and I already know I can provision certificates against their LiveDNS API :). We do still have to get a few other domains off of this infrastructure though, I'll do the ones we control (Divunal) and contact folks at the others. Longer-term, I would like to host some DNS, since I want to make sure that twisted.names does not bit-rot. However, perhaps a prerequisite to setting this up again will be (A) containerizing it and (B) providing something like Sentry for us to track exceptions, so that the dog-fooding here is interesting and not just a tedious obligation that produces gigabytes full of tracback-laden log files that nobody ever looks at or fixes.

* Move Trac Wiki to GitHub Wiki

The trac wiki's main purpose is not really to serve as a wiki-style editable document repository, but rather to host the website. I think migrating it to something like Pelican, or some other static-site management thing might make more sense.

* Move Trac Tickets to GitHub Issues

Sure, sure, as soon as we figure out a way to preserve all the links :).

* Move static file hosting to GitHub pages

I wonder if it would make sense for www.twistedmatrix.com <http://www.twistedmatrix.com/> itself to be hosted by

* Move highscore to a container and update it to pull info from GitHub hooks

+1. This thing might be interesting <https://github.com/leereilly/github-high-scores <https://github.com/leereilly/github-high-scores>> in that it might be a better starting point than our own code (which is not even open source because, among other reasons, it is such trash) this version gets the point of the scoring wrong, which is to weight

* IRC bot to some sort of integration to Gitter and then IRC bridge

Given that this is already broken right now, I'd say "don't bother migrating the IRC bot right away". This also means we don't need amptrac, and we can rebuild the IRC bot in a more sustainable way, inside a container, later

* Retire speed tests and buildbot

Buildbot yes, speed tests no. I agree that speedcenter is not super well maintained, but it does at least give us some sense of performance, and right now it's all we've got. (If you can find some cloud CI thing like Github Actions which does anything with perf we should destroy speedcenter immediately, but given that PyPy still uses it, I have to imagine it's the best thing going. We should absolutely put it in a container though. We should expect all of our relative perf data to become garbage when we move to new hosting hardware, of course.

From past experience, I would say that most active Twisted contributors are software developers and not sysadmin. So we don't have the resources to keep custom Twisted infrastructure. And with the limited resources that we have, I would like to see Twisted contributors work at doing code review, fixing bugs rather than doing infrastructure work.

Absolutely.

So I am +1 for outsourcing as many Twisted services as possible.

My only caveat to this is that I think it would be interesting to be able to host fun stuff that uses Twisted. Some kind of goofy webchat thing or minimal real-time game to show off event-driven communication. (Also maybe we should have first-party websockets support, but I digress.) Maybe some public IMAP server that you can connect a mail client to that does something weird that isn't quite email. Maybe a logbot that can talk to the IRC channel. These things should all be containerized, and should not be permanently required for the project's workflows to work (writing such a toy should not be the same as committing to maintain it forever; if it breaks, it can break and we let it go), but we shouldn't aspire to pare down to nothing running in the cloud. That said, to have the energy to write, set up, and maintain such things, we do have to discard the perpetual albatross of propping up a years-old non-containerized system with tons of mystery miserable mutable state in it. (I really think relieving ourselves of Mailman will get rid of most of the stuff that makes this difficult, though, so I'm quite excited to get that migration underway.)

I think that we have already outsourced a few services:

* Automated testing to Azure Pipelines and GitHub Actions * Git Hosting to GitHub * Some Twisted sub-projects use GitHub Issues * Python package downloads to PyPi * Documentation hosting to Read The Docs

Don't forget moving our mail hosting & forwarding to Mailgun (which mailgun donates for free, thank you mailgunners)!