Hi Craig, Thanks for this, sharing your past experience is invaluable :) I've gone through and thought about it a bit, and rewritten it into https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.... -- it is basically your plan, with some added notes about Twisted Infra specific parts. I've skimmed over the specific details, since I feel that going too in-depth in such a plan will just be wasted effort as unknown issues arise, but with enough structure that we have a clear set of overarching goals for each step. The migration will have a handful of policy changes that we will have to resolve -- such as ensuring that all merges have a topfile -- which aren't possible under a GitHub based system. I think these issues will just involve a lot of scrutiny and double checking during the transitional period until we are confident that we are enforcing our existing quality and process standards. If anyone has any suggestions, or any more invaluable experience to share, please do :) - Amber

On 18 Nov 2015, at 07:48, Craig Rodrigues wrote:

On Tue, Nov 17, 2015 at 8:57 AM, Adi Roiban wrote:

For now, the funds were raised to migrate to GitHub, so we can not use them to do other things.

We will stay on Trac for issues... at least for now.

I have no idea how we can migrate to any issue tracker without losing data if we don't have full access to the database.

It is possible to migrate to another issue tracker and not lose data. I've done Trac -> Redmine, and it works, but there was an existing migration script I could use. For migrating to a cloud based bug tracker, you would need to take every user in the existing Trac database, and see if there would be a way to map the existing users to the cloud database, such as GitHub. It's a lot of work, but possible. However, for the scope of this project, if staying with Trac for issues is what is required, that is fine.

We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages

OK.

So based on what you have listed, I would say that most of the work will be working with Git post commit hooks.

I would say the plan should be something like this.

A.1 https://github.com/twisted/twisted will be the "repository of truth" for Twisted. -> Twisted releases will be done from GitHub -> the Twisted developers who are now "core committers" for SVN, must be given access to be "core committers" to https://github.com/twisted/twisted

A.2 On the Trac server, a local git mirror of the GitHub must be set up. Every bug tracker I've seen that integrates with git needs a local mirror of the repo in order to parse the git history in order to update the bug database. This mirror should be read-only, and the only thing updating this repo should be the Trac GitHub plugin.

A.3 On the Trac server, this plugin must be installed: https://github.com/trac-hacks/trac-github

A.4 On the GitHub server, a post-commit web hook must be configured. The workflow will be this:

[core committer does push to https://github.com/twisted/twisted] -> [post commit GitHub hook will be called to poke the Trac GitHub plugin] -> [Trac GitHub plugin will update the local git repo on the Trac server] -> [Trac GitHub will parse the git history for new commits and update tickets]

I would recommend that steps (1) - (4) be made to work in a staging environment, with a separate GitHub repo, and a separate copy of the Trac database. That way, you can test things out without derailing Twisted developers. When you are confident that this workflow works, then the transition plan will be something like the following.

B.1 Send an e-mail to the mailing list and pick one day for the maintenance window. This will warn folks when they should take a holiday from Twisted work. :)

B.2 When maintenance is about to begin, send a [HEADSUP] mail saying that repo will be unavailable.

B.3 Create Subversion pre-commit hook to disable all commits to Subversion: http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-reposito...

B.4 Set up steps A.1 - A.4

B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to https://github.com/twisted/twisted, and make sure that Trac works.

B.6 Once the Twisted core team are satisified that everything works, send an e-mail to the mailing list that the maintenance window is over, and GitHub is now where the action is!

B.7 Update all wiki documentation to change all references to getting code from Subversion, to getting code from GitHub.

B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.

-- Craig

-- Craig _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Nov 29, 2015, at 08:20, Adi Roiban wrote:

Hi Craig,

Sorry for the delay and many thanks for your plan.

I have also sent your plan to the Unofficial Twisted Software Foundation.

Just for everyone's information, the real name for the relevant group here is the "Twisted Project Leadership Committee for the Software Freedom Conservancy". The reason that the relevant mailing list is titled "unofficial twisted software foundation" is that we started off investigating if we could start our own foundation, and later, when we opted to have the Software Freedom Conservancy as our fiscal sponsor instead, the name "twisted software foundation" became "unofficial" because there is no such legal entity.

From what I can see we are stuck in bureaucratic process.

We are stuck at the point before the bureaucratic process :). The bureaucratic process is that the PLC votes to approve the plan.

The plan needs to be approved ... and [the PLC] want to have a single plan submitted for approval.

The reason the plan needs to be approved is that the plans for the fellowship are documented here - https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan - and that says "The maintainer will develop a plan for migration of development to GitHub, and once it has been approved implement the plan".

We now have 3 plans : Amber's, Craig's and mine..... and we see which plan to be sent to [the PLC].

Also [the PLC] will only consider plans for GitHub.com.

This is also because it's written in https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan, so it was decided before the fellowship began. However, lots of the steps to move to GitHub.com http://github.com/ are also necessary prerequisites to use any of those sites - getting rid of subversion and reducing the amount of infrastructure we are operating. Moving from Github somewhere else ought to be radically simpler than the process we've been undertaking to move to Github.

We can try to keep tickets/wiki/website on Trac and only move the main repo + PR + hooks to GitHub.

This sounds good to me.

This should allow us to get rid of SVN and build the infrastructure on web hooks. Later we can consider migration to other tools... or extending the GitHub.com usage to Issues / Wiki / GitHub Pages...etc

Until now we failed to coordinate toward creating a single plan and besides Glyph's comments on IRC, I have not received any feedback from [the PLC] for any of the plans.

The PLC is unlikely to be involved. Personally, I'm severely overcommitted; most of the other PLC members have a very low level of involvement with the project. Not going to point fingers specifically, but some hardly answer their email :-). While I would prefer it if the PLC were a bit more involved, it should not be much of an issue in this case (assuming that I can herd the cats in the right direction when it's time for a vote). The PLC's job in this case is only to provide oversight, just to verify that the plan is a proper investment of the SLC's financial resources. In the same way that reviewers should not participate too closely in the authorship of patches they review, it should be fine if the PLC is not involved in the planning process. (If we had the personal resources to do so, we probably wouldn't have needed to put it into the fellowship plan!) The PLC won't decide between a set of different plans; it will just give final approval to the proposed one.

Can we plan an IRC/Google Hangouts meeting to discuss the plan?

I've not had good luck with Hangouts, personally, but scheduling time on IRC sounds like a good idea.

If you would like to see Twisted on Git and GitHub please consider joining the meeting.

I have created this Doodle poll to help us schedule the date and time for the meeting - http://doodle.com/poll/4ys8m8qakav9u9f9 http://doodle.com/poll/4ys8m8qakav9u9f9 Ultimately, it is Amber and Adi who need to coordinate on this. But I would strongly encourage anyone with an interest in this migration to participate in the planning process; clearly we need some help nailing down the specifics :).

-glyph

On Tue, 17 Nov 2015 at 23:48 Craig Rodrigues wrote:

On Tue, Nov 17, 2015 at 8:57 AM, Adi Roiban wrote:

...

For now, the funds were raised to migrate to GitHub, so we can not use them to do other things.

We will stay on Trac for issues... at least for now.

I have no idea how we can migrate to any issue tracker without losing data if we don't have full access to the database.

[snip]

B.7 Update all wiki documentation to change all references to getting code from Subversion, to getting code from GitHub.

Probably would be a good idea to have a list of such changes *before* the migration.

B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.

Couldn't we do this before the migration anyway? At least partly? jml

Glyph Lefkowitz writes:

...

Probably would be a good idea to have a list of such changes *before* the migration.

Yes. Everything should be written up and reviewed beforehand, and

There has been a lot of words written talking about coming up with a plan for the migration, but I have yet to see a concrete plan. I think this (a list of all the things that depend on SVN) followed by plans to address each of the, is probably a sensible first step. I suspect that none of this will be contreviersial, but it seems that the disucssion keeps dancing around things needing to be done, but nobody has taken the time to actually come up with a list. Tom

On Dec 3, 2015, at 3:53 PM, Craig Rodrigues wrote:

On Tue, Dec 1, 2015 at 5:01 PM, Glyph Lefkowitz mailto:glyph@twistedmatrix.com> wrote:

One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.

I worked with Amber and Adi and we have the checklist: https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.... https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration....

Thank you very much for your participation in this. The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened? -glyph

On Thu, Dec 3, 2015 at 7:07 PM, Glyph Lefkowitz wrote:

The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened?

The checklist is written using the GitHub flavored markdown syntax for checklists: https://github.com/blog/1375%0A-task-lists-in-gfm-issues-pulls-comments so when the tasks are completed, this checklist will be updated by changing: [ ] task item foobar to [X] task item foobar and committing the updated checklist to GitHub. Adi and Amber are on board with this, so it should be fine. -- Craig