[Twisted-Python] Suggested plan for GitHub migration
On Tue, Nov 17, 2015 at 8:57 AM, Adi Roiban adi@roiban.ro wrote:
For now, the funds were raised to migrate to GitHub, so we can not use them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing data if we don't have full access to the database.
It is possible to migrate to another issue tracker and not lose data. I've done Trac -> Redmine, and it works, but there was an existing migration script I could use. For migrating to a cloud based bug tracker, you would need to take every user in the existing Trac database, and see if there would be a way to map the existing users to the cloud database, such as GitHub. It's a lot of work, but possible. However, for the scope of this project, if staying with Trac for issues is what is required, that is fine.
We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages
OK.
So based on what you have listed, I would say that most of the work will be working with Git post commit hooks.
I would say the plan should be something like this.
A.1 https://github.com/twisted/twisted will be the "repository of truth" for Twisted. -> Twisted releases will be done from GitHub -> the Twisted developers who are now "core committers" for SVN, must be given access to be "core committers" to https://github.com/twisted/twisted
A.2 On the Trac server, a local git mirror of the GitHub must be set up. Every bug tracker I've seen that integrates with git needs a local mirror of the repo in order to parse the git history in order to update the bug database. This mirror should be read-only, and the only thing updating this repo should be the Trac GitHub plugin.
A.3 On the Trac server, this plugin must be installed: https://github.com/trac-hacks/trac-github
A.4 On the GitHub server, a post-commit web hook must be configured. The workflow will be this:
[core committer does push to https://github.com/twisted/twisted]
-> [post commit GitHub hook will be called to poke the Trac GitHub plugin] -> [Trac GitHub plugin will update the local git repo on the Trac server] -> [Trac GitHub will parse the git history for new commits and update tickets]
I would recommend that steps (1) - (4) be made to work in a staging environment, with a separate GitHub repo, and a separate copy of the Trac database. That way, you can test things out without derailing Twisted developers. When you are confident that this workflow works, then the transition plan will be something like the following.
B.1 Send an e-mail to the mailing list and pick one day for the maintenance window. This will warn folks when they should take a holiday from Twisted work. :)
B.2 When maintenance is about to begin, send a [HEADSUP] mail saying that repo will be unavailable.
B.3 Create Subversion pre-commit hook to disable all commits to Subversion:
http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-reposito...
B.4 Set up steps A.1 - A.4
B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to https://github.com/twisted/twisted, and make sure that Trac works.
B.6 Once the Twisted core team are satisified that everything works, send an e-mail to the mailing list that the maintenance window is over, and GitHub is now where the action is!
B.7 Update all wiki documentation to change all references to getting code from Subversion, to getting code from GitHub.
B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.
-- Craig
-- Craig
Hi Craig,
Sorry for the delay and many thanks for your plan.
I have also sent your plan to the Unofficial Twisted Software Foundation.
From what I can see we are stuck in bureaucratic process.
The plan needs to be approved by Unofficial Twisted Software Foundation and the Unofficial Twisted Software Foundation want to have a single plan submitted for approval.
We now have 3 plans : Amber's, Craig's and mine..... and we see which plan to be sent to the Unofficial Twisted Software Foundation.
Also the Unofficial Twisted Software Foundation will only consider plans for GitHub.com.
No GitLab/Bitbucket/Jira/Redmime....etc
We can try to keep tickets/wiki/website on Trac and only move the main repo + PR + hooks to GitHub.
This should allow us to get rid of SVN and build the infrastructure on web hooks. Later we can consider migration to other tools... or extending the GitHub.com usage to Issues / Wiki / GitHub Pages...etc
Until now we failed to coordinate toward creating a single plan and besides Glyph's comments on IRC, I have not received any feedback from the Unofficial Twisted Software Foundation for any of the plans.
Can we plan an IRC/Google Hangouts meeting to discuss the plan?
It would be great if someone from the Unofficial Twisted Software Foundation could also join.
If you would like to see Twisted on Git and GitHub please consider joining the meeting.
I have created this Doodle poll to help us schedule the date and time for the meeting - http://doodle.com/poll/4ys8m8qakav9u9f9
Thanks!
On Nov 29, 2015, at 08:20, Adi Roiban adi@roiban.ro wrote:
Hi Craig,
Sorry for the delay and many thanks for your plan.
I have also sent your plan to the Unofficial Twisted Software Foundation.
Just for everyone's information, the real name for the relevant group here is the "Twisted Project Leadership Committee for the Software Freedom Conservancy". The reason that the relevant mailing list is titled "unofficial twisted software foundation" is that we started off investigating if we could start our own foundation, and later, when we opted to have the Software Freedom Conservancy as our fiscal sponsor instead, the name "twisted software foundation" became "unofficial" because there is no such legal entity.
From what I can see we are stuck in bureaucratic process.
We are stuck at the point before the bureaucratic process :). The bureaucratic process is that the PLC votes to approve the plan.
The plan needs to be approved ... and [the PLC] want to have a single plan submitted for approval.
The reason the plan needs to be approved is that the plans for the fellowship are documented here - https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan - and that says "The maintainer will develop a plan for migration of development to GitHub, and once it has been approved implement the plan".
We now have 3 plans : Amber's, Craig's and mine..... and we see which plan to be sent to [the PLC].
Also [the PLC] will only consider plans for GitHub.com.
This is also because it's written in https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan, so it was decided before the fellowship began. However, lots of the steps to move to GitHub.com http://github.com/ are also necessary prerequisites to use any of those sites - getting rid of subversion and reducing the amount of infrastructure we are operating. Moving from Github somewhere else ought to be radically simpler than the process we've been undertaking to move to Github.
We can try to keep tickets/wiki/website on Trac and only move the main repo + PR + hooks to GitHub.
This sounds good to me.
This should allow us to get rid of SVN and build the infrastructure on web hooks. Later we can consider migration to other tools... or extending the GitHub.com usage to Issues / Wiki / GitHub Pages...etc
Until now we failed to coordinate toward creating a single plan and besides Glyph's comments on IRC, I have not received any feedback from [the PLC] for any of the plans.
The PLC is unlikely to be involved. Personally, I'm severely overcommitted; most of the other PLC members have a very low level of involvement with the project. Not going to point fingers specifically, but some hardly answer their email :-).
While I would prefer it if the PLC were a bit more involved, it should not be much of an issue in this case (assuming that I can herd the cats in the right direction when it's time for a vote). The PLC's job in this case is only to provide oversight, just to verify that the plan is a proper investment of the SLC's financial resources. In the same way that reviewers should not participate too closely in the authorship of patches they review, it should be fine if the PLC is not involved in the planning process. (If we had the personal resources to do so, we probably wouldn't have needed to put it into the fellowship plan!) The PLC won't decide between a set of different plans; it will just give final approval to the proposed one.
Can we plan an IRC/Google Hangouts meeting to discuss the plan?
I've not had good luck with Hangouts, personally, but scheduling time on IRC sounds like a good idea.
If you would like to see Twisted on Git and GitHub please consider joining the meeting.
I have created this Doodle poll to help us schedule the date and time for the meeting - http://doodle.com/poll/4ys8m8qakav9u9f9 http://doodle.com/poll/4ys8m8qakav9u9f9
Ultimately, it is Amber and Adi who need to coordinate on this. But I would strongly encourage anyone with an interest in this migration to participate in the planning process; clearly we need some help nailing down the specifics :).
-glyph
Hi Craig,
Thanks for this, sharing your past experience is invaluable :)
I've gone through and thought about it a bit, and rewritten it into https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.... -- it is basically your plan, with some added notes about Twisted Infra specific parts. I've skimmed over the specific details, since I feel that going too in-depth in such a plan will just be wasted effort as unknown issues arise, but with enough structure that we have a clear set of overarching goals for each step.
The migration will have a handful of policy changes that we will have to resolve -- such as ensuring that all merges have a topfile -- which aren't possible under a GitHub based system. I think these issues will just involve a lot of scrutiny and double checking during the transitional period until we are confident that we are enforcing our existing quality and process standards.
If anyone has any suggestions, or any more invaluable experience to share, please do :)
- Amber
On 18 Nov 2015, at 07:48, Craig Rodrigues rodrigc@crodrigues.org wrote:
On Tue, Nov 17, 2015 at 8:57 AM, Adi Roiban adi@roiban.ro wrote:
For now, the funds were raised to migrate to GitHub, so we can not use them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing data if we don't have full access to the database.
It is possible to migrate to another issue tracker and not lose data. I've done Trac -> Redmine, and it works, but there was an existing migration script I could use. For migrating to a cloud based bug tracker, you would need to take every user in the existing Trac database, and see if there would be a way to map the existing users to the cloud database, such as GitHub. It's a lot of work, but possible. However, for the scope of this project, if staying with Trac for issues is what is required, that is fine.
We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages
OK.
So based on what you have listed, I would say that most of the work will be working with Git post commit hooks.
I would say the plan should be something like this.
A.1 https://github.com/twisted/twisted will be the "repository of truth" for Twisted. -> Twisted releases will be done from GitHub -> the Twisted developers who are now "core committers" for SVN, must be given access to be "core committers" to https://github.com/twisted/twisted
A.2 On the Trac server, a local git mirror of the GitHub must be set up. Every bug tracker I've seen that integrates with git needs a local mirror of the repo in order to parse the git history in order to update the bug database. This mirror should be read-only, and the only thing updating this repo should be the Trac GitHub plugin.
A.3 On the Trac server, this plugin must be installed: https://github.com/trac-hacks/trac-github
A.4 On the GitHub server, a post-commit web hook must be configured. The workflow will be this:
[core committer does push to https://github.com/twisted/twisted] -> [post commit GitHub hook will be called to poke the Trac GitHub plugin] -> [Trac GitHub plugin will update the local git repo on the Trac server] -> [Trac GitHub will parse the git history for new commits and update tickets]I would recommend that steps (1) - (4) be made to work in a staging environment, with a separate GitHub repo, and a separate copy of the Trac database. That way, you can test things out without derailing Twisted developers. When you are confident that this workflow works, then the transition plan will be something like the following.
B.1 Send an e-mail to the mailing list and pick one day for the maintenance window. This will warn folks when they should take a holiday from Twisted work. :)
B.2 When maintenance is about to begin, send a [HEADSUP] mail saying that repo will be unavailable.
B.3 Create Subversion pre-commit hook to disable all commits to Subversion: http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-reposito...
B.4 Set up steps A.1 - A.4
B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to https://github.com/twisted/twisted, and make sure that Trac works.
B.6 Once the Twisted core team are satisified that everything works, send an e-mail to the mailing list that the maintenance window is over, and GitHub is now where the action is!
B.7 Update all wiki documentation to change all references to getting code from Subversion, to getting code from GitHub.
B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.
-- Craig
-- Craig _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 30 November 2015 at 16:49, Amber "Hawkie" Brown < hawkowl@atleastfornow.net> wrote:
Hi Craig,
Thanks for this, sharing your past experience is invaluable :)
I've gone through and thought about it a bit, and rewritten it into https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.... -- it is basically your plan, with some added notes about Twisted Infra specific parts. I've skimmed over the specific details, since I feel that going too in-depth in such a plan will just be wasted effort as unknown issues arise, but with enough structure that we have a clear set of overarching goals for each step.
[snit]
+1 for "master" as the main branch.
+1 for GitHub logins but I have no idea how we could migrate existing accounts.
This is more like a Git migration plan with main repo on GitHub, but I think that this should be the first step...and focus on this, before looking at PR and GitHub issues.
I don't see any note about the code browser and how to migrate narrative and apidocs source code links.
The Internet is full of links like http://twistedmatrix.com/trac/browser/tags/releases/twisted-8.2.0/twisted/we...
I think that tracext.github.GitHubBrowser only redirects changeset ... and from the docs only new changesets
I was hoping that as part of the git migration, we can implement a custom Twisted web resource which will handle all the redirections to GitHub code browser
Since Twisted trunk merges are not busy, I don't think that we need to worry to much about breaking the dev process... I am more worried about failing to gather the merges required to create the waiting/testing queue.
-- Adi
On 18 Nov 2015, at 07:48, Craig Rodrigues rodrigc@crodrigues.org
wrote:
On Tue, Nov 17, 2015 at 8:57 AM, Adi Roiban adi@roiban.ro wrote:
For now, the funds were raised to migrate to GitHub, so we can not use them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing data if we don't have full access to the database.
It is possible to migrate to another issue tracker and not lose data. I've done Trac -> Redmine, and it works, but there was an
existing migration
script I could use. For migrating to a cloud based bug tracker, you would need to take every
user
in the existing Trac database, and see if there would be a way to map the existing users to the cloud database, such as GitHub. It's a lot of
work, but possible.
However, for the scope of this project, if staying with Trac for issues
is what is required, that is fine.
We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages
OK.
So based on what you have listed, I would say that most of the work will
be
working with Git post commit hooks.
I would say the plan should be something like this.
A.1 https://github.com/twisted/twisted will be the "repository of
truth"
for Twisted. -> Twisted releases will be done from GitHub -> the Twisted developers who are now "core committers" forSVN, must be
given access to be "core committers" tohttps://github.com/twisted/twisted
A.2 On the Trac server, a local git mirror of the GitHub must be set up. Every bug tracker I've seen that integrates with git needs a
local mirror of the repo
in order to parse the git history in order to update the bugdatabase.
This mirror should be read-only, and the only thing updatingthis repo should be the Trac GitHub plugin.
A.3 On the Trac server, this plugin must be installed:
https://github.com/trac-hacks/trac-github
A.4 On the GitHub server, a post-commit web hook must be configured.
The workflow will be this:
[core committer does push tohttps://github.com/twisted/twisted]
-> [post commit GitHub hook will be called to poke the TracGitHub plugin]
-> [Trac GitHub plugin will update the local git repo onthe Trac server]
-> [Trac GitHub will parse the git history for new commitsand update tickets]
I would recommend that steps (1) - (4) be made to work in a staging
environment, with a separate
GitHub repo, and a separate copy of the Trac database. That way, you
can test things out without derailing
Twisted developers. When you are confident that this workflow works,
then the transition plan will be something
like the following.
B.1 Send an e-mail to the mailing list and pick one day for the
maintenance window.
This will warn folks when they should take a holiday fromTwisted work. :)
B.2 When maintenance is about to begin, send a [HEADSUP] mail saying
that repo will be unavailable.
B.3 Create Subversion pre-commit hook to disable all commits to Subversion:
http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-reposito...
B.4 Set up steps A.1 - A.4
B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to
https://github.com/twisted/twisted, and
make sure that Trac works.B.6 Once the Twisted core team are satisified that everything works,
send an e-mail to the mailing list
that the maintenance window is over, and GitHub is now where theaction is!
B.7 Update all wiki documentation to change all references to getting
code from Subversion,
to getting code from GitHub.B.8 Update all systems which used Subversion to use GitHub. For
example, buildbots.
-- Craig
-- Craig _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, 30 Nov 2015 at 16:50 Amber "Hawkie" Brown hawkowl@atleastfornow.net wrote:
The migration will have a handful of policy changes that we will have to resolve -- such as ensuring that all merges have a topfile -- which aren't possible under a GitHub based system.
You could make master/trunk/whatever a protected branch, and have a required status check for this, but that would require some external CI thing that actually performs the status check.
On Tue, 17 Nov 2015 at 23:48 Craig Rodrigues rodrigc@crodrigues.org wrote:
On Tue, Nov 17, 2015 at 8:57 AM, Adi Roiban adi@roiban.ro wrote:
For now, the funds were raised to migrate to GitHub, so we can not use them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing data if we don't have full access to the database.
[snip]
B.7 Update all wiki documentation to change all references to getting code from Subversion, to getting code from GitHub.
Probably would be a good idea to have a list of such changes *before* the migration.
B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.
Couldn't we do this before the migration anyway? At least partly?
jml
On Dec 1, 2015, at 10:36 AM, Jonathan Lange jml@mumak.net wrote:
B.7 Update all wiki documentation to change all references to getting code from Subversion, to getting code from GitHub.
Probably would be a good idea to have a list of such changes *before* the migration.
Yes. Everything should be written up and reviewed beforehand, and
B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.
Couldn't we do this before the migration anyway? At least partly?
Well, I mean... this is "the migration" itself. So this should happen earlier in the process, particularly earlier than we start committing directly to git? Much of this work is already done: the buildbots are already getting the source from git, not SVN.
One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.
-glyph
Glyph Lefkowitz glyph@twistedmatrix.com writes:
Probably would be a good idea to have a list of such changes *before* the migration.
Yes. Everything should be written up and reviewed beforehand, and
There has been a lot of words written talking about coming up with a plan for the migration, but I have yet to see a concrete plan. I think this (a list of all the things that depend on SVN) followed by plans to address each of the, is probably a sensible first step. I suspect that none of this will be contreviersial, but it seems that the disucssion keeps dancing around things needing to be done, but nobody has taken the time to actually come up with a list.
Tom
On Dec 1, 2015, at 10:09 PM, Tom Prince tom.prince@ualberta.net wrote:
Glyph Lefkowitz glyph@twistedmatrix.com writes:
Probably would be a good idea to have a list of such changes *before* the migration.
Yes. Everything should be written up and reviewed beforehand, and
There has been a lot of words written talking about coming up with a plan for the migration, but I have yet to see a concrete plan. I think this (a list of all the things that depend on SVN) followed by plans to address each of the, is probably a sensible first step. I suspect that none of this will be contreviersial, but it seems that the disucssion keeps dancing around things needing to be done, but nobody has taken the time to actually come up with a list.
This checklist is actually what will become the official plan, because investigating the exact steps required to satisfy each step is part of the plan itself :).
-glyph
On Tue, Dec 1, 2015 at 5:01 PM, Glyph Lefkowitz glyph@twistedmatrix.com wrote:
One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.
I worked with Amber and Adi and we have the checklist: https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration....
-- Craig
On Dec 3, 2015, at 3:53 PM, Craig Rodrigues rodrigc@crodrigues.org wrote:
On Tue, Dec 1, 2015 at 5:01 PM, Glyph Lefkowitz <glyph@twistedmatrix.com mailto:glyph@twistedmatrix.com> wrote:
One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.
I worked with Amber and Adi and we have the checklist: https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.... https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.md
Thank you very much for your participation in this.
The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened?
-glyph
On 4 Dec 2015, at 11:07, Glyph Lefkowitz glyph@twistedmatrix.com wrote:
On Dec 3, 2015, at 3:53 PM, Craig Rodrigues rodrigc@crodrigues.org wrote:
On Tue, Dec 1, 2015 at 5:01 PM, Glyph Lefkowitz glyph@twistedmatrix.com wrote:
One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.
I worked with Amber and Adi and we have the checklist: https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration....
Thank you very much for your participation in this.
The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened?
-glyph
When it's accepted, we can make it a GItHub issue, and then we can, well, click on the checkbox. :)
- Amber
On Thu, Dec 3, 2015 at 7:07 PM, Glyph Lefkowitz glyph@twistedmatrix.com wrote:
The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened?
The checklist is written using the GitHub flavored markdown syntax for checklists:
https://github.com/blog/1375%0A-task-lists-in-gfm-issues-pulls-comments
so when the tasks are completed, this checklist will be updated by changing:
[ ] task item foobar
to
[X] task item foobar
and committing the updated checklist to GitHub.
Adi and Amber are on board with this, so it should be fine. -- Craig
participants (7)
-
Adi Roiban -
Amber "Hawkie" Brown -
Craig Rodrigues -
Glyph Lefkowitz -
Jonathan Lange -
Tom Prince -
Tristan Seligmann