[core-workflow] We will be moving to GitHub

[Ezio Melotti]

On Sat, Jan 2, 2016 at 8:45 PM, Brett Cannon <brett at python.org> wrote:
> I was purposefully trying to avoid having this discussion start until
> Monday, but since Ezio sort has a deadline for issue-related stuff we can
> start on that side now (I still have an email planned on Monday to outline
> the initial steps to moving things over and the very first hurdle to work
> through).
>

No hurry, there's still a week before the sprint so take your time.
I also still have to merge all the GSoC improvements before looking at
these other issues.

> On Fri, 1 Jan 2016 at 17:57 Ezio Melotti <ezio.melotti at gmail.com> wrote:
>>
>> Hi,
>>
>> On Fri, Jan 1, 2016 at 9:21 PM, Brett Cannon <brett at python.org> wrote:
>> > ...
>> > We can start the discussion of how we want to handle the transition next
>> > week, but I'm going to try and step away from this whole workflow topic
>> > until Monday so I can spend the last couple of days of my vacation not
>> > thinking about this stuff. :)
>> >
>>
>> This will likely require a new PEP, that should cover:
>>   1) the new workflow (including how to handle reviews -- see below);
>>   2) the steps required for the migration and a timeline;
>>   3) a list of things that will break and/or that will need to be
>> added/replaced before/during/after the migration;
>>   4) the fate of hg.python.org;
>
>
> Yes to all of that.
>
>>
>>
>> Some of these things might already be covered by existing PEPs, but I
>> don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost
>> among all the competing PEPs and multiple threads across at least a
>> couple different MLs :).
>
>
> It will be either be a new PEP or the GitHub PEP will simply be rewritten.
>

OK.

>>
>>
>> Above you said that GitHub will be used for reviews but we will keep
>> using our bug tracker.
>
>
> Yes, we are not moving to GitHub's issue tracker.
>
>>
>> This leads to the following questions:
>>   * Do GitHub reviews only work for pull requests?
>
>
> What do you mean by this? Do you mean by patch upload?
>

With reviews here I mean inline comments similar to the ones you can
add on Rietveld.
I haven't used GitHub yet, but I assume this can be done for pull
requests (same as with BitBucket) and perhaps for patches attached to
the GitHub bug tracker.
Since we are not going to use the GitHub bug tracker, only pull
requests can be reviewed on GitHub.
If someone uploads a patch on the bug tracker it won't be possible to
review it unless we convert it to a pull request (or unless we keep
Rietveld around, but we don't want that).


>>
>>   * Are we still going to support uploading diffs/patches to the
>> tracker (short term and long term)?
>
>
> Well, "support" as in "allow". We won't be keeping Rietveld around (part of
> this move is so we can get off of Rietveld).
>

Only supporting pull requests will force all contributors to use
GitHub (unless GitHub supports "external" pull requests).  I think we
should keep supporting plain patches as an alternative, and write
tools to convert those to pull requests.  If GitHub doesn't support
"external" pull requests, we might also be able to write additional
tools to do so from the tracker directly.

>>
>>   * If so, how/where are we going to review those diffs/patches?
>
>
> I don't know. Is it possible to have the bot create a PR for an uploaded
> patch?
>

It should be possible -- we will need an account for b.p.o and then
b.p.o can commit the patches on its clone (perhaps using a different
branch for issue/patch) and send a pull request to the main clone (for
the last bit we might need to use the GitHub API, unless there are
other ways).

>>
>>   * Will patches be automatically converted to pull requests, or pull
>> requests converted to patches and attached to b.p.o issues?
>
>
> I don't think we need to upload a patch file if we already have a PR,

The goal is to have everything in the same place, even if
contributions are made both via pull requests and via patches uploaded
to b.p.o.
I think the best option is to convert all patches uploaded to b.p.o to
pull requests, so that all the reviews happen on GitHub.

> but we
> should either have a PR -> issue or issue -> PR mapping somehow. This can
> either be via bot command or specifying a PR # in the issue tracker.

b.p.o should automatically link to PRs when "PR num" appears in a
message (this can be done easily).
GitHub should also link to b.p.o issues when an issue number appears
in the PR (I don't know if/how this can be done).
For patches uploaded to b.p.o, a link to the generated PR can be added
where the "review" link is now.

> I also
> have no issue if people want to make PRs generate patches for the issue as
> well for backup purposes, but that might get a bit noisy for those following
> both an issue and a PR.
>

I'm not sure we need to generate a diff and attach it to the issue,
but PRs related to the issue should probably be listed on b.p.o where
the list of patches currently is.
One possible reason to add patches is the new patch analysis tool
written during GSoC that can extract metadata from patches (affected
files, whether the patch has tests/docs, etc.) which in turn can be
used during searches.  The tool can also be adapted to generate the
diff dynamically and extract metadata from there though.

>>
>>   * Are there plans to migrate existing Rietveld reviews to GitHub and
>> shut Rietveld down for good?
>
>
> Yes, there are plans to shut Rietveld down. I suspect we will leave it up
> until GitHub is running in an equivalent fashion to our current workflow,

Good to know.

> then we will have a deadline to work through any outstanding reviews and
> then close it up (and if we want to be thorough, set Rietveld so that it
> won't work on new issues and only pre-existing ones that already have a
> patch).
>

Or we could just copy-paste the reviews as messages on the tracker so
that languishing reviews don't hold up the Rietveld demise.

>>
>>   * If not, will Rietveld stay around and be read-only?  Or will it
>> still be used for patches uploaded to b.p.o?
>
>
> We are trying to cut down on the infrastructure we maintain, so I don't want
> Rietveld sticking around indefinitely.
>

Agreed.

>>
>>   * What else is needed to integrate GitHub and b.p.o?
>
>
> Basically a way to map an issue to a PR (or vice-versa). Probably the
> simplest solution is to allow pasting in a GitHub PR URL or the PR # to make
> the association. The other option is for the bot to accept a command to make
> the association. No reason we can't have both, though. But either way we
> should have a way to connect PRs to issues.
>

I'll have to think a bit about the best way to implement this and the
UI (I already have some ideas).

> If a bot can create PRs from a patch, then that would be nice for folks who
> don't want to use GitHub, but I don't consider that a priority. I'm happy to
> try to be accommodating to people who don't want to use GitHub for whatever
> reason, there is a limit to how much energy we should put into supporting
> that scenario past uploading a patch.
>

If we want to support this (and I think we should) we will either need
a bot (actually a Roundup detector) or a human to do it, and the
former sounds better.

>>
>>
>> About points 3 of the initial list, these are some examples of things
>> that will need to be added/updated/replaced:
>>   * the hgroundup hook[2] that post messages to b.p.o when a commit
>> includes an issue number needs to be replaced;
>>   * the hgirker hook[3] used for deadparrot on #python-dev needs to be
>> replaced (probably there is already an irker-git hook that can be
>> used/adapted);
>>   * the hgbuildbot[4] hook that triggers the buildbots on commit needs
>> to be replaced;
>>   * other hg hooks[5] might need to be rewritten/replaced;
>>   * the script[6] that generates bug tracker links to hg.p.o needs to
>> be updated (for both cs ids and paths);
>>   * the hg code that converts issue links in commit messages to b.p.o
>> links needs to be replaced;
>>   * other places where issue numbers appears on GitHub should generate
>> links to b.p.o;
>>   * the hg-cpydev hg extension [7] should be ported to git (optional);
>>   * the buildbots need to be updated if they are going to pull the
>> source from github and use git;
>>   * the bug tracker will need to be updated to interact with github;
>>   * the devguide needs to be updated (both to cover the new workflow
>> and update links/commands);
>
>
> Yes to all of these (I think; depends if we change how something operates in
> terms of associations).
>
>>
>>
>> FWIW next weekend (9-10 January) we are organizing a sprint in
>> Helsinki, and I'm planning to work on the bug tracker, so I might be
>> able to start addressing some of these points.
>
>
> I think deciding how to associate issues with PRs would be a great thing to
> work through. Otherwise we need to decide if we want to go with an issue
> tracker solution to the NEWS file or if we want a individual file solution
> (which happens to be bot-friendly). Those two things are probably the most
> critical starting points.
>

There's also another point that I haven't seen being discussed: how we
will handle the author and the committer fields.
Currently we only know the committer so, while migrating to git, this
can either be duplicated in the author field or just preserved in the
committer field leaving the author field empty.  Afterwards we can
start using the two fields for what they are meant for.
A possible problem is how to update them when commits/merges are
performed by bots (e.g. if b.p.o automatically converts a patch into a
pull request, or if a core dev merges a patch by clicking on a
button).  Perhaps someone more familiar with Git/GitHub can clarify
this (I think it's possible to specify author/committer manually, but
I don't know how GitHub handles it on its side).

Best Regards,
Ezio Melotti

> -Brett
>
>>
>>
>> Best Regards,
>> Ezio Melotti
>>
>> P.S. enjoy your last few days of vacation while you can ;)
>>
>> [0]: https://www.python.org/dev/peps/pep-0507/
>> [1]: https://www.python.org/dev/peps/pep-0481/
>> [2]: https://hg.python.org/hooks/file/tip/hgroundup.py
>> [3]: https://hg.python.org/hooks/file/tip/hgirker.py
>> [4]: https://hg.python.org/hooks/file/tip/hgbuildbot.py
>> [5]: https://hg.python.org/hooks/file/tip
>> [6]:
>> https://hg.python.org/tracker/python-dev/file/tip/extensions/local_replace.py
>> [7]: https://bitbucket.org/introom/hg-cpydev