On 17 January 2016 at 23:20, Oleg Broytman <phd@phdru.name> wrote:

On Sun, Jan 17, 2016 at 08:48:42PM +0000, Brett Cannon <brett@python.org> wrote:

...

Rejected Ideas ============== Commit multi-release changes in bugfix branch first --------------------------------------------------- As the current development process has changes committed in the oldest branch first and then merged up to the default branch, the question came up as to whether this workflow should be perpetuated. In the end it was decided that committing in the newest branch and then cherrypicking changes into older branches would work best as

That's a rather strange workflow. Gitworkflows recommands against it [1], people recommends against it [2], [3].

...

most people will instinctively work off the newest branch and it is a more common workflow when using Git [#git]_.

Most people commit to master because most [visible] repositories are for web sites/services/applications that don't have many branches. But for a project with a lot of release branches merge workflow is usually recommended.

1. https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html 2. https://stackoverflow.com/a/1241829 3. http://dan.bravender.net/2011/10/20/Why_cherry-picking_should_not_be_part_of...

In theory I would also prefer sticking with the current process of merging the 3.x branches, and only cherry-picking for the 2.7 branch. Are there any more details on this decision? When making a patch for review I currently do it off the “default” branch, because then one patch will usually encompass all the changes needed (but a patch against 3.5 for instance may miss an API added in 3.6). But when actually making a commit I think it is better to commit first to 3.5 (for instance). For handling pull requests from others, I can see that having the pull request against master would be easier for the original contributor. On the other hand, if the contributor were able to pre-arrange all the merges between branches (including resolving conflicts), it might be help the person doing the final push. But maybe this is too complicated for all parties; I dunno.

On Sun, 17 Jan 2016 at 15:20 Oleg Broytman <phd@phdru.name> wrote:

Hi!

On Sun, Jan 17, 2016 at 08:48:42PM +0000, Brett Cannon <brett@python.org> wrote:

...

Change sys._mercurial ''''''''''''''''''''' Once Python is no longer kept in Mercurial, the ``sys._mercurial`` attribute will need to be removed. An equivalent ``sys._git`` attribute will be needed to take its place.

Shouldn't there be a sys._vcs with possible values 'subversion', 'mercurial' or 'git'? Well, currently it can be only 'git', of course, but in the future it will change.

This was discussed already the first time the attribute was added for Subversion and the decision that a VCS-specific attribute was preferred.

...

Rejected Ideas ============== Commit multi-release changes in bugfix branch first --------------------------------------------------- As the current development process has changes committed in the oldest branch first and then merged up to the default branch, the question came up as to whether this workflow should be perpetuated. In the end it was decided that committing in the newest branch and then cherrypicking changes into older branches would work best as

That's a rather strange workflow. Gitworkflows recommands against it [1], people recommends against it [2], [3].

...

most people will instinctively work off the newest branch and it is a more common workflow when using Git [#git]_.

Most people commit to master because most [visible] repositories are for web sites/services/applications that don't have many branches. But for a project with a lot of release branches merge workflow is usually recommended.

1. https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html 2. https://stackoverflow.com/a/1241829 3. http://dan.bravender.net/2011/10/20/Why_cherry-picking_should_not_be_part_of...

I'm personally not going to argue about this and change the current proposal right now, but if people want to start a new thread to discuss whether merging up or cherry-picking down is the better workflow they can and if consensus can be reached I will change the PEP.

On Sun, 17 Jan 2016 at 16:30 Ezio Melotti <ezio.melotti@gmail.com> wrote:

On Sun, Jan 17, 2016 at 10:48 PM, Brett Cannon <brett@python.org> wrote:

...

Here is the first complete draft of the PEP laying out what needs to happen to migrate to GitHub (along with some extras that would be nice). The PEP can also be viewed at

...

or at https://www.python.org/dev/peps/ once the PEP index is updated.

My hope is that I captured all of the requirements for the various repositories. If we can agree that is the case then we can start work on

https://github.com/brettcannon/github-transition-pep/blob/master/pep-0512.rs... the

...

various requirements and start the migration work!

---------- PEP: 512 Title: Migrating from hg.python.org to GitHub Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett@python.org> Status: Active Type: Process Content-Type: text/x-rst Created: Post-History: 17-Jan-2015

Abstract ======== This PEP outlines the steps required to migrate Python's development process from Mercurial [#hg]_ as hosted at hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting the minimum goals of this PEP should allow for the development process of Python to be as productive as it currently is, and meeting its extended goals should improve it.

Rationale ========= In 2014, it became obvious that Python's custom development process was becoming a hindrance. As an example, for an external contributor to submit a fix for a bug that eventually was committed, the basic steps were:

1. Open an issue for the bug at bugs.python.org [#b.p.o]_. 2. Checkout out the CPython source code from hg.python.org [#h.p.o]_. 3. Make the fix. 4. Upload a patch. 5. Have a core developer review the patch using our fork of the Rietveld code review tool [#rietveld]_. 6. Download the patch to make sure it still applies cleanly. 7. Run the test suite manually. 8. Commit the change manually. 9. If the change was for a bugfix release, merge into the in-development branch. 10. Run the test suite manually again. 11. Commit the merge. 12. Push the changes.

This is a very heavy, manual process for core developers. Even in the simple case, you could only possibly skip the code review step, as you would still need to build the documentation. This led to patches languishing on the issue tracker due to core developers not being able to work through the backlog fast enough to keep up with submissions. In turn, that led to a side-effect issue of discouraging outside contribution due to frustration from lack of attention, which is dangerous problem for an open source project as it runs counter to having a viable future for the project. Simply accepting patches uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an external contributor, it is as slow and burdensome as it gets for a core developer to work with.

Hence the decision was made in late 2014 that a move to a new development process was needed. A request for PEPs proposing new workflows was made, in the end leading to two: PEP 481 and PEP 507 proposing GitHub [#github]_ and GitLab [#gitlab]_, respectively.

The year 2015 was spent off-and-on working on those proposals and trying to tease out details of what made them different from each other on the core-workflow mailing list [#core-workflow]_. PyCon US 2015 also showed that the community was a bit frustrated with our process due to both cognitive overhead for new contributors and how long it was taking for core developers to look at a patch (see the end of Guido van Rossum's keynote at PyCon US 2015 [#guido-keynote]_ as an example of the frustration).

On January 1, 2016, the decision was made by Brett Cannon to move the development process to GitHub. The key reasons for choosing GitHub were [#reasons]_:

* Maintaining custom infrastructure has been a burden on volunteers (e.g., a custom fork of Rietveld [#rietveld]_ that is not being maintained is currently being used). * The custom workflow is very time-consuming for core developers (not enough automated tooling built to help support it). * The custom workflow is a hindrance to external contributors (acts as a barrier of entry due to time required to ramp up on development process). * There is no feature differentiating GitLab from GitHub beyond GitLab being open source. * Familiarity with GitHub is far higher amongst core developers and external contributors than with GitLab. * Our BDFL prefers GitHub (who would be the first person to tell you that his opinion shouldn't matter, but the person making the decision felt it was important that the BDFL feel comfortable with the workflow of his own programming language to encourage his continued participation).

There's even already an unofficial image to use to represent the migration to GitHub [#pythocat]_.

The overarching goal of this migration is to improve the development process to the extent that a core developer can go from external contribution submission through all the steps leading to committing said contribution all from within a browser on a tablet with WiFi. All of this will be done in such a way that if an external contributor chooses not to use GitHub then they will continue to have that option.

Repositories to Migrate ======================= While hg.python.org [#h.p.o]_ hosts many repositories, there are only six key repositories that must move:

1. devinabox [#devinabox-repo]_ 2. benchmarks [#benchmarks-repo]_ 3. tracker [#tracker-repo]_

I don't think the tracker repo /must/ move (at least not immediately).

I don't think so either, just that it should probably happen at some point, especially if hg.python.org gets shut down.

The tracker repo includes 7 subrepos (our own forks of Roundup, Rietveld, and django-gae2django,

Is that being used by anyone?

and the instances for b.p.o, the meta tracker, the Jython tracker,

They might not stick with their issue tracker (the Jython team knows what's going on and they are going to have a discussion over what they will end up doing).

and the setuptools tracker

Is that being used? https://bitbucket.org/pypa/setuptools/issues seems to be actively used.

) and migrating it now means that:

1) The total number of repos to be converted will be 12* instead of 5; 2) I (and RDM/MvL) will have to learn a new workflow while updating the tracker for the migration;

* 10 if we exclude Rietveld and gae2django (but that means holding up the migration until they are removed)

I think it would be best to: 1) migrate the other repos; 2) work on the b.p.o/github integration using HG; 3) remove Rietveld; 4) (possibly) migrate the 4 tracker instances to GitHub

That's fine by me. As I said, migration is dependent on what happens to hg.python.org as to whether it has to happen or not.

Our Roundup fork is also an HG clone of upstream Roundup (with a separate branch with our changes), so migrating that to git, while doable, might be problematic. FWIW when we switched from SVN to HG it took over a year and half before I migrated the tracker repos, and I don't remember anyone being particularly concerned that we were still using SVN or relieved when we eventually moved it to HG. Personally I wouldn't mind if we kept them on HG (RDM/MvL might disagree though), but if it's a burden for the infra-team to keep hg.python.org alive, I would prefer if we moved them later rather than sooner.

Another option is to move it to Bitbucket if you want to keep it on Mercurial if hg.python.org doesn't stick around.

...

4. peps [#peps-repo]_ 5. devguide [#devguide-repo]_ 6. cpython [#cpython-repo]_

The devinabox, benchmarksm and tracker repositories are code-only.

Typo: benchmarks

...

The peps and devguide repositories involve the generation of webpages. And the cpython repository has special requirements for integration with bugs.python.org [#b.p.o]_.

Migration Plan ============== The migration plan is separated into sections based on what is required to migrate the repositories listed in the `Repositories to Migrate`_ section. Completion of requirements outlined in each section should unblock the migration of the related repositories. The sections are expected to be completed in order, but not necessarily the requirements within a section.

Requirements for Code-Only Repositories --------------------------------------- Completion of the requirements in this section will allow the devinabox, benchmarks, and tracker repositories to move to GitHub. While devinabox has a sufficiently descriptive name, the benchmarks repository does not; therefore, it will be named "python-benchmark-suite". The tracker repo also has a misleading name and will be renamed "bugs.python.org".

As mentioned above, tracker contains 7 subrepos, and the bugs.python.org instance is tracker/python-dev. Roundup (tracker/roundup) is running 4 tracker instances (tracker/python-dev, tracker/meta for the metatracker, tracker/jython for the Jython tracker, and tracker/setuptools for https://bugs.python.org/setuptools/ (now hosted on BitBucket but kept around for historical reasons)).

...

Create a 'python-dev' team '''''''''''''''''''''''''' To manage permissions, a 'python-dev' team will be created as part of the python organization [#github-python-org]_. Any repository that is moved will have the 'python-dev' team added to it with write permissions [#github-org-perms]_. Anyone who previously had rights to manage SSH keys on hg.python.org will become a team maintainer for the 'python-dev' team.

Define commands to move a Mercurial repository to Git ''''''''''''''''''''''''''''''''''''''''''''''''''''' Since moving to GitHub also entails moving to Git [#git]_, we must decide what tools and commands we will run to translate a Mercurial repository to Git. The exact tools and steps to use are an open issue; see `Tools and commands to move from Mercurial to Git`_.

CLA enforcement ''''''''''''''' A key part of any open source project is making sure that its source code can be properly licensed. This requires making sure all people making contributions have signed a contributor license agreement (CLA) [#cla]_. Up until now, enforcement of CLA signing of contributed code has been enforced by core developers checking whether someone had an ``*`` by their username on bugs.python.org [#b.p.o]_. With this migration, the plan is to start off with automated checking and enforcement of contributors signing the CLA.

Adding GitHub username support to bugs.python.org +++++++++++++++++++++++++++++++++++++++++++++++++ To keep tracking of CLA signing under the direct control of the PSF, tracking who has signed the PSF CLA will be continued by marking that fact as part of someone's bugs.python.org user profile. What this means is that an association will be needed between a person's bugs.python.org [#b.p.o]_ account and their GitHub account, which will be done through a new field in a user's profile.

We have to decide how to deal with users that don't have a b.p.o account. The two options that we discussed in the previous mails are: 1) require them to create a b.p.o account; 2) allow them to log in to b.p.o using their github account; (see also next section)

Both require creating an account, it just varies whether they log in using GitHub or not. I don't see how we can avoid that if we are going to continue to own the CLA dataset. Honestly I don't see this as a big issue as we have not seemed to have any issues with people creating accounts up to this point.

...

A bot to enforce CLA signing ++++++++++++++++++++++++++++ With an association between someone's GitHub account and their bugs.python.org [#b.p.o]_ account, which has the data as to whether someone has signed the CLA, a bot can monitor pull requests on GitHub and denote whether the contributor has signed the CLA.

If the user has signed the CLA, the bot will add a positive label to the issue to denote the pull request has no CLA issues (e.g., a green label stating, "CLA: ✓"). If the contributor has not signed a CLA, a negative label will be added to the pull request will be blocked using GitHub's status API (e.g., a red label stating, "CLA: ✗"). If a contributor lacks a bugs.python.org account, that will lead to another label (e.g., "CLA: ✗ (no account)"). Using a label for both positive and negative cases provides a fallback notification if the bot happens to fail, preventing potential false-positives or false-negatives. It also allows for an easy way to trigger the bot again by simply removing a CLA-related label.

Requirements for Web-Related Repositories ----------------------------------------- Due to their use for generating webpages, the devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need their respective processes updated to pull from their new Git repositories.

The devguide repository might also need to be named ``python-devguide`` to make sure the repository is not ambiguous when viewed in isolation from the python organization [#github-python-org]_.

Requirements for the cpython Repository --------------------------------------- Obviously the most active and important repository currently hosted at hg.python.org [#h.p.o]_ is the cpython repository [#cpython-repo]_. Because of its importance and high- frequency use, it requires more tooling before being moved to GitHub compared to the other repositories mentioned in this PEP.

Document steps to commit a pull request ''''''''''''''''''''''''''''''''''''''' During the process of choosing a new development workflow, it was decided that a linear history is desired. This means that the convenient "Merge" button in GitHub pull requests is undesireable, as

Typo: undesirable

...

it creates a merge commit along with all of the contributor's individual commits (this does not affect the other repositories where the desire for a linear history doesn't exist).

Luckily, Git [#git]_ does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history.

A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [#b.p.o]_. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author.

The exact sequence of commands that will be given as guidelines to core developers is an open issue: `Git CLI commands for committing a pull request to cpython`_.

Handling Misc/NEWS '''''''''''''''''' Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic for changes which spanned Python releases. Often times there will be merge conflicts when committing a change between e.g., 3.5 and 3.6 only in the ``Misc/NEWS`` file. It's so common, in fact, that the example instructions in the devguide explicitly mention how to resolve conflicts in the ``Misc/NEWS`` file [#devguide-merge-across-branches]_. As part of our tool modernization, working with the ``Misc/NEWS`` file will be simplified.

There are currently two competing approaches to solving the ``Misc/NEWS`` problem which are discussed in an open issue: `How to handle the Misc/NEWS file`_.

Handling Misc/ACKS '''''''''''''''''' Traditionally the ``Misc/ACKS`` file [#acks-file]_ has been managed by hand. But thanks to Git supporting an ``author`` value as well as a ``committer`` value per commit, authorship of a commit can be part of the history of the code itself.

As such, manual management of ``Misc/ACKS`` will become optional. A script will be written that will collect all author and committer names and merge them into ``Misc/ACKS`` with all of the names listed prior to the move to Git. Running this script will become part of the release process.

Linking pull requests to issues ''''''''''''''''''''''''''''''' Historically, external contributions were attached to an issue on bugs.python.org [#b.p.o]_ thanks to the fact that all external contributions were uploaded as a file. For changes committed by a core developer who committed a change directly, the specifying of an issue number in the commit message of the format ``Issue #`` at the start of the message led to a comment being posted to the issue linking to the commit.

Linking a pull request to an issue ++++++++++++++++++++++++++++++++++ An association between a pull request and an issue is needed to track when a fix has been proposed. The association needs to be many-to-one as there can take multiple pull requests to solve a single issue (technically it should be a many-to-many association for when a single fix solves multiple issues, but this is fairly rare and issues can be merged into one using the ``Superceder`` field on the issue tracker).

Association between a pull request and an issue will be done based on detecting the regular expression``[Ii]ssue #(?P<bpo_id>\d+)``. If this is specified in either the title or in the body of a message on a pull request then connection will be made on bugs.python.org [#b.p.o]_. A label will also be added to the pull request to signify that the connection was made successfully. This could lead to incorrect associations if the wrong issue or referencing another issue was done, but these are rare occasions.

Is there a way to associate the PR to an issue (in case the user forgot) or change association (in case it got the wrong number) after the creation of the PR?

You tell me. :) I assume any bot we write to handle this will monitor PR-level comments since GitHub doesn't notify on title changes. But we can choose any workflow we want, so if we want it to be an explicit command like `/bot issue 12345` then we can do that instead.

...

Notify the issue if the pull request is committed +++++++++++++++++++++++++++++++++++++++++++++++++ Once a pull request is closed (merged or not), the issue should be updated to reflect this fact.

Update linking service for mapping commit IDs to URLs ''''''''''''''''''''''''''''''''''''''''''''''''''''' Currently you can use https://hg.python.org/lookup/ with a revision ID from either the Subversion or Mercurial copies of the cpython repo [#cpython-repo]_ to get redirected to the URL for that revision in the Mercurial repository. The URL rewriter will need to be updated to redirect to the Git repository and to support the new revision IDs created for the Git repository.

Create https://git.python.org ''''''''''''''''''''''''''''' Just as hg.python.org [#h.p.o]_ currently points to the Mercurial repository for Python, git.python.org should do the equivalent for the Git repository.

Is this simply a redirect to GitHub or something else? We might also want to encourage the use of git.python.org in the Devguide and elsewhere to make future migrations simpler.

From my understanding it can't be any else but a redirect. But yes, we will encourage using git.python.org to keep the coupling to GitHub loose for when we make this kind of change again in the future.

...

Backup of pull request data ''''''''''''''''''''''''''' Since GitHub [#github]_ is going to be used for code hosting and code review, those two things need to be backed up. In the case of code hosting, the backup is implicit as all non-shallow Git [#git]_ clones contain the full history of the repository, hence there will be many backups of the repository.

If possible I would still prefer an "official" backup. I don't think we want to go around asking who has the latest copy of the repo in case something happens to GitHub.

I would be shocked if a core developer doesn't have an up-to-date repository (then again I don't expect to have to flee GitHub overnight, giving us time to update). But if you want to make it this an optional feature then I'm fine with that (I don't think the PSF infrastructure team will have issue with a regularly updated clone of the repository).

...

The code review history does not have the same implicit backup mechanism as the repository itself. That means a daily backup of code review history should be done so that it is not lost in case of any issues with GitHub. It also helps guarantee that a migration from GitHub to some other code review system is feasible were GitHub to disappear overnight.

Is there an API to export this or do we need to write another bot and decide how to collect/store the reviews?

I don't know; maybe someone else can speak up on this topic who has experience. I know for a fact that the webhooks can be used, but whether there is some specific API to get a dump I don't know.

...

Change sys._mercurial ''''''''''''''''''''' Once Python is no longer kept in Mercurial, the ``sys._mercurial`` attribute will need to be removed. An equivalent ``sys._git`` attribute will be needed to take its place.

Optional, Planned Features -------------------------- Once the cpython repository [#cpython-repo]_ is migrated, all repositories will have been moved to GitHub [#github]_ and the development process should be on equal footing as before. But a key reason for this migration is to improve the development process, making it better than it has ever been. This section outlines some plans on how to improve things.

It should be mentioned that overall feature planning for bugs.python.org [#b.p.o]_ -- which includes plans independent of this migration -- are tracked on their own wiki page [#tracker-plans]_.

Bot to handle pull request merging '''''''''''''''''''''''''''''''''' As stated in the section entitled "`Document steps to commit a pull request`_", the desire is to maintain a linear history for cpython. Unfortunately, Github's [#github]_ web-based workflow does not support a linear history. Because of this, a bot should be written to substitute for GitHub's in-browser commit abilities.

To start, the bot should accept commands to commit a pull request against a list of branches. This allows for committing a pull request that fixes a bug in multiple versions of Python.

More advanced features such as a commit queue can come later. This would linearly apply accepted pull requests and verify that the commits did not interfere with each other by running the test suite and backing out commits if the test run failed. To help facilitate the speed of testing, all patches committed since the last test run can be applied and run in a single test run as the optimistic assumption is that the patches will work in tandem.

Inspiration or basis of the bot could be taken from pre-existig bots such as Homu [#homu]_ or Zuul [#zuul]_.

The name given to this bot in order to give it commands is an open issue: `Naming the commit bot`_.

Continuous integration per pull request ''''''''''''''''''''''''''''''''''''''' To help speed up pull request approvals, continuous integration testing should be used. This helps mitigate the need for a core developer to download a patch simply to run the test suite against the patch.

Which free CI service to use is an open issue: `Choosing a CI service`_.

Test coverage report '''''''''''''''''''' Getting an up-to-date test coverage report for Python's standard library would be extremely beneficial as generating such a report can take quite a while to produce.

There are a couple pre-existing services that provide free test coverage for open source projects. Which option is best is an open issue: `Choosing a test coverage service`_.

Do we want to eventually request that all new code introduced is fully covered by tests?

I have always told sprinters that 80% is a good guideline. I wouldn't ever want a rule, though. Obviously lowering coverage would not be great and should be avoided, but if

I think having an indication of how much code in a PR is covered by tests would be useful regardless of the answer to the previous question.

I know Coveralls already does this; don't know about Codecov.

...

Notifying issues of pull request comments ''''''''''''''''''''''''''''''''''''''''' The current development process does not include notifying an issue on bugs.python.org [#b.p.o]_ when a review comment is left on Rietveld [#rietveld]_. It would be nice to fix this so that people can subscribe only to comments at bugs.python.org and not GitHub [#github]_ and yet still know when something occurs on GitHub in terms of review comments on relevant pull requests. Current thinking is to post a comment to bugs.python.org to the relevant issue when at least one review comment has been made over a certain period of time (e.g., 15 or 30 minutes). This keeps the email volume down for those that receive both GitHub and bugs.python.org email notifications while still making sure that those only following bugs.python.org know when there might be a review comment to address.

Allow bugs.python.org to use GitHub as a login provider ''''''''''''''''''''''''''''''''''''''''''''''''''''''' As of right now, bugs.python.org [#b.p.o]_ allows people to log in using Google, Launchpad, or OpenID credentials. It would be good to expand this to GitHub credentials.

Web hooks for re-generating web content ''''''''''''''''''''''''''''''''''''''' The content at https://docs.python.org/, https://docs.python.org/devguide, and https://www.python.org/dev/peps/ are all derived from files kept in one of the repositories to be moved as part of this migration. As such, it would be nice to set up appropriate webhooks to trigger rebuilding the appropriate web content when the files they are based on change instead of having to wait for, e.g., a cronjob to trigger.

Link web content back to files that it is generated from '''''''''''''''''''''''''''''''''''''''''''''''''''''''' It would be helpful for people who find issues with any of the documentation that is generated from a file to have a link on each page which points back to the file on GitHub [#github]_ that stores the content of the page. That would allow for quick pull requests to fix simple things such as spelling mistakes.

Here you are talking about PEPs/devguide/docs.p.o, right?

Yes.

FWIW this was problem was supposed to be fixed with pootle, but that project seems dead (not sure if due to technical reasons, or simply because no one had time to integrate it).

...

Splitting out parts of the documentation into their own repositories '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' While certain parts of the documentation at https://docs.python.org change with the code, other parts are fairly static and are not tightly bound to the CPython code itself. The following sections of the documentation fit this category of slow-changing, loosely-coupled:

* `Tutorial <https://docs.python.org/3/tutorial/index.html>`__ * `Python Setup and Usage <https://docs.python.org/3/using/index.html `__ * `HOWTOs <https://docs.python.org/3/howto/index.html>`__ * `Installing Python Modules <https://docs.python.org/3/installing/index.html>`__ * `Distributing Python Modules <https://docs.python.org/3/distributing/index.html>`__ * `Extending and Embedding <https://docs.python.org/3/extending/index.html>`__ * `FAQs <https://docs.python.org/3/faq/index.html>`__

These parts of the documentation could be broken out into their own repositories to simplify their maintenance and to expand who has commit rights to them to ease in their maintenance.

I would still consider these somewhat dynamic (especially between Py 2 and Py 3). There are other documents that are more version-independent, such as the whatsnew pages,

The What's New docs get updated with changes so that can't be pulled out from the cpython repo (especially if my dream ever comes true of making people be better about making sure that document gets updated with changes that warrant being mentioned there).

the meta information section at the bottom of docs.p.o/index (that includes pages such as https://docs.python.org/3/bugs.html ) and a new page with the current status of the releases proposed in http://bugs.python.org/issue25296

...

Status ====== Requirements for migrating the devinabox [#devinabox-repo]_, benchmarks [#benchmarks-repo]_, and tracker [#tracker-repo]_ repositories:

* Not started

- `Create a 'python-dev' team`_ - `Define commands to move a Mercurial repository to Git`_ - `Adding GitHub username support to bugs.python.org`_ - `A bot to enforce CLA signing`_

* In progress

- None

* Completed

- None

Repositories whose build steps need updating:

* Not started

- peps [#peps-repo]_ - devguide [#devguide-repo]_

* In progress

- None

* Completed

- None

Requirements to move over the cpython repo [#cpython-repo]_:

* Not started

- `Document steps to commit a pull request`_ - `Handling Misc/NEWS`_ - `Handling Misc/ACKS`_ - `Linking a pull request to an issue`_ - `Notify the issue if the pull request is committed`_ - `Update linking service for mapping commit IDs to URLs`_ - `Create https://git.python.org`_ - `Backup of pull request data`_ - `Change sys._mercurial`_

* In progress

- None

* Completed

- None

Optional features:

* Not started

- `Bot to handle pull request merging`_ - `Continuous integration per pull request`_ - `Test coverage report`_ - `Notifying issues of pull request comments`_ - `Allow bugs.python.org to use GitHub as a login provider`_ - `Web hooks for re-generating web content`_ - `Link web content back to files that it is generated from`_ - `Splitting out parts of the documentation into their own

repositories`_

...

* In progress

- None

* Completed

- None

Open Issues =========== For this PEP, open issues are ones where a decision needs to be made to how to approach or solve a problem. Open issues do not entail coordination issues such as who is going to write a certain bit of code.

The fate of hg.python.org ------------------------- With the code repositories moving over to Git [#git]_, there is no technical need to keep hg.python.org [#h.p.o]_ running. Having said that, some in the community would like to have it stay functioning as a Mercurial [#hg]_ mirror of the Git repositories. Others have said that they still want a mirror, but one using Git.

As maintaining hg.python.org is not necessary, it will be up to the PSF infrastructure committee to decide if they want to spend the time and resources to keep it running. They may also choose whether they want to host a Git mirror on PSF infrastructure.

+1 to keep a read-only mirror. Also see above about the tracker repo and our Roundup fork.

...

Tools and commands to move from Mercurial to Git ------------------------------------------------ A decision needs to be made on exactly what tooling and what commands involving those tools will be used to convert a Mercurial repository to Git. Currently a suggestion has been made to use https://github.com/frej/fast-export. Another suggestion is to use https://github.com/felipec/git-remote-hg. Finally, http://hg-git.github.io/ has been suggested.

Git CLI commands for committing a pull request to cpython --------------------------------------------------------- Because Git [#git]_ may be a new version control system for core developers, the commands people are expected to run will need to be written down. These commands also need to keep a linear history while giving proper attribution to the pull request author.

Another set of commands will also be necessary for when working with a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear history will be kept implicitly, but it will need to make sure to keep/add attribution.

Nick Coghlan, Pierre-Yves David (a Mercurial dev), and Shiyao Ma (one of our GSoC student) have been working on an HG extension that simplifies interaction with the bug tracker (see the list of patches, download/apply them, upload new patches): https://bitbucket.org/introom/hg-cpydev In a previous email, someone mentioned an alias that allows an easier interaction with PRs. Would it make sense to write and distribute an official git extension that provides extra commands/aliases for these set of commands? (I guess the answer depends on how many tasks we have and how straightforward it is to do with plain git commands.)

It quite possibly might be. Otherwise shell commands could be written and kept in Tools/. I major perk IMO with Git over Mercurial is Git Bash comes with GIt and that gives you Bash on Windows. That makes writing cross-platform shell scripts to help with this sort of thing easy without leaving Windows users stranded.

...

How to handle the Misc/NEWS file -------------------------------- There are two competing approaches to handling ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on bugs.python.org [#b.p.o]_. This would mean an issue that is marked as "resolved" could not be closed until a news entry is added in the "news" field in the issue tracker. The benefit of tying the news entry to the issue is it makes sure that all changes worthy of a news entry have an accompanying issue. It also makes classifying a news entry automatic thanks to the Component field of the issue. The Versions field of the issue also ties the news entry to which Python releases were affected. A script would be written to query bugs.python.org for relevant new entries for a release and to produce the output needed to be checked into the code repository. This approach is agnostic to whether a commit was done by CLI or bot.

The competing approach is to use an individual file per news entry, containg the text for the entry. In this scenario each feature

Typo: containing

...

release would have its own directory for news entries and a separate file would be created in that directory that was either named after the issue it closed or a timestamp value (which prevents collisions). Merges across branches would have no issue as the news entry file would still be uniqeuely named and in the directory of the latest

Typo: uniquely

...

version that contained the fix. A script would collect all news entry files no matter what directory they reside in and create an appropriate news file (the release directory can be ignored as the mere fact that the file exists is enough to represent that the entry belongs to the release). Classification can either be done by keyword in the new entry file itself or by using subdirectories representing each news entry classification in each release directory (or classification of news entries could be dropped since critical information is captured by the "What's New" documents which are organized). The benefit of this approach is that it keeps the changes with the code that was actually changed. It also ties the message to being part of the commit which introduced the change. For a commit made through the CLI, a script will be provided to help generate the file. In a bot-driven scenario, the merge bot will have a way to specify a specific news entry and create the file as part of its flattened commit (while most likely also supporting using the first line of the commit message if no specific news entry was specified). Code for this approach has been written previously for the Mercurial workflow at http://bugs.python.org/issue18967. There is also a tool from the community at https://pypi.python.org/pypi/towncrier.

Does using git (and fast-forward merges, rebases, etc.) still create conflicts? (I guess the answer is yes, but perhaps we should double-check.)

I believe so, but I guess I could be wrong since I have not explicitly tested it.

If it does, there's also a third option: writing a merge script. I wrote a basic one for hg and it seemed to work decently, perhaps with git it's even easier.

I'll mention it, but i suspect the file-based solution will win out based on the feeling I have gotten from people when this topic has come up before.

...

Naming the commit bot --------------------- As naming things can lead to bikeshedding of epic proportions, Brett Cannon will choose the final name of the commit bot (the name of the project for the bot itself can be anything, this is purely for the name used in giving commands to the bot). The name will come from Monty Python, which is only fitting since Python is named after the comedy troupe. It will most likely come from 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be how Brett was introduced to Monty Python). Current ideas on the name include:

"Black Knight" sketch [#black-knight-sketch]_:

* black-knight * none-shall-pass * just-a-flesh-wound

"Bridge of Death" sketch [#bridge-of-death-sketch]_:

* bridge-keeper * man-from-scene-24 * five-questions * what-is-your-quest * blue-no-green * air-speed-velocity * your-favourite-colour (and that specific spelling; Monty Python is British, after all)

"Killer rabbit" sketch [#killer-rabbit-sketch]_:

* killer-rabbit * holy-hand-grenade * 5-is-right-out

"Witch Village" sketch [#witch-village-sketch]_:

* made-of-wood * burn-her

"French Taunter" sketch [#french-taunter-sketch]_:

* elderberries * kanigget

"Constitutional Peasants" sketch [#constitutional-peasants-sketch]_:

* dennis * from-the-masses * watery-tart

"Knights Who Say Ni" sketch [#ni-sketch]_:

* shubbery * ni

From "Monty Python and the Holy Grail" in general:

* brave-sir-robin

FWIW the bot that currently reports HG commits on IRC is called deadparrot.

I'll take that into consideration. :) -Brett

Best Regards, Ezio Melotti

...

Choosing a CI service --------------------- There are various CI services that provide free support for open source projects hosted on GitHub [#github]_. Two such examples are Travis [#travis]_ and Codeship [#codeship]_. Whatever solution is chosen will need to not time out in the time it takes to execute Python's test suite. It should optimally provide access to multiple C compilers for more thorough testing. Network access is also beneficial.

The current CI service for Python is Pypatcher [#pypatcher]_. A request can be made in IRC to try a patch from bugs.python.org [#b.p.o]_. The results can be viewed at https://ci.centos.org/job/cPython-build-patch/ .

Choosing a test coverage service -------------------------------- Getting basic test coverage of Python's standard library can be created simply by using coverage.py [#coverage]_. Getting thorough test coverage is actually quite tricky, with the details outlined in the devinabox's README [#devinabox-repo]_. It would be best if a service could be found that would allow for thorough test coverage, but it might not be feasible.

Free test coverage services include Coveralls [#coveralls]_ and Codecov [#codecov]_.

Rejected Ideas ============== Separate Python 2 and Python 3 repositories ------------------------------------------- It was discussed whether separate repositories for Python 2 and Python 3 were desired. The thinking was that this would shrink the overall repository size which benefits people with slow Internet connections or small bandwidth caps.

In the end it was decided that it was easier logistically to simply keep all of CPython's history in a single repository.

Commit multi-release changes in bugfix branch first --------------------------------------------------- As the current development process has changes committed in the oldest branch first and then merged up to the default branch, the question came up as to whether this workflow should be perpetuated. In the end it was decided that committing in the newest branch and then cherrypicking changes into older branches would work best as most people will instinctively work off the newest branch and it is a more common workflow when using Git [#git]_.

Deriving ``Misc/NEWS`` from the commit logs ------------------------------------------- As part of the discussion surrounding `Handling Misc/NEWS`_, the suggestion has come up of deriving the file from the commit logs itself. In this scenario, the first line of a commit message would be taken to represent the news entry for the change. Some heuristic to tie in whether a change warranted a news entry would be used, e.g., whether an issue number is listed.

This idea has been rejected due to some core developers preferring to write a news entry separate from the commit message. The argument is the first line of a commit message compared to that of a news entry have different requirements in terms of brevity, what should be said, etc.

References ========== .. [#h.p.o] https://hg.python.org

.. [#GitHub] GitHub (https://github.com)

.. [#hg] Mercurial (https://www.mercurial-scm.org/)

.. [#git] Git (https://git-scm.com/)

.. [#b.p.o] https://bugs.python.org

.. [#rietveld] Rietveld (https://github.com/rietveld-codereview/rietveld )

.. [#gitlab] GitLab (https://about.gitlab.com/)

.. [#core-workflow] core-workflow mailing list (https://mail.python.org/mailman/listinfo/core-workflow)

.. [#guido-keynote] Guido van Rossum's keynote at PyCon US (https://www.youtube.com/watch?v=G-uKNd5TSBw)

.. [#reasons] Email to core-workflow outlining reasons why GitHub was selected

( https://mail.python.org/pipermail/core-workflow/2016-January/000345.html)

.. [#benchmarks-repo] Mercurial repository for the Unified Benchmark Suite (https://hg.python.org/benchmarks/)

.. [#devinabox-repo] Mercurial repository for devinabox (https://hg.python.org/devinabox/)

.. [#peps-repo] Mercurial repository of the Python Enhancement Proposals (https://hg.python.org/peps/)

.. [#tracker-repo] bugs.python.org code repository (https://hg.python.org/tracker/python-dev/)

.. [#devguide-repo] Mercurial repository for the Python Developer's Guide (https://hg.python.org/devguide/)

.. [#cpython-repo] Mercurial repository for CPython (https://hg.python.org/cpython/)

.. [#github-python-org] Python organization on GitHub (https://github.com/python)

.. [#github-org-perms] GitHub repository permission levels

( https://help.github.com/enterprise/2.4/user/articles/repository-permission-l... )

.. [#cla] Python Software Foundation Contributor Agreement (https://www.python.org/psf/contrib/contrib-form/)

.. [#news-file] ``Misc/NEWS`` (https://hg.python.org/cpython/file/default/Misc/NEWS)

.. [#acks-file] ``Misc/ACKS`` (https://hg.python.org/cpython/file/default/Misc/ACKS)

.. [#devguide-merge-across-branches] Devguide instructions on how to merge across branches

( https://docs.python.org/devguide/committing.html#merging-between-different-b... )

.. [#pythocat] Pythocat (https://octodex.github.com/pythocat/)

.. [#tracker-plans] Wiki page for bugs.python.org feature development (https://wiki.python.org/moin/TrackerDevelopmentPlanning)

.. [#black-knight-sketch] The "Black Knight" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=dhRUe-gz690)

.. [#bridge-of-death-sketch] The "Bridge of Death" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=cV0tCphFMr8)

.. [#holy-grail] "Monty Python and the Holy Grail" sketches

( https://www.youtube.com/playlist?list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp)

.. [#killer-rabbit-sketch] "Killer rabbit" sketch from "Monty Python and the Holy Grail"

( https://www.youtube.com/watch?v=Nvs5pqf-DMA&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=11 )

.. [#witch-village-sketch] "Witch Village" from "Monty Python and the Holy Grail"

( https://www.youtube.com/watch?v=k3jt5ibfRzw&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=12 )

.. [#french-taunter-sketch] "French Taunter" from "Monty Python and the Holy Grail"

( https://www.youtube.com/watch?v=A8yjNbcKkNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=13 )

.. [#constitutional-peasants-sketch] "Constitutional Peasants" from "Monty Python and the Holy Grail"

( https://www.youtube.com/watch?v=JvKIWjnEPNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=14 )

.. [#ni-sketch] "Knights Who Say Ni" from "Monty Python and the Holy Grail"

( https://www.youtube.com/watch?v=zIV4poUZAQo&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=15 )

.. [#homu] Homu (http://homu.io/)

.. [#zuul] Zuul (http://docs.openstack.org/infra/zuul/)

.. [#travis] Travis (https://travis-ci.org/)

.. [#codeship] Codeship (https://codeship.com/)

.. [#coverage] coverage.py (https://pypi.python.org/pypi/coverage)

.. [#coveralls] Coveralls (https://coveralls.io/)

.. [#codecov] Codecov (https://codecov.io/)

.. [#pypatcher] Pypatcher (https://github.com/kushaldas/pypatcher)

Copyright =========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

_______________________________________________ core-workflow mailing list core-workflow@python.org https://mail.python.org/mailman/listinfo/core-workflow This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct

[just an FYI to everyone: replying without trimming the PEP will lead to moderation, so please cut out stuff that doesn't matter when replying] On Sun, 17 Jan 2016 at 21:37 Ezio Melotti <ezio.melotti@gmail.com> wrote:

On Mon, Jan 18, 2016 at 4:33 AM, Brett Cannon <brett@python.org> wrote:

...

On Sun, 17 Jan 2016 at 16:30 Ezio Melotti <ezio.melotti@gmail.com>

wrote: [SNIP]

...

...

...

Adding GitHub username support to bugs.python.org +++++++++++++++++++++++++++++++++++++++++++++++++ To keep tracking of CLA signing under the direct control of the PSF, tracking who has signed the PSF CLA will be continued by marking that fact as part of someone's bugs.python.org user profile. What this means is that an association will be needed between a person's bugs.python.org [#b.p.o]_ account and their GitHub account, which will be done through a new field in a user's profile.

We have to decide how to deal with users that don't have a b.p.o account. The two options that we discussed in the previous mails are: 1) require them to create a b.p.o account; 2) allow them to log in to b.p.o using their github account; (see also next section)

Both require creating an account, it just varies whether they log in using GitHub or not. I don't see how we can avoid that if we are going to continue to own the CLA dataset. Honestly I don't see this as a big issue as we have not seemed to have any issues with people creating accounts up to this point.

It's not a big issue, but there will be people that only have a github account, and we will need to explain them that in order to accept their PRs we need them to sign the CLA, and in order to sign the CLA we need them to create a b.p.o account and link it to their github username.

I expect this to be the common case. I wish there was a way to avoid it, but if we want people to participate on the issue tracker they will need the account anyway. Plus it's no worse than it is today as I expect it's already the case most people have a GItHub account but not a b.p.o one.

...

...

...

Linking a pull request to an issue ++++++++++++++++++++++++++++++++++ An association between a pull request and an issue is needed to track when a fix has been proposed. The association needs to be many-to-one as there can take multiple pull requests to solve a single issue (technically it should be a many-to-many association for when a single fix solves multiple issues, but this is fairly rare and issues can be merged into one using the ``Superceder`` field on the issue tracker).

Association between a pull request and an issue will be done based on detecting the regular expression``[Ii]ssue #(?P<bpo_id>\d+)``. If this is specified in either the title or in the body of a message on a pull request then connection will be made on bugs.python.org [#b.p.o]_. A label will also be added to the pull request to signify that the connection was made successfully. This could lead to incorrect associations if the wrong issue or referencing another issue was done, but these are rare occasions.

Is there a way to associate the PR to an issue (in case the user forgot) or change association (in case it got the wrong number) after the creation of the PR?

You tell me. :) I assume any bot we write to handle this will monitor PR-level comments since GitHub doesn't notify on title changes. But we can choose any workflow we want, so if we want it to be an explicit command like `/bot issue 12345` then we can do that instead.

I was asking about the github side. On b.p.o it's not a problem adding PRs either automatically or manually, but I don't know if the same can be done from github (it already happens when a commit message includes the wrong issue number -- the commit can not be changed and the notification is sent to the wrong issue on b.p.o). Even if it's not possible, I guess we could still add a comment to github with the correct issue number, and add the PR to the issue and the people to the nosy list manually.

So are you asking about a b.p.o -> GH association, so that you can make the association on b.p.o and have it show up somehow? If that's your question then we can add a comment like we do on b.p.o.

[SNIP]

...

...

...

Backup of pull request data ''''''''''''''''''''''''''' Since GitHub [#github]_ is going to be used for code hosting and code review, those two things need to be backed up. In the case of code hosting, the backup is implicit as all non-shallow Git [#git]_ clones contain the full history of the repository, hence there will be many backups of the repository.

If possible I would still prefer an "official" backup. I don't think we want to go around asking who has the latest copy of the repo in case something happens to GitHub.

I would be shocked if a core developer doesn't have an up-to-date repository (then again I don't expect to have to flee GitHub overnight, giving us time to update). But if you want to make it this an optional feature then I'm fine with that (I don't think the PSF infrastructure team will have issue with a regularly updated clone of the repository).

Yes, this is not a particularly realistic issue -- even without an official backup we won't lose any code. It's mostly to have an official backup and procedure to restore the repo instead of having to figure out what to do when it happens (even if it probably never will). The infra team can decide if this is a reasonable request or if it's not worth the extra effort.

+1

[SNIP]

...

...

...

Test coverage report '''''''''''''''''''' Getting an up-to-date test coverage report for Python's standard library would be extremely beneficial as generating such a report can take quite a while to produce.

There are a couple pre-existing services that provide free test coverage for open source projects. Which option is best is an open issue: `Choosing a test coverage service`_.

Do we want to eventually request that all new code introduced is fully covered by tests?

I have always told sprinters that 80% is a good guideline. I wouldn't ever want a rule, though. Obviously lowering coverage would not be great and should be avoided, but if

Maybe we could use red/yellow/green labels to indicate the coverage level. That alone might lead contributors to aim for the green label without having to create and enforce any rule.

To give you an idea of at least how Coveralls integration looks, https://github.com/python-modernize/python-modernize/pull/117

...

...

I think having an indication of how much code in a PR is covered by tests would be useful regardless of the answer to the previous question.

I know Coveralls already does this; don't know about Codecov.

...

...

Link web content back to files that it is generated from '''''''''''''''''''''''''''''''''''''''''''''''''''''''' It would be helpful for people who find issues with any of the documentation that is generated from a file to have a link on each page which points back to the file on GitHub [#github]_ that stores the content of the page. That would allow for quick pull requests to fix simple things such as spelling mistakes.

Here you are talking about PEPs/devguide/docs.p.o, right?

Yes.

FWIW the docs.python.org pages already have a "report a bug" link in the sidebar and also in the footer, but they both just redirect to https://docs.python.org/3/bugs.html .

Yep, which is what made me think that it would be nice if we could direct people directly to the actual page instead of having to figure it out. But then again, if we think a lot of drive-by PRs will be doc-based and from people who have never contributed before, then we will have to wait for them to sign the CLA anyway, so maybe it isn't worth it? I guess the direct link to the underlying content is only useful if people who have signed the CLA will use it the most.

...

...

FWIW this was problem was supposed to be fixed with pootle, but that project seems dead (not sure if due to technical reasons, or simply because no one had time to integrate it).

...

Splitting out parts of the documentation into their own repositories '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' While certain parts of the documentation at https://docs.python.org change with the code, other parts are fairly static and are not tightly bound to the CPython code itself. The following sections of the documentation fit this category of slow-changing, loosely-coupled:

* `Tutorial <https://docs.python.org/3/tutorial/index.html>`__ * `Python Setup and Usage <https://docs.python.org/3/using/index.html>`__ * `HOWTOs <https://docs.python.org/3/howto/index.html>`__ * `Installing Python Modules <https://docs.python.org/3/installing/index.html>`__ * `Distributing Python Modules <https://docs.python.org/3/distributing/index.html>`__ * `Extending and Embedding <https://docs.python.org/3/extending/index.html>`__ * `FAQs <https://docs.python.org/3/faq/index.html>`__

These parts of the documentation could be broken out into their own repositories to simplify their maintenance and to expand who has commit rights to them to ease in their maintenance.

I would still consider these somewhat dynamic (especially between Py 2 and Py 3). There are other documents that are more version-independent, such as the whatsnew pages,

The What's New docs get updated with changes so that can't be pulled out from the cpython repo (especially if my dream ever comes true of making people be better about making sure that document gets updated with changes that warrant being mentioned there).

The reason to keep them in a separate repo is that they are identical copies of the same document. If you fix a typo in the whatsnew/2.7, you have to fix the same typo in all branches, and in theory the page should always be identical in each branch.

Sure, but I try to make it a habit to update What's New at the same time as committing something that warrants a mention. Pulling that out will become a bigger chore. Having said that, if we want core developers to be the ones that author those kinds of changes then having it be a separate repo won't necessarily be as critical. We would just probably need to add a "needs What's New mention" label or something to keep track of what needs to be added so people didn't forget.

Howtos, FAQs, etc. might differ between major and even minor versions as new features are added and old features deprecated, so it makes sense to have (slightly) different versions in each branch (unless you want to move them outside the cpython repo and still keep separate branches). For example https://docs.python.org/3/faq/programming.html#is-there-an-equivalent-of-c-s... had a different answer before 2.5 :)

I suspect that when that kind of situation occurred that I version-specific branch would be created and then merged in once a release went out.

Also moving them might make things more complicated (from simply having to find/clone another repo to building docs for docs.python.org from different sources).

It's possible, but I doubt it. We could have a docs.python.org repo that contains everything but the docs carried in the cpython repo. Or we can have these other doc repos be Git submodules of cpython that you can check out if you want, but it isn't necessary to. What I do know is this idea has come up many times in the past from the perspective of being able to give out commit rights to the docs much more readily than with the cpython repo and I think that's a good idea.

[SNIP]

...

...

...

Git CLI commands for committing a pull request to cpython --------------------------------------------------------- Because Git [#git]_ may be a new version control system for core developers, the commands people are expected to run will need to be written down. These commands also need to keep a linear history while giving proper attribution to the pull request author.

Another set of commands will also be necessary for when working with a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear history will be kept implicitly, but it will need to make sure to keep/add attribution.

Nick Coghlan, Pierre-Yves David (a Mercurial dev), and Shiyao Ma (one of our GSoC student) have been working on an HG extension that simplifies interaction with the bug tracker (see the list of patches, download/apply them, upload new patches): https://bitbucket.org/introom/hg-cpydev In a previous email, someone mentioned an alias that allows an easier interaction with PRs. Would it make sense to write and distribute an official git extension that provides extra commands/aliases for these set of commands? (I guess the answer depends on how many tasks we have and how straightforward it is to do with plain git commands.)

It quite possibly might be. Otherwise shell commands could be written and kept in Tools/. I major perk IMO with Git over Mercurial is Git Bash comes with GIt and that gives you Bash on Windows. That makes writing cross-platform shell scripts to help with this sort of thing easy without leaving Windows users stranded.

Isn't that also possible with HG, since extensions are written in Python?

I mean .sh files, not extensions, e.g., we have a shell script in Tools that calls Git directly.

(BTW, is it possible/reasonable to write git extensions/hooks in Python?)

I believe so. From my understanding, Git doesn't go with an API solution like Mercurial and instead prefers a way to simply specify a naming scheme that will call commands on your $PATH. See https://www.atlassian.com/git/articles/extending-git/ as an example.

...

...

...

How to handle the Misc/NEWS file -------------------------------- There are two competing approaches to handling ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on bugs.python.org [#b.p.o]_. This would mean an issue that is marked as "resolved" could not be closed until a news entry is added in the "news" field in the issue tracker. The benefit of tying the news entry to the issue is it makes sure that all changes worthy of a news entry have an accompanying issue. It also makes classifying a news entry automatic thanks to the Component field of the issue. The Versions field of the issue also ties the news entry to which Python releases were affected. A script would be written to query bugs.python.org for relevant new entries for a release and to produce the output needed to be checked into the code repository. This approach is agnostic to whether a commit was done by CLI or bot.

The competing approach is to use an individual file per news entry, containg the text for the entry. In this scenario each feature

Typo: containing

...

release would have its own directory for news entries and a separate file would be created in that directory that was either named after the issue it closed or a timestamp value (which prevents collisions). Merges across branches would have no issue as the news entry file would still be uniqeuely named and in the directory of the latest

Typo: uniquely

...

version that contained the fix. A script would collect all news entry files no matter what directory they reside in and create an appropriate news file (the release directory can be ignored as the mere fact that the file exists is enough to represent that the entry belongs to the release). Classification can either be done by keyword in the new entry file itself or by using subdirectories representing each news entry classification in each release directory (or classification of news entries could be dropped since critical information is captured by the "What's New" documents which are organized). The benefit of this approach is that it keeps the changes with the code that was actually changed. It also ties the message to being part of the commit which introduced the change. For a commit made through the CLI, a script will be provided to help generate the file. In a bot-driven scenario, the merge bot will have a way to specify a specific news entry and create the file as part of its flattened commit (while most likely also supporting using the first line of the commit message if no specific news entry was specified). Code for this approach has been written previously for the Mercurial workflow at http://bugs.python.org/issue18967. There is also a tool from the community at https://pypi.python.org/pypi/towncrier.

Does using git (and fast-forward merges, rebases, etc.) still create conflicts? (I guess the answer is yes, but perhaps we should double-check.)

I believe so, but I guess I could be wrong since I have not explicitly tested it.

...

If it does, there's also a third option: writing a merge script. I wrote a basic one for hg and it seemed to work decently, perhaps with git it's even easier.

I'll mention it, but i suspect the file-based solution will win out based on the feeling I have gotten from people when this topic has come up before.

I was re-reading the issue, and found due interesting links: 1) https://mail.python.org/pipermail/python-dev/2014-December/137393.html (Pierre-Yves David actually convinced me to write the merge script and helped me) 2) https://github.com/twisted/newsbuilder (this can be another option that can be added to the PEP)

Thanks, I'll add them.

A few additional points that you might want to add to the PEP: * A new workflow for releasing Python should also be defined, and PEP 101 updated accordingly.

I view this as implicit.

* The devguide also needs to be updated.

Implicit, but I can call this out. I assume it will end up with a github-migration branch for a while to store updates until the migration occurs.

* We should decide what to do with all the other repos at hg.python.org

They are all personal repos, so people can do what they want with them. -Brett

On Sun, 17 Jan 2016 at 17:37 Martin Panter <vadmium+py@gmail.com> wrote:

On 17 January 2016 at 20:48, Brett Cannon <brett@python.org> wrote:

...

Rationale ========= . . . The overarching goal of this migration is to improve the development process to the extent that a core developer can go from external contribution submission through all the steps leading to committing said contribution all from within a browser on a tablet with WiFi.

Later on, at least for the main “cpython” repository, you mention maintaining a linear history, avoiding the automatic Git Hub merge button, and cherry-picking between branches. I guess you would need the commit bot to help do all this from a browser.

Yes.

If so, mention that, otherwise this gives the impression that people will Git Hub’s merge button, which seems to contradict later sections.

Well, you get it for some of the repositories, just not cpython. And I don't entirely agree that the sentence suggests GitHub's built-in browser workflow is the answer, just that *some* in-browser workflow is desired.

...

Document steps to commit a pull request ''''''''''''''''''''''''''''''''''''''' During the process of choosing a new development workflow, it was decided that a linear history is desired. This means that the convenient "Merge" button in GitHub pull requests is undesireable, as it creates a merge commit along with all of the contributor's individual commits (this does not affect the other repositories where the desire for a linear history doesn't exist).

Please clarify whether this linear history is just avoiding the suprious merge commits that Git Hub can add for each pull request (merging a pull request with its parent commit), or whether we are going to stop merging maintainence branches into master, and do cherry-picks instead.

Both is the current plan.

...

Linking a pull request to an issue ++++++++++++++++++++++++++++++++++ . . . (technically it should be a many-to-many association for when a single fix solves multiple issues, but this is fairly rare and issues can be merged into one using the ``Superceder`` field on the issue tracker).

Spelling: ``Super[s]eder``

...

Status ====== . . . Repositories whose build steps need updating:

There is some stray indentation starting here which seems to be messing with the markup.

...

The fate of hg.python.org ------------------------- With the code repositories moving over to Git [#git]_, there is no technical need to keep hg.python.org [#h.p.o]_ running.

I presume this means other repositories will be migrated somewhere else too? For instance, I recently had to update the “pythontestdotnet” repository.

The assumption is everything will move long-term instead of fragmenting between GitHub and hg.python.org.

...

Tools and commands to move from Mercurial to Git ------------------------------------------------ A decision needs to be made on exactly what tooling and what commands involving those tools will be used to convert a Mercurial repository to Git. Currently a suggestion has been made to use https://github.com/frej/fast-export. Another suggestion is to use https://github.com/felipec/git-remote-hg. Finally, http://hg-git.github.io/ has been suggested.

The migration to Git might be a good opportunity to clean up some of the older Subversion history. I would like to investigate this. In particular, rewriting svnmerge commits as proper Git merges, and restoring the missing history for merges and imports from feature branches.

On the other hand, there is already a Git mirror <https://github.com/python/cpython> which many seem to be already using. Perhaps it would be good to ensure the commit hashes used there are kept current.

I'm not going to worry about that if the mirror was not done accurately.

Which tool is used to update that mirror? Are there any deficiencies that would mean we need to use a different tool, possibly affecting the commit hashes?

I don't know. Eli Bendersky set up that mirror and he has said previously that he just did the bare minimum to get it working and didn't worry too much about correctness, etc. -Brett

...

Separate Python 2 and Python 3 repositories ------------------------------------------- It was discussed whether separate repositories for Python 2 and Python 3 were desired. The thinking was that this would shrink the overall repository size which benefits people with slow Internet connections or small bandwidth caps.

In the end it was decided that it was easier logistically to simply keep all of CPython's history in a single repository.

Agreed. If people really only want one of the branches, they can set Git to only download from that branch. And Git supports shallow clones (only download the most recent commit, or N commits deep), although some functions may not work in this mode (but apparently things have improved since I learnt about it).

On Sun, 17 Jan 2016 at 18:58 Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Jan 18, 2016 at 7:48 AM, Brett Cannon <brett@python.org> wrote:

...

Luckily, Git [#git]_ does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history.

A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [#b.p.o]_. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author.

Point to note: If you want to make use of git's separate author and committer records (so the author is the person who wrote the original patch, and the committer is the core dev who actually pushed it), you'll forfeit the identifiable hashes on commits. Every commit will have to be rebased (or amended, which is a short-hand form of rebase) to change its committer, which creates a brand new commit with a new hash. GitHub won't recognize the push as incorporating the original commit, and neither will people's tools elsewhere.

IMO this is a worthwhile trade-off, but it is a cost, and may be worth mentioning in the PEP. It's no worse than the current state (where a Mercurial commit completely loses track of its original author, unless it's mentioned in the human-readable message), but it's perhaps not what people will be expecting.

I don't quite follow this. If you do a ff + rebase for the final commit how does that affect the hash of the final commit? Or what if you take a patch, apply it, and as part of the `git commit` command you specify the author on the command-line? I understand how it would change things if we were updating a pre-existing Git repository, but I'm only talking about future commits and not necessarily trying to retroactively do this for the direct migration of a repository. -Brett

(Also, when it comes time to write up the actual commands for people, it'll need to be made clear that an actual fast-forward isn't acceptable, as it won't update the committer. Amending or rebasing MUST be done.)

ChrisA _______________________________________________ core-workflow mailing list core-workflow@python.org https://mail.python.org/mailman/listinfo/core-workflow This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct

On Mon, 18 Jan 2016 at 11:03 Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:

On Jan 18, 2016, at 10:45, Brett Cannon <brett@python.org> wrote:

On Sun, 17 Jan 2016 at 18:58 Chris Angelico <rosuav@gmail.com> wrote:

...

On Mon, Jan 18, 2016 at 7:48 AM, Brett Cannon <brett@python.org> wrote:

...

Luckily, Git [#git]_ does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history.

A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [#b.p.o]_. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author.

Point to note: If you want to make use of git's separate author and committer records (so the author is the person who wrote the original patch, and the committer is the core dev who actually pushed it), you'll forfeit the identifiable hashes on commits. Every commit will have to be rebased (or amended, which is a short-hand form of rebase) to change its committer, which creates a brand new commit with a new hash. GitHub won't recognize the push as incorporating the original commit, and neither will people's tools elsewhere.

IMO this is a worthwhile trade-off, but it is a cost, and may be worth mentioning in the PEP. It's no worse than the current state (where a Mercurial commit completely loses track of its original author, unless it's mentioned in the human-readable message), but it's perhaps not what people will be expecting.

I don't quite follow this. If you do a ff + rebase for the final commit how does that affect the hash of the final commit? Or what if you take a patch, apply it, and as part of the `git commit` command you specify the author on the command-line? I understand how it would change things if we were updating a pre-existing Git repository, but I'm only talking about future commits and not necessarily trying to retroactively do this for the direct migration of a repository.

I think what is meant here is that GitHub has some features that allows you to go from commit number to which PR it originate from. When you rebase to avoid merge commit, you will generate new commit hashes, which will make these functionality not work.

You can see an example here[1], for example the the indicator `master (#9124)`, indicate that this commit was merged into master by PR #9124. this is typically useful to get back to the discussion/review of the PR.

The other thing you might lose is the auto closing of Pull-requests once the commit.

You will lose also other nice things like intermediate CI status on commit, but that’s less interesting that above features. Agreed that might seem not that useful, but greatly improve some tasks from time to time.

That's all true and a consequence of a linear history. We will probably need to get into the habit of pasting the resulting commit into the PR when it gets closed.

On 19 January 2016 at 05:53, Brett Cannon <brett@python.org> wrote:

On Mon, 18 Jan 2016 at 11:03 Matthias Bussonnier <bussonniermatthias@gmail.com> wrote:

...

I think what is meant here is that GitHub has some features that allows you to go from commit number to which PR it originate from. When you rebase to avoid merge commit, you will generate new commit hashes, which will make these functionality not work.

You can see an example here[1], for example the the indicator `master (#9124)`, indicate that this commit was merged into master by PR #9124. this is typically useful to get back to the discussion/review of the PR.

The other thing you might lose is the auto closing of Pull-requests once the commit.

You will lose also other nice things like intermediate CI status on commit, but that’s less interesting that above features. Agreed that might seem not that useful, but greatly improve some tasks from time to time.

That's all true and a consequence of a linear history. We will probably need to get into the habit of pasting the resulting commit into the PR when it gets closed.

It also falls into the category of problem that a workflow bot addresses - it just becomes a requirement for the bot to comment on and close the original PR with a reference to the commit, in addition to actually doing the commit itself. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia