It seems like last year already since the release of 3.6.0. I guess that's because it was last year, 2016-12-22 to be exact! Now we're approaching the end of the first quarter and, according to PEP 494, it's time to start producing the first maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-02-27 UTC. As was the case with the 3.6.0 release cycle, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.1 merged in before 2017-02-27. I will send out another reminder a couple of days before. The 3.6.1 final is planned for two weeks following rc1, that is, on 2017-03-13. I expect the next 3.6 maintenance release (3.6.2) will follow about 3 months later, so most likely in 2017-06 after PyCon US.
3.6.1 will be the first release using our new GitHub-based development process (thanks, Brett and team!). If you are planning to push something for 3.6.1 and haven't yet tried out the new workflow or are not yet familiar with GitHub pull requests, you should probably give yourself some extra time. As always, the Developer's Guide is the primary reference for the development workflow; not surprisingly, with such a major change, there are likely still some parts of the guide that could use further changes and clarifications. You can help by reviewing the devguide's open issues and pull requests in its repository and adding to them as you work through issues. If you have comments on or improvement suggestions for the new workflow, the place to discuss them is on the core-workflow mailing list.
Thanks again for all of your efforts in bringing 3.6.0 into the world and for helping now to make it even better!
https://www.python.org/dev/peps/pep-0494/http://cpython-devguide.readthedocs.iohttps://mail.python.org/mailman/listinfo/core-workflow
--
Ned Deily
nad(a)python.org -- []
I don't know about the rest of you but I can't believe it's only been two
weeks since we moved over to GitHub! Hopefully people are still viewing it
in a positive light. :)
One complaint I have heard consistently is the required review sign-off on
PRs. To do some A/B testing of this, I have switched that requirement off
but switched on the requirement that Travis tests be passing.
In two weeks I will be starting a conversation about the new workflow to
find out what people think we need to tweak, add/create, or remove in the
process. At that point we can make a final call if this situation is better
than requiring reviews, or if we want both CI and reviews for each PR.
Hi,
I'm going to book my hotel and flight for Pycon US, but I don't know
the date of the Language Summit and I would prefer to not miss it if
possible :-)
Does someone organize it?
Victor
Hi,
I clicked on a Mercurial commit number from:
http://bugs.python.org/issue18383#msg249581
It points me to:
http://hg.python.org/lookup/c1396d28c440
... which displays short error message:
---
Usage: /lookup/GITHEXHASH or gitGITHEXHASH (10, 11, or 40 hex characters)
/lookup/HGHEXNODE or hgHGHEXNODE (12 or 40 hex characters)
/lookup/rSVNREVISION
---
"c1396d28c440" hash is 12 hex characters long.
Victor
With the move to Github, I'm seeing a move away from
discussions on the issue tracker towards discussions on
pull requests.
Example: https://github.com/python/cpython/pull/4
Is this something we should encourage or better ask the OPs
to open a ticket on the tracker first and reference the PR
there ?
Discussion on the PRs would then only be for code review
aspects.
Thanks,
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Experts (#1, Feb 13 2017)
>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> Python Database Interfaces ... http://products.egenix.com/
>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
________________________________________________________________________
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/http://www.malemburg.com/
Hi,
Would it be possible to modify the workflow of GitHub pull requests to
allow to click on Merge, but only merge a PR once tests complete and
only if tests pass?
If some tests start to become too annoying for the pre-commit CI, we
can try to fix them, or even disable them in the CI to only rely on
post-commit buildbots.
Victor
Hi,
I don't get the value of code coverage on a CI. I *like* running code
coverage sometimes, but I don't like being annoying by "test failed:
coverage -0,01%" by a change completely unrelated to code (note: this
issue has been fixed be using a threshold of 1%). I'm annoyed by all
these Codecov messages. Berker just told me that he even created a
Gmail filter to drop all these email notifications...
What do you think of keeping this very useful feature, but not use it
on pull requests: only run it on the branches (once changes are
merged), to not "pollute" pull requests?
Victor
[rendered version:
https://paper.dropbox.com/doc/CPython-workflow-changes-mx1k8G6M0rg5JLy80F1r6
]
# CPython workflow changes
After more than two years, our new GitHub workflow is ready to accept
changes (you can look back to my first “[vision statement](
https://mail.python.org/pipermail/python-dev/2014-December/137472.html)” on
changing our workflow to see how things have changed since I started
working on this)! I hope you are all excited to see this finished; I know
my wife is very excited as she’s tired of listening to me talk about it for
a third of our marriage. ;)
# Thanks
First and foremost, I want to thank everyone who helped with this. Thanks
to Donald and Barry for writing the initial PEPs proposing [GitHub](
https://www.python.org/dev/peps/pep-0481/) and [GitLab](
https://www.python.org/dev/peps/pep-0507/) and Nick [pre-proposing
RhodeCode](https://www.python.org/dev/peps/pep-0474/). Thanks to everyone
on [core-workflow](https://mail.python.org/mailman/listinfo/core-workflow)
for helping out with various discussion (and which will continue to host
discussions on future improvements on our workflow). Thanks to Ezio,
Maciej, and Anish Shah for helping with the changes required to
bugs.python.org in order to keep the issue tracker around. Thanks to the
infrastructure team for helping deal with the migration of the [peps](
https://github.com/python/peps) and [devguide](
https://github.com/python/devguide) repos (especially Donald and Ernest).
Thanks to Senthil for doing the conversion of the repo itself. Thanks to
Benjamin for helping with hg.python.org stuff. Thanks to Zach for helping
with the buildbots (and the devguide). Thanks to Mariatta, Carol Willing,
Berker, Oleg, and Stéphane Wirtel for helping with the devguide. There are
also plenty of other people who have provided feedback over the past 2
years on mailing lists and in-person.
# What has changed
The documentation in the [devguide](https://cpython-devguide.readthedocs.io)
should be up-to-date, so don’t worry about keeping this around as a
reference. Consider this more of an announcement letter to get people
quickly up-to-speed and excited about the new workflow.
## The location of the code repository
CPython’s code now lives at https://github.com/python/cpython .
hg.python.org/cpython is and will stay read-only (no determination has been
made as to how long the Mercurial repository will be kept running). It
should also be mentioned that we are doing squash commits and not rebased
commits or merge commits as some projects on GitHub do. This basically
means we will continue to have a single contribution lead to a single
commit, keeping our history linear and compact.
To up the bus factor on the new repository, I have set up a team for
[Python release managers](
https://github.com/orgs/python/teams/python-release-managers) and made them
administrators on the repository. I don’t necessarily expect RMs to use
this power, but it’s there in case any of them need to change a setting in
order to get a release out (to be upfront about it, I’m also in the team as
its creator, but I have administrative privileges for the [Python team](
https://github.com/python) on GitHub so it doesn’t change what I’m able to
do).
## Specifying issue numbers
Traditionally we have specified issues as “Issue #NNNN”. The problem with
this format is that [GitHub automatically links](
https://help.github.com/articles/autolinked-references-and-urls/#issues-and…)
text in this format to GitHub issues and pull requests. While our
repository will initially have no issues or PRs to automatically link to,
this might not be true long-term (GitHub does the automatic linking eagerly
at push time, so there’s no worry for older commit messages; we actually
almost mutated the history to match the new format but in the end I made
the decision not to as I didn’t consult with python-committers prior to the
migration to make sure such a change was acceptable to everyone).
To avoid this issue we are going to start specifying issues as “bpo-NNNN”.
This clearly delineates that issue numbers directly relate to
bugs.python.org since “[namespaces are one honking great idea](
https://www.python.org/dev/peps/pep-0020/)”. This is also a format that
GitHub supports — “GH-NNNN” — as well as JIRA who [lets you specify the
prefix](
https://confluence.atlassian.com/adminjiraserver072/changing-the-project-ke…),
so there’s precedent for choosing it. This change applies both to
`[Misc/NEWS](
https://cpython-devguide.readthedocs.io/committing.html#what-s-new-and-news…
and in [commit messages](
https://cpython-devguide.readthedocs.io/committing.html#commit-messages).
Mentioning an issue this way in a pull request title or comment will
connect the PR to the corresponding issue on bugs.python.org. Mentioning an
issue this way in a commit message will cause a comment to show up in the
issue relating to the commit.
## Cherry-picking instead of merging
When a patch has spanned more than one version/branch we have always done a
forward merge. The common issue with this, though, is it leads to racing
with other committers from when you make to your initial commit in the
oldest version/branch to pushing to hg.python.org on the newest
version/branch. There was also the problem of having to remember that
Python 2.7 is a special branch which was never merged forward.
To deal with these issues we will use [cherry-picking](
https://cpython-devguide.readthedocs.io/committing.html#backporting-changes…)
going forward. This allows changes to be pulled into other branches as
independent commits. This prevents any commit races with other core
developers as we have traditionally needed to deal with when doing forward
merges that span e.g. three branches. It also allows using CI to easily
verify a change works if each cherry-pick is done as a separate pull
request. The Python 2.7 branch also stops being a special case when
backporting. It also prevents potential issues stemming from contributors
submitting pull requests against the master branch by default and not the
oldest branch a change should be applied to. Finally, this also removes the
discussion of whether a change should be backported or not from blocking
the commit into master to begin with.
Labels will be provided for people to use to help track any cherry-picking
that needs to occur for a pull request (e.g. “backport to 3.6”). I also
left the “bug” and “enhancement” labels to help classify PRs (adding more
labels is easy so we can do that as our experience and workflow organically
converge towards common practices).
## Protected branches
All feature branches have been marked as [protected](
https://help.github.com/articles/about-protected-branches/). This means
that feature branches cannot be deleted nor can they be pushed to directly.
The latter will be the biggest change as it means all changes must go
through a pull request. This helps make sure that there are no accidental
breakage of code (I know I have done this multiple times when in a rush and
I didn’t take my time when preparing a commit). This also means that all
core developers follow the same development workflow as any other
contributor. This not only allows all core developers to be able to help
any other contributors with our workflow, but it also helps make sure we
are aware of any sticking points in the contributor process so that we can
all work towards resolving them for everyone’s benefit. (If experience
shows that this is too much overhead we can turn this off.)
All feature branches that are in security-only mode are locked down so that
only [Python release managers](
https://github.com/orgs/python/teams/python-release-managers) can approve
pull requests to them. For all branches that have reached EOL, no one is
able to push to them. I expect that RMs will also use this feature when
they are ready to gate all commits to a branch on their approval (e.g. when
a release reaches RC, maybe even beta if they choose to go that far).
# What has improved
## Accepting PRs through GitHub’s web UI
While using hg.python.org, all commits had to be done through Mercurial’s
CLI. With the move to GitHub we gain the ability to accept pull requests
through a web UI. While this will only accept the change into the branch it
was submitted against (which can be changed in the web UI), for situations
where a change does not need to be backported it will allow for easier
acceptance of a change. (When a change does need to be backported this is
when you need to cherry-pick and that requires using the git CLI). If a
change does need to be cherry-picked into an older branch you can either
wait to accept the PR when you have a clone to work with or accept the
change into master now and then cherry-pick later when you have a clone
available.
## Continuous integration
Previously changes required running the test suite manually along with
verifying various other things like the documentation building. Moving to
GitHub allows us to leverage the Travis continuous integration service to
test several things in parallel automatically for each pull request:
1. Debug build under gcc
2. Debug build under clang
3. Documentation is valid and has no stale links
4. Python.h C++ compatibility
While this doesn’t solve all testing scenarios (e.g. this doesn’t test a
macOS or Windows-related change due to the added hours it take for a PR to
be “green” when run on Travis for macOS or AppVeyor for Windows), it does
help with the common case of a cross-platform change. (There is an [open
issue](https://github.com/python/core-workflow/issues/14) to add some code
so that these tests only run when appropriate files have changed so that
e.g. fixing a spelling mistake doesn’t run the test suite.)
It should be mentioned that status checks on issues are **not** required
prior to committing a pull request. While this may be a good idea
long-term, until we know that our test suite is stable enough to not have
regular flaky tests this would be more trouble than it’s worth (GitHub does
visibly show, though, when not all status checks have passed so you won’t
easily ignore this situation either).
## Code coverage
Traditionally the code coverage of our tests was only known when someone
ran the test suite manually under something like [coverage.py](
https://coverage.readthedocs.io). Even when someone did generate a coverage
report it was generally not shared with other developers, and so it wasn’t
widely known if a pull request increased or lowered test coverage.
With the move to GitHub we are able to use [Codecov](https://codecov.io/)
to calculate code coverage for each pull request. This also implicitly
tests a non-debug build as that’s used to make the coverage results run
faster. It should be noted, though, that some tests are skipped due to them
holding up the coverage run from completing. (There is an [open issue](
https://github.com/python/core-workflow/issues/18) to use coverage.py’s
fullcoverage hack so that the coverage report can even be accurate for
modules imported during interpreter startup.)
## CLA enforcement
To tell if someone has signed the PSF contributor license agreement you
have to look to see if they have an asterisk by their name on the issue
tracker. Unfortunately this is a passive thing to need to check for and is
easily forgotten. Thanks to GitHub’s webhook events and developer API we
now have a bot which checks if the contributor(s) to a pull request have
signed the CLA, adding an appropriate label to the PR to signal the CLA
signing status (the bot is named [The Knights Who Say Ni](
https://github.com/python/the-knights-who-say-ni)). If the contributor(s)
have not signed the CLA then a message is left on the PR explaining how to
rectify the issue (it’s either they need to connect their GitHub account to
their bugs.python.org account or they need to sign the CLA; there’s also is
an easter egg that occasionally appears in the message).
If a contributor does end up fixing the issue that leads to the bot
thinking the contributor had not signed the CLA, you can remove the “CLA
not signed” label and the bot will recheck the PR and add the appropriate
label (this also happens automatically if any code changes are made to the
PR). If for some the reason the bot has a hiccup then no label will be
applied (this is to act as a safeguard against false-negatives and to make
it easy to spot when something has gone wrong due to the absence of either
a “CLA signed” or “CLA not signed” label). To trigger the bot again you can
simply apply the “CLA not signed” label and then remove it.
## Contribution guidelines
There is now a `[.github/CONTRIBUTING.rst](
https://github.com/python/cpython/blob/master/.github/CONTRIBUTING.rst)`
file which gives general info on contributing. Primarily it has all the
various build status badges and links, link to the devguide and an overview
of how we differ from most GitHub projects, and a mention of the CoC. For
core devs I suspect the badge list for all active branches will be useful
to know when something is broken (I am only listing the test coverage for
the master branch to prevent encouraging people spending time trying to
increase test coverage for bugfix-only branches) .
# The future
This isn’t here for discussion per-se, but to let people know what I am
thinking should change next. If you want to help or discuss anything in
this section, please subscribe to core-workflow and participate there.
Ideas are also tracked on the [core-workflow issue tracker](
https://github.com/python/core-workflow/issues) (where there are other
ideas for improvements beyond the two listed below).
First, we need to solve the Misc/NEWS problem. The [plan](
https://github.com/python/core-workflow/issues/6) is to move to individual
files per NEWS entry and then have a script to roll them all up into the
NEWS file at release time. This will do away with merge conflicts which has
always been the biggest hassle when a change spanned versions.
Second, we will probably develop some solution to [automatically generate
cherry-picking PRs](https://github.com/python/core-workflow/issues/8). This
plus the Misc/NEWS solution should be enough to allow most patches to be
accepted through the web UI entirely.
All the other changes are nice-to-have, but I think these two will lead to
the greatest improvement to our workflow and get us far beyond the workflow
we had on hg.python.org.
