Split Misc/NEWS into individual files
Hi, As an user, I like Misc/NEWS changelog. I like reading https://docs.python.org/dev/whatsnew/changelog.html#changelog after a minor Python release to see all bugfixes. As a developer, I deeply hate this file :-) It's very painful to: * find the latest Python version in this giant file * find the right section * ... handle merge conflicts!!! * understand how this thing works on multiple Python versions The last point was always a mystery for me. If we make a change in Python 3.5.3, merge it into the 3.6 branch: should we document the change in 3.5.3 and 3.6.0 changelog? What if 3.6.0 is released *before* 3.5.3? What if 3.6.0 is released *after* 3.5.3? IMHO developers should not have to think about the release date when writing a changelog, we need a tool doing that for us. OpenStack happily fixed this issue one or two years ago with "reno": https://pypi.python.org/pypi/reno It's a simple tool building a changelog from a list of files: each file describe a change of a commit. The filename is the commit identifier. Simple, isn't it? OpenStack has something like 100x more commits than CPython (the overall OpenStack project). It's just a giant project and so many people and pull requests which can remain open for months. Very obvious win of individual files: no more merge conflict in two concurrent changes, since each commit adds its own file. My idea is not new, it's already discussed in the Python core-workflow: https://github.com/python/core-workflow/issues/6 I'm opening a thread on python-dev because Misc/NEWS quickly became a blocker issue with the Python new workflow. What do you think of the overall idea? Victor
On 2/22/17 7:27 AM, Victor Stinner wrote:
OpenStack happily fixed this issue one or two years ago with "reno": https://pypi.python.org/pypi/reno
It's a simple tool building a changelog from a list of files: each file describe a change of a commit. The filename is the commit identifier. Simple, isn't it?
I've read through the reno design document, and I think it would work for Python: https://docs.openstack.org/developer/reno/design.html
What do you think of the overall idea?
I'm for it. I'd be interested in hearing what current and former release managers say. Eric.
On 02/22/2017 08:26 AM, Eric V. Smith wrote:
On 2/22/17 7:27 AM, Victor Stinner wrote:
OpenStack happily fixed this issue one or two years ago with "reno": https://pypi.python.org/pypi/reno [...] What do you think of the overall idea? I'm for it. I'd be interested in hearing what current and former release managers say.
I'm for it too. Fixing up Misc/NEWS is probably the trickiest and most error-prone part of cutting a release. We've discussed this before, somewhere on bpo though I don't have the issue no. handy. I actually wrote a tool specifically to solve this problem at the time: https://bitbucket.org/larry/mergenews Obviously it's out of date as it's designed to work with Mercurial. I don't know how such a tool would be integrated into a Git-based workflow. While I'm sure we can learn a lot by examining "reno", my suspicion is we'll want our own custom tool. I don't think the hard part is writing the tool--the hard part is figuring out what we want and how it should be integrated into our workflow. Actually writing such a tool is easy and fun, and I'm sure our community would fall all over themselves to write it if asked. In fact, I'll kind of pre-volunteer to write it right now, as long as I can write it in Python 3 ;-) //arry/
On Feb 24, 2017, at 11:40 PM, Larry Hastings wrote:
I'm for it too. Fixing up Misc/NEWS is probably the trickiest and most error-prone part of cutting a release.
Not to mention cherry picking for older releases.
While I'm sure we can learn a lot by examining "reno", my suspicion is we'll want our own custom tool. I don't think the hard part is writing the tool--the hard part is figuring out what we want and how it should be integrated into our workflow. Actually writing such a tool is easy and fun, and I'm sure our community would fall all over themselves to write it if asked. In fact, I'll kind of pre-volunteer to write it right now, as long as I can write it in Python 3 ;-)
What else would you write it in?! :) Let's start with a simple question: do we like the general layout of the existing Misc/NEWS file? I do, and I *really* like it containing issue numbers, because that makes it relatively easy to search to see if a particular bug was fixed in a particular version (which is also why I wish all pull requests were tied to a bug). Other things I like about the file: * It's easy to search for a particular Python release, just by searching for "What's New". * I like having a release date easily visible with the Python version heading. * I like the bullet list format of the file, and I like the alphabetized category headings. What I don't like: * Lines that are too long ;) * All the inherent conflicts in editing a single file. * Misc/NEWS (on master) doesn't go back farther than 2 Python versions and 2 years, so it's harder to search for earlier changes. * It's in Misc/ not Doc/ Can we agree then that Misc/NEWS is the target output format for this tool? Maybe it's not the *only* target output format? Now, what's the input? The other thing is that I'm not sure design-by-committee is going to work so well here. There are several competing interests here. Release managers, non-core devs (drive-by contributors needing sponsorship), core devs (back porting/cherry picking changes), downstream consumers. Cheers, -Barry
On Feb 22, 2017, at 01:27 PM, Victor Stinner wrote:
OpenStack happily fixed this issue one or two years ago with "reno": https://pypi.python.org/pypi/reno
reno looks interesting but there is at least one thing I think would need changing for us. Aside: I'm not super psyched about its requiring YAML files for its input. Their structure can be fairly arcane for drive-by contributors, but I suppose with good documentation and a CI gate, we can live with that. I'm also not thrilled with having to learn yet another command line tool to submit changes to Python. The main dissonance for us is I think the sections that reno supports https://docs.openstack.org/developer/reno/usage.html Our release notes have different sections: * Core and Builtins * Extension Modules * Library * Windows * C API * Documentation * Build * Tools/Demos * Tests (others?) I don't know if those are configurable in reno or it would require a fork, but I'd like to preserve that organizational structure. If reno can also help wrap long lines, enforce/encourage bpo-* mentions, and clean up whitespace, then I'm for trying it out. -Barry
Excerpts from Barry Warsaw's message of 2017-02-22 11:42:57 -0500:
On Feb 22, 2017, at 01:27 PM, Victor Stinner wrote:
OpenStack happily fixed this issue one or two years ago with "reno": https://pypi.python.org/pypi/reno
reno looks interesting but there is at least one thing I think would need changing for us. Aside: I'm not super psyched about its requiring YAML files for its input. Their structure can be fairly arcane for drive-by contributors, but I suppose with good documentation and a CI gate, we can live with that.
I'm also not thrilled with having to learn yet another command line tool to submit changes to Python.
The main dissonance for us is I think the sections that reno supports
https://docs.openstack.org/developer/reno/usage.html
Our release notes have different sections:
* Core and Builtins * Extension Modules * Library * Windows * C API * Documentation * Build * Tools/Demos * Tests (others?)
I don't know if those are configurable in reno or it would require a fork, but I'd like to preserve that organizational structure. If reno can also help wrap long lines, enforce/encourage bpo-* mentions, and clean up whitespace, then I'm for trying it out.
-Barry
FWIW, I've been planning to work on a demonstration patch using reno for cpython so we can identify feature gaps like this. I have permission to work on this as part of my day job, to make reno suitable for both communities. I'm at a conference this week, but will try to put something together after I get back to work, and definitely before the language summit at PyCon. We have a patch up for review to support configuring the sections, including the keys within the input file and the titles in the output, so that should take care of that gap. The use of lists within each section is also something I think we could either make smart or explicitly configurable, and that would simplify the file format to just section keys and an RST blob. The "report" command assembles the RST embedded in the YAML and then passes it through docutils to produce text output. In the process, it does wrap lines, standardize bullets for lists, etc. For an example of the output, you can look at the release announcement for nova from earlier today [1]. The content of that email up to the git log output at the end was produced by reno based on the input files in [2]. Doug [1] http://lists.openstack.org/pipermail/release-announce/2017-February/000784.h... [2] http://git.openstack.org/cgit/openstack/nova/tree/releasenotes/notes
2017-02-22 17:42 GMT+01:00 Barry Warsaw <barry@python.org>:
Our release notes have different sections:
* Core and Builtins * Extension Modules * Library * Windows * C API * Documentation * Build * Tools/Demos * Tests (others?)
While we are talking about changing things... I would propose to require to specify a module name if a change is in the Library section, and then maybe even group changes per module. Victor
On 2/22/2017 5:53 PM, Serhiy Storchaka wrote:
On 22.02.17 22:59, Victor Stinner wrote:
While we are talking about changing things... I would propose to require to specify a module name if a change is in the Library section, and then maybe even group changes per module.
Some changes can be related to a couple of modules.
There should be a 'General' categories for changes not limited to modules within a grouping (see previous message). Many of your multi-module patches would fit in 'General'. -- Terry Jan Reedy
On 2/22/2017 3:59 PM, Victor Stinner wrote:
2017-02-22 17:42 GMT+01:00 Barry Warsaw <barry@python.org>:
Our release notes have different sections:
* Core and Builtins * Extension Modules * Library
* IDLE
* Windows * C API * Documentation * Build * Tools/Demos * Tests (others?)
While we are talking about changing things... I would propose to require to specify a module name if a change is in the Library section, and then maybe even group changes per module.
+1 I would like tkinter changes grouped together. There have been similar requests to more finely divide tracker issues. Since there are about 400 modules, an alternative would be to group by package or doc chapter. The 30 chapters after the 5 built-in chapters are a much more manageable number for both the tracker and NEWS. Having IDLE issues (which groups 60 related modules) split out tremendously helps IDLE users. But it need not be the only such grouping. -- Terry Jan Reedy
On Feb 22, 2017, at 07:27, Victor Stinner <victor.stinner@gmail.com> wrote:
[...]My idea is not new, it's already discussed in the Python core-workflow: https://github.com/python/core-workflow/issues/6
I'm opening a thread on python-dev because Misc/NEWS quickly became a blocker issue with the Python new workflow.
What do you think of the overall idea?
I think we clearly need to do something about Misc/NEWS but this issue is already under discussion in the core-workflow working group and I think we shouldn't be moving it to python-dev until there is a final concrete proposal. I know it is high on Brett's list but I believe he's taking a few days of holiday right so he may not be able to respond immediately. Let's move this discussion over to the core-workflow mailing list. It can wait a few days. https://mail.python.org/mailman/listinfo/core-workflow -- Ned Deily nad@python.org -- []
On Wed, 22 Feb 2017 at 13:30 Ned Deily <nad@python.org> wrote:
On Feb 22, 2017, at 07:27, Victor Stinner <victor.stinner@gmail.com> wrote:
[...]My idea is not new, it's already discussed in the Python core-workflow: https://github.com/python/core-workflow/issues/6
I'm opening a thread on python-dev because Misc/NEWS quickly became a blocker issue with the Python new workflow.
What do you think of the overall idea?
I think we clearly need to do something about Misc/NEWS but this issue is already under discussion in the core-workflow working group and I think we shouldn't be moving it to python-dev until there is a final concrete proposal. I know it is high on Brett's list but I believe he's taking a few days of holiday right so he may not be able to respond immediately. Let's move this discussion over to the core-workflow mailing list. It can wait a few days.
Ned is right that I was on holiday for the past week and am slowly digging myself out from underneath my pile of email. Ned is also right that I view the Misc/NEWS issue the most important thing to solve next for our workflow as we have been putting up with merge conflicts on this file for long enough. Since this is going to be a somewhat protracted discussion as there are multiple tools consider (including writing our own), I think we should keep the discussion at https://github.com/python/core-workflow/issues/6.
tools, projects, changelog filenames x-posting from https://github.com/python/core-workflow/issues/6#issuecomment-282529194 : ... changelog filenames - CHANGELOG.rst - HISTORY.rst - whatsnew.rst - Misc/NEWS ### ``Escaping M``arkup NOTE: commit logs may contain (executable) markup - **https://wiki.python.org/moin/EscapingHtml** - https://github.com/westurner/dotfiles/blob/develop/scripts/git-changelog.py - ``rst_escape()`` (!?) Tools - mergenews - Src: https://bitbucket.org/larry/mergenews - splitnews - mergenews - pyci - hg-specific - Reno - Src: https://github.com/openstack/reno - Src: https://github.com/openstack/reno/blob/master/reno/formatter.py - Src: https://github.com/openstack/reno/blob/master/reno/sphinxext.py - PyPI: https://pypi.org/project/reno - Docs: https://docs.openstack.org/developer/reno/usage.html#editing-a-release-note - [x] ENH: "make sections configurable" https://github.com/openstack/reno/commit/081a4145e18c82acba877ee22c180b3428c... - Towncrier - Src: https://github.com/hawkowl/towncrier - PyPI: https://pypi.org/project/towncrier Projects: - CPython - https://github.com/python/cpython/blob/master/Misc/NEWS - IPython - https://github.com/ipython/ipython/tree/master/docs/source/whatsnew/pr#docum... - https://github.com/ipython/ipython/tree/master/docs/source/whatsnew - Src: https://github.com/ipython/ipython/blob/master/tools/update_whatsnew.py - Matplotlib - https://github.com/matplotlib/matplotlib/blob/master/doc/users/whats_new.rst - https://github.com/matplotlib/matplotlib/tree/master/doc/users/prev_whats_ne... - https://github.com/matplotlib/matplotlib/tree/master/doc/users/whats_new README.rst - scikit-learn - ENH, BUG, DOC, MRG - https://github.com/scikit-learn/scikit-learn/blob/master/doc/whats_new.rst - statsmodels - ENH, BUG, DOC, MRG - https://github.com/statsmodels/statsmodels/tree/master/docs/source/release - OpenStack Nova - reno - https://git.openstack.org/cgit/openstack/nova/tree/releasenotes/notes - https://github.com/openstack/nova/tree/master/releasenotes/notes - http://lists.openstack.org/pipermail/release-announce/2017-February/000784.h... On Sat, Feb 25, 2017 at 1:10 PM, Brett Cannon <brett@python.org> wrote:
On Wed, 22 Feb 2017 at 13:30 Ned Deily <nad@python.org> wrote:
On Feb 22, 2017, at 07:27, Victor Stinner <victor.stinner@gmail.com> wrote:
[...]My idea is not new, it's already discussed in the Python core-workflow: https://github.com/python/core-workflow/issues/6
I'm opening a thread on python-dev because Misc/NEWS quickly became a blocker issue with the Python new workflow.
What do you think of the overall idea?
I think we clearly need to do something about Misc/NEWS but this issue is already under discussion in the core-workflow working group and I think we shouldn't be moving it to python-dev until there is a final concrete proposal. I know it is high on Brett's list but I believe he's taking a few days of holiday right so he may not be able to respond immediately. Let's move this discussion over to the core-workflow mailing list. It can wait a few days.
Ned is right that I was on holiday for the past week and am slowly digging myself out from underneath my pile of email.
Ned is also right that I view the Misc/NEWS issue the most important thing to solve next for our workflow as we have been putting up with merge conflicts on this file for long enough.
Since this is going to be a somewhat protracted discussion as there are multiple tools consider (including writing our own), I think we should keep the discussion at https://github.com/python/core-workflow/issues/6.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ wes.turner%40gmail.com
participants (10)
-
Barry Warsaw -
Brett Cannon -
Doug Hellmann -
Eric V. Smith -
Larry Hastings -
Ned Deily -
Serhiy Storchaka -
Terry Reedy -
Victor Stinner -
Wes Turner