[issue23921] Standardize documentation whitespace, formatting
New submission from James Edwards: I realize this is a huge patch, I'd be happy to split it to multiple little patches (one per file, one per documentation directory, etc.) to make things easier. Just let me know. The patch attempts to do a few things (with exceptions, as noted below): 1. Standardize leading whitespace in block bodies (4 spaces per level) 2. Enforce >= 2 spaces between code and inline comment 3. In interactive interpreter snippets, ensures lines meant to be entered are preceded by '>>> ' or '... ' 4. Inline whitespace standardization (following a liberal / lenient (not strict) PEP-8 testing) Scanning the documentation, extracting the code snippets, and testing them for PEP-8 compliance was done with a script, but all adjustments were done by hand. This means that there remain some code that fails the lenient PEP-8 checks that I used. For example, code at [1] ("How to write obfuscated one liners") obviously threw errors, but because of the context of the snippet and the use it serves in the documentation, it was left it unchanged. Similarly, there are (intentionally) poorly formatted snippets in [2] ("Lexical Analysis") that were also left unchanged. Since these changes were applied by hand, I was able to ignore situations where things that would normally raise warnings. I erred on the side of leaving the documentation examples unchanged, and strived to make only innocuous changes. I made no attempt to change the functionality or semantics of any of the snippets. The only changes I made were "harmless" formatting. None of the changes will affect the function or output of the snippets. What the changes do, however, is give a consistency to the documentation that will allow readers to become more comfortable with the structure of the language and improve readability. [1] https://docs.python.org/3/faq/programming.html#is-it-possible-to-write-obfus... [2] https://docs.python.org/3/reference/lexical_analysis.html#indentation [3] http://pep8.readthedocs.org/en/latest/intro.html In addition to the checks that are ignored by default by the pep8[3] module for not being unanimously accepted: E121 # continuation line under-indented for hanging indent E123 # closing bracket does not match indentation of opening bracket’s line E126 # continuation line over-indented for hanging indent E133 # closing bracket is missing indentation E226 # missing whitespace around arithmetic operator E241 # multiple spaces after ‘,’ E242 # tab after ‘,’ E704 # multiple statements on one line (def) The following checks were "globally" ignored in my script, though many others were conditionally ignored by the script or by myself. ignore.append('W292') # no newline at end of file ignore.append('E302') # expected 2 blank lines, found 0 ignore.append('E401') # multiple imports on one line ignore.append('W391') # blank line at end of file ignore.append('E231') # missing whitespace after ',' ignore.append('E114') # indentation is not multiple of four (comment) ignore.append('E116') # unexpected indentation (comment) While the patch diffstat is: 67 files changed, 450 insertions(+), 412 deletions(-) Ignoring all whitespace changes, the diffstat is only: 10 files changed, 118 insertions(+), 114 deletions(-) The majority of these remaining changes fix the inconsistency of interactive interpreter snippets, where, within the same snippet, some lines have '>>> ' while others are missing it, or the '... ' continuation prompt. Let me know if you need anything from me (such as splitting this patch up) to help get it merged. Thanks. ---------- assignee: docs@python components: Documentation files: docsdiff.diff keywords: patch messages: 240540 nosy: docs@python, jedwards priority: normal severity: normal status: open title: Standardize documentation whitespace, formatting type: enhancement Added file: http://bugs.python.org/file38903/docsdiff.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Changes by James Edwards <jheiv@jheiv.com>: ---------- versions: +Python 3.6 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: ---------- nosy: +r.david.murray, serhiy.storchaka stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Berker Peksag added the comment: Patch looks good to me. Thanks! :) I left a couple of comments on Rietveld: http://bugs.python.org/review/23921/ We can probably ignore the following type of changes too: - if pid == 0: # In a child process + if pid == 0: # In a child process - status = '200 OK' # HTTP Status - headers = [('Content-type', 'text/plain')] # HTTP Headers + status = '200 OK' # HTTP Status + headers = [('Content-type', 'text/plain')] # HTTP Headers - dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name + dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name Also, perhaps we can add your script to Tools/scripts and integrate it with the "make patchcheck" command. ---------- nosy: +berker.peksag versions: +Python 3.4, Python 3.5 -Python 3.6 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
R. David Murray added the comment: I will have comments on this, but it may be a bit before I get to it. In general I'm not sure of the value, though in the little bit I scanned so far there are at least a few changes that look worthwhile. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
James Edwards added the comment: Thanks Berker, I responded to most of your comments in rietveld. A few of your comments suggested "we should get rid of X", and while I can't say I disagree, I really tried to limit the scope of the changes to whitespace and formatting. As far as the script goes, it still requires significant human interaction. Of the 1983 code snippets it extracts[1], 346 still raise warnings and require human verification. The majority of these snippets aren't actually intended to be Python code. Some of the snippets are C code, others are optparse/argparse command line examples, configparser config files, or just sections where the author wanted to use a literal block in the documentation.[2] Of course, when the script triggers on one of these blocks, I ignore it. Eventually, my goal was to have the script also verify interactive code blocks, much like doctest, but I thought a good intermediary step was to unify the whitespace. As for the value, I understand the concern one might have before embarking on the process of standardizing the whitespace and formatting -- but given that it's mostly done and comes neatly packaged in a patch, I'm not sure I understand any remaining concerns. In other words, if it's not something you would spend time on personally, because you have better ways to contribute/ spend your time, I absolutely understand that. But given that the majority of the leg work is done by the patch, I hope you'll consider merging the changes. In general the goal was to make the snippets, especially in the tutorial-type sections consistent, to allow the reader to become comfortable with the language instead of being distracted by varied formatting and to facilitate readability. I hope this explains the motivation behind the effort. [1] Some files were ignored from future runs after the snippets were validated, because they contained too many false positives, e.g. ctypes.rst. So this count doesn't reflect the blocks in those ignored files. [2] I wonder if explicitly indicating which rst literal blocks are intended to be python code would be of some benefit. For example, adding such a tag could trigger the automatic formatting checks and, for interactive blocks, run the automatic doctest-type verification I was envisioning. I may think more about the feasibility of this and bring it to python-ideas/python-dev. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Serhiy Storchaka added the comment: The patch mostly LGTM. Most changes are trivial, but the patch also fixes few bugs. It is nice that the patch adds empty lines after blocks, this makes copying the code to interactive interpreter easier. I left comments on Rietveld. The most questionable changes are adding spaces before comments. But these changes are simple and can slightly increase readability. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
James Edwards added the comment: Thanks for the review Serhiy. I'll prepare a revised patchset, given the comments from you and Berker and have it uploaded today. Thanks again. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
James Edwards added the comment: Attaching revised patch per reviews. Notable changes: * Reverted howto/curses.rst multiple inline statements -> multi-target assignment (curses.rst is now unchanged) * Reverted library/subprocess.rst "==>" to "# ==>" changes since there are other outstanding issues in the file. (subprocess.rst is now unchanged) ---------- Added file: http://bugs.python.org/file39013/docsdiff.r2.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Martin Panter added the comment: I do prefer extra space before a comment, otherwise it looks too much like some kind of “x # y” binary operator. So I think those changes are worthwhile. I left some review comments, mainly minor suggestions. Most of the other changes look worthwhile. Others I don’t mind either way, but I can understand others may prefer them, such as lining up hanging indents for fixed-width fonts. David: did you end up looking this? Are there any particular bits you think are not worth it? ---------- nosy: +martin.panter _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Changes by R. David Murray <rdmurray@bitdance.com>: ---------- nosy: -r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
R. David Murray added the comment: Sorry, I'm not going to have time to look at it. ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Changes by R. David Murray <rdmurray@bitdance.com>: ---------- nosy: -r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Serhiy Storchaka added the comment: James, could you please update your patch to address Martin's comments? In general the patch LGTM and I hope to commit it in short time. ---------- nosy: +georg.brandl versions: +Python 3.6 -Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Georg Brandl added the comment: I had no idea we had so many cases of wonky indentation :) This looks good to me, apart from a few comments I left on rietveld. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: ---------- assignee: docs@python -> serhiy.storchaka _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Roundup Robot added the comment: New changeset 2b492ea961c1 by Serhiy Storchaka in branch '3.5': Issue #23921: Standardized documentation whitespace formatting. https://hg.python.org/cpython/rev/2b492ea961c1 New changeset 909099686e6e by Serhiy Storchaka in branch 'default': Issue #23921: Standardized documentation whitespace formatting. https://hg.python.org/cpython/rev/909099686e6e ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Roundup Robot added the comment: New changeset be85897e6d58 by Serhiy Storchaka in branch '2.7': Issue #23921: Standardized documentation whitespace formatting. https://hg.python.org/cpython/rev/be85897e6d58 ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
Serhiy Storchaka added the comment: Addressed Martin's Georg's and my comments, resolved conflicts, stripped trailing spaces, backported to 2.7. ---------- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 2.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue23921> _______________________________________
participants (7)
-
Berker Peksag -
Georg Brandl
-
James Edwards
-
Martin Panter
-
R. David Murray
-
Roundup Robot
-
Serhiy Storchaka