[Python-ideas] Line continuations with comments

[Ron Adam]

There were a few people who liked the idea of having comments after a line 
continuation.

I was able to make a small patch that removed the some of the restrictions 
on the '\' for testing some ideas which does the following.

     * Allow a line to continue on the same line.

     * Skips comments before checking for the new line
       after a back slash.


Here are some examples...


These are technically the same.

 >>> 'aaa' \
... 'bbb' \
... 'ccc'
'aaabbbccc'

 >>> 'aaa' \ 'bbb' \ 'ccc'
'aaabbbccc'

Yes there is't much need for this, but I wanted to see if it would work and 
if the test suit passes.  It does. ;-)


You can put a comment after a line continuation.

 >>> 'aaa'    \# one
... 'bbb'    \# two
... 'ccc'     # three
'aaabbbccc'


Works with expressions too.

 >>> result = \
...        + 111  \# A
...        + 222  \# B
...        + 333  \# C
...        + 444   # D
 >>> result
1110


But if it has a space between the \ and the #, the line is continued on the 
same line instead of the following line.

 >>> 'aaa' \ #comment
'aaa'

The reason \# works, but not \ #, is when the comment comes directly after 
the back slash, it's removed and leaves a (backslash + new-line) pair.

Removing the white space before the new line check caused some errors in 
the test suite.  I haven't figured out why yet.  So this doesn't do that 
for now.



Currently you get this if you try any of these examples.

 >>> 'abc' \#comment
   File "<stdin>", line 1
     'abc' \#comment
                   ^
SyntaxError: unexpected character after line continuation character




Only one of pythons tests fail, and I don't think it's related.

     test test_urllib2_localnet failed



See the diff below if you want to play with it.  It's not big.

Cheers,
     Ron




diff -r 155e6fb309f5 Parser/tokenizer.c
--- a/Parser/tokenizer.c	Tue May 21 21:02:04 2013 +0200
+++ b/Parser/tokenizer.c	Tue May 21 22:10:31 2013 -0500
@@ -1391,18 +1391,31 @@

   again:
      tok->start = NULL;
+
+    c = tok_nextc(tok);
+
+    /* Check if continuing line */
+    if (tok->cont_line == 1 && c == '\n') {
+        tok->cont_line = 0;
+        c = tok_nextc(tok);
+    }
+
      /* Skip spaces */
-    do {
+    while (c == ' ' || c == '\t' || c == '\014') {
          c = tok_nextc(tok);
-    } while (c == ' ' || c == '\t' || c == '\014');
+        tok->cont_line = 0;
+    }

      /* Set start of current token */
      tok->start = tok->cur - 1;

      /* Skip comment */
-    if (c == '#')
+    if (c == '#') {
          while (c != EOF && c != '\n')
              c = tok_nextc(tok);
+        tok_backup(tok, c);
+        goto again;
+    }

      /* Check for EOF and errors now */
      if (c == EOF) {
@@ -1641,12 +1654,6 @@

      /* Line continuation */
      if (c == '\\') {
-        c = tok_nextc(tok);
-        if (c != '\n') {
-            tok->done = E_LINECONT;
-            tok->cur = tok->inp;
-            return ERRORTOKEN;
-        }
          tok->cont_line = 1;
          goto again; /* Read next line */
      }