[Python-checkins] r61980 - in python/branches/release25-maint: Lib/test/output/test_tokenize Lib/test/tokenize_tests.txt Lib/tokenize.py Misc/NEWS

[amaury.forgeotdarc]

Author: amaury.forgeotdarc
Date: Fri Mar 28 00:41:59 2008
New Revision: 61980

Modified:
   python/branches/release25-maint/Lib/test/output/test_tokenize
   python/branches/release25-maint/Lib/test/tokenize_tests.txt
   python/branches/release25-maint/Lib/tokenize.py
   python/branches/release25-maint/Misc/NEWS
Log:
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
"" "" becomes """", which is invalid code.

Backport of r61979.


Modified: python/branches/release25-maint/Lib/test/output/test_tokenize
==============================================================================
--- python/branches/release25-maint/Lib/test/output/test_tokenize	(original)
+++ python/branches/release25-maint/Lib/test/output/test_tokenize	Fri Mar 28 00:41:59 2008
@@ -656,4 +656,10 @@
 177,11-177,15:	NAME	'pass'
 177,15-177,16:	NEWLINE	'\n'
 178,0-178,1:	NL	'\n'
-179,0-179,0:	ENDMARKER	''
+179,0-179,13:	COMMENT	'# Issue 2495\n'
+180,0-180,1:	NAME	'x'
+180,2-180,3:	OP	'='
+180,4-180,6:	STRING	"''"
+180,7-180,9:	STRING	"''"
+180,9-180,10:	NEWLINE	'\n'
+181,0-181,0:	ENDMARKER	''

Modified: python/branches/release25-maint/Lib/test/tokenize_tests.txt
==============================================================================
--- python/branches/release25-maint/Lib/test/tokenize_tests.txt	(original)
+++ python/branches/release25-maint/Lib/test/tokenize_tests.txt	Fri Mar 28 00:41:59 2008
@@ -176,3 +176,5 @@
 @staticmethod
 def foo(): pass
 
+# Issue 2495
+x = '' ''

Modified: python/branches/release25-maint/Lib/tokenize.py
==============================================================================
--- python/branches/release25-maint/Lib/tokenize.py	(original)
+++ python/branches/release25-maint/Lib/tokenize.py	Fri Mar 28 00:41:59 2008
@@ -171,11 +171,12 @@
         t1 = [tok[:2] for tok in generate_tokens(f.readline)]
         newcode = untokenize(t1)
         readline = iter(newcode.splitlines(1)).next
-        t2 = [tok[:2] for tokin generate_tokens(readline)]
+        t2 = [tok[:2] for tok in generate_tokens(readline)]
         assert t1 == t2
     """
 
     startline = False
+    prevstring = False
     indents = []
     toks = []
     toks_append = toks.append
@@ -185,6 +186,14 @@
         if toknum in (NAME, NUMBER):
             tokval += ' '
 
+        # Insert a space between two consecutive strings
+        if toknum == STRING:
+            if prevstring:
+                tokval = ' ' + tokval
+            prevstring = True
+        else:
+            prevstring = False
+
         if toknum == INDENT:
             indents.append(tokval)
             continue

Modified: python/branches/release25-maint/Misc/NEWS
==============================================================================
--- python/branches/release25-maint/Misc/NEWS	(original)
+++ python/branches/release25-maint/Misc/NEWS	Fri Mar 28 00:41:59 2008
@@ -27,6 +27,10 @@
 Library
 -------
 
+- Issue #2495: tokenize.untokenize now inserts a space between two consecutive
+  string literals; previously, ["" ""] was rendered as [""""], which is
+  incorrect python code.
+
 - Issue #2482: Make sure that the coefficient of a Decimal is always
   stored as a str instance, not as a unicode instance.  This ensures
   that str(Decimal) is always an instance of str.  This fixes a