[pypy-svn] r77734 - pypy/branch/fast-forward/pypy/interpreter/pyparser

[afa at codespeak.net]

Author: afa
Date: Fri Oct  8 22:41:29 2010
New Revision: 77734

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py
   pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py
Log:
Allow 'whitespace' to form a valid token, when nothing more can match.

This gives a better error position: when given "if ?:", the parser
will return whitespace as a token, and stop on the ?.
Previously, the error would be at the space character,
and one test in interpreter/test/test_syntax.py failed.

This effectively reverts the only effective change in r77650,
which I did not understand correctly at the time.


Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py	Fri Oct  8 22:41:29 2010
@@ -182,6 +182,7 @@
     pseudoToken = chain(states,
                         makeWhitespace(),
                         group(states,
+                              newArcPair(states, EMPTY),
                               pseudoExtras, number, funny, contStr, name))
     dfaStates, dfaAccepts = nfaToDfa(states, *pseudoToken)
     return DFA(dfaStates, dfaAccepts)

Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py	Fri Oct  8 22:41:29 2010
@@ -23,7 +23,7 @@
 # ______________________________________________________________________
 # Automatically generated DFA's
 
-accepts = [False, True, True, True, True, True, True, True,
+accepts = [True, True, True, True, True, True, True, True,
            True, True, False, True, True, True, False, False,
            False, False, True, True, True, True, True, False,
            True, False, True, False, True, False, False,