[Python-bugs-list] [ python-Bugs-678518 ] Another parsermodule validation error

[SourceForge.net]

Bugs item #678518, was opened at 2003-02-01 02:23
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=678518&group_id=5470

Category: Parser/Compiler
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: logistix (logistix)
Assigned to: Michael Hudson (mwh)
Summary: Another parsermodule validation error

Initial Comment:
The following code generates an validation error.  It's an 
invalid ast tree where the global_stmt doesn't have the 
keyword 'global' as a child.

I tracked down the fix to validate_global_stmt() this time.

Python 2.3a1 (#38, Dec 31 2002, 17:53:59) [MSC 
v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import symbol,token,parser
>>> ast = [symbol.file_input,
...  [symbol.stmt, [symbol.simple_stmt, 
[symbol.small_stmt, [symbol.global_stmt,
 [1, 'foo']]], [token.NEWLINE, '']]],
...  [token.NEWLINE, ''],
...  [token.ENDMARKER, '']]
>>> a = parser.sequence2ast(ast)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
parser.ParserError: VALIDATION FAILURE: report this 
to the maintainer!
>>>

----------------------------------------------------------------------

>Comment By: Michael Hudson (mwh)
Date: 2003-02-07 12:08

Message:
Logged In: YES 
user_id=6656

Pff, I was right, but logistix's patch fix it.

logistix, did you forget to attach the encoding patch?  I
still get errors in module's with a cookie.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-07 11:58

Message:
Logged In: YES 
user_id=6656

Patches as such are probably fine.  The reason I haven't
checked them in is a wierdness that I may or may not be
related, but is bothering me.  Consider:

->> import parser, sys,glob,os, traceback
/>> for path in sys.path:
|..     for fil in glob.glob(os.path.join(path,"*.py")):
|..       try:
|..         try:
|..             a = parser.suite(file(fil).read())
|..             b = parser.ast2tuple(a)
|..             c = parser.sequence2ast(b)
|..         except parser.ParserError:
|..             print "Parse of %s" % fil, "FAILED"
|..             traceback.print_exc()
|..             print '----------------------------'
|..       except:
|..          print "waaa?"
|.. else:
|..     print "what?"
\__ 
Parse of /home/mwh/src/sf/python/dist/src/Lib/tarfile.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/heapq.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/getopt.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/inspect.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/pydoc.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/lib-tk/Tix.py
FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Traceback (most recent call last):
  File "<input>", line 2, in ?
ParserError: Illegal terminal: expected "__assert__"
->> 

Notice that an exception seems to have made it through the
except clauses!  The scary bit is that a (seemingly) random
number of parser errors *will* caught, and then one will get
through.

The only explanation I have for this is some kind of
returning non-NULL with exception set kind of bug.

Off to play with a debug build, I guess.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2003-02-07 01:17

Message:
Logged In: YES 
user_id=33168

Michael were these patches ok to check in?

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-03 11:00

Message:
Logged In: YES 
user_id=6656

Thanks for doing this.  I'll probably check these in after
I've finished crunching my way through monday morning email...

----------------------------------------------------------------------

Comment By: logistix (logistix)
Date: 2003-02-01 21:42

Message:
Logged In: YES 
user_id=699438

And I found a few more by doing a roundtrip 
ast2Tuple/tuple2ast on all files in pythonpath.  These actually 
affect valid AST's, so they're probably a bigger deal.

Valid assert statements were failing.  The patch was trivial, 
so its attached.

Also, files that took advantage of encoding_decl (from PEP 
263) were failing.  I couldn't tell if PEP 263 was offically 
closed, and didn't know what the offical grammar was, so I 
didn't do anything there.

I'll also attach the test script that turned up these errors.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=678518&group_id=5470