[Python-bugs-list] [ python-Bugs-678518 ] Another parsermodule validation error

Fri, 07 Feb 2003 14:51:50 -0800

Bugs item #678518, was opened at 2003-01-31 20:23
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=678518&group_id=5470

Category: Parser/Compiler
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: logistix (logistix)
Assigned to: Michael Hudson (mwh)
Summary: Another parsermodule validation error

Initial Comment:
The following code generates an validation error.  It's an 
invalid ast tree where the global_stmt doesn't have the 
keyword 'global' as a child.

I tracked down the fix to validate_global_stmt() this time.

Python 2.3a1 (#38, Dec 31 2002, 17:53:59) [MSC 
v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import symbol,token,parser
>>> ast = [symbol.file_input,
...  [symbol.stmt, [symbol.simple_stmt, 
[symbol.small_stmt, [symbol.global_stmt,
 [1, 'foo']]], [token.NEWLINE, '']]],
...  [token.NEWLINE, ''],
...  [token.ENDMARKER, '']]
>>> a = parser.sequence2ast(ast)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
parser.ParserError: VALIDATION FAILURE: report this 
to the maintainer!
>>>

----------------------------------------------------------------------

>Comment By: logistix (logistix)
Date: 2003-02-07 16:51

Message:
Logged In: YES 
user_id=699438

You're right, the assert statement was the unhandled 
exception.  It was raising an error but returning true on 
the "__assert__" check.

Just posted the whole patch

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-07 08:38

Message:
Logged In: YES 
user_id=6656

> At the time I didn't have an encoding patch and punted.

Oh, I misread you then.

> I did figure it out last night though.  I'll post it later
today 
> since I'm at the office now.

Ta.

> And yes, this is the error that'll print out "XXX 
> unhandled exception" out of ceval.c when you run
> the test from debug mode.

Are you sure?  It seems that was the assert one.

> There was some code added to add the encoding 
> type in the ast2sequence functions,

By me :-)  I didn't really think about the sequence2ast 
functions as all I was doing was trying to get the compiler
package to work...

> Also, as a matter of style, should I add the stuff in 
> my test script to the test suite or is that overkill? Should
> I just have it check 4 or 5 random files?

It might be best to get it to check a few files we knew there
were problems with -- test_pep263, maybe.  Checking the
whole lot is probably overkill.

----------------------------------------------------------------------

Comment By: logistix (logistix)
Date: 2003-02-07 08:23

Message:
Logged In: YES 
user_id=699438

Michael,

At the time I didn't have an encoding patch and punted.  I did 
figure it out last night though.  I'll post it later today since I'm 
at the office now.

And yes, this is the error that'll print out "XXX unhandled 
exception" out of ceval.c when you run the test from debug 
mode.

It comes down to the bastardized way that encoding is 
implemented.  It doesn't fit the current definition of a token or 
a symbol.  There was some code added to add the encoding 
type in the ast2sequence functions, but not the 
sequence2ast functions, so when it hit scripts with 
encoding_decl's, it bombed big time.

Also, as a matter of style, should I add the stuff in my test 
script to the test suite or is that overkill? Should I just have it 
check4 or 5 random files?

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-07 06:08

Message:
Logged In: YES 
user_id=6656

Pff, I was right, but logistix's patch fix it.

logistix, did you forget to attach the encoding patch?  I
still get errors in module's with a cookie.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-07 05:58

Message:
Logged In: YES 
user_id=6656

Patches as such are probably fine.  The reason I haven't
checked them in is a wierdness that I may or may not be
related, but is bothering me.  Consider:

->> import parser, sys,glob,os, traceback
/>> for path in sys.path:
|..     for fil in glob.glob(os.path.join(path,"*.py")):
|..       try:
|..         try:
|..             a = parser.suite(file(fil).read())
|..             b = parser.ast2tuple(a)
|..             c = parser.sequence2ast(b)
|..         except parser.ParserError:
|..             print "Parse of %s" % fil, "FAILED"
|..             traceback.print_exc()
|..             print '----------------------------'
|..       except:
|..          print "waaa?"
|.. else:
|..     print "what?"
\__ 
Parse of /home/mwh/src/sf/python/dist/src/Lib/tarfile.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/heapq.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/getopt.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/inspect.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/pydoc.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/lib-tk/Tix.py
FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Traceback (most recent call last):
  File "<input>", line 2, in ?
ParserError: Illegal terminal: expected "__assert__"
->> 

Notice that an exception seems to have made it through the
except clauses!  The scary bit is that a (seemingly) random
number of parser errors *will* caught, and then one will get
through.

The only explanation I have for this is some kind of
returning non-NULL with exception set kind of bug.

Off to play with a debug build, I guess.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2003-02-06 19:17

Message:
Logged In: YES 
user_id=33168

Michael were these patches ok to check in?

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-03 05:00

Message:
Logged In: YES 
user_id=6656

Thanks for doing this.  I'll probably check these in after
I've finished crunching my way through monday morning email...

----------------------------------------------------------------------

Comment By: logistix (logistix)
Date: 2003-02-01 15:42

Message:
Logged In: YES 
user_id=699438

And I found a few more by doing a roundtrip 
ast2Tuple/tuple2ast on all files in pythonpath.  These actually 
affect valid AST's, so they're probably a bigger deal.

Valid assert statements were failing.  The patch was trivial, 
so its attached.

Also, files that took advantage of encoding_decl (from PEP 
263) were failing.  I couldn't tell if PEP 263 was offically 
closed, and didn't know what the offical grammar was, so I 
didn't do anything there.

I'll also attach the test script that turned up these errors.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=678518&group_id=5470