[Patches] [ python-Patches-1070218 ] Add BLANK_LINE to token.py

[SourceForge.net]

Patches item #1070218, was opened at 2004-11-20 22:58
Message generated for change (Comment added) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1070218&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Library (Lib)
Group: None
>Status: Closed
>Resolution: Rejected
Priority: 5
Private: No
Submitted By: Grant Olson (logistix)
Assigned to: Jeremy Hylton (jhylton)
Summary: Add BLANK_LINE to token.py

Initial Comment:
The parser module generates an ast node number 54 for 
an entirely blank line.  These lines are apparently ignored 
by the real parser, and don't have a real definition in 
include\token.h.  Presumably because of that they don't 
have an entry in token.py.

However, this broke an app where I hardcoded the 
number and recent language features have changed the 
mapping of numbers to token names.

I think there should be a BLANK_LINE defined in token.py 
so that it can be referred to by name, even if it's not 
used in the real parser.  While dealing with parsermodule 
generated ast-trees, "symbol.sym_name[node] or 
token.tok_name[node]" should always be true.

----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2006-11-19 09:33

Message:
Logged In: YES 
user_id=21627
Originator: NO

As you found yourself, the patch is wrong. This token type is already
exposed from tokenize as tokenize.NL; if you use tokenize, you should use
its list of token types, not the one of token.py.

Closing this patch as rejected.

----------------------------------------------------------------------

Comment By: Grant Olson (logistix)
Date: 2004-11-22 01:41

Message:
Logged In: YES 
user_id=699438

A few clarifications:

On further review, the node number 54 comes from 
tokenize.py, not the parsermodule.

I also meant to say that "symbol.sym_name.has_key(node) or 
token.tok_name.has_key(node)" should always be true.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1070218&group_id=5470