[Python-bugs-list] [ python-Bugs-410155 ] tokenize.NL undocumented

[noreply@sourceforge.net]

Bugs item #410155, was updated on 2001-03-20 13:28
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=410155&group_id=5470

Category: Documentation
Group: None
Status: Open
Priority: 5
Submitted By: Jürgen Hermann (jhermann)
Assigned to: Guido van Rossum (gvanrossum)
Summary: tokenize.NL undocumented

Initial Comment:
In section 17.5 of the Python 2.0 Library Reference, 
the constant "NL" is not documented.

----------------------------------------------------------------------

>Comment By: Jürgen Hermann (jhermann)
Date: 2001-03-20 19:21

Message:
Logged In: YES 
user_id=39128

tokenize.NL seems to be an empty line, as opposed to 
token.NEWLINE which is a logical line end. Insofar NL is 
comparable to COMMENT, since it has no semantic content and 
is ignorable by a parser.

1 a = \
2 2
3
4 b=4

emits these tokens:

1,0-1,1:        NAME    'a'
1,2-1,3:        OP      '='
2,0-2,1:        NUMBER  '2'
2,1-2,2:        NEWLINE '\012'
3,0-3,1:        NL      '\012'
4,0-4,1:        NAME    'b'
4,1-4,2:        OP      '='
4,2-4,3:        NUMBER  '4'
4,3-4,4:        NEWLINE '\012'
5,0-5,0:        ENDMARKER       ''

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-03-20 18:22

Message:
Logged In: YES 
user_id=31435

Yes, it should be, but Ping doesn't know about NL:  Guido 
added that one.  Python's real tokenizer has no such token 
type.  It was added because-- unlike Python's real 
tokenizer --tokenizer.py passes *everything* on to the 
client, including blank lines and comments, so passes on 
newlines that aren't NEWLINEs too.  Reassigned to Guido in 
case he remembers more about the details ...

----------------------------------------------------------------------

Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2001-03-20 14:58

Message:
Logged In: YES 
user_id=3066

Ping, should NL be documented?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=410155&group_id=5470