[ python-Bugs-1230484 ] tokenize bug
SourceForge.net
noreply at sourceforge.net
Fri Jul 15 11:00:12 CEST 2005
Bugs item #1230484, was opened at 2005-06-30 12:35
Message generated for change (Comment added) made by rhettinger
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230484&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Eduardo Aguiar (aguiar)
>Assigned to: Tim Peters (tim_one)
Summary: tokenize bug
Initial Comment:
hi,
I have found a bug in 'tokenize' module: it is merging
a COMMENT and a NL token for lines that start with a
comment.
I have made a fell changes and it seems to be working
fine. Follows a patch:
*** /usr/lib/python2.4/tokenize.py 2005-01-02
03:34:20.000000000 -0200
--- tokenize.py 2005-06-30 14:31:19.000000000 -0300
***************
*** 216,223 ****
pos = pos + 1
if pos == max: break
! if line[pos] in '#\r\n': # skip
comments or blank lines
! yield ((NL, COMMENT)[line[pos] ==
'#'], line[pos:],
(lnum, pos), (lnum,
len(line)), line)
continue
--- 216,235 ----
pos = pos + 1
if pos == max: break
! if line[pos] == '#': # skip
comments
! end = len(line) - 1
! while end > pos and line[end] in '\r\n':
! end = end - 1
! end = end + 1
!
! yield (COMMENT, line[pos:end],
! (lnum, pos), (lnum, end),
line)
! yield (NL, line[end:],
! (lnum, end), (lnum,
len(line)), line)
! continue
!
! if line[pos] in '\r\n': # skip
blank lines
! yield (NL, line[pos:],
(lnum, pos), (lnum,
len(line)), line)
continue
Best regards,
Eduardo Aguiar (eaguiar at programmer.net)
----------------------------------------------------------------------
>Comment By: Raymond Hettinger (rhettinger)
Date: 2005-07-15 04:00
Message:
Logged In: YES
user_id=80475
Tim, this affects a line you checked-in, 1.23 on 6/18/2001.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230484&group_id=5470
More information about the Python-bugs-list
mailing list