[New-bugs-announce] [issue15054] bytes literals erroneously tokenized

[Florent Xicluna]

New submission from Florent Xicluna <florent.xicluna at gmail.com>:

With Python 2.7, both b'hello' and br'hello' are wrong.
With Python 3.3, b'hello' is wrong.



$ python2.7 -m tokenize <<<"'hello', u'hello', ur'hello', b'hello', br'hello'"
1,0-1,7:	STRING	"'hello'"
1,7-1,8:	OP	','
1,9-1,17:	STRING	"u'hello'"
1,17-1,18:	OP	','
1,19-1,28:	STRING	"ur'hello'"
1,28-1,29:	OP	','
1,30-1,31:	NAME	'b'
1,31-1,38:	STRING	"'hello'"
1,38-1,39:	OP	','
1,40-1,42:	NAME	'br'
1,42-1,49:	STRING	"'hello'"
1,49-1,50:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

$ python3.3 -m tokenize <<<"'hello', u'hello', ur'hello', b'hello', br'hello', rb'hello'"
1,0-1,7:            STRING         "'hello'"      
1,7-1,8:            OP             ','            
1,9-1,17:           STRING         "u'hello'"     
1,17-1,18:          OP             ','            
1,19-1,28:          STRING         "ur'hello'"    
1,28-1,29:          OP             ','            
1,30-1,31:          NAME           'b'            
1,31-1,38:          STRING         "'hello'"      
1,38-1,39:          OP             ','            
1,40-1,49:          STRING         "br'hello'"    
1,49-1,50:          OP             ','            
1,51-1,60:          STRING         "rb'hello'"    
1,60-1,61:          NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''

----------
components: Library (Lib)
messages: 162705
nosy: flox
priority: normal
severity: normal
status: open
title: bytes literals erroneously tokenized
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue15054>
_______________________________________