[New-bugs-announce] [issue23894] lib2to3 doesn't recognize rb'...' as a raw byte string in Python 3

[Eli Bendersky]

New submission from Eli Bendersky:

lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc')

This is because pgen2/tokenize.py doesn't list rb'' as a viable prefix for a string, even though according to https://docs.python.org/3/reference/lexical_analysis.html it is

----------
keywords: easy
messages: 240322
nosy: benjamin.peterson, eli.bendersky
priority: normal
severity: normal
status: open
title: lib2to3 doesn't recognize rb'...' as a raw byte string in Python 3
versions: Python 3.5, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23894>
_______________________________________