I fed this into tokenize.tokenize():
b''' x = "\u1234" '''
I was a bit surprised to see \Uxxxx in the output. Particularly
because the output (t.string) was a *string* and not *bytes*.
It turns out, Python's tokenizer ignores escape sequences. All it
does is ignore the next character so that \" does the proper thing.
But it doesn't do any substitutions. The escape sequences are only
handled when the AST node is created for the literal string!
Maybe I'm making a parade of my ignorance, but I assumed that string
literals were parsed by the parser--just like everything else is
parsed by the parser, hey it seems like a good place for it--and in
particular that the escape sequence substitutions would be done in
the tokenizer. Having stared at it a little, I now detect a whiff
of "this design solved a real problem". So... what was the problem,
and how does this design solve it?
BTW, my use case is that I hoped to use CPython's tokenizer to parse
some Python-ish-looking text and handle double-quoted strings for
me. *Especially* all the escape sequences--leveraging all CPython's
support for funny things like \U{penguin}. The current behavior of
the tokenizer makes me think it'd be easier to roll my own!
/arry