[issue3353] make built-in tokenizer available via Python C API

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

IMO the "struct tok_state" should not be part of the API, it contains
too many implementation details. Or maybe as an opaque structure.

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3353>
_______________________________________