[issue3353] make built-in tokenizer available via Python C API

[Jim Fasarakis-Hilliard]

Jim Fasarakis-Hilliard added the comment:

Okay, I'll take a look at it over the next days and try and submit a PR after fixing any issues that might be present.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3353>
_______________________________________