[issue3353] make built-in tokenizer available via Python C API

[Fredrik Lundh]

Fredrik Lundh <effbot at users.sourceforge.net> added the comment:

That's should be all that's needed to expose the existing API, as is. 
If you want to verify the build, you can grab the pytoken.c and setup.py
files from this directory, and try building the module.

http://svn.effbot.org/public/stuff/sandbox/pytoken/

Make sure you remove the local copy of "tokenizer.h" that's present in
that directory before you build.  If that module builds, all's well.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3353>
_______________________________________