tokenizer..

[Jürgen Hermann]

"John" <john.thai at dspfactory.com> schrieb im Newsbeitrag
news:fhxT6.245523$Z2.2804247 at nnrp1.uunet.ca...
>     Is there a built in function or module which allows me to get tokens
> from a string?
> I know there's the tokenize module but that needs a callable object and it
> seems to loop forever...

See http://aspn.activestate.com/ASPN/Python/Cookbook/Recipe/52298 for an
example.