generic tokenizer

Angus Mackay yeah at
Tue Aug 31 20:30:26 CEST 2004

I remember python having a generic tokenizer in the library. all I want 
is to set a list of token seperators and then read tokens out of a 
stream, the token seperators should be returned as themselves.

is there anything like this?

cheers, Angus.

More information about the Python-list mailing list