Basic tokenizer

Dale Strickland-Clak dale at
Wed Sep 1 15:05:49 CEST 2004

I'm looking for a simple string tokenizer. The Python tokenize module is 
too specific and is geared to Python syntax.

I just want something that can be setup to read the basic stuff like 
. identifiers 
. integers (and/or reals)
. strings
. special characters such as operators and brackets.

Is anyone aware of such a module?


Dale Strickland-Clark
Riverhall Systems Ltd,

More information about the Python-list mailing list