Basic tokenizer

Dale Strickland-Clak dale at riverhall.NOSPAM.co.uk
Wed Sep 1 15:05:49 CEST 2004


I'm looking for a simple string tokenizer. The Python tokenize module is 
too specific and is geared to Python syntax.

I just want something that can be setup to read the basic stuff like 
. identifiers 
. integers (and/or reals)
. strings
. special characters such as operators and brackets.

Is anyone aware of such a module?

Thanks.

-- 
Dale Strickland-Clark
Riverhall Systems Ltd, www.riverhall.co.uk



More information about the Python-list mailing list