tokenise a string

M.E.Farmer mefjr75 at
Sun Oct 17 00:05:13 CEST 2004

Matthias Teege <matthias-dated at> wrote in message 
> I have a input string like this:
> "name=matthias;count>10" or this "location!=thisone".

> So I must parse the input, build tokens and map the fieldnames. Is
> there a special modul which I can use or are the standard string
> functions adequate?
> Many thanks
> Matthias

This might be helpful.
Shlex can do much more, but I have not had the time to explore it.
#python shell#
>>> import shlex
>>> import cStringIO
>>> myFilelikeString = cStringIO.StringIO("name=matthias;count>10")
>>> lexer = shlex.shlex(myFilelikeString)
>>> while 1:
...     token = lexer.get_token()
...     if token:
...         print token
...     else:
...         break

Also look at PySourceColor that I posted it covers token and tokenize.

More information about the Python-list mailing list