Howto or Tutorial for tokenize module for a newbie?

[sreekant]

TY wrote:
> Hi,
> 
> Can someone point me to a Howto or Tutorial for tokenize module for a
> newbie?  I read the documentation but it doesn't have a lot of info...
>  Thanks!
> 
Hi there

I don't know if I got your requirement. But I used a own version of 
tokenizer which can use more than one delimiters. It is below. Hopefully 
of some use to you.




def gettokens(dat):
     delims={'=':'',';':',','=':'','(':'',')':'',':':'','[':'',']':''}
     wlist=[]
     appended=0
     for n in string.split(dat,'\n'):
         word=''
         for m in n:
             appended=0
             if delims.has_key(m):
                 wlist.append(word)
                 wlist.append(m)
                 word=''
                 appended=1
             else:
                 word=word+m
         if appended==0:
             wlist.append(word)
             appended=1
         wlist.append("\n")
     return wlist

I am sure there are plenty of ways to write it better than that.

Good luck
sree