Tokenizing a string

[Gordon McMillan]

Michael Dartt wrote:

> I've got a string I'd like to tokenize, but it's not in a file, and it'd
> be rather inefficient to write it to a file just to tokenize it.  Is
> there any function I can use to pass this string to
> tokenize.tokenize()?  

You just need to pass tokenize a function that returns a line. If 
your text is in a list, you could use 
 tokenize(lambda txt=txt: txt.pop(0))

StringIO or cStringIO is another good possibility.


- Gordon