how to determine an 'open' string?

[holger krekel]

Bernhard Herzog wrote:
> holger krekel <pyth at devel.trillke.net> writes:
> 
> > def open_quote(text, rex=re.compile('"""|\'\'\'|"|\'')):
> >     """ return the open quote at the end of text.
> >         if all string-quotes are matched, return the
> >         empty string. thanks to Harvey Thomas&John La Roy.
> >     """
> >     rfunc = lambda x,y: x=='' and y  or  not y.startswith(x) and x   or ''
> >     quotes = rex.findall(text)
> >     return quotes and reduce(rfunc,quotes) or ''
> 
> This doesn't even look at backslashes in the string.
> 
> >>> open_quote("'\\'")   # should return "'"
> ''

i have a somewhat corrected version but ...

> Since you're trying to parse python source code: why don't you try the
> tokenize module?

this seems like a good idea! I didn't know that tokenize does this
(in other contexts like c++/java 'tokenize' was a more general thing not
directly capable of parsing quoted strings correctly etc.)

Preliminary tests show that python's tokenize is relatively easy to handle
although i already found some gotchas. E.g. tokenize produces for 

    print "

first the ERRORTOKEN ' ' and then the ERRORTOKEN '"'. (strange, not?)
I guess i have to understand 'tokenize' better to have it under full control.

i knew there must be a right way :-)

thanks again,

    holger