[Python-Dev] tokenize string literal problem

[C or L Smith]

C or L Smith wrote:
> PROBLEM
>     I need to find code snippets which are located in docstrings.
> Docstrings, being string literals should be able to be parsed out
> with tokenize. But tokenize is giving the wrong results (or I am
> doing something wrong) for this (pathological) case:   
> 
> foo.py:
> +----
> def bar():
>     """
>     A quoted triple quote is not a closing
>     of this docstring:
>     >>> print '"""'
>     """
>     """ # <-- this is the closing quote
>     pass
> +----
> 

I now see that I've created a code snippet that is invalid. Myopia. The thing that pythonWin was doing correctly was displaying my sample STRING not code. I had delimited the code with triple-single-quotes so it showed up correctly. In fact, if entered as code it would show the need to delimit the docstring contents with ''' rather than """.

Sorry!
/c