[Python-Dev] tokenize string literal problem

C or L Smith smiles at worksmail.net
Sat Oct 24 08:33:10 CEST 2009

C or L Smith wrote:
>     I need to find code snippets which are located in docstrings.
> Docstrings, being string literals should be able to be parsed out
> with tokenize. But tokenize is giving the wrong results (or I am
> doing something wrong) for this (pathological) case:   
> foo.py:
> +----
> def bar():
>     """
>     A quoted triple quote is not a closing
>     of this docstring:
>     >>> print '"""'
>     """
>     """ # <-- this is the closing quote
>     pass
> +----

I now see that I've created a code snippet that is invalid. Myopia. The thing that pythonWin was doing correctly was displaying my sample STRING not code. I had delimited the code with triple-single-quotes so it showed up correctly. In fact, if entered as code it would show the need to delimit the docstring contents with ''' rather than """.


More information about the Python-Dev mailing list