[Tutor] how to cope with text litterals by code parsing?

[denis]

----- Original Message -----
From: Rich Krauter <rmkrauter at yahoo.com>
To: <tutor at python.org>
Sent: Tuesday, May 18, 2004 1:21 AM
Subject: Re: [Tutor] how to cope with text litterals by code parsing?


> On Mon, 2004-05-17 at 11:12, denis wrote:
> > Hello,
> >
> > In a module that parses Python code, I have a symbol_row class that
builds a
> > list of meaningful symbols (about what Guido van Rossum calls 'tokens'
in
> > the language reference) out of a line of code. It splits the line and
> > defines each symbol's nature (e.g. keyword) and role (e.g. operator).
> > Everything works fine, but I'm not satisfied of how it's done.

<snipped explainatons>

> Hi Denis,
>
> Do you have specific requirements that call for you to do everything
> yourself?
Well, yes and no, this is rather a first step on a personal project. There's
no requirement else than mine! ;-) But the final purpose isn't to parse
Python, I do it as training, because it's clear and stable.

> If not, there is a standard module called tokenize:
> http://docs.python.org/lib/module-tokenize.html

I didn't even look for a module. Stupid. Anyway, thank you very much for the
tip. It will probably find there many right ways to do all the stuff. I'll
watch that carefully.
[Funny that I first called the main function 'tokenize', too.]

denis

<snipped example>