string-based tokenizer?

[The Blue Wizard]

Tim Peters <tim_one at email.msn.com> wrote in article
<000401be95df$3a86a1e0$dca22299 at tim>...
> [The Blue Wizard]
> > I was looking for a quick way to tokenize the Python and Python-like
> > expressions, and I found the tokenizer.py.  It is quite nice, but there
> > is one little ugly fact:  it is dependent on datastream.
> 
> Nope:  it depends on something (anything!) that "acts like a stream". 
With
> enough Pythonistic cleverness, you can feed it a disguised integer
<wink>.

Uh...yeah...but it would be far better not to be so dependent on
"cleverness"
that is so endemic to many computer languages.  Thank goodness Python
is largely free of that!

> > I want to be able to just type  tokenize('a cute string known as 3+4')
> > and get a series of tokens directly from it.
> 
> Well, that's not what it does-- so you write a class to take what it does
do
> and turn that into what you want it to do:
> 
> [Python code snipped]

Thanks!  I tried something vaguely along this line (without class wrapping
trick though), but it didn't work at all, so I gave up on that avenue.

> If you write a different TokenWrapper class for each string, you're
missing
> the point <wink>.

I need to do it this way, because I need to type in arbitrary string and
get the
resulting list (actually, that list would be fed to another routine to
produce the
desired output, but that is another story entirely), so I can then
cut-n-paste
the output to another window.  I can't just prepare a file and feed the
tokenize
routine; I need to do it interactively to see the result and decide on it.

For you in the know, it is for coding the COBOL's MOVE xxx TO yyy
statement...I know COBOL sucks, but I need to code all those error
and other messages at my work, hence this headache.

Cheers!  (albeit with lots of aspirins ;)

The Blue Wizard