<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1><pre><font size=2 face="Courier New"><span
style='font-size:10.0pt'>Aaran,<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Thanks for your input. Your examples gave me other alternatives for what I wanted to do and it seems to work.<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Thanks all for your help.<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>On Sep 16, 2:48 pm, "Karl Kobata" <<a
href="http://mail.python.org/mailman/listinfo/python-list">karl.kob... at syncira.com</a>> wrote:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Hi Fredrik,<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> This is exactly what I need. Thank you.<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> I would like to do one additional function. I am not using the tokenizer to<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> parse python code. It happens to work very well for my application.<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> However, I would like either or both of the following variance:<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> 1) I would like to add 2 other characters as comment designation<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> 2) write a module that can readline, modify the line as required, and<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> finally, this module can be used as the argument for the tokenizer.<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Def modifyLine( fileHandle ):<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> # readline and modify this string if required<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> ...<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> For token in tokenize.generate_tokens( modifyLine( myFileHandle ) ):<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Print token<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Anxiously looking forward to your thoughts.<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> karl<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> -----Original Message-----<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> From: python-list-bounces+kkobata=<a
href="http://mail.python.org/mailman/listinfo/python-list">syncira.... at python.org</a><o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> [mailto:python-list-bounces+kkobata=<a
href="http://mail.python.org/mailman/listinfo/python-list">syncira.... at python.org</a>] On Behalf Of<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Fredrik Lundh<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Sent: Monday, September 15, 2008 2:04 PM<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> To: <a
href="http://mail.python.org/mailman/listinfo/python-list">python-l... at python.org</a><o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Subject: Re: ka-ping yee tokenizer.py<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> Karl Kobata wrote:<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> > I have enjoyed using ka-ping yee's tokenizer.py. I would like to<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> > replace the readline parameter input with my own and pass a list of<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> > strings to the tokenizer. I understand it must be a callable object and<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> > iteratable but it is obvious with errors I am getting, that this is not<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> > the only functions required.<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> not sure I can decipher your detailed requirements, but to use Python's<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> standard "tokenize" module (written by ping) on a list, you can simple<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> do as follows:<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> import tokenize<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> program = [ ... program given as list ... ]<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> for token in tokenize.generate_tokens(iter(program).next):<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> print token<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> another approach is to turn the list back into a string, and wrap that<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> in a StringIO object:<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> import tokenize<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> import StringIO<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> program = [ ... program given as list ... ]<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> program_buffer = StringIO.StringIO("".join(program))<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> for token in tokenize.generate_tokens(program_buffer.readline):<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> print token<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> </F><o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'> --<a
href="http://mail.python.org/mailman/listinfo/python-list">http://mail.python.org/mailman/listinfo/python-list</a><o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>><i><span
style='font-style:italic'><o:p> </o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>This is an interesting construction:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>>>><i><span
style='font-style:italic'> a= [ 'a', 'b', 'c' ]<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>>>><i><span
style='font-style:italic'> def moditer( mod, nextfun ):<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>... while 1:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>... yield mod( nextfun( ) )<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>...<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>>>><i><span
style='font-style:italic'> list( moditer( ord, iter( a ).next ) )<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>[97, 98, 99]<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Here's my point:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>>>><i><span
style='font-style:italic'> a= [ 'print a', 'print b', 'print c' ]<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>>>><i><span
style='font-style:italic'> tokenize.generate_tokens( iter( a ).next )<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><generator object at 0x009FF440><o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>>>><i><span
style='font-style:italic'> tokenize.generate_tokens( moditer( lambda s: s+ '#', iter( a ).next ).next )<o:p></o:p></span></i></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>It adds a '#' to the end of every line, then tokenizes.<o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<!--endarticle--></div>
</body>
</html>