[Tutor] need a tutorial in tokens ,different types

Danny Yoo dyoo at hkn.eecs.berkeley.edu
Mon Nov 21 06:46:49 CET 2005

Taking python-list out of CC yet again.

Enas, please fix your email client setup.  I really don't want to
threaten, but as a power-hungry dictatorial list admin, I have an
responsibility to help maintain a healthy mailing list.

You been warned twice now not to crosspost.  You continue to crosspost the
same question to several mailing lists, not only to comp.lang.python, but
also to NLTK's developer list.


so I doubt your behavior is accidental now: it looks pretty deliberate to

Enas, please don't push me into the Bad Cop role.  Change your behavior:
you've violating community norms.  If you don't correct yourself, I will
moderate your messsages.  I hate threatening you like this.  Don't make me
follow through with the threat.

On Sun, 20 Nov 2005, Alan Gauld wrote:

> >  could any one suggest me tutorials in different tokenizations and
> > clear describtion of how can i use token type and assign diff
> > attributes to tokens
> What kind of tokens? Are we talking lexical tokens in a parser or
> security tokens or what?

Anyway, I think we are talking about the tokens defined by the NLTK
natural language toolkit.  I'm not sure we can do much better than
recommend the tutorials that come with NLTK:


They go through several examples of assigning attributes to tokens.  If
you has questions about NLTK, again, Enas may be better off asking on an
NLTK-specific mailing list.  There's one specifically set up for


> > also good tutorials in diff data types in python
> The official documentaton explains most of it.
> You can try the Raw Materials topic in my tutorial if you like.

Those tutorials are here:


They should give enough grounding to understand how NLTK works better.
Again, if you has any questions relating to learning how to program, we're
glad to be of help.

But take some time to learn how to use your email client.  Also, if people
have not recommend it, read through:


so that you have a sense of what kind of community norms are expected of
you when you ask for help.  Violating them impovershes the entire
community, which is why I take this so seriously.

More information about the Tutor mailing list