[spambayes-dev] RE: [Spambayes-checkins] spambayes/spambayes Options.py, 1.90, 1.91 UserInterface.py, 1.35, 1.36 classifier.py, 1.11, 1.12

[Tim Peters]

> *** UserInterface.py	11 Dec 2003 18:44:23 -0000	1.35
> --- UserInterface.py	16 Dec 2003 02:03:31 -0000	1.36
> ***************
> *** 306,309 ****
> --- 306,313 ----
>               for tok in tokens:
>                   clues.append((tok, None))
> +             # Need to regenerate the tokens (is there a way to
> +             # 'rewind' or copy a generator?  Would that be
> +             # more effecient?
> +             tokens = tokenizer.tokenize(message)
>               probability = self.classifier.spamprob(tokens)
>               cluesTable = self._fillCluesTable(clues)

Change the first line of the function to:

    tokens = list(tokenizer.tokenize(message))

There's no need to tokenize again, then.  The construction of clues can be
the one-liner:

    clues = [(tok, None) for tok in tokens]