[spambayes-bugs] [ spambayes-Bugs-1175439 ] UnicodeEncodeError raised for bogus Content-Type header

[SourceForge.net]

Bugs item #1175439, was opened at 2005-04-03 05:09
Message generated for change (Comment added) made by anadelonbrin
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=498103&aid=1175439&group_id=61702

Category: None
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Jim Correia (correia)
>Assigned to: Tony Meyer (anadelonbrin)
Summary: UnicodeEncodeError raised for bogus Content-Type header

Initial Comment:
When using sb_mboxtrain.py or sb_filter.py on the following 
message (which is a contrived sample based on an actual message 
in my mail spool from March) a UnicodeEncodeError is raised due to 
the bogus Content-Type header.

I'm using the sb 1.0.4 release.

Exception Backtrace:

$ ./scripts/sb_filter.py ~/Desktop/msg.txt 
Traceback (most recent call last):
  File "./scripts/sb_filter.py", line 257, in ?
    main()
  File "./scripts/sb_filter.py", line 248, in main
    action(msg)
  File "./scripts/sb_filter.py", line 180, in filter
    return self.h.filter(msg)
  File "/usr/local/lib/python2.4/site-packages/spambayes/
hammie.py", line 109, in filter
    prob, clues = self._scoremsg(msg, True)
  File "/usr/local/lib/python2.4/site-packages/spambayes/
hammie.py", line 38, in _scoremsg
    return self.bayes.spamprob(tokenize(msg), evidence)
  File "/usr/local/lib/python2.4/site-packages/spambayes/
classifier.py", line 190, in chi2_spamprob
    clues = self._getclues(wordstream)
  File "/usr/local/lib/python2.4/site-packages/spambayes/
classifier.py", line 492, in _getclues
    for word in Set(wordstream):
  File "/usr/local/lib/python2.4/sets.py", line 429, in __init__
    self._update(iterable)
  File "/usr/local/lib/python2.4/sets.py", line 383, in _update
    for element in iterable:
  File "/usr/local/lib/python2.4/site-packages/spambayes/
tokenizer.py", line 1224, in tokenize
    for tok in self.tokenize_headers(msg):
  File "/usr/local/lib/python2.4/site-packages/spambayes/
tokenizer.py", line 1235, in tokenize_headers
    for w in crack_content_xyz(x):
  File "/usr/local/lib/python2.4/site-packages/spambayes/
tokenizer.py", line 823, in crack_content_xyz
    for x in msg.get_charsets(None):
  File "/usr/local/lib/python2.4/email/Message.py", line 804, in 
get_charsets
    return [part.get_content_charset(failobj) for part in self.walk()]
  File "/usr/local/lib/python2.4/email/Message.py", line 784, in 
get_content_charset
    charset = unicode(charset[2], pcharset).encode('us-ascii')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in 
position 0: ordinal not in range(128)

===

Sample message:

>From ???@??? Sat Mar  5 09:56:25 2005 -0500
Content-Type: text/plain; charset*=ISO-8859
-1''%E6N%C0%00%00%00%00%15;
	format=flowed
To: user at example.com
From: User <user at example.com>
Subject: example
Date: Sat, 5 Mar 2005 09:55:34 -0500

The contents of the body don't matter, just the bogus Content-Type 
header.


----------------------------------------------------------------------

>Comment By: Tony Meyer (anadelonbrin)
Date: 2005-04-04 11:31

Message:
Logged In: YES 
user_id=552329

Thanks.  Fixed in tokenizer.py 1.35.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=498103&aid=1175439&group_id=61702