[Bug 1462754] [NEW] qrunner crashes on invalid unicode sequence

[Thijs Kinkhorst]

Public bug reported:

When a message contains an invalud unicode sequence in its header,
qrunner flat out crashes on that:

May 17 15:32:20 2015 (981) Uncaught runner exception: 'utf8' codec can't decode byte
0xe9 in position 18: invalid continuation byte
May 17 15:32:20 2015 (981) Traceback (most recent call last):
  File "/var/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
    self._onefile(msg, msgdata)
  File "/var/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/var/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose
    more = self._dopipeline(mlist, msg, msgdata, pipeline)
  File "/var/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline
    sys.modules[modname].process(mlist, msg, msgdata)
  File "/var/lib/mailman/Mailman/Handlers/CookHeaders.py", line 239, in process
    i18ndesc = uheader(mlist, mlist.description, 'List-Id', maxlinelen=998)
  File "/var/lib/mailman/Mailman/Handlers/CookHeaders.py", line 65, in uheader
    return Header(s, charset, maxlinelen, header_name, continuation_ws)
  File "/usr/lib/python2.7/email/header.py", line 183, in __init__
    self.append(s, charset, errors)
  File "/usr/lib/python2.7/email/header.py", line 267, in append
    ustr = unicode(s, incodec, errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 18: invalid
continuation byte

May 17 15:32:20 2015 (981) SHUNTING:
1431869540.389822+156779307d54473d0eb732994bb67eee95733285


A solution for this specific case is to have Mailman/Handlers/CookHeaders.py pass the erorrs='replace' parameter.

I would say that this is actually a bug in python-email, since I think it doesn't make sense to set errors to "strict" rather than something like "replace" when the intention is to parse stuff so free-formed, under-specd
and user-controlled as email. Nonetheless, Mailman already sets errors='replace' in some places so it might aswell add it here.

** Affects: mailman
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1462754

Title:
  qrunner crashes on invalid unicode sequence

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1462754/+subscriptions