[Mailman-Developers] Mix of 8-bit ascii and unicode in Archive marshals

[Mads Kiilerich]

Hi,

I just upgraded from 2.1.5 to 2.1.7, using a Fedora beta rpm. 
http://sourceforge.net/tracker/index.php?func=detail&aid=1405790&group_id=103&atid=300103 
has been applied.

Now I see crashes in Archiver like shown below. My problem is similar to 
but different from 
http://archives.free.net.ph/message/20060116.022354.167a1fae.en.html

It turns out to be because both 8-bit ascii and unicode strings appears 
as first element in key tuples in the 2006-February-author marshal. When 
sort tries to compare these unrelated encodings python don't know what 
to do. Neither do I...

How should the problem be solved? Should keys be stored as 8-bit ascii 
or as unicode?

If anybody can give a hint I'll try to come up with a solution ;-)

/Mads


Feb 18 17:57:44 2006 (29051) Uncaught runner exception: 'ascii' codec 
can't decode byte 0xe5 in position 1: ordinal not in range(128)
Feb 18 17:57:44 2006 (29051) Traceback (most recent call last):
   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
     self._onefile(msg, msgdata)
   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
     keepqueued = self._dispose(mlist, msg, msgdata)
   File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose
     mlist.ArchiveMail(msg)
   File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 217, in 
ArchiveMail
     h.close()
   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 327, in close
     self.update_dirty_archives()
   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 543, in 
update_dirty_archives
     self.update_archive(i)
   File "/usr/lib/mailman/Mailman/Archiver/HyperArch.py", line 1125, in 
update_archive
     self.__super_update_archive(archive)
   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 447, in 
update_archive
     self._update_simple_index(hdr, archive, arcdir)
   File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 460, in 
_update_simple_index
     msgid = self.database.first(archive, hdr)
   File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 296, 
in first
     key, msgid = index.first()
   File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 113, 
in first
     self.__sort() # guarantee that the list is sorted
   File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 73, 
in __sort
     self.sorted.sort()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 1: 
ordinal not in range(128)

Feb 18 17:57:44 2006 (29051) SHUNTING: 
1140281864.286916+117720f8b561f5c15f1f799c2dd1fce5881078cb