[Spambayes-checkins] spambayes/scripts sb_imapfilter.py,1.51,1.52
Tony Meyer
anadelonbrin at users.sourceforge.net
Thu Jan 13 22:47:29 CET 2005
Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv9538/scripts
Modified Files:
sb_imapfilter.py
Log Message:
Try and work around the reasonably common problems with imaplib and OS X. A MemoryError
gets generated when a malloc fails trying to do a read on the socket that is too
large. As a workaround, only read in manageable chunks and then join all the data
together.
I hope this will work, but I can't be sure. Maybe the malloc fail will simply move
to the join call. I don't think this will make any noticeable difference in speed,
but if it does, then MAXIMUM_SAFE_READ can be increased to something larger.
If anyone out there uses sb_imapfilter from CVS with a OS X, it would be great if
you could try and test this out before 1.1 and 1.0.2 are released :)
Index: sb_imapfilter.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_imapfilter.py,v
retrieving revision 1.51
retrieving revision 1.52
diff -C2 -d -r1.51 -r1.52
*** sb_imapfilter.py 4 Jan 2005 00:19:34 -0000 1.51
--- sb_imapfilter.py 13 Jan 2005 21:47:25 -0000 1.52
***************
*** 43,55 ****
Warnings:
! o This is beta software. While we are reasonably confident that it
! should work as designed, there is a lot of difference between IMAP
! servers, and we can only test with a few. We never delete mail,
! unless you use the -e/purge option, but we do mark a lot as deleted,
! and your mail client might remove that for you. We try to only mark
! as deleted once the moved/altered message is correctly saved, but
! things might go wrong. We *strongly* recommend that you try this
! script out on mail that you can recover from somewhere else, at least
! at first.
"""
--- 43,52 ----
Warnings:
! o We never delete mail, unless you use the -e/purge option, but we do
! mark a lot as deleted, and your mail client might remove that for
! you. We try to only mark as deleted once the moved/altered message
! is correctly saved, but things might go wrong. We *strongly*
! recommend that you try this script out on mail that you can recover
! from somewhere else, at least at first.
"""
***************
*** 65,69 ****
"""
! # This module is part of the SpamBayes project, which is Copyright 2002-4
# The Python Software Foundation and is covered by the Python Software
# Foundation license.
--- 62,66 ----
"""
! # This module is part of the SpamBayes project, which is Copyright 2002-5
# The Python Software Foundation and is covered by the Python Software
# Foundation license.
***************
*** 162,165 ****
--- 159,170 ----
self.current_folder = None
+ # We override the base read so that we only read a certain amount
+ # of data at a time. OS X and Python has problems with getting
+ # large amounts of memory at a time, so maybe this will be a way we
+ # can work around that (I don't know, and don't have a mac to test,
+ # but we need to try something).
+ self._read = self.read
+ self.read = self.safe_read
+
def readline_timeout(self):
"""Read line from remote, possibly timing out."""
***************
*** 397,400 ****
--- 402,419 ----
return data
+ # Maximum amount of data that will be read at any one time.
+ MAXIMUM_SAFE_READ = 4096
+ def safe_read(self, size):
+ """Read data from remote, but in manageable sizes."""
+ data = []
+ while size > 0:
+ if size < self.MAXIMUM_SAFE_READ:
+ to_collect = size
+ else:
+ to_collect = self.MAXIMUM_SAFE_READ
+ data.append(self._read(to_collect))
+ size -= self.MAXIMUM_SAFE_READ
+ return "".join(data)
+
class IMAPMessage(message.SBHeaderMessage):
More information about the Spambayes-checkins
mailing list