[Spambayes-checkins] spambayes/scripts sb_imapfilter.py,1.51,1.52

Tony Meyer anadelonbrin at users.sourceforge.net
Thu Jan 13 22:47:29 CET 2005


Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv9538/scripts

Modified Files:
	sb_imapfilter.py 
Log Message:
Try and work around the reasonably common problems with imaplib and OS X.  A MemoryError
 gets generated when a malloc fails trying to do a read on the socket that is too
 large.  As a workaround, only read in manageable chunks and then join all the data
 together.

I hope this will work, but I can't be sure.  Maybe the malloc fail will simply move
 to the join call.  I don't think this will make any noticeable difference in speed,
 but if it does, then MAXIMUM_SAFE_READ can be increased to something larger.

If anyone out there uses sb_imapfilter from CVS with a OS X, it would be great if
 you could try and test this out before 1.1 and 1.0.2 are released :)

Index: sb_imapfilter.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_imapfilter.py,v
retrieving revision 1.51
retrieving revision 1.52
diff -C2 -d -r1.51 -r1.52
*** sb_imapfilter.py	4 Jan 2005 00:19:34 -0000	1.51
--- sb_imapfilter.py	13 Jan 2005 21:47:25 -0000	1.52
***************
*** 43,55 ****
  
  Warnings:
!     o This is beta software.  While we are reasonably confident that it
!       should work as designed, there is a lot of difference between IMAP
!       servers, and we can only test with a few.  We never delete mail,
!       unless you use the -e/purge option, but we do mark a lot as deleted,
!       and your mail client might remove that for you.  We try to only mark
!       as deleted once the moved/altered message is correctly saved, but
!       things might go wrong.  We *strongly* recommend that you try this
!       script out on mail that you can recover from somewhere else, at least
!       at first.
  """
  
--- 43,52 ----
  
  Warnings:
!     o We never delete mail, unless you use the -e/purge option, but we do
!       mark a lot as deleted, and your mail client might remove that for
!       you.  We try to only mark as deleted once the moved/altered message
!       is correctly saved, but things might go wrong.  We *strongly*
!       recommend that you try this script out on mail that you can recover
!       from somewhere else, at least at first.
  """
  
***************
*** 65,69 ****
  """
  
! # This module is part of the SpamBayes project, which is Copyright 2002-4
  # The Python Software Foundation and is covered by the Python Software
  # Foundation license.
--- 62,66 ----
  """
  
! # This module is part of the SpamBayes project, which is Copyright 2002-5
  # The Python Software Foundation and is covered by the Python Software
  # Foundation license.
***************
*** 162,165 ****
--- 159,170 ----
          self.current_folder = None
  
+         # We override the base read so that we only read a certain amount
+         # of data at a time.  OS X and Python has problems with getting 
+         # large amounts of memory at a time, so maybe this will be a way we
+         # can work around that (I don't know, and don't have a mac to test,
+         # but we need to try something).
+         self._read = self.read
+         self.read = self.safe_read
+ 
      def readline_timeout(self):
          """Read line from remote, possibly timing out."""
***************
*** 397,400 ****
--- 402,419 ----
          return data
  
+     # Maximum amount of data that will be read at any one time.
+     MAXIMUM_SAFE_READ = 4096
+     def safe_read(self, size):
+         """Read data from remote, but in manageable sizes."""
+         data = []
+         while size > 0:
+             if size < self.MAXIMUM_SAFE_READ:
+                 to_collect = size
+             else:
+                 to_collect = self.MAXIMUM_SAFE_READ
+             data.append(self._read(to_collect))
+             size -= self.MAXIMUM_SAFE_READ
+         return "".join(data)
+ 
  
  class IMAPMessage(message.SBHeaderMessage):



More information about the Spambayes-checkins mailing list