Bugs item #647102, was opened at 2002-12-02 11:08 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_id=103

Category: Pipermail Group: 2.0.x Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: MemoryError when generating indexes

Initial Comment: Vernier Networks has a relatively small/medium mailing list archive (only ~238megs) that can not be regenerated in 512megs of memory. (Once the machine needs a reboot, this will increase to 1gig.) It dies with a MemoryError supplied below: Updating index files for archive [2002-May] Traceback (most recent call last): File "/home/mailman/bin/arch", line 129, in ? main() File "/home/mailman/bin/arch", line 119, in main archiver.close() File "/home/mailman/Mailman/Archiver/pipermail.py", line 304, in close self.update_dirty_archives() File "/home/mailman/Mailman/Archiver/HyperArch.py", line 934, in update_dirty_archives self.__super_update_dirty_archives() File "/home/mailman/Mailman/Archiver/pipermail.py", line 513, in update_dirty_archives self.update_archive(i) File "/home/mailman/Mailman/Archiver/HyperArch.py", line 937, in update_archive self.__super_update_archive(archive) File "/home/mailman/Mailman/Archiver/pipermail.py", line 417, in update_archive self.__set_parameters(archive) File "/home/mailman/Mailman/Archiver/pipermail.py", line 332, in __set_parameters firstdate = self.database.firstdate(archive) File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 207, in firstdate self.__openIndices(archive) File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 238, in __openIndices self.__closeIndices() File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 259, in __closeIndices index.close() File "/home/mailman/Mailman/Archiver/HyperDatabase.py", line 186, in close fp.write(marshal.dumps(self.dict)) MemoryError

and I also have an IOError on a lock file.

I will be looking at it, but since I do not know Pipermail, I won't progress very far.

Comment By: John-Mark Gurney (jmg) Date: 2002-12-02 15:56

Message: Logged In: YES user_id=27410

Found that in pipermail.T.processUnixMailbox, it will read in ALL articles w/ their bodies attached. I have fixed pipermail.Article to use an instance of a new class I wrote for body. The class will emulate a simple sequence and an interator object and just store the offsets to the lines, and reread the lines as necessary. I'm waiting the nightly run to see how things will go.

The patches currently work for small archives.

You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_id=103