[Mailman-Developers] [ mailman-Bugs-647102 ] MemoryError when generating indexes

[noreply at sourceforge.net]

Bugs item #647102, was opened at 2002-12-02 11:08
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_id=103

Category: Pipermail
Group: 2.0.x
Status: Open
Resolution: None
Priority: 5
Submitted By: John-Mark Gurney (jmg)
Assigned to: Nobody/Anonymous (nobody)
Summary: MemoryError when generating indexes

Initial Comment:
Vernier Networks has a relatively small/medium mailing
list archive (only ~238megs) that can not be
regenerated in 512megs of memory.  (Once the machine
needs a reboot, this will increase to 1gig.)   It dies
with a MemoryError supplied below:
Updating index files for archive [2002-May]
Traceback (most recent call last):
  File "/home/mailman/bin/arch", line 129, in ?
    main()
  File "/home/mailman/bin/arch", line 119, in main
    archiver.close()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 304, in close
    self.update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 934, in update_dirty_archives
    self.__super_update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 513, in update_dirty_archives
    self.update_archive(i)
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 937, in update_archive
    self.__super_update_archive(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 417, in update_archive
    self.__set_parameters(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 332, in __set_parameters
    firstdate = self.database.firstdate(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
207, in firstdate
    self.__openIndices(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
238, in __openIndices
    self.__closeIndices()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
259, in __closeIndices
    index.close() 
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
186, in close
    fp.write(marshal.dumps(self.dict))
MemoryError

and I also have an IOError on a lock file.

I will be looking at it, but since I do not know
Pipermail, I won't progress very far.

----------------------------------------------------------------------

>Comment By: John-Mark Gurney (jmg)
Date: 2002-12-02 15:56

Message:
Logged In: YES 
user_id=27410

Found that in pipermail.T.processUnixMailbox, it will read
in ALL articles w/ their bodies attached.  I have fixed
pipermail.Article to use an instance of a new class I wrote
for body.  The class will emulate a simple sequence and an
interator object and just store the offsets to the lines,
and reread the lines as necessary.  I'm waiting the nightly
run to see how things will go.

The patches currently work for small archives.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=100103&aid=647102&group_id=103