[Mailman-Users] problem in archiving the large mbox file
brad at shub-internet.org
Tue Jul 17 23:45:33 CEST 2007
On 7/18/07, alpesh gajbe wrote:
> I am not sure whether this mail belongs to the users or developers list so
> I am sending to both. Apologies.
This sounds to me more like a question for -users.
> I have a large 1.2 GB mbox file which i need to archive, the messages in
> mbox typically have jpeg image attachments
> whose approximate image sizes is 4 KB.
4KB JPEGs? That's pretty small for a JPEG. You have over 300,000 of
these messages per mailbox file?
> when i run arch script in /usr/lib/mailman/bin/ on my ubuntu 7.04 I get the
> following error
> *File "/usr/lib/python2.5/os.py", line 172, in makedirs mkdir(name, mode)
> OSError: [Errno 31] Too many links:
That sounds like a directory problem, and not a file problem. With
as large a mailbox file as you're talking about, and as many messages
as you're talking about, you probably need to break your archives
more frequently than once a month. Try breaking them weekly instead.
Alternatively, try changing your underlying filesystem to one that
supports large numbers of files in a single directory, and preferably
does so with an internal hashed directory/inode structure (e.g., XFS).
Also keep in mind that you'll want to make sure that your OS is built
to support large files (files over 2GB). Many Linux distributions
are not built out-of-the-box to support large files.
Finally, the version of Python that is recommended for use with the
latest release version of Mailman, is Python 2.4.x for Mailman 2.1.9,
for whatever the most recent version of Python 2.4.x (currently
2.4.3, I believe). More recent versions of Python may or may not
work with Mailman 2.1.9, and almost certainly will not work correctly
with earlier versions of Mailman.
> *My basic objective is to archive 250GB of mails every month for two years .
> The size of which would exceed 7 Tera Bytes approx. Would this be a feasible
> option using mailman.(out of curiosity !!)
That should be possible. We have 4GB worth of archives for
python-list at python.org going back to 1999, and I'm pretty sure those
messages are text-only or text+code fragments.
> What has been the largest mail archive volume anyone has ever deployed into
> mailman ? (out of curiosity !!)
We've got some information in the FAQ Wizard about large mailing
lists servers with regards to numbers of subscribers or numbers of
messages, but I don't know that anyone has tried to gather any
specific information with regards to large sizes of archives.
The largest archives I am personally aware of are the ones we have
for python-list, but I'd love to hear any information that anyone
else has about any others.
Brad Knowles <brad at shub-internet.org>, Consultant & Author
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
Slides from Invited Talks: <http://tinyurl.com/tj6q4>
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
More information about the Mailman-Users