Backing up Mailman list installations

J C Lawrence

29 Nov 2001 29 Nov '01

5:34 a.m.

A moderately frequent request on these lists has been, "How can I back up my Mailman lists/membership/etc?" I happened to be in the area today touching up the scripts I use. Please feel free to adapt freely.

Note: You'll need to have a reasonably recent version of nmh installed as the scripts rely on the MH tools to build a MIME message:

http://www.mhost.com/nmh/

The scripts: ~/bin/mimemail: --<cut>-- #!/bin/bash #set -x

subject=$1
file=$2
addr=$3

echo "To: ${addr}
From: nobody
Subject: ${subject}

#application/octet-stream [${subject}] ${file}

" | /usr/bin/mh/mhbuild - > /tmp/mimemail.tmp.${$}
/usr/lib/mh/post -verbose -watch /tmp/mimemail.tmp.${$}
rm /tmp/mimemail.tmp.${$}
--<cut>--

~/bin/mailman.backup --<cut>-- #!/bin/bash #set -x

datestr=`date +%Y%m%d`
host=`hostname -f`
file=mailman.lists.${datestr}.tar.gz
filepath=~/backups/${file}
sendto="root"

cd /var/lib/mailman
tar zcf ${filepath} lists
cd ~/backups
mimemail ${file} ${filepath} ${sendto}
#rm ${file}
--<cut>-

Yeah, there are some hard coded paths in there. Sue me. I never said they were pretty, merely that they worked. Brief explanation:

mimemail takes three arguments: the Subject: of the message its going to send, the file it is going to send, and the address it is going to send it to. mimemail depends on the nmh tools to build the MIME message.

mailman.backup takes no arguments. It creates a compressed tarball of ~mailman/lists in the file ~/backups/mailman.lists.YYMMDD.tar.gz

eg: mailman.lists.20011128.tar.gz

And then uses mimemail to send that file to root@localhost (edit/change if you want) before deleting it.

To use drop them in a cronjob something like:

0 6 * * 0 /home/archiver/bin/mailman.backup

A sample execution should look something like:

$ ./mailman.backup -- Posting for All Recipients -- -- Local Recipients -- root: address ok -- Recipient Copies Posted -- Message Processed

The result will be that every time the cronjob runs it will send you (in this case root@host.dom) a message looking something like:

Subject: mailman.lists.20011128.tar.gz From: nobody@kanga.nu Date: Wed, 28 Nov 2001 21:12:55 -0800 To: root@kanga.nu

Where the MIME attachment is the tarball constructed by ~/bin/mailman.backup. Edit the value of sendto in mailman.backup if you want them going somewhere else.

I've added this to FAQ at:

http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.006.htp

-- J C Lawrence
---------(*) Satan, oscillate my metallic sonatas. claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

Show replies by date

Jon Carnes

29 Nov 29 Nov

7:09 a.m.

Cool scripts. I used to do something similar, and then I moved over to using "ncftp" to drop the tarball off on my backup-server. A matter of preference, but I like that better than mailing myself the tarball.

Jon Carnes ----- Original Message ----- From: "J C Lawrence" claw@kanga.nu To: Mailman-Users@python.org Cc: mailman-developers@python.org Sent: Thursday, November 29, 2001 12:34 AM Subject: [Mailman-Users] Backing up Mailman list installations

...

<>

A moderately frequent request on these lists has been, "How can I back up my Mailman lists/membership/etc?" I happened to be in the area today touching up the scripts I use. Please feel free to adapt freely.

Note: You'll need to have a reasonably recent version of nmh installed as the scripts rely on the MH tools to build a MIME message:

http://www.mhost.com/nmh/

The scripts:

~/bin/mimemail: --<cut>-- #!/bin/bash #set -x
subject=$1
file=$2
addr=$3

echo "To: ${addr}
From: nobody
Subject: ${subject}

#application/octet-stream [${subject}] ${file}

" | /usr/bin/mh/mhbuild - > /tmp/mimemail.tmp.${$}
/usr/lib/mh/post -verbose -watch /tmp/mimemail.tmp.${$}
rm /tmp/mimemail.tmp.${$}
--<cut>--
~/bin/mailman.backup --<cut>-- #!/bin/bash #set -x
datestr=`date +%Y%m%d`
host=`hostname -f`
file=mailman.lists.${datestr}.tar.gz
filepath=~/backups/${file}
sendto="root"

cd /var/lib/mailman
tar zcf ${filepath} lists
cd ~/backups
mimemail ${file} ${filepath} ${sendto}
#rm ${file}
--<cut>-
Yeah, there are some hard coded paths in there. Sue me. I never said they were pretty, merely that they worked. Brief explanation:

mimemail takes three arguments: the Subject: of the message its going to send, the file it is going to send, and the address it is going to send it to. mimemail depends on the nmh tools to build the MIME message.

mailman.backup takes no arguments. It creates a compressed tarball of ~mailman/lists in the file ~/backups/mailman.lists.YYMMDD.tar.gz
eg: mailman.lists.20011128.tar.gz
And then uses mimemail to send that file to root@localhost (edit/change if you want) before deleting it.

To use drop them in a cronjob something like:

0 6 * * 0 /home/archiver/bin/mailman.backup

A sample execution should look something like:

$ ./mailman.backup -- Posting for All Recipients -- -- Local Recipients -- root: address ok -- Recipient Copies Posted -- Message Processed

The result will be that every time the cronjob runs it will send you (in this case root@host.dom) a message looking something like:

Subject: mailman.lists.20011128.tar.gz From: nobody@kanga.nu Date: Wed, 28 Nov 2001 21:12:55 -0800 To: root@kanga.nu

Where the MIME attachment is the tarball constructed by ~/bin/mailman.backup. Edit the value of sendto in mailman.backup if you want them going somewhere else.

I've added this to FAQ at:

http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.006.htp

-- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw@kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

Mailman-Users maillist - Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users

Thomas T. Veldhouse

2:08 p.m.

My only problem with this is that it still leaves config.db for each list in a binary format. So, I can not simply place it on a Linux platform and move the list from a FreeBSD platform which was running a different version of Python and a different version of Berkley DB. So -- I need a script to extract the information from these DB files and to be able to reload it again.

Essentially, it would be nice to have a script that was like this:

% mm_bkup <listname> <filename> % mm_restore <listname> <filename>

The <filename> would contain the archives, members, passwords, member settings and all list setting. It would also be platform independent, so it wouldn't matter what version of Python, Berkley DB or the operation system mailman is running on.

I don't know Python (although I do know C++ and Java quite well), and I really don't want to learn it for this singular use. I was hoping somebody had already written such a script.

Thanks in advance,

Tom Veldhouse veldy@veldy.net

----- Original Message ----- From: "J C Lawrence" claw@kanga.nu To: Mailman-Users@python.org Cc: mailman-developers@python.org Sent: Wednesday, November 28, 2001 11:34 PM Subject: [Mailman-Users] Backing up Mailman list installations

...

<>

A moderately frequent request on these lists has been, "How can I back up my Mailman lists/membership/etc?" I happened to be in the area today touching up the scripts I use. Please feel free to adapt freely.

Note: You'll need to have a reasonably recent version of nmh installed as the scripts rely on the MH tools to build a MIME message:

http://www.mhost.com/nmh/

The scripts:

~/bin/mimemail: --<cut>-- #!/bin/bash #set -x
subject=$1
file=$2
addr=$3

echo "To: ${addr}
From: nobody
Subject: ${subject}

#application/octet-stream [${subject}] ${file}

" | /usr/bin/mh/mhbuild - > /tmp/mimemail.tmp.${$}
/usr/lib/mh/post -verbose -watch /tmp/mimemail.tmp.${$}
rm /tmp/mimemail.tmp.${$}
--<cut>--
~/bin/mailman.backup --<cut>-- #!/bin/bash #set -x
datestr=`date +%Y%m%d`
host=`hostname -f`
file=mailman.lists.${datestr}.tar.gz
filepath=~/backups/${file}
sendto="root"

cd /var/lib/mailman
tar zcf ${filepath} lists
cd ~/backups
mimemail ${file} ${filepath} ${sendto}
#rm ${file}
--<cut>-
Yeah, there are some hard coded paths in there. Sue me. I never said they were pretty, merely that they worked. Brief explanation:

mimemail takes three arguments: the Subject: of the message its going to send, the file it is going to send, and the address it is going to send it to. mimemail depends on the nmh tools to build the MIME message.

mailman.backup takes no arguments. It creates a compressed tarball of ~mailman/lists in the file ~/backups/mailman.lists.YYMMDD.tar.gz
eg: mailman.lists.20011128.tar.gz
And then uses mimemail to send that file to root@localhost (edit/change if you want) before deleting it.

To use drop them in a cronjob something like:

0 6 * * 0 /home/archiver/bin/mailman.backup

A sample execution should look something like:

$ ./mailman.backup -- Posting for All Recipients -- -- Local Recipients -- root: address ok -- Recipient Copies Posted -- Message Processed

The result will be that every time the cronjob runs it will send you (in this case root@host.dom) a message looking something like:

Subject: mailman.lists.20011128.tar.gz From: nobody@kanga.nu Date: Wed, 28 Nov 2001 21:12:55 -0800 To: root@kanga.nu

Where the MIME attachment is the tarball constructed by ~/bin/mailman.backup. Edit the value of sendto in mailman.backup if you want them going somewhere else.

I've added this to FAQ at:

http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.006.htp

-- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw@kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

Mailman-Users maillist - Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users

Greg Ward

3:03 p.m.

On 29 November 2001, Thomas T. Veldhouse said:

...

My only problem with this is that it still leaves config.db for each list in a binary format. So, I can not simply place it on a Linux platform and move the list from a FreeBSD platform which was running a different version of Python and a different version of Berkley DB.

Mailman ".db" files, despite the name, are *not* Berkeley DB files. They are Python marshall files. (In Mailman 2.1, they will be called ".pck" and become Python pickle files.)

In any event, I'm pretty sure they are platform independent.

Also, since when are Berkeley DB files platform-dependent?

    Greg

-- Greg Ward - software developer gward@mems-exchange.org MEMS Exchange http://www.mems-exchange.org

Thomas T. Veldhouse

3:28 p.m.

...

Mailman ".db" files, despite the name, are *not* Berkeley DB files. They are Python marshall files. (In Mailman 2.1, they will be called ".pck" and become Python pickle files.)

In any event, I'm pretty sure they are platform independent.

If that is the case, then are these files portable between Python implementations and versions? I had a horrible problem moving a mailman 2.0.2 using Python 1.52 on FreeBSD to Mailman 2.0.3 using Python 1.6 on Linux (2.2) about one year ago. The files could NOT be read by mailman. I was forced to add all the users to the new list by hand and thus reset all the passwords and settings for each user (most unhappy about that). It was also a bit of a pain to get the archives working again so that users could browse them on the web.

...

Also, since when are Berkeley DB files platform-dependent?

I am not an expert with them -- I avoid them in favor of true RDBMS. I was under and assumption that they were in platform specific binary files (though the file layout maybe the same on each platform, byte ordering could make a difference).

Tom Veldhouse veldy@veldy.net

...

    Greg

Greg Ward

3:58 p.m.

On 29 November 2001, Thomas T. Veldhouse said:

...

If that is the case, then are these files portable between Python implementations and versions?

Yes, with the proviso that the format for Python bytecode (which is also stored in marshal files -- you see them as .pyc files) usually changes with each new Python version. But Mailman doesn't marshal bytecode, just lists and dictionaries.

Here's proof -- and, at the same time, a handy method for dumping Mailman config.db files to human-readable form. I wrote this little script, called pprint_marshal:

import sys from marshal import load from pprint import pprint

fn = sys.argv[1] f = open(fn) d = load(f) pprint(d)

Then I transferred the ~mailman/lists tree from our mail server to my development machine. (Both are running Debian Linux on x86 processors, so that's not much of a portability challenge.) I ran the script on a particular config.db with Python 1.5.2, 2.0, 2.1.1, and 2.2b1; it worked just fine on all of them. Eg.

$ python1.5 pprint_marshal mailman/lists/test/config.db $ python2.0 pprint_marshal mailman/lists/test/config.db $ python2.1 pprint_marshal mailman/lists/test/config.db $ python2.2 pprint_marshal mailman/lists/test/config.db

Except for some minor changes in how Python formats strings, I got the same result each time.

Then I copied everything to our old mail server, a Sun running Solaris 2.6. Only Python 1.5 and 1.6 were available there, and again I got the same results.

Oh yeah, we moved all our Mailman lists from that Sun machine (big-endian, Solaris) to an x86 Linux machine (little-endian) a few months ago, and had no problems whatsoever. It never even occurred to me that Mailman might use a non-portable format for config.db! (C'mon, give the Mailman cabal a little more credit...)

...

I had a horrible problem moving a mailman 2.0.2 using Python 1.52 on FreeBSD to Mailman 2.0.3 using Python 1.6 on Linux (2.2) about one year ago. The files could NOT be read by mailman. I was forced to add all the users to the new list by hand and thus reset all the passwords and settings for each user (most unhappy about that). It was also a bit of a pain to get the archives working again so that users could browse them on the web.

That sounds awful -- my sympathies. I have no idea what could have gone wrong, but I'm quite confident that config.db files are portable across architectures and Python versions.

...

I am not an expert with them -- I avoid them in favor of true RDBMS. I was under and assumption that they were in platform specific binary files (though the file layout maybe the same on each platform, byte ordering could make a difference).

Not that this has *anything* to do with Mailman, but: Berkeley DB files have a byte-order, but are platform-independent. Using a little-endian file on a big-endian machine obviously incurs a runtime penalty, and vice-versa. See http://www.sleepycat.com/docs/ref/am_conf/byteorder.html

    Greg

-- Greg Ward - software developer gward@mems-exchange.org MEMS Exchange http://www.mems-exchange.org

barry＠zope.com

4:14 p.m.

...

...
...
...
...
"GW" == Greg Ward gward@mems-exchange.org writes:

GW> Mailman ".db" files, despite the name, are *not* Berkeley DB
GW> files.  They are Python marshall files.  (In Mailman 2.1, they
GW> will be called ".pck" and become Python pickle files.)

GW> In any event, I'm pretty sure they are platform independent.

Correct. They are platform independent.

Note that Python marshals are not guaranteed to be compatible across Python releases, although in practice I don't think we've ever broken this, at least for the data types that Mailman stores. However, this was one of the factors for moving to Python pickles in Mailman 2.1, which /are/ guaranteed to be compatible across Python versions, platforms, endian-ness, etc. etc.

You may still want human readable versions of the database files though, for other reasons. It's not a priority of mine to provide that, but it wouldn't be hard to write, e.g. a pickle <-> XML conversion tool. Contributions are welcome <wink>.

-Barry

Jon Carnes

3:35 p.m.

Feel free to create such a script. It should be fairly easy - if tedious to do:

../bin/config_list will easily dump out the entire list configuration in text format, and it will also easily dump it right back into a list...

../bin/list_members will let you dump out the user lists into groups that recieve regular or digest mail

../bin/add_members will let you add users to a list and select whether they are "digest" or "regular" mail users.

And that gives you everything but the Membership options for each individual user: hide, nomail, ack, and notmetoo

../bin/dumpdb will let you dump out everything! This lets you dump out the config of the list plus all the users and their options. The options are in the section:
'user_options': { 'email address': option_binary, 'email address': option_binary, 'email address': option_binary, ... The email address is followed by a number that corresponds to the options for the user:

1 = nomail (+digest) 2 = notmetoo (+digest) 4 = ack (+digest) 8 = plain 16 = hide (+digest)

9 = nomail + plain 12 = ack + plain 24 = hide + plain 27 = hide + nomail + notmetoo + plain

Using this, you can record the users individual options.

Now here is the *hard* part (okay so it's not that easy...). The only way to get the options back into a database (using available tools) seems to be via the ../bin/withlist command. I'll leave this to someone with more time to figure it out...

Jon

On Thursday 29 November 2001 09:08, Thomas T. Veldhouse wrote:

...

My only problem with this is that it still leaves config.db for each list in a binary format. So, I can not simply place it on a Linux platform and move the list from a FreeBSD platform which was running a different version of Python and a different version of Berkley DB. So -- I need a script to extract the information from these DB files and to be able to reload it again.

Essentially, it would be nice to have a script that was like this:

% mm_bkup <listname> <filename> % mm_restore <listname> <filename>

The <filename> would contain the archives, members, passwords, member settings and all list setting. It would also be platform independent, so it wouldn't matter what version of Python, Berkley DB or the operation system mailman is running on.

I don't know Python (although I do know C++ and Java quite well), and I really don't want to learn it for this singular use. I was hoping somebody had already written such a script.

Thanks in advance,

Tom Veldhouse veldy@veldy.net

8181

Age (days ago)

8181

Last active (days ago)

List overview

Download

7 comments

6 participants

participants (6)

barry＠zope.com
Greg Ward
J C Lawrence
Jon Carnes
Jon Carnes
Thomas T. Veldhouse

Backing up Mailman list installations

J C Lawrence

Jon Carnes

Thomas T. Veldhouse

Greg Ward

Thomas T. Veldhouse

Greg Ward

barry＠zope.com

Jon Carnes

1 = nomail (+digest) 2 = notmetoo (+digest) 4 = ack (+digest) 8 = plain 16 = hide (+digest)

tags

participants (6)