On 29 November 2001, Thomas T. Veldhouse said:
If that is the case, then are these files portable between Python implementations and versions?
Yes, with the proviso that the format for Python bytecode (which is also stored in marshal files -- you see them as .pyc files) usually changes with each new Python version. But Mailman doesn't marshal bytecode, just lists and dictionaries.
Here's proof -- and, at the same time, a handy method for dumping Mailman config.db files to human-readable form. I wrote this little script, called pprint_marshal:
import sys from marshal import load from pprint import pprint
fn = sys.argv[1] f = open(fn) d = load(f) pprint(d)
Then I transferred the ~mailman/lists tree from our mail server to my development machine. (Both are running Debian Linux on x86 processors, so that's not much of a portability challenge.) I ran the script on a particular config.db with Python 1.5.2, 2.0, 2.1.1, and 2.2b1; it worked just fine on all of them. Eg.
$ python1.5 pprint_marshal mailman/lists/test/config.db $ python2.0 pprint_marshal mailman/lists/test/config.db $ python2.1 pprint_marshal mailman/lists/test/config.db $ python2.2 pprint_marshal mailman/lists/test/config.db
Except for some minor changes in how Python formats strings, I got the same result each time.
Then I copied everything to our old mail server, a Sun running Solaris 2.6. Only Python 1.5 and 1.6 were available there, and again I got the same results.
Oh yeah, we moved all our Mailman lists from that Sun machine (big-endian, Solaris) to an x86 Linux machine (little-endian) a few months ago, and had no problems whatsoever. It never even occurred to me that Mailman might use a non-portable format for config.db! (C'mon, give the Mailman cabal a little more credit...)
I had a horrible problem moving a mailman 2.0.2 using Python 1.52 on FreeBSD to Mailman 2.0.3 using Python 1.6 on Linux (2.2) about one year ago. The files could NOT be read by mailman. I was forced to add all the users to the new list by hand and thus reset all the passwords and settings for each user (most unhappy about that). It was also a bit of a pain to get the archives working again so that users could browse them on the web.
That sounds awful -- my sympathies. I have no idea what could have gone wrong, but I'm quite confident that config.db files are portable across architectures and Python versions.
I am not an expert with them -- I avoid them in favor of true RDBMS. I was under and assumption that they were in platform specific binary files (though the file layout maybe the same on each platform, byte ordering could make a difference).
Not that this has *anything* to do with Mailman, but: Berkeley DB files have a byte-order, but are platform-independent. Using a little-endian file on a big-endian machine obviously incurs a runtime penalty, and vice-versa. See http://www.sleepycat.com/docs/ref/am_conf/byteorder.html
Greg-- Greg Ward - software developer gward@mems-exchange.org MEMS Exchange http://www.mems-exchange.org