After migrating list to new server, config.pck database gets truncated by web UI

(Note PS at bottom!)
Hi. I'm prepping to migrate a bunch of lists (one at a time, due to huge number of lists and huge size of archives) from one server to another, and I've hit a snag with the first list I'm trying. After migrating the list (as described below), I can go to the lists admindb page on the new server and get the list of pending requests that was on the old server, but immediately the request database gets truncated. (It stays a valid pickle file, it just gets all the requests emptied out of it, so the file itself is much shorter but non-zero length.)
This happens *when I load* the admindb page, not when I submit the form. That seems really weird to me, since I'd expect a page load would only read the pickle, not write it.
The old server is running Debian and Mailman 2.1.13 (from the Debian package). The new server is running Ubuntu and Mailman 2.1.16 (from the Ubuntu Trusty package; we need to run Trusty for now for complex and uninteresting reasons; I'd rather run 2.1.18, and may look into running that on Trusty once I get the basic migration issues resolved).
Relevant UIDs and GIDs (www-data:www-data and list:list) are the same on both systems.
Short version: I rsync -aSHov /var/lib/mailman/lists/$listname/ new-server:/var/lib/mailman/lists/$listname and similarly copy the public and private archives (preserving symlinks as needed). check_perms on both systems reveals similar errors which look cosmetic (things like rotated logs, temporary directories where I've copied things, and the like), but I haven't yet let it run to completion because of the volume of our archives. Then I change host_name via the web interface and m.web_page_url interactively with withlist (using fix_url seems not to work when changing http: to https:) and m.Save().
One *possibly* relevant detail is that the new host doesn't currently have a valid certificate. (It's using the old host's cert, and I manually allow the exception in my web browser for testing.) But for Mailman 2, the only http{,s} traffic should be sent from my browser, right?
This kind of has the feel of a permissions problem, but clearly the CGI scripts can read from and write to the request.pck database. (And changes to the list config data in config.pck seem to be working normally.) As I said, check_perms hasn't run to completion yet because it's plowing through the (already pre-rsync'ed) archives, but it got through the things in /var/lib/mailman/lists and didn't find anything wrong with this list.
There's nothing interesting in the Mailman logs (which Debian/Ubuntu put in /var/log/mailman), and the only thing in the Apache error logs is a warning that the cert it has configured doesn't match its hostname.
Anybody have any ideas?
Jay
PS -- I composed this all last night. Today, the behavior has changed: This morning, a new message was received by the list (forwarded from the old list server to the new list server, and added to request.pck on the new server by the new Mailman installation). Now, when I load the admindb page, the old requests (which were in the request.pck copied from the old server) are all immediately thrown away (although displayed in the admindb form) but the new request which came in this morning remains. So it kind of looks like something about the old requests causes the list to think they're invalid and discard them when it loads them. I initially saw this behavior with "require_explicit_destination" on and "acceptable_aliases" empty, but turning off "require_explicit_destination" and putting just the local part of the list address in "acceptable_aliases" doesn't make any difference.

Naturally, I was fighting with this for hours, and answered my own question a few minutes after sending my query. It looks like I need to copy the relevant heldmsg-$LISTNAME-* files from /var/lib/mailman/data along with the request.pck database. Apparently, when the CGI script queries the request database, it gets and displays all the entries there, but also notices which entries don't have corresponding heldmsg files and deletes them.
Oh, well, maybe this question being in the list archives will mean the next person will have more success than I did googling to troubleshoot.
With those files copied as well, everything's looking OK.
Jay

[changed the subject] Hi,
I realize this wasn't part of your question, but-
On 12/29/2016 8:16 AM, Jay Sekora wrote:
The old server is running Debian and Mailman 2.1.13 (from the Debian package). The new server is running Ubuntu and Mailman 2.1.16 (from the Ubuntu Trusty package; we need to run Trusty for now for complex and uninteresting reasons; I'd rather run 2.1.18, and may look into running that on Trusty once I get the basic migration issues resolved).
Install 2.1.23 from source? I seems like most linux packages are older versions (and the list archive has many questions about them). I assume there are reasons, but I'm not that deep into linux to know what they are.
Later,
z! (in the land of BSD)

On 12/29/2016 11:51 AM, Carl Zwanzig wrote:
[changed the subject] Hi,
I realize this wasn't part of your question, but-
On 12/29/2016 8:16 AM, Jay Sekora wrote:
The old server is running Debian and Mailman 2.1.13 (from the Debian package). The new server is running Ubuntu and Mailman 2.1.16 (from the Ubuntu Trusty package; we need to run Trusty for now for complex and uninteresting reasons; I'd rather run 2.1.18, and may look into running that on Trusty once I get the basic migration issues resolved).
Install 2.1.23 from source? I seems like most linux packages are older versions (and the list archive has many questions about them). I assume there are reasons, but I'm not that deep into linux to know what they are.
Sure, that's a good way to go if freshness is more important than consistent packaging. The advantage of sticking with distro packages is that (1) somebody else is on top of the security updates (and with Debian in particular, a lot of attention is often devoted to automating upgrade-related maintenance) and (2) there's coordination with other packages on the system (so for instance if a security upgrade to Python requires a minor tweak to Mailman I'll get both at the same time), and log-rotation and cron jobs and the like are handled in a consistent way across packages, and you can expect the Debian/Ubuntu Mailman package to work smoothly with the Debian/Ubuntu SpamAssassin and Exim packages, for instance.
In this particular case, I have seen reports that the Xenial (16.04) Mailman packages install and run cleanly on Trusty (14.04), and another approach would be to rebuild the package with newer source, but of course those impact the above advantages to some degree (as does just installing from upstream source).
Jay
PS -- One other advantage to installing from distro packages, especially on a widely deployed platform like Ubuntu, Fedora, or RHEL, is that a large fraction of the other people out on the web are doing the same thing, so if you run into something peculiar, some other package-using newbie on the net may have posted about the exact same problem. Obviously on a Mailman-specific list like this (or a Drupal-specific list or a ClamAV-specific list, or whatever) the ratio of people installing from source is likely to be higher, though, and of course that's where to look for the highest-quality answers.

On 12/29/2016 09:20 AM, Jay Sekora wrote:
PS -- One other advantage to installing from distro packages, especially on a widely deployed platform like Ubuntu, Fedora, or RHEL, is that a large fraction of the other people out on the web are doing the same thing, so if you run into something peculiar, some other package-using newbie on the net may have posted about the exact same problem. Obviously on a Mailman-specific list like this (or a Drupal-specific list or a ClamAV-specific list, or whatever) the ratio of people installing from source is likely to be higher, though, and of course that's where to look for the highest-quality answers.
But don't forget https://wiki.list.org/x/12812344.

On 12/29/2016 08:51 AM, Carl Zwanzig wrote:
I realize this wasn't part of your question, but-
On 12/29/2016 8:16 AM, Jay Sekora wrote:
The old server is running Debian and Mailman 2.1.13 (from the Debian package). The new server is running Ubuntu and Mailman 2.1.16 (from the Ubuntu Trusty package; we need to run Trusty for now for complex and uninteresting reasons; I'd rather run 2.1.18, and may look into running that on Trusty once I get the basic migration issues resolved).
Install 2.1.23 from source? I seems like most linux packages are older versions (and the list archive has many questions about them). I assume there are reasons, but I'm not that deep into linux to know what they are.
I agree with Carl that installing Mailman from source is almost always the best option (of course *I* would), but I understand the appeal of packages and as I have posted before, there is a 2.1.23 Ubuntu package at http://packages.ubuntu.com/zesty/mailman and Debian at https://packages.debian.org/stretch/mailman.
участники (3)
-
Carl Zwanzig
-
Jay Sekora
-
Mark Sapiro