----- Original Message ----- From: "Mark Sapiro" <msapiro@value.net> To: "John Fleming" <john@wa9als.com>; <mailman-users@python.org> Sent: Friday, March 02, 2007 10:17 PM Subject: Re: [Mailman-Users] Urgent newbie help - "Hit a bug" page

Mark Sapiro wrote:

...

Try the following shell script

#! /bin/sh for list in /path/to/bin/list_lists --bare do echo ---------------------------------- echo list: $list /path/to/bin/dumpdb /path/to/lists/$list/config.pck | grep advertised echo ----------------------------------- done

BTW, this will only identify the problem list. If the only thing wrong with it is the missing advertised attribute, you can fix it with withlist.

bin/withlist -l listname ... ...

...

...

...

m.advertised = 1 m.Save()

...

The last input is control-D (eof). If the list should not be advertised, then set m.advertised = 0.

1. I must be doing something wrong - Running that script only gives me:

Luke:/var/lib/mailman# ./johntest <----- the script sh-2.05b#

2. There are other atribute errors in the log - I see name and language, to name 2. (I haven't changed anything, so why are the atributes missing anyway?)

3. Luke:/var/lib/mailman/bin# ./withlist -l sked Loading list sked (locked) The variable `m' is the sked MailList instance

...

...

Unlocking (but not saving) list: sked Finalizing Luke:/var/lib/mailman/bin#

----- Original Message ----- From: "Mark Sapiro" <msapiro@value.net> To: "John Fleming" <john@wa9als.com>; <mailman-users@python.org> Sent: Friday, March 02, 2007 11:11 PM Subject: Re: [Mailman-Users] Urgent newbie help - "Hit a bug" page

John Fleming wrote:

...

1. I must be doing something wrong - Running that script only gives me:

Luke:/var/lib/mailman# ./johntest <----- the script sh-2.05b#

Yes. something is wrong because apparently, bin/sh is trying to read its input from the terminal instead of the script. What is the exact contents of johntest? What if you run

bin/sh ./johntest

instead?

Same thing as before. Here's the script:

Luke:/var/lib/mailman# cat johntest ! /bin/sh for list in /var/lib/mailman/bin/list_lists --bare do echo ---------------------------------- echo list: $list /var/lib/mailman/bin/dumpdb /var/lib/mailman/lists/$list/config.pck | grep advertised echo ----------------------------------- doneLuke:/var/lib/mailman#

I don't know what I'm missing about running this script. Looking through the .pck files manually, they all seem to have the attributes real_name, language, and advertised - things that appeared in the error log. Some are advertised and some are not, but they all have that entry.

I have no idea how it got broken. Did you have a system crash? power failure? some similar event?

No, really. Server on UPS. There could have been a brief burp, but UPS -should've- protected me from that.

I received an email from a list member that a list wasn't working. I restarted qrunner to no avail, rebooted the server to no avail, and then restarted qrunner again and then the lists all started working again. The "We hit a bug" webpage problem persisted throughout.

Thanks for your patience - really. If I resort to my month-old backup, should I just replace the /lists directory? - John

----- Original Message ----- From: "Mark Sapiro" <msapiro@value.net> To: "John Fleming" <john@wa9als.com>; <mailman-users@python.org> Sent: Saturday, March 03, 2007 9:43 AM Subject: Re: [Mailman-Users] Urgent newbie help - "Hit a bug" page

John Fleming wrote:

...

Same thing as before. Here's the script:

Luke:/var/lib/mailman# cat johntest ! /bin/sh

This line needs to begin with a '#'

#! /bin/sh

Duh - thanks.

The output I get now I think is the same as in the error log:

Luke:/var/lib/mailman# ./johntest Traceback (most recent call last): File "/var/lib/mailman/bin/list_lists", line 122, in ? main() File "/var/lib/mailman/bin/list_lists", line 102, in main longest = max(len(mlist.real_name), longest) File "/var/lib/mailman/Mailman/MailList.py", line 144, in __getattr__ raise AttributeError, name AttributeError: real_name Luke:/var/lib/mailman#

• but I don't see any list-specific info!

And, here's the result of bin/list_lists: (same as above)

Maybe there's a list you're not looking at. Does 'bin/list_lists' or 'ls -l /var/mailman/lists/' show anything you're not looking at?

list_lists gives me the same error:

Luke:/var/lib/mailman/bin# ./list_lists Traceback (most recent call last): File "./list_lists", line 122, in ? main() File "./list_lists", line 102, in main longest = max(len(mlist.real_name), longest) File "/var/lib/mailman/Mailman/MailList.py", line 144, in __getattr__ raise AttributeError, name AttributeError: real_name Luke:/var/lib/mailman/bin#

There were a couple of lists I hadn't looked at (getting tedious to do it manually) because they hadn't been used for a year or more. However, they also look OK to me. (They are not corrupted files, and the real_name values are correct, and nothing else seems amiss to me.)

We're still not getting the expected results from the script though...??

Was Mailman running when you rebooted?

The first thing I did was restart mailman, and I got the initial message "qrunner not running", so no, it had stopped. It restarted without further error, but the web problem was still there and the lists apparently were not working. Same thing after the reboot. But then after another qrunner restart, then I started receiving a backlog of messages from various lists (plural).

Also, is it possible you had

multiple instances of Mailman running when you rebooted so that a shutdown script if any didn't stop them all? See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.068.htp>

• not for the problem, but for the solution.

Do ps auxww | egrep 'p[y]thon' or ps -fAww | egrep 'p[y]thon' or however you might spell it on your system to see that only one mailmanctl process and one of each of eight qrunner processes are running.

Yup:

Luke:/var/lib/mailman# ps auxww | egrep 'p[y]thon' list 11070 0.0 0.5 7792 5416 ? Ss Mar02 0:00 /usr/bin/python /usr/lib/mailman/bin/mailmanctl start list 11086 0.0 0.5 7592 5288 ? S Mar02 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s list 11087 0.0 0.6 8188 6284 ? S Mar02 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s list 11088 0.0 0.5 7620 5300 ? S Mar02 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=CommandRunner:0:1 -s list 11089 0.0 0.9 11900 9800 ? S Mar02 0:03 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s list 11090 0.0 0.5 7600 5328 ? S Mar02 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=NewsRunner:0:1 -s list 11091 0.0 0.6 8280 6320 ? S Mar02 0:01 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s list 11092 0.0 0.5 8264 6192 ? S Mar02 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=VirginRunner:0:1 -s list 11093 0.0 0.5 7508 5288 ? S Mar02 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=RetryRunner:0:1 -s Luke:/var/lib/mailman#

There is probably only one bad list, so no, don't replace the directory. Find the bad list first and then perhaps replace its list/listname directory or, if possible, use the config.pck.last or fix the config.pck.

OK - Comments on the script that is not giving list-specific info?

And, I assume we're still on the same page about the problem: The list overview web page gives an error when trying to view all of the advertised lists on -ANY- of my domains (all hosted on the same server) e.g. www.one.domain/mailman/. (As you know, if there are no advertised lists on a domain, we should get a nice web page stating that fact - or the advertised lists etc.) If I ask about a -specific- list www.one.domain/mailman/listinfo/<listname>, I get the expected web page without error.

• John

----- Original Message ----- From: "Mark Sapiro" <msapiro@value.net> To: "John Fleming" <john@wa9als.com>; <mailman-users@python.org> Sent: Saturday, March 03, 2007 10:45 AM Subject: Re: [Mailman-Users] Urgent newbie help - "Hit a bug" page

Replace the line

for list in /var/lib/mailman/bin/list_lists --bare

with

for list in ls /var/lib/mailman/lists

<snip>

OK, finally, that gives the expected output (below) - Only it just lists all of my lists (only 12) with their expected advertised attributes, some 0 and some 1. No errors about any of them being missing. I also ran it grepping for "real_name" and also got the expected output for all 12 lists. This doesn't surprise me because I knew from looking manually that they all had the "advertised" and "real_name" attributes. I'm also not too excited that simply restoring backup .pck files are going to fix the problem at this point...?

Luke:/var/lib/mailman# ./johntest

list: 40days 'advertised': 1,

list: adboard 'advertised': 1,

list: communications 'advertised': 1,

list: flemings 'advertised': 0,

<snip> Luke:/var/lib/mailman#

----- Original Message ----- From: "Mark Sapiro" <msapiro@value.net>

Try the following:

and add the line ' print n' so it becomes

mlists = [] longest = 0 for n in names: print n mlist = MailList.MailList(n, lock=0) if advertised and not mlist.advertised: continue if vhost and mm_cfg.VIRTUAL_HOST_OVERVIEW and
vhost.find(mlist.web_page_url) == -1 and
mlist.web_page_url.find(vhost) == -1: continue mlists.append(mlist) longest = max(len(mlist.real_name), longest)

Make sure that 'print' is indented exactly 8 spaces. Then run bin/my_list_lists.

This should print list names one by one until it throws the exception and prints the traceback. The last name printed has a problem. Temporarily move this list aside

mv /var/lib/mailman/thislist /somewhere/outside/mailman/thislist

and then run bin/my_list_lists again to see if there are any problems. If not, you've identified the one problem list. There is something wrong with this list's config.pck. There is something mysterious if various software says this list doesn't have an 'advertised' or 'real_name' attribute, yet bin/dumpdb says it does.

OTOH, maybe bin/my_list_lists fails on the first list and then when you move that list aside, it fails on the next one and so on. This would incriminate the software somehow and not a list.

BINGO! The results listed several lists without error, and then there was the attribute error for real_name. The preceeding list was the mailman list! So with appropriate backups, I renamed the mailman list's config.pck.last file to config.pck and ran mylist_lists again, and it ran without error, listing all of my lists! And guess what, the list overview web page works as expected now!

Luke:/var/lib/mailman/bin# ./mylist_lists <listname> <listname> <listname> <listname> mailman Traceback (most recent call last): File "./mylist_lists", line 123, in ? main() File "./mylist_lists", line 103, in main longest = max(len(mlist.real_name), longest) File "/var/lib/mailman/Mailman/MailList.py", line 144, in __getattr__ raise AttributeError, name AttributeError: real_name Luke:/var/lib/mailman/bin#

Do you agree that there was something VERY subtle about what was wrong with it? Here's a snippet from the original config.pck:

'preferred_language': 'en',
'private_roster': 1,
'real_name': 'Mailman',
'reject_these_nonmembers': [],
'reply_goes_to_list': 0,

The real_name -looks- OK to a text editor, no?

Here's how it looks now:

'preferred_language': 'en',
'private_roster': 1,
'real_name': 'Mailman',
'reject_these_nonmembers': [],
'reply_goes_to_list': 0,

Running diff adds to the confusion, and it's getting difficult to explain.

If I diff the present (working) config.pck (the renamed config.pck.last file) and config-orig.pck (the problem file), there is NO DIFFERENCE. However, if I diff config-orig.pck (problem file) and config.pck.last.orig (the original pck.last file), there IS a difference!! So it seems that simply renaming the "same" (according to diff) file fixed the problem?! Some very subtle binary difference not detectable in the text output from dumpdb??

Luke:/var/lib/mailman/lists/mailman# diff config.pck config-orig.pck Luke:/var/lib/mailman/lists/mailman#

Luke:/var/lib/mailman/lists/mailman# diff config-orig.pck config.pck.last.orig Binary files config-orig.pck and config.pck.last.orig differ

============================================== A special THANK YOU to Mark Shapiro for his patient and detailed help on this problem. Hopefully something in this thread has helped someone else now or in the future! - John

----- Original Message ----- From: "Mark Sapiro" <msapiro@value.net>

And yes, this looks the same. One thing I thought of - are both of these excerpts from something identified by bin/dumpdb as 'object 1' in the file. i.e. directly below the lines

[----- start pickle file -----] <----- start object 1 ----->

with no intervening

<----- start object 2 ----->

-YES-

...

Running diff adds to the confusion, and it's getting difficult to explain.

If I diff the present (working) config.pck (the renamed config.pck.last file) and config-orig.pck (the problem file), there is NO DIFFERENCE.

Have the timestamps changed? maybe Mailman Saved the list in the mean time. However, that would not explain why diff says that config-orig.pck, and the current config.pck are the same, yet one works and the other didn't. That is the real mystery, and I have no answer for that.

...

So it seems that simply renaming the "same" (according to diff) file fixed the problem?! Some very subtle binary difference not detectable in the text output from dumpdb??

Still, diff 'should' detect a difference. Are the files the same length? What does

cmp config.pck config-orig.pck

Luke:/var/lib/mailman/lists/mailman# cmp config.pck config-orig.pck config.pck config-orig.pck differ: char 46, line 2 Luke:/var/lib/mailman/lists/mailman#

Line 2 in the dumpdb output does not have 46 characters, so I assume this refers to the raw file??

Now look at this:

Luke:/var/lib/mailman/lists/mailman# cmp config-orig.pck config.pck.last.orig config-orig.pck config.pck.last.orig differ: char 46, line 2 Luke:/var/lib/mailman/lists/mailman#

So the first and third cmp results above are consistent and show the same difference.

***** And if you're not already going crazy, this will do it:

Earlier I sent you this diff result:

Luke:/var/lib/mailman/lists/mailman# diff config.pck config-orig.pck Luke:/var/lib/mailman/lists/mailman#

But now when I run it, I get:

Luke:/var/lib/mailman/lists/mailman# diff config.pck config-orig.pck Binary files config.pck and config-orig.pck differ Luke:/var/lib/mailman/lists/mailman#

So I think we've finally beaten the horse to death and I can barely keep it straight anymore! These are all cut and paste results, so it's not just my memory!

I don't under stand why they diff'ed the same this morning and not now!

Anyway, we know something was wrong with config.pck and using the config.pck.last file fixed it.

• John