Debugging mass invitations using web page interface...
Dear Mailman Cognoscenti,
I'm helping one of my list owners send out 5K plus invitations to students to subscribe to his mailing list. Our current configuration:
Mailman v2.1.20
RHEL v5.11
Semdmail v8.13.8
Apache v2.2.3Since this was the first time doing this, I suggested breaking the batch input into 3 groups, 50, 500, and the rest. The 50 went fine, as did the 500, but the largest batch gave him a generic web server error:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, root@conundrum.unh.edu and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
I looked at the logs and I couldn't find anything that hinted at what went wrong. So I asked the owner to send me the last back and I'd give it a try. I wrote a script that removed folks already subscribed to his list and split the remaining subscribers up into 6 files with a thousand records each. I just tried uploading the 1st batch of 1K, with the following options:
Subscribe these users now... (*) Invite
Send welcome message... (*) No
Send notifications... (*) NoAnd entered a 7 line paragraph explaining the invitation.
I ended up having the same error happen. Looking at the Mailman logs, I can't see any difference before or after my submission. In the HTTPD logs, I see:
[Fri Aug 26 19:59:23 2016] [warn] [client 132.177.215.132] Timeout waiting for output from CGI script /usr/local/mailman/cgi-bin/admin, referer: https://lists.unh.edu/mailman/admin/campus.connection/members/add
[Fri Aug 26 19:59:23 2016] [error] [client 132.177.215.132] Premature end of script headers: admin, referer: https://lists.unh.edu/mailman/admin/campus.connection/members/add
So is there an inherent limit to the number of invites that can be submitted via the web form?
As a work around, how would I do large invites on behalf of the owner from the command line, including the 'extra text' that is allowed via the web interface?
-- Cordially, the UNH Mailing List Server Admins Bill Costa, senior admin (603) 862-3056
On 08/26/2016 05:24 PM, The Mailing List Server Admin wrote:
I ended up having the same error happen. Looking at the Mailman logs, I can't see any difference before or after my submission. In the HTTPD logs, I see:
[Fri Aug 26 19:59:23 2016] [warn] [client 132.177.215.132] Timeout waiting for output from CGI script /usr/local/mailman/cgi-bin/admin, referer: https://lists.unh.edu/mailman/admin/campus.connection/members/add
[Fri Aug 26 19:59:23 2016] [error] [client 132.177.215.132] Premature end of script headers: admin, referer: https://lists.unh.edu/mailman/admin/campus.connection/members/add
So is there an inherent limit to the number of invites that can be submitted via the web form?
The admin CGI is processing the mass subscribe/invite list. While it is doing this it is not generating any output. If it takes 'too long' the web server will time out waiting for output from the CGI. That's what's happening here.
The amount of time is set by apache's TimeOut directive <http://httpd.apache.org/docs/current/mod/core.html#timeout>
As a work around, how would I do large invites on behalf of the owner from the command line, including the 'extra text' that is allowed via the web interface?
You'd need a custom script to do the 'extra text'. I have added an invite_members script I use at <https://www.msapiro.net/scripts/invite_members> and <https://fog.ccsf.edu/~msapiro/scripts/invite_members>. This can send invitations including the 'extra text'.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark, once again thank you for your valued expertise...
With regards to the failed thousand invitations submission via the Web UI:
The admin CGI is processing the mass subscribe/invite list. While it is doing this it is not generating any output. If it takes 'too long' the web server will time out waiting for output from the CGI.
Makes perfect sense. (I wonder if MM3 uses Ajax or other web techniques to provide some sort of progress indicator to prevent this from happening.)
So when Apache timed out, would it have killed the admin CGI on the way out? In other words, would the admin script still run to completion and send out all the invitations from that batch? Or is it most likely to have been a partial run? Am I correct that the only place I will find a record of the invitations sent will be in the Sendmail mail logs? I'd like to avoid sending dup invites if possible.
Thanks for pointer to the command line invitation script. I'll add it to my growing arsenal.
-- Cordially, the UNH Mailing List Server Admins Bill Costa, senior admin (603) 862-3056
On 08/27/2016 03:16 AM, The Mailing List Server Admin wrote:
So when Apache timed out, would it have killed the admin CGI on the way out? In other words, would the admin script still run to completion and send out all the invitations from that batch? Or is it most likely to have been a partial run? Am I correct that the only place I will find a record of the invitations sent will be in the Sendmail mail logs? I'd like to avoid sending dup invites if possible.
When Apache times out, it kills the CGI process. The process meanwhile has been queueing the invitations to be sent, so a bunch will be sent.
You could use the script at <https://www.msapiro.net/scripts/list_pending> to list the pending subscriptions. Some may have been accepted already, so the fact that one isn't there doesn't mean it wasn't sent, but if you know the order from the original input, you can probably figure how far it got.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro recently wrote (in part):
You could use the script ... to list the pending subscriptions. Some may have been accepted already, so the fact that one isn't there doesn't mean it wasn't sent, but if you know the order from the original input, you can probably figure how far it got.
Between being able to see who is now already subscribed from my original list, and looking at the pending subscription requests, I should be able to figure out who we've missed. Thanks for pointing out yet another useful MM2 utility!
-- Cordially, the UNH Mailing List Server Admins Bill Costa, senior admin (603) 862-3056
Way way back in August 2016, we had a problem with submitting too many members via the web interface. As a solution, the following was suggested:
You'd need a custom script to do the 'extra text'. I have added an invite_members script I use at <https://www.msapiro.net/scripts/invite_members> and <https://fog.ccsf.edu/~msapiro/scripts/invite_members> This can send invitations including the 'extra text'.
We used this script with great success in Aug 2017 and 2018. But this year we've run into a problem. When I submit the script it appears to hang waiting for some kind of lock. I can see this when I eventually CTRL-C out.
Traceback (most recent call last):
File "/home/mladmin/Util/invite_members", line 236, in ?
main()
File "/home/mladmin/Util/invite_members", line 197, in main
mlist = MailList.MailList(listname)
File "/usr/local/mailman/Mailman/MailList.py", line 129, in
__init__
self.Lock()
File "/usr/local/mailman/Mailman/MailList.py", line 162, in
Lock
self.__lock.lock(timeout)
File "/usr/local/mailman/Mailman/LockFile.py", line 312, in
lock
self.__sleep()
File "/usr/local/mailman/Mailman/LockFile.py", line 496, in
__sleep
time.sleep(interval)
KeyboardInterruptIn the past we've used the script from a non-priv'd account with no problems. When it failed this time, I tried running it as 'mailman' but it still hangs on the lock. Any suggestions? (BTW, the line numbers will be different from the original script because of some debug print statements and additional comments added.)
-- Cordially, the UNH Mailing List Server Admins Bill Costa, Adjunct Admin (603) 862-3056
On 8/26/19 3:28 PM, The Mailing List Server Admin wrote:
Way way back in August 2016, we had a problem with submitting too many members via the web interface. As a solution, the following was suggested:
You'd need a custom script to do the 'extra text'. I have added an invite_members script I use at <https://www.msapiro.net/scripts/invite_members> and <https://fog.ccsf.edu/~msapiro/scripts/invite_members> This can send invitations including the 'extra text'.
We used this script with great success in Aug 2017 and 2018. But this year we've run into a problem. When I submit the script it appears to hang waiting for some kind of lock. I can see this when I eventually CTRL-C out.
Traceback (most recent call last): File "/home/mladmin/Util/invite_members", line 236, in ? main() File "/home/mladmin/Util/invite_members", line 197, in main mlist = MailList.MailList(listname) File "/usr/local/mailman/Mailman/MailList.py", line 129, in __init__ self.Lock() File "/usr/local/mailman/Mailman/MailList.py", line 162, in Lock self.__lock.lock(timeout) File "/usr/local/mailman/Mailman/LockFile.py", line 312, in lock self.__sleep() File "/usr/local/mailman/Mailman/LockFile.py", line 496, in __sleep time.sleep(interval) KeyboardInterrupt
The list is locked. See <https://wiki.list.org/x/17891756> for more about locks and removing stale locks.
I have looked at the script and I see an issue. It can leave the list locked if you specify a message file and that file cannot be read.
I have updated the script to fix this.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Mark Sapiro -
The Mailing List Server Admin