Our last mailman crash was at 11:47pm on Nov 7. Here is the qrunner log:
I looked at our vps to see if we were hitting any resource limits, but I haven't found any problems.
Nov 07 23:47:30 2010 (23778) OutgoingRunner qrunner caught SIGTERM.
Stopping.
Nov 07 23:47:30 2010 (23771) CommandRunner qrunner caught SIGTERM.
Stopping.
Nov 07 23:47:31 2010 (23778) OutgoingRunner qrunner exiting.
Nov 07 23:47:31 2010 (23771) CommandRunner qrunner exiting.
Nov 07 23:47:30 2010 (23776) IncomingRunner qrunner caught SIGTERM.
Stopping.
Nov 07 23:47:30 2010 (23744) Master watcher caught SIGTERM. Exiting.
Nov 07 23:47:30 2010 (23777) NewsRunner qrunner caught SIGTERM. Stopping.
Nov 07 23:47:30 2010 (23779) VirginRunner qrunner caught SIGTERM. Stopping.
Nov 07 23:47:31 2010 (23776) IncomingRunner qrunner exiting.
Nov 07 23:47:31 2010 (23777) NewsRunner qrunner exiting.
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23771, sig: None, sts: 15, class: CommandRunner, slice: 1/1)
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23778, sig: None, sts: 15, class: OutgoingRunner, slice: 1/1)
Nov 07 23:47:31 2010 (23779) VirginRunner qrunner exiting.
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23777, sig: None, sts: 15, class: NewsRunner, slice: 1/1)
Nov 07 23:47:30 2010 (23780) RetryRunner qrunner caught SIGTERM. Stopping.
Nov 07 23:47:30 2010 (23770) BounceRunner qrunner caught SIGTERM. Stopping.
Nov 07 23:47:30 2010 (23767) ArchRunner qrunner caught SIGTERM. Stopping.
Nov 07 23:47:31 2010 (23780) RetryRunner qrunner exiting.
Nov 07 23:47:31 2010 (23767) ArchRunner qrunner exiting.
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23776, sig: None, sts: 15, class: IncomingRunner, slice: 1/1)
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23779, sig: None, sts: 15, class: VirginRunner, slice: 1/1)
Nov 07 23:47:31 2010 (23770) BounceRunner qrunner exiting.
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23767, sig: None, sts: 15, class: ArchRunner, slice: 1/1)
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23780, sig: None, sts: 15, class: RetryRunner, slice: 1/1)
Nov 07 23:47:31 2010 (23744) Master qrunner detected subprocess exit
(pid: 23770, sig: None, sts: 15, class: BounceRunner, slice: 1/1)
Nov 07 23:49:12 2010 (28002) ArchRunner qrunner started.
Nov 07 23:49:12 2010 (28007) NewsRunner qrunner started.
Nov 07 23:49:12 2010 (28005) IncomingRunner qrunner started.
Nov 07 23:49:12 2010 (28010) VirginRunner qrunner started.
Nov 07 23:49:12 2010 (28011) RetryRunner qrunner started.
Nov 07 23:49:12 2010 (28004) CommandRunner qrunner started.
Nov 07 23:49:12 2010 (28003) BounceRunner qrunner started.
Nov 07 23:49:13 2010 (28009) OutgoingRunner qrunner started.
Nov 08 00:32:15 2010 (28002) ArchRunner qrunner caught SIGTERM. Stopping.
Nov 08 00:32:15 2010 (28002) ArchRunner qrunner exiting.
Nov 08 00:32:15 2010 (27975) Master watcher caught SIGTERM. Exiting.
Nov 08 00:32:15 2010 (28003) BounceRunner qrunner caught SIGTERM. Stopping.
Nov 08 00:32:15 2010 (28007) NewsRunner qrunner caught SIGTERM. Stopping.
Nov 08 00:32:15 2010 (28007) NewsRunner qrunner exiting.
Nov 08 00:32:15 2010 (28005) IncomingRunner qrunner caught SIGTERM.
Stopping.
Nov 08 00:32:15 2010 (28009) OutgoingRunner qrunner caught SIGTERM.
Stopping.
Nov 08 00:32:15 2010 (28005) IncomingRunner qrunner exiting.
Nov 08 00:32:15 2010 (28009) OutgoingRunner qrunner exiting.
Nov 08 00:32:15 2010 (28010) VirginRunner qrunner caught SIGTERM. Stopping.
Nov 08 00:32:15 2010 (28010) VirginRunner qrunner exiting.
Nov 08 00:32:15 2010 (28004) CommandRunner qrunner caught SIGTERM.
Stopping.
Nov 08 00:32:15 2010 (28004) CommandRunner qrunner exiting.
Nov 08 00:32:15 2010 (28011) RetryRunner qrunner caught SIGTERM. Stopping.
Nov 08 00:32:15 2010 (28011) RetryRunner qrunner exiting.
Nov 08 00:32:15 2010 (28003) BounceRunner qrunner exiting.
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28002, sig: None, sts: 15, class: ArchRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28004, sig: None, sts: 15, class: CommandRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28005, sig: None, sts: 15, class: IncomingRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28009, sig: None, sts: 15, class: OutgoingRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28010, sig: None, sts: 15, class: VirginRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28011, sig: None, sts: 15, class: RetryRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28007, sig: None, sts: 15, class: NewsRunner, slice: 1/1)
Nov 08 00:32:15 2010 (27975) Master watcher caught SIGTERM. Exiting.
Nov 08 00:32:15 2010 (27975) Master qrunner detected subprocess exit
(pid: 28003, sig: 15, sts: None, class: BounceRunner, slice: 1/1)
The error log was similar to the last one:
Nov 08 00:36:20 2010 mailmanctl(14175): No child with pid: 27975
Nov 08 00:36:20 2010 mailmanctl(14175): [Errno 3] No such process
Nov 08 00:36:20 2010 mailmanctl(14175): Stale pid file removed.
Nov 08 00:36:20 2010 mailmanctl(14185): Traceback (most recent call last):
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl", line 555, in ?
Nov 08 00:36:20 2010 mailmanctl(14185): main()
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl", line 367, in main
Nov 08 00:36:20 2010 mailmanctl(14185): check_for_site_list()
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl", line 277, in
check_for_site_list
Nov 08 00:36:20 2010 mailmanctl(14185): sitelist =
MailList(sitelistname, lock=0)
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/local/cpanel/3rdparty/mailman/Mailman/MailList.py", line 115, in
__init__
Nov 08 00:36:20 2010 mailmanctl(14185): execfile(filename, dict)
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/local/cpanel/3rdparty/mailman/lists/mailman/extend.py", line 5, in ?
Nov 08 00:36:20 2010 mailmanctl(14185): from
Mailman.MysqlMemberships import MysqlMemberships
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line
49, in ?
Nov 08 00:36:20 2010 mailmanctl(14185): import MySQLdb
Nov 08 00:36:20 2010 mailmanctl(14185): File
"build/bdist.linux-x86_64/egg/MySQLdb/__init__.py", line 19, in ?
Nov 08 00:36:20 2010 mailmanctl(14185): File
"build/bdist.linux-x86_64/egg/_mysql.py", line 7, in ?
Nov 08 00:36:20 2010 mailmanctl(14185): File
"build/bdist.linux-x86_64/egg/_mysql.py", line 4, in __bootstrap__
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 881, in resource_filename
Nov 08 00:36:20 2010 mailmanctl(14185): return
get_provider(package_or_requirement).get_resource_filename(
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 1351, in get_resource_filename
Nov 08 00:36:20 2010 mailmanctl(14185):
self._extract_resource(manager, self._eager_to_zip(name))
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 1372, in _extract_resource
Nov 08 00:36:20 2010 mailmanctl(14185): real_path =
manager.get_cache_path(
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 962, in get_cache_path
Nov 08 00:36:20 2010 mailmanctl(14185): self.extraction_error()
Nov 08 00:36:20 2010 mailmanctl(14185): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 928, in extraction_error
Nov 08 00:36:20 2010 mailmanctl(14185): raise err
Nov 08 00:36:20 2010 mailmanctl(14185): pkg_resources . ExtractionError
: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg cache:
[Errno 13] Permission denied: '/root/.python-eggs'
The Python egg cache directory is currently set to:
/root/.python-eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
Nov 08 00:36:20 2010 mailmanctl(14185):
Nov 08 00:37:46 2010 mailmanctl(19488): PID unreadable in:
/usr/local/cpanel/3rdparty/mailman/data/master-qrunner.pid
Nov 08 00:37:46 2010 mailmanctl(19488): [Errno 2] No such file or
directory: '/usr/local/cpanel/3rdparty/mailman/data/master-qrunner.pid'
Nov 08 00:37:46 2010 mailmanctl(19488): Is qrunner even running?
Nov 08 00:37:48 2010 mailmanctl(19567): Traceback (most recent call last):
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl", line 555, in ?
Nov 08 00:37:48 2010 mailmanctl(19567): main()
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl", line 367, in main
Nov 08 00:37:48 2010 mailmanctl(19567): check_for_site_list()
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl", line 277, in
check_for_site_list
Nov 08 00:37:48 2010 mailmanctl(19567): sitelist =
MailList(sitelistname, lock=0)
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/local/cpanel/3rdparty/mailman/Mailman/MailList.py", line 115, in
__init__
Nov 08 00:37:48 2010 mailmanctl(19567): execfile(filename, dict)
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/local/cpanel/3rdparty/mailman/lists/mailman/extend.py", line 5, in ?
Nov 08 00:37:48 2010 mailmanctl(19567): from
Mailman.MysqlMemberships import MysqlMemberships
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line
49, in ?
Nov 08 00:37:48 2010 mailmanctl(19567): import MySQLdb
Nov 08 00:37:48 2010 mailmanctl(19567): File
"build/bdist.linux-x86_64/egg/MySQLdb/__init__.py", line 19, in ?
Nov 08 00:37:48 2010 mailmanctl(19567): File
"build/bdist.linux-x86_64/egg/_mysql.py", line 7, in ?
Nov 08 00:37:48 2010 mailmanctl(19567): File
"build/bdist.linux-x86_64/egg/_mysql.py", line 4, in __bootstrap__
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 881, in resource_filename
Nov 08 00:37:48 2010 mailmanctl(19567): return
get_provider(package_or_requirement).get_resource_filename(
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 1351, in get_resource_filename
Nov 08 00:37:48 2010 mailmanctl(19567):
self._extract_resource(manager, self._eager_to_zip(name))
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 1372, in _extract_resource
Nov 08 00:37:48 2010 mailmanctl(19567): real_path =
manager.get_cache_path(
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 962, in get_cache_path
Nov 08 00:37:48 2010 mailmanctl(19567): self.extraction_error()
Nov 08 00:37:48 2010 mailmanctl(19567): File
"/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r80622-py2.4.egg/pkg_resources.py",
line 928, in extraction_error
Nov 08 00:37:48 2010 mailmanctl(19567): raise err
Nov 08 00:37:48 2010 mailmanctl(19567): pkg_resources . ExtractionError
: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg cache:
[Errno 13] Permission denied: '/root/.python-eggs'
The Python egg cache directory is currently set to:
/root/.python-eggs
Perhaps your account does not have write access to this directory? You can change the cache directory by setting the PYTHON_EGG_CACHE environment variable to point to an accessible directory. Nov 08 00:37:48 2010 mailmanctl(19567):
Why do you have /usr/local/cpanel/3rdparty/mailman/lists/mailman/extend.py which imports MysqlMemberships? As far as I know, this is not something that cPanel Mailman does by default? I'm using the mailman mysql adaptor so the members are stored in mysql
On 10/15/2010 8:53 AM, Mark Sapiro wrote:
Perhaps your account does not have write access to this directory? You can change the cache directory by setting the PYTHON_EGG_CACHE environment variable to point to an accessible directory. [...]
The underlying issue here is that Python's MySQLdb was apparently installed by root in a way that the the Python egg cache directory is not writeable by the Mailman process.
You can get rid of the above errors by simply removing /usr/local/cpanel/3rdparty/mailman/lists/mailman/extend.py and perhaps any other /usr/local/cpanel/3rdparty/mailman/lists/*/extend.py files, but this will defeat your use of MysqlMemberships.py, but since it doesn't work anyway, and maybe is only installed for the site list, that shouldn't be too much of an issue.
Also, please read the FAQ at <http://wiki.list.org/display/DOC/Mailman+and+CPanel>.
Finally, it is not clear to me that the above error is from the crash. It seems to me it would occur whenever mailmanctl starts. Is this all that's in Mailman's error log? What's in Mailman's qrunner log?
-- Energy Justice Communities Map Developer - http://www.energyjustice.net/map