Mailman MySQL adaptor - MySQLdb module error

I'm trying to get the Mailman MySQL adaptor to work, while we're waiting for Mailman 3 to be finished. I'd love some help!
I followed the instructions on: http://loeki.tv/log/archives/81-Setting-up-Mailman-to-store-members-in-a-MyS...
I have a VPS (virtual private server) running Centos, cPanel, and Mailman 2.1.13.
I create a new list to test the adaptor. When I go to the list's page it gives me an error. Mailman's error log says "ImportError: No module named MySQLdb".
I've installed the MySQLdb module and am able to run a simple test script at the command line (that connects to the mysql database), without a problem. So what could be the problem?
My setup has two versions of python running - one in /usr/bin/python and another in /usr/bin/python2.4, however both of them run the test script without a problem. I'm wondering if having two versions might be the cause? Or could there be a user permissions issue?
Test Script:
import MySQLdb
# Open database connection db = MySQLdb.connect("localhost","testuser","test123456","TESTDB" )
# prepare a cursor object using cursor() method cursor = db.cursor()
# execute SQL query using execute() method. cursor.execute("SELECT VERSION()")
# Fetch a single row using fetchone() method. data = cursor.fetchone()
print "Database version : %s " % data
# disconnect from server db.close()

Aaron Kreider wrote:
Did you install Mailman yourself or are you using cPanel's Mailman. If the latter, did you modify the paths in the above instructions to match cPanel.
See the FAQ at <http://wiki.list.org/x/sYA9>.
In a typical CentOS RPM install, /usr/bin/python and /usr/bin/python2.4 are hard links to the same file.
How many /usr/lib/python* directories do you have?
I'm wondering if having two versions might be the cause? Or could there be a user permissions issue?
How was Mailman configured? Having two or even more versions of Python should not be a problem as long as the one Mailman was configured to use has MySQLdb installed.
Is Mailman running? What happens if you send a 'help' command by email to the new list's -request address?
I don't think this is a permissions issue as MySQLdb should be world searchable/readable, but I suppose it could be a SELinux issue if SELinux is enabled.
If you run
strings /path/to/mailman/cgi-bin/listinfo
The output should contain amongst much other stuff the path of the python used by the CGI scripts.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 8/6/2010 1:34 AM, Mark Sapiro wrote:
Only one directory. /usr/lib/python2.4
I changed /user/local/cpanel/3rdparty/mailman/Mailman/mm_cfg.py to have my MySQL database settings:
# Put YOUR site-specific settings below this line. MYSQL_MEMBER_DB_NAME = "the_db" MYSQL_MEMBER_DB_USER = "the_user" MYSQL_MEMBER_DB_PASS = "topsecret MYSQL_MEMBER_DB_HOST = "localhost" MYSQL_MEMBER_TABLE_TYPE = "wide"
Is Mailman running? What happens if you send a 'help' command by email to the new list's -request address?
Sending "help" does not work for the new list.
It does work for old lists (non-mysql based ones).
This gives me an error message: Content-type: text/html <head> </head><body> <h1>Mailman CGI error!!!</h1> <pre> </pre> PYTHONPATH= --with-cgi-gid --with-mail-gid mail PYTHONHOME= <title>Mailman CGI error!!!</title> The Mailman CGI wrapper encountered a fatal error. This entry is being stored in your syslog: Failure to find group name for GID %d. Mailman expected the %s wrapper to be executed as group "%s", but the system's %s server executed the wrapper as GID %d for which the name could not be found. Try adding GID %d to your system as "%s", or tweak your %s server to run the wrapper as group "%s". Group mismatch error. Mailman expected the %s wrapper script to be executed as group "%s", but the system's %s server executed the %s script as group "%s". Try tweaking the %s server to run the script as group "%s", or re-run configure, providing the command line option `%s=%s'. /usr/local/cpanel/3rdparty/mailman/scripts/ /usr/local/cpanel/3rdparty/mailman /usr/local/cpanel/3rdparty/bin/python driver listinfo nobody Mailman cgi-wrapper (listinfo)

Aaron Kreider wrote:
OK
What's in Mailman's error log. Do you see the same ImportError causing the message to be shunted?
It looks as if you ran
/path/to/mailman/cgi-bin/listinfo
not
strings /path/to/mailman/cgi-bin/listinfo
as I asked. I don't want you to run the listinfo wrapper, I want to see what Python it is invoking. I am guessing there is a python in /user/local/cpanel/3rdparty/ and that that is the Python being used by Mailman. I'm trying to confirm that or at least confirm that it is other than /usr/bin/python(2.4). The wrappers contain the path to Python as a string, and that's what I want to see.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark,
No. I see it has an error about the table not being created. So that must mean it is correctly importing the module? Why does it work here and not in other cases? I might need to manually create the tables (one for each list). I'm fine with doing that.
Aug 06 15:18:00 2010 (12201) Uncaught runner exception: (1103, "Incorrect table name 'dbtest8_energyjustice.net'") Aug 06 15:18:00 2010 (12201) Traceback (most recent call last): File "/usr/local/cpanel/3rdparty/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop self._onefile(msg, msgdata) File "/usr/local/cpanel/3rdparty/mailman/Mailman/Queue/Runner.py", line 166, in _onefile mlist = self._open_list(listname) File "/usr/local/cpanel/3rdparty/mailman/Mailman/Queue/Runner.py", line 211, in _open_list mlist = MailList.MailList(listname, lock=False) File "/usr/local/cpanel/3rdparty/mailman/Mailman/MailList.py", line 125, in __init__ func(self) File "/usr/local/cpanel/3rdparty/mailman/lists/dbtest8_energyjustice.net/extend.py", line 4, in extend list._memberadaptor = MysqlMemberships(list) File "/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line 89, in __init__ self.createTable() File "/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line 205, in createTable self.query ( File "/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line 241, in query return mm_cfg.cursor.execute (query) File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 174, in execute File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler ProgrammingError: (1103, "Incorrect table name 'dbtest8_energyjustice.net'")
Ok this is what I get.
Content-type: text/html <head> </head><body> <h1>Mailman CGI error!!!</h1> <pre> </pre> PYTHONPATH= --with-cgi-gid --with-mail-gid mail PYTHONHOME= <title>Mailman CGI error!!!</title> The Mailman CGI wrapper encountered a fatal error. This entry is being stored in your syslog: Failure to find group name for GID %d. Mailman expected the %s wrapper to be executed as group "%s", but the system's %s server executed the wrapper as GID %d for which the name could not be found. Try adding GID %d to your system as "%s", or tweak your %s server to run the wrapper as group "%s". Group mismatch error. Mailman expected the %s wrapper script to be executed as group "%s", but the system's %s server executed the %s script as group "%s". Try tweaking the %s server to run the script as group "%s", or re-run configure, providing the command line option `%s=%s'. /usr/local/cpanel/3rdparty/mailman/scripts/ /usr/local/cpanel/3rdparty/mailman /usr/local/cpanel/3rdparty/bin/python driver listinfo nobody Mailman cgi-wrapper (listinfo)
BTW this file is a symbolic link:- /usr/local/cpanel/3rdparty/bin/python It links to /usr/bin/python2.4
Thanks for your help so far!
Aaron

Aaron Kreider wrote:
I'm not sure. see below.
[...]
Sorry. I don't know why I didn't see it the first time. You had done what I asked, and I got confused by Content-Type: being the first line of output (and maybe by your referring to an error). The following is the output I wanted.
This is what I suspected, but if it's a symlink to /usr/bin/python, that isn't the issue.
So it appears that the wrappers are using the correct Python, so maybe it is permissions. The qrunners are (or should be) running as the mailman user:group, and the web CGI's should run as the mailman group, but run as the web server user, so that's a difference. Look at the permissions on the /usr/lib/python2.4/site-packages/MySQLdb directory and its subordinates. Directories should be 0755 (drwxr-xr-x) and files 0644 (-rw-r--r--). Is that the case?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Aaron Kreider wrote:
I think that's the problem. Did you install MySQLdb from source or via some RPM or ?
The fact that you have an egg and not a MySQLdb directory implies that this MySQLdb was installed with Python 2.5+, not Python 2.4
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark Sapiro wrote:
Or maybe not. The name of the egg indicates it was built with/for Python 2.4. Is it world readable?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

I found a solution!
Mark Sapiro deserves all the credit for helping me out a TON =)
Here is my documentation in the hope that it will help someone in the future.
Mark writes: The basic problem is that the python library or at least the one containing the site-packages/ into which MySQLdb was installed is /usr/lib64/python2.4 rather than /usr/lib/python2.4.
Mailman starts a lot of Python processes with the -S option which bypasses importing the site module at startup and this is what normally puts the path to site-packages in sys.path. Mailman attempts to do this itself in its own paths module which everything imports, but it assumes the path begins with /usr/lib/pythonv.v so it never got /usr/lib64/python2.4/site-packages into sys.path.
The bit added to extend.py does this when it replaces the MemberAdaptor, so MysqlMemberships can successfully import MySQLdb.
Aaron adds:
We changed extend.py to be:
import sys if '/usr/lib64/python2.4/site-packages' not in sys.path: sys.path.append('/usr/lib64/python2.4/site-packages')
from Mailman.MysqlMemberships import MysqlMemberships
def extend(list): list._memberadaptor = MysqlMemberships(list)
To track down the fact that the module's directory was wrong we had to fix a bug in Mailman's error logging:
There is a bug in the error logging code in /usr/local/cpanel/3rdparty/mailman/scripts/driver
Edit that file. Find
def print_environment(logfp=None): if logfp is None: logfp = sys.__stderr__
try:
import os
except ImportError:
os = None
# Write some information about our Python executable to the log
file. print>> logfp, '[----- Python Information -----]' print>> logfp, 'sys.version =', sys.version print>> logfp, 'sys.executable =', sys.executable print>> logfp, 'sys.prefix =', sys.prefix print>> logfp, 'sys.exec_prefix =', sys.exec_prefix print>> logfp, 'sys.path =', sys.exec_prefix print>> logfp, 'sys.platform =', sys.platform
Change the next to last line of that from
print>> logfp, 'sys.path =', sys.exec_prefix
to
print>> logfp, 'sys.path =', sys.path
and then see what's reported for sys.path in the traceback.
Finally because my Cpanel installation of mailman was creating mailing lists that had "." in their name, and MySQL does not create tables with a period in the name I modified the code for MysqlMemberships.py
A global replace of
self.__mlist.internal_name()
with
self.__mlist.internal_name().replace('.', '_')

So it appears that the wrappers are using the correct Python, so maybe
It looks like my webhost's support installed MySQL-python-1.2.3 in the root directory of my entire server, and it is owned by a user called "500"!
500 is not the permissions level, that is the name of the owner. Should this value be changed?
Directory permissions are 775. Files are 664.

Aaron Kreider wrote:
Did you install Mailman yourself or are you using cPanel's Mailman. If the latter, did you modify the paths in the above instructions to match cPanel.
See the FAQ at <http://wiki.list.org/x/sYA9>.
In a typical CentOS RPM install, /usr/bin/python and /usr/bin/python2.4 are hard links to the same file.
How many /usr/lib/python* directories do you have?
I'm wondering if having two versions might be the cause? Or could there be a user permissions issue?
How was Mailman configured? Having two or even more versions of Python should not be a problem as long as the one Mailman was configured to use has MySQLdb installed.
Is Mailman running? What happens if you send a 'help' command by email to the new list's -request address?
I don't think this is a permissions issue as MySQLdb should be world searchable/readable, but I suppose it could be a SELinux issue if SELinux is enabled.
If you run
strings /path/to/mailman/cgi-bin/listinfo
The output should contain amongst much other stuff the path of the python used by the CGI scripts.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 8/6/2010 1:34 AM, Mark Sapiro wrote:
Only one directory. /usr/lib/python2.4
I changed /user/local/cpanel/3rdparty/mailman/Mailman/mm_cfg.py to have my MySQL database settings:
# Put YOUR site-specific settings below this line. MYSQL_MEMBER_DB_NAME = "the_db" MYSQL_MEMBER_DB_USER = "the_user" MYSQL_MEMBER_DB_PASS = "topsecret MYSQL_MEMBER_DB_HOST = "localhost" MYSQL_MEMBER_TABLE_TYPE = "wide"
Is Mailman running? What happens if you send a 'help' command by email to the new list's -request address?
Sending "help" does not work for the new list.
It does work for old lists (non-mysql based ones).
This gives me an error message: Content-type: text/html <head> </head><body> <h1>Mailman CGI error!!!</h1> <pre> </pre> PYTHONPATH= --with-cgi-gid --with-mail-gid mail PYTHONHOME= <title>Mailman CGI error!!!</title> The Mailman CGI wrapper encountered a fatal error. This entry is being stored in your syslog: Failure to find group name for GID %d. Mailman expected the %s wrapper to be executed as group "%s", but the system's %s server executed the wrapper as GID %d for which the name could not be found. Try adding GID %d to your system as "%s", or tweak your %s server to run the wrapper as group "%s". Group mismatch error. Mailman expected the %s wrapper script to be executed as group "%s", but the system's %s server executed the %s script as group "%s". Try tweaking the %s server to run the script as group "%s", or re-run configure, providing the command line option `%s=%s'. /usr/local/cpanel/3rdparty/mailman/scripts/ /usr/local/cpanel/3rdparty/mailman /usr/local/cpanel/3rdparty/bin/python driver listinfo nobody Mailman cgi-wrapper (listinfo)

Aaron Kreider wrote:
OK
What's in Mailman's error log. Do you see the same ImportError causing the message to be shunted?
It looks as if you ran
/path/to/mailman/cgi-bin/listinfo
not
strings /path/to/mailman/cgi-bin/listinfo
as I asked. I don't want you to run the listinfo wrapper, I want to see what Python it is invoking. I am guessing there is a python in /user/local/cpanel/3rdparty/ and that that is the Python being used by Mailman. I'm trying to confirm that or at least confirm that it is other than /usr/bin/python(2.4). The wrappers contain the path to Python as a string, and that's what I want to see.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark,
No. I see it has an error about the table not being created. So that must mean it is correctly importing the module? Why does it work here and not in other cases? I might need to manually create the tables (one for each list). I'm fine with doing that.
Aug 06 15:18:00 2010 (12201) Uncaught runner exception: (1103, "Incorrect table name 'dbtest8_energyjustice.net'") Aug 06 15:18:00 2010 (12201) Traceback (most recent call last): File "/usr/local/cpanel/3rdparty/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop self._onefile(msg, msgdata) File "/usr/local/cpanel/3rdparty/mailman/Mailman/Queue/Runner.py", line 166, in _onefile mlist = self._open_list(listname) File "/usr/local/cpanel/3rdparty/mailman/Mailman/Queue/Runner.py", line 211, in _open_list mlist = MailList.MailList(listname, lock=False) File "/usr/local/cpanel/3rdparty/mailman/Mailman/MailList.py", line 125, in __init__ func(self) File "/usr/local/cpanel/3rdparty/mailman/lists/dbtest8_energyjustice.net/extend.py", line 4, in extend list._memberadaptor = MysqlMemberships(list) File "/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line 89, in __init__ self.createTable() File "/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line 205, in createTable self.query ( File "/usr/local/cpanel/3rdparty/mailman/Mailman/MysqlMemberships.py", line 241, in query return mm_cfg.cursor.execute (query) File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 174, in execute File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler ProgrammingError: (1103, "Incorrect table name 'dbtest8_energyjustice.net'")
Ok this is what I get.
Content-type: text/html <head> </head><body> <h1>Mailman CGI error!!!</h1> <pre> </pre> PYTHONPATH= --with-cgi-gid --with-mail-gid mail PYTHONHOME= <title>Mailman CGI error!!!</title> The Mailman CGI wrapper encountered a fatal error. This entry is being stored in your syslog: Failure to find group name for GID %d. Mailman expected the %s wrapper to be executed as group "%s", but the system's %s server executed the wrapper as GID %d for which the name could not be found. Try adding GID %d to your system as "%s", or tweak your %s server to run the wrapper as group "%s". Group mismatch error. Mailman expected the %s wrapper script to be executed as group "%s", but the system's %s server executed the %s script as group "%s". Try tweaking the %s server to run the script as group "%s", or re-run configure, providing the command line option `%s=%s'. /usr/local/cpanel/3rdparty/mailman/scripts/ /usr/local/cpanel/3rdparty/mailman /usr/local/cpanel/3rdparty/bin/python driver listinfo nobody Mailman cgi-wrapper (listinfo)
BTW this file is a symbolic link:- /usr/local/cpanel/3rdparty/bin/python It links to /usr/bin/python2.4
Thanks for your help so far!
Aaron

Aaron Kreider wrote:
I'm not sure. see below.
[...]
Sorry. I don't know why I didn't see it the first time. You had done what I asked, and I got confused by Content-Type: being the first line of output (and maybe by your referring to an error). The following is the output I wanted.
This is what I suspected, but if it's a symlink to /usr/bin/python, that isn't the issue.
So it appears that the wrappers are using the correct Python, so maybe it is permissions. The qrunners are (or should be) running as the mailman user:group, and the web CGI's should run as the mailman group, but run as the web server user, so that's a difference. Look at the permissions on the /usr/lib/python2.4/site-packages/MySQLdb directory and its subordinates. Directories should be 0755 (drwxr-xr-x) and files 0644 (-rw-r--r--). Is that the case?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Aaron Kreider wrote:
I think that's the problem. Did you install MySQLdb from source or via some RPM or ?
The fact that you have an egg and not a MySQLdb directory implies that this MySQLdb was installed with Python 2.5+, not Python 2.4
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark Sapiro wrote:
Or maybe not. The name of the egg indicates it was built with/for Python 2.4. Is it world readable?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

I found a solution!
Mark Sapiro deserves all the credit for helping me out a TON =)
Here is my documentation in the hope that it will help someone in the future.
Mark writes: The basic problem is that the python library or at least the one containing the site-packages/ into which MySQLdb was installed is /usr/lib64/python2.4 rather than /usr/lib/python2.4.
Mailman starts a lot of Python processes with the -S option which bypasses importing the site module at startup and this is what normally puts the path to site-packages in sys.path. Mailman attempts to do this itself in its own paths module which everything imports, but it assumes the path begins with /usr/lib/pythonv.v so it never got /usr/lib64/python2.4/site-packages into sys.path.
The bit added to extend.py does this when it replaces the MemberAdaptor, so MysqlMemberships can successfully import MySQLdb.
Aaron adds:
We changed extend.py to be:
import sys if '/usr/lib64/python2.4/site-packages' not in sys.path: sys.path.append('/usr/lib64/python2.4/site-packages')
from Mailman.MysqlMemberships import MysqlMemberships
def extend(list): list._memberadaptor = MysqlMemberships(list)
To track down the fact that the module's directory was wrong we had to fix a bug in Mailman's error logging:
There is a bug in the error logging code in /usr/local/cpanel/3rdparty/mailman/scripts/driver
Edit that file. Find
def print_environment(logfp=None): if logfp is None: logfp = sys.__stderr__
try:
import os
except ImportError:
os = None
# Write some information about our Python executable to the log
file. print>> logfp, '[----- Python Information -----]' print>> logfp, 'sys.version =', sys.version print>> logfp, 'sys.executable =', sys.executable print>> logfp, 'sys.prefix =', sys.prefix print>> logfp, 'sys.exec_prefix =', sys.exec_prefix print>> logfp, 'sys.path =', sys.exec_prefix print>> logfp, 'sys.platform =', sys.platform
Change the next to last line of that from
print>> logfp, 'sys.path =', sys.exec_prefix
to
print>> logfp, 'sys.path =', sys.path
and then see what's reported for sys.path in the traceback.
Finally because my Cpanel installation of mailman was creating mailing lists that had "." in their name, and MySQL does not create tables with a period in the name I modified the code for MysqlMemberships.py
A global replace of
self.__mlist.internal_name()
with
self.__mlist.internal_name().replace('.', '_')

So it appears that the wrappers are using the correct Python, so maybe
It looks like my webhost's support installed MySQL-python-1.2.3 in the root directory of my entire server, and it is owned by a user called "500"!
500 is not the permissions level, that is the name of the owner. Should this value be changed?
Directory permissions are 775. Files are 664.
participants (3)
-
Aaron Kreider
-
Dale Newfield
-
Mark Sapiro