Re: [Mailman-Developers] More on Mysql MemberAdaptor?
On Wed, 5 Nov 2003 16:31:51 +0800 "Yuan, Cain" <cain.yuan@intel.com> wrote:
Hi Kev, I had a look at the patch and have some questions here: why you establish a connection to MySQL and then close it in each function? AFAIK this will put much presure on MySQL if there are some people to subscribe/unscribe from the list. Why not just establish a ststic link to MySQL and then use it every time you need to access the DB?
Seemed like a good idea at the time ;*)
But seriously, doing it that way means I don't really have to worry about connection timeouts, etc. Maybe it can be rewritten later to work differently (like when I learn to write better python ;*), but for the most part, it will work now. I have cut down the number of connect() calls a little already though, but not as far as perhaps I could go.
I've updated the whole thing a lot since both of those emails last night.
The remaining problems, though, are that I still haven't cracked {get,set}DeliveryStatus(for the reasons previously detailed), and{get,set}BounceInfo, which is another problem.
In the MySQL database I end up with the following string:
<bounce info for member bounceybouncey@orenet.co.uk current score: 1.0 last bounce date: (2003, 11, 4) email notices left: 3 last notice date: (1970, 1, 2) confirmation cookie: a9e8f042d93d3777da981f353f2e00ade42f6f
I'm putting the "info" parameter from setBounceInfo directly into the database, which I think is an array itself, not a single value, and the above doesn't look like Python's just traversing an array, and dumping it into the database(the LHS names don't tie up with what I think are the keys for the subelements of "info"), so it looks like I'll have to take a "best guess" at how to implement this.
If anyone has any useful suggestions, I'd appreciate them.
K.
-----Original Message----- From: mailman-developers-bounces+cain.yuan=intel.com@python.org [mailto:mailman-developers-bounces+cain.yuan=intel.com@python.org]On Behalf Of Kyrian (List) Sent: Wednesday, November 05, 2003 2:04 AM To: mailman-developers@python.org Subject: [Mailman-Developers] More on Mysql MemberAdaptor?
Further to my previous message, I've now got it reading config from mm_cfg.py, verifying the connectivity to the mysql database on the __init__ call, rather than throwing up(numerous) errors later on, updated the README to reflect relevant changes, and just generally cleaned things up a bit. That's RCS revision 1.13.
The only major problem I have is the return types for {get,set}DeliveryStatus. I can't work out how I'm suppose to return a tuple of values, and what they should be.
If someone could assist I'd be grateful :-)
K.
Begin forwarded message:
Date: Tue, 4 Nov 2003 15:27:45 +0000 From: Kyrian (List) <kyrian-list@ore.org> To: mailman-developers@python.org Subject: Mysql MemberAdaptor?
Hi All,
I'm presuming this is the appropriate list to post this to...
If anyone cares, I've written a Mysql MemberAdaptor based on the OldStyleMemberships.py module, which seems to work ok. I've not done much large scale testing as yet, though.
I've put it up at http://kyrian.ore.org/MailmanMysql/
Although I could use some pointers on the following:
How to incorporate exception handling in python to trap DB errors, and stop Mailman choking on them.
How to incorporate some better configuration (you currently would have to edit the module file directly to specify the database parameters)
How to properly incorporate it into mailman (if nobody minds that ;), as it currently seems to require modifying MemberAdaptor.py directly to activate it.
Whether I've actually done it even half way right?
Either way, if anyone has anything to say about it, please go easy, I kinda needed this thing, and delved into Python for the first time to do so.
Oh, and I know the MySQL data structure I'm using is pretty atrocious, as it was a best-guess, though I can always clean it up later...
I do hope I've not just spent several days reinventing the wheel here, though... ;*)
Yours,
Kev.
-- Kev Green, aka Kyrian. "Be excellent to each other" -- Bill & Ted. Email: kyrian@ore.org Web: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/-- Kev Green, aka Kyrian. "Be excellent to each other" -- Bill & Ted. Email: kyrian@ore.org Web: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
-- Kev Green, aka Kyrian. E: kyrian@ore.org WWW: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/ "A superficial episode, as life continues to unfold" -- Bad Religion
Kev Green, aka Kyrian. "Be excellent to each other" -- Bill & Ted. Email: kyrian@ore.org Web: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/
I'm trying to catch up on this thread, so apologies if these questions are already answered...
On Wed, 2003-11-05 at 06:11, Kyrian wrote:
But seriously, doing it that way means I don't really have to worry about connection timeouts, etc. Maybe it can be rewritten later to work differently (like when I learn to write better python ;*), but for the most part, it will work now. I have cut down the number of connect() calls a little already though, but not as far as perhaps I could go.
I've been using the MySQLdb wrapper for Python recently in a different project and here's how I've dealt with connection timeouts. I have a "doquery" method which essentially calls connection.query() with the SQL statement. Wrap that call in try/except for MySQLdb.OperationalError, catch the exception object and test the error code against MySQLdb.constants.CR.SERVER_LOST. If they match, just chuck and re-open the connection. If they don't, just re-raise the exception.
Untested code example:
from MySQLdb import OperationalError from MySQLdb.constants.CR import SERVER_LOST
def _doquery(self, sqlcmd):
try:
self._conn.query(sqlcmd)
except OperationalError, e:
code, msg = e
if code <> SERVER_LOST: raise
self._open()<bounce info for member bounceybouncey@orenet.co.uk current score: 1.0 last bounce date: (2003, 11, 4) email notices left: 3 last notice date: (1970, 1, 2) confirmation cookie: a9e8f042d93d3777da981f353f2e00ade42f6f
I'm putting the "info" parameter from setBounceInfo directly into the database, which I think is an array itself, not a single value, and the above doesn't look like Python's just traversing an array, and dumping it into the database(the LHS names don't tie up with what I think are the keys for the subelements of "info"), so it looks like I'll have to take a "best guess" at how to implement this.
My suggestion would be to pickle the BounceInfo object on the way into the database, and unpickle it on the way out.
-Barry
(Moved just onto the developers list, in case people are getting annoyed with the duplicate messages, and on the assumption we're all on it)
On Mon, 10 Nov 2003 11:36:38 -0500 Barry Warsaw <barry@python.org> wrote:
I'm trying to catch up on this thread, so apologies if these questions are already answered... np.
Had me worried there for a while though.
Untested code example:
from MySQLdb import OperationalError from MySQLdb.constants.CR import SERVER_LOST
def _doquery(self, sqlcmd): try: self._conn.query(sqlcmd) except OperationalError, e: code, msg = e if code <> SERVER_LOST: raise self._open()
Heh.
Excceptions make me nervous.
FWIW. I think the ping() based implementation is probably the best one to go for, because it doesn't raise any exceptions (so say the MySQL[db] docs, anyway), and it automagically reconnects internally to MySQL's own code if it can. No point in making work for ourselves, eh? ;-)
Although, importing the MySQLdb errorcodes is a good idea, and perhaps I'll make use of them in a future version.
All of which reminds me, must document the fact that if you change the MySQL connection details, (I think) you need to restart Mailman.
Anyways, thanks for the responses.
If you're not totally appauled by the code quality, and you think it'll work, let me know, and (if necessary) I'll upload it to the sourceforge site patches section, from which you may apply a GPL license to it (as far as I'm concerned, and I think it's safe from the perspective of the related software, etc.), and do what you wish with it, as long as it's appropriately credited, and, for preference continues to include a link to http://www.orenet.co.uk/ in it.
K.
-- Kev Green, aka Kyrian. "Be excellent to each other" -- Bill & Ted. Email: kyrian@ore.org Web: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/
On Mon, 2003-11-10 at 12:22, Kyrian wrote:
Excceptions make me nervous.
Exceptions are your friend! They're so warm and cuddly they shouldn't frighten anyone. :)
FWIW. I think the ping() based implementation is probably the best one to go for, because it doesn't raise any exceptions (so say the MySQL[db] docs, anyway), and it automagically reconnects internally to MySQL's own code if it can. No point in making work for ourselves, eh? ;-)
It all depends. Catching the exception might actually be less work because you're assuming that most of the time the connection is good, so you don't need the extra ping. You only need it when the connection is timed out.
If you're not totally appauled by the code quality, and you think it'll work, let me know, and (if necessary) I'll upload it to the sourceforge site patches section, from which you may apply a GPL license to it (as far as I'm concerned, and I think it's safe from the perspective of the related software, etc.), and do what you wish with it, as long as it's appropriately credited, and, for preference continues to include a link to http://www.orenet.co.uk/ in it.
Cool, thanks. Do upload it. If we decide to use it, we'll probably need some kind of more formal disclaimer. In the meantime, it's a useful place for others to find it.
-Barry
Excceptions make me nervous.
Exceptions are your friend! They're so warm and cuddly they shouldn't frighten anyone. :)
FWIW. I think the ping() based implementation is probably the best one to go for, because it doesn't raise any exceptions (so say the MySQL[db] docs, anyway), and it automagically reconnects internally to MySQL's own code if it can. No point in making work for ourselves, eh? ;-)
It all depends. Catching the exception might actually be less work because you're assuming that most of the time the connection is good, so you don't need the extra ping. You only need it when the connection is timed out.
Swings and roundabouts I guess. ping() is a pretty low overhead call as I understand it, so there's not much in it.
Without connection cacheing though, it takes three times longer to initialize a list and start mailing (that's for a 60,000 member list).
If you're not totally appauled by the code quality, and you think it'll work, let me know, and (if necessary) I'll upload it to the sourceforge site patches section, from which you may apply a GPL license to it (as far as I'm concerned, and I think it's safe from the perspective of the related software, etc.), and do what you wish with it, as long as it's appropriately credited, and, for preference continues to include a link to http://www.orenet.co.uk/ in it.
Cool, thanks. Do upload it. Done.
It's in as "[ 839386 ] MySQL MemberAdaptor for Mailman 2.1".
If we decide to use it, we'll probably need some kind of more formal disclaimer. Shouldn't be a problem, as long as I don't have to guarantee it'll work flawlessly ;-)
IIRC the only code that I've used in anything approaching a direct cut+paste job came from Mailman's CVS anyway, so it should all be fine.
In the meantime, it's a useful place for others to find it. Indeed.
K.
-- Kev Green, aka Kyrian. E: kyrian@ore.org WWW: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/ "Love is that condition in which the happiness of another person is essential to your own." R. Heinlein, Stranger in a Strange Land, 1961.
Kev Green, aka Kyrian. "Be excellent to each other" -- Bill & Ted. Email: kyrian@ore.org Web: http://kyrian.ore.org/ ISP/Perl/PHP/Linux/Security Contractor, via http://www.orenet.co.uk/
participants (2)
-
Barry Warsaw -
Kyrian