Storm based MemberAdaptor for Mailman
Hi all
I am working on writing a storm based member adaptor for mailman so that the mailman membership data can be stored in a database instead of .pck files. The reason for choosing storm is that it provides an abstraction to use any underlying db language- either mysql, postgresql or sqllite.
I have created a branch on launchpad for this adaptor and would be great if I can get a code review of the adaptor ,t he schema of the db that I am using for storing memberships data and an opinion of whether such a member adaptor is desired at all. Presently the storm adaptor is written for Postgresql but it will support mysql and sqllite with slight modifications.
Please take a look at *Mailman/PgsqlMemberships.py* at * https://code.launchpad.net/~malveeka/+junk/StormMemberAdaptor* under the first revision.
The work is under progress and for now, I using to storm to create a database with memberships data alongwith the pickle files. The Mailman/PgsqlMemberships.py subclasses OldStyleMemberships.py and still uses the accessor methods in OldStyleMemberships instead of accessing the data from the postgres db.
The ideal case would be to use only a database and no pickle files for Memberships data but I have not reached there. I had tried to read and use the data from the database instead of pickle files and that had broken my Mailman which leaves me with few questions
In OldStyleMemberships.py the lower cased email address is used as a key for accessing the membership properties. However in my schema, I am using the (listname, case preserved email address) as the PK. Is it possible that not storing and using LCE as a key might break something.
I also want to make sure that in the database I am caturing all the Memberships data. Presently my database uses the following class as a storm abstraction for the database. Do I need to add/remove anything?
class PgsqlMembers(object): __storm_table__ = "mailman_test" __storm_primary__ = "listname","address"
listname = Unicode()
address = Unicode()
password = Unicode()
lang = Unicode()
name = Unicode()
digest = Unicode()
delivery_status = Int()
user_options = Int()
topics_userinterest = Unicode()
bounce_info = Unicode()
delivery_status_timestamp = Unicode()Looking forward to your reviews and finally getting the thing in place!!
Thanks a lot all Malveeka
On Aug 8, 2009, at 4:25 AM, Malveeka Tewari wrote:
I am working on writing a storm based member adaptor for mailman so
that the mailman membership data can be stored in a database instead of .pck
files. The reason for choosing storm is that it provides an abstraction to
use any underlying db language- either mysql, postgresql or sqllite.
I'm a big fan of Storm. It's the ORM used in Mailman 3 and I've had a
lot of good success with it.
Please note that this new member adapter cannot go officially into
Mailman 2.1, but I do think it might be useful for Mailman 2.2,
probably as unofficial contrib, though Mark may want to weigh in on
that. Because the schema is so different in Mailman 3 and we already
use Storm, this won't be relevant for that branch.
The ideal case would be to use only a database and no pickle files for Memberships data but I have not reached there. I had tried to read and use the data from the database instead of
pickle files and that had broken my Mailman which leaves me with few
questions
Are you using Pickle() fields for non-scalar data types? (i.e. lists
and dictionaries). Of course, you don't gain from relational data
that way, but it's still useful.
In OldStyleMemberships.py the lower cased email address is used as a
key for accessing the membership properties. However in my schema, I am using the (listname, case preserved email address) as the PK. Is it possible that not storing and using LCE as a key might break something.
From OldStyleMembership's (very loose) contract, I think the answer
is "yes". It doesn't matter so much how you store things in the
database, but you will have to honor the contract that the
MemberAdapter.py promises.
I also want to make sure that in the database I am caturing all the Memberships data. Presently my database uses the following class as
a storm abstraction for the database. Do I need to add/remove anything?class PgsqlMembers(object): __storm_table__ = "mailman_test" __storm_primary__ = "listname","address"
listname = Unicode() address = Unicode() password = Unicode() lang = Unicode() name = Unicode() digest = Unicode() delivery_status = Int() user_options = Int() topics_userinterest = Unicode() bounce_info = Unicode() delivery_status_timestamp = Unicode()
When I was creating the schemas for MM3, I basically had to inspect
OldStyleMembership.py to see what fields it requires, then fix things
as tests broke. MM2 has the great disadvantage that there isn't a
usable test suite. This is fixed in MM3. So I think you're left to
manual testing until it basically works for you. :(
-Barry
Barry Warsaw quoted:
On Aug 8, 2009, at 4:25 AM, Malveeka Tewari wrote:
I also want to make sure that in the database I am caturing all the Memberships data. Presently my database uses the following class as
a storm abstraction for the database. Do I need to add/remove anything?class PgsqlMembers(object): __storm_table__ = "mailman_test" __storm_primary__ = "listname","address"
listname = Unicode() address = Unicode() password = Unicode() lang = Unicode() name = Unicode() digest = Unicode() delivery_status = Int() user_options = Int() topics_userinterest = Unicode() bounce_info = Unicode()
This has caused problems with the MySQL MemberAdaptor in the past. bounce_info is a _BounceInfo class instance that is supposed to be opaque to the MemberAdaptor. The issue with the MySQL MemberAdaptor was that it 'knew' what the attributes of this instance were and stored them separately in the database, but then the class grew another attribute and the MemberAdaptor failed.
The problem is that the methods setBounceInfo() and getBounceInfo() have to know something about the _BounceInfo class in order to be able to store something in the database that can later be returned as a class instance.
The MySQL MemberAdaptor does this by hving setBounceInfo() store the attributes separately and getBounceInfo() instantiate the class with the retrieved attributes, but this fails if the number of arguments expected by _BounceInfo.__init__() changes.
An alternative is to pickle the class instance into a string to be stored in the database, but this then loses the ability to see the various bounce attributes in the database.
delivery_status_timestamp = Unicode()
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Aug 10, 2009, at 11:55 AM, Mark Sapiro wrote:
Barry Warsaw quoted:
On Aug 8, 2009, at 4:25 AM, Malveeka Tewari wrote:
I also want to make sure that in the database I am caturing all the Memberships data. Presently my database uses the following class as a storm abstraction for the database. Do I need to add/remove anything?
class PgsqlMembers(object): __storm_table__ = "mailman_test" __storm_primary__ = "listname","address"
listname = Unicode() address = Unicode() password = Unicode() lang = Unicode() name = Unicode() digest = Unicode() delivery_status = Int() user_options = Int() topics_userinterest = Unicode() bounce_info = Unicode()
This has caused problems with the MySQL MemberAdaptor in the past. bounce_info is a _BounceInfo class instance that is supposed to be opaque to the MemberAdaptor. The issue with the MySQL MemberAdaptor was that it 'knew' what the attributes of this instance were and stored them separately in the database, but then the class grew another attribute and the MemberAdaptor failed.
The problem is that the methods setBounceInfo() and getBounceInfo() have to know something about the _BounceInfo class in order to be able to store something in the database that can later be returned as a class instance.
The MySQL MemberAdaptor does this by hving setBounceInfo() store the attributes separately and getBounceInfo() instantiate the class with the retrieved attributes, but this fails if the number of arguments expected by _BounceInfo.__init__() changes.
This has been a PITA in MM3 too, and not something I've totally
unfscked yet.
An alternative is to pickle the class instance into a string to be stored in the database, but this then loses the ability to see the various bounce attributes in the database.
The nice thing is that Storm has a Pickle() type that makes it easy.
But you're right, doing this is only a stopgap and makes the bounce
information opaque to other proceses.
-Barry
participants (3)
-
Barry Warsaw -
Malveeka Tewari -
Mark Sapiro