Mailman MySQL backend migration question
Hello,
I am attempting to migrate from using Mailman's flat file to store information about members towards a MySQL database backend instead. I have successfully configured it so that when I add a new member or change a member, it changes appropriately with the database table. The problem is that I have 1500+ people who already exist within the flat file and have preferences stored in that file. I need to migrate the data from the flat file to the mySQL database so that all of the preferences are preserved.
I have looked a little bit into .json files and importing the information in that manner, but I have not found a solution that works for me.
Does anyone have any advice for me on this one?
Thanks in advance!
-Kevin
View this message in context: http://old.nabble.com/Mailman-MySQL-backend-migration-question-tp32000121p32... Sent from the Mailman - Users mailing list archive at Nabble.com.
kphamilton wrote:
I am guessing that by 'flat file' you mean membership data as stored in the list object by the default OldStyleMemberships.py member adaptor.
Again, assuming you have done this in the 'normal' way and have implemented a replacement MemberAdaptor such as one of the MysqlMemberships.py MemberAdaptors from <http://www.orenet.co.uk/opensource/> or <http://trac.rezo.net/trac/rezo/browser/Mailman>, I suggest you create a withlist script similar to the following outline.
from Mailman import mm_cfg from Mailman import OldStyleMemberships as OSM
def move_members(mlist): if not mlist.Locked(): mlist.Lock() for member in OSM.getMembers(mlist): if mlist.isMember(member): continue cpmember = OSM.getMemberCPAddress(mlist, member) digest = OSM.getMemberOption(mlist, member, mm_cfg.Digests) password = OSM.getMemberPassword(mlist, member) language = OSM.getMemberLanguage(mlist, member) realname = OSM.getMemberName(mlist, member) mlist.addNewMember(cpmember, digest=digest, password=password, language=language, realname=realname) mlist.setDeliveryStatus(member, OSM.getDeliveryStatus(mlist, member)) for flag in (mm_cfg.DontReceiveOwnPosts, mm_cfg.AcknowledgePosts, mm_cfg.DisableMime, mm_cfg.ConcealSubscription, mm_cfg.SuppressPasswordReminder, mm_cfg.ReceiveNonmatchingTopics, mm_cfg.Moderate, mm_cfg.DontReceiveDuplicates, ): mlist.setMemberOption(member, flag, OSM.getMemberOption(mlist, member, flag)) mlist.setMemberTopics(member, OSM.getMemberTopics(mlist, member)) OSM.removeMember(mlist, member) mlist.Save() mlist.Unlock()
The above intentionally does not move bounce_info, but could if desired.
The OSM.removeMember(mlist, member) is optional and assumes that none of this data is used. If you don't do it, the list's config.pck will still retain all this information which unnecessarily increases its size an potentially causes confusion
The script isn't too robust. Probably at a minimum, everything from for member ... through mlist.Save() should be in a try: finally: with the mlist.Unlock() in the finally:.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
I forgot to note that the above was untested and without warranty. In particular, the last two lines
mlist.Save()
mlist.Unlock()should only be indented 4 spaces, not 8 as in
mlist.Save()
mlist.Unlock()That change will at least give it a chance of processing more than one old member before dying.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I have actually accomplished this! Woohoo! I did upgrade as you said to MysqlMemberships.py.
I accomplished the rest in a different manner than you suggested, so I will share if others attempt this.
I used export in the /bin folder to export all list data in xml format.
I then wrote a python script that parsed through the xml file for each list and added appropriate members to the appropriate MySQL table. This proved to be fairly tricky as some of the attributes in the tables are not actually what are used to store the information (mostly dealing with the preferences).
While there are fields in the tables for preferences like nomail, ack, plain, etc, these are unused fields. The field that controls most of the preferences is user_options. Thanks to a few tutorials (I am at home now so I do not have any URLs from work), I was able to decipher the values required to save preferences.
Although I used a different strategy, I thank you Mark Sapiro for your attempted help. You actually did help me on this project because you were the poster in a lot of the tutorials I referenced throughout the process.
Thanks again! -Kevin
Mark Sapiro-3 wrote:
-- View this message in context: http://old.nabble.com/Mailman-MySQL-backend-migration-question-tp32000121p32... Sent from the Mailman - Users mailing list archive at Nabble.com.
kphamilton wrote:
But you must have somehow disabled the use of MysqlMemberships.py as the MemberAdaptor for the running of bin/export.py or you would have only gotten those members and settings that were already in the MySQL database since bin/export.py uses the list's MemberAdaptor methods to retrieve these things.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I used the bin/export before I added the extend.py to any of the lists and it retrieved all of the necessary information.
Mark Sapiro-3 wrote:
will
-- View this message in context: http://old.nabble.com/Mailman-MySQL-backend-migration-question-tp32000121p32... Sent from the Mailman - Users mailing list archive at Nabble.com.
kphamilton wrote:
I used the bin/export before I added the extend.py to any of the lists and it retrieved all of the necessary information.
OK. That makes sense.
BTW, just for curiousity, I actually tried a partial test on the withlist suggestion I posted, and I found problems. In case anyone finds this in the archives and wants to try it, here's a script that will be a little closer to working.
from Mailman import mm_cfg from Mailman.OldStyleMemberships import OldStyleMemberships
def move_members(mlist): OSM = OldStyleMemberships(mlist) if not mlist.Locked(): mlist.Lock() for member in OSM.getMembers(): if mlist.isMember(member): continue cpmember = OSM.getMemberCPAddress(member) digest = OSM.getMemberOption(member, mm_cfg.Digests) password = OSM.getMemberPassword(member) language = OSM.getMemberLanguage(member) realname = OSM.getMemberName(member) mlist.addNewMember(cpmember, digest=digest, password=password, language=language, realname=realname) mlist.setDeliveryStatus(member, OSM.getDeliveryStatus(member)) for flag in (mm_cfg.DontReceiveOwnPosts, mm_cfg.AcknowledgePosts, mm_cfg.DisableMime, mm_cfg.ConcealSubscription, mm_cfg.SuppressPasswordReminder, mm_cfg.ReceiveNonmatchingTopics, mm_cfg.Moderate, mm_cfg.DontReceiveDuplicates, ): mlist.setMemberOption(member, flag, OSM.getMemberOption(member, flag)) mlist.setMemberTopics(member, OSM.getMemberTopics(member)) OSM.removeMember(member) mlist.Save() mlist.Unlock()
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
kphamilton wrote:
I am guessing that by 'flat file' you mean membership data as stored in the list object by the default OldStyleMemberships.py member adaptor.
Again, assuming you have done this in the 'normal' way and have implemented a replacement MemberAdaptor such as one of the MysqlMemberships.py MemberAdaptors from <http://www.orenet.co.uk/opensource/> or <http://trac.rezo.net/trac/rezo/browser/Mailman>, I suggest you create a withlist script similar to the following outline.
from Mailman import mm_cfg from Mailman import OldStyleMemberships as OSM
def move_members(mlist): if not mlist.Locked(): mlist.Lock() for member in OSM.getMembers(mlist): if mlist.isMember(member): continue cpmember = OSM.getMemberCPAddress(mlist, member) digest = OSM.getMemberOption(mlist, member, mm_cfg.Digests) password = OSM.getMemberPassword(mlist, member) language = OSM.getMemberLanguage(mlist, member) realname = OSM.getMemberName(mlist, member) mlist.addNewMember(cpmember, digest=digest, password=password, language=language, realname=realname) mlist.setDeliveryStatus(member, OSM.getDeliveryStatus(mlist, member)) for flag in (mm_cfg.DontReceiveOwnPosts, mm_cfg.AcknowledgePosts, mm_cfg.DisableMime, mm_cfg.ConcealSubscription, mm_cfg.SuppressPasswordReminder, mm_cfg.ReceiveNonmatchingTopics, mm_cfg.Moderate, mm_cfg.DontReceiveDuplicates, ): mlist.setMemberOption(member, flag, OSM.getMemberOption(mlist, member, flag)) mlist.setMemberTopics(member, OSM.getMemberTopics(mlist, member)) OSM.removeMember(mlist, member) mlist.Save() mlist.Unlock()
The above intentionally does not move bounce_info, but could if desired.
The OSM.removeMember(mlist, member) is optional and assumes that none of this data is used. If you don't do it, the list's config.pck will still retain all this information which unnecessarily increases its size an potentially causes confusion
The script isn't too robust. Probably at a minimum, everything from for member ... through mlist.Save() should be in a try: finally: with the mlist.Unlock() in the finally:.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
I forgot to note that the above was untested and without warranty. In particular, the last two lines
mlist.Save()
mlist.Unlock()should only be indented 4 spaces, not 8 as in
mlist.Save()
mlist.Unlock()That change will at least give it a chance of processing more than one old member before dying.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I have actually accomplished this! Woohoo! I did upgrade as you said to MysqlMemberships.py.
I accomplished the rest in a different manner than you suggested, so I will share if others attempt this.
I used export in the /bin folder to export all list data in xml format.
I then wrote a python script that parsed through the xml file for each list and added appropriate members to the appropriate MySQL table. This proved to be fairly tricky as some of the attributes in the tables are not actually what are used to store the information (mostly dealing with the preferences).
While there are fields in the tables for preferences like nomail, ack, plain, etc, these are unused fields. The field that controls most of the preferences is user_options. Thanks to a few tutorials (I am at home now so I do not have any URLs from work), I was able to decipher the values required to save preferences.
Although I used a different strategy, I thank you Mark Sapiro for your attempted help. You actually did help me on this project because you were the poster in a lot of the tutorials I referenced throughout the process.
Thanks again! -Kevin
Mark Sapiro-3 wrote:
-- View this message in context: http://old.nabble.com/Mailman-MySQL-backend-migration-question-tp32000121p32... Sent from the Mailman - Users mailing list archive at Nabble.com.
kphamilton wrote:
But you must have somehow disabled the use of MysqlMemberships.py as the MemberAdaptor for the running of bin/export.py or you would have only gotten those members and settings that were already in the MySQL database since bin/export.py uses the list's MemberAdaptor methods to retrieve these things.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I used the bin/export before I added the extend.py to any of the lists and it retrieved all of the necessary information.
Mark Sapiro-3 wrote:
will
-- View this message in context: http://old.nabble.com/Mailman-MySQL-backend-migration-question-tp32000121p32... Sent from the Mailman - Users mailing list archive at Nabble.com.
kphamilton wrote:
I used the bin/export before I added the extend.py to any of the lists and it retrieved all of the necessary information.
OK. That makes sense.
BTW, just for curiousity, I actually tried a partial test on the withlist suggestion I posted, and I found problems. In case anyone finds this in the archives and wants to try it, here's a script that will be a little closer to working.
from Mailman import mm_cfg from Mailman.OldStyleMemberships import OldStyleMemberships
def move_members(mlist): OSM = OldStyleMemberships(mlist) if not mlist.Locked(): mlist.Lock() for member in OSM.getMembers(): if mlist.isMember(member): continue cpmember = OSM.getMemberCPAddress(member) digest = OSM.getMemberOption(member, mm_cfg.Digests) password = OSM.getMemberPassword(member) language = OSM.getMemberLanguage(member) realname = OSM.getMemberName(member) mlist.addNewMember(cpmember, digest=digest, password=password, language=language, realname=realname) mlist.setDeliveryStatus(member, OSM.getDeliveryStatus(member)) for flag in (mm_cfg.DontReceiveOwnPosts, mm_cfg.AcknowledgePosts, mm_cfg.DisableMime, mm_cfg.ConcealSubscription, mm_cfg.SuppressPasswordReminder, mm_cfg.ReceiveNonmatchingTopics, mm_cfg.Moderate, mm_cfg.DontReceiveDuplicates, ): mlist.setMemberOption(member, flag, OSM.getMemberOption(member, flag)) mlist.setMemberTopics(member, OSM.getMemberTopics(member)) OSM.removeMember(member) mlist.Save() mlist.Unlock()
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
kphamilton -
Mark Sapiro