structural problem with MemberAdapter
Good news, bad news: good news is I've implemented a LDAP based membership adapter that implements the complete MemberAdapter interface (the previous LDAP adapter was incomplete and limited in application). In stand alone testing my adapter seems to work quite nicely.
Skip to the last paragraph if you don't need or want the details.
Now the bad news, I'm having trouble integrating it because of either structural problems with adapters or a lack of understanding of how adapters are supposed to work. I'm looking for guidance or insight. Here are the issues I've run into:
- Adapters are discovered by the presence of extend.py in the list's directory. How does extend.py get into the list directory? The only way I could figure was manual copying after list creation, this isn't a viable mechanism for a running site.
Also, while I can see the value of per list adapters that seems like a niche case, a more common scenario would be every list resident at a site would use the same adapter module. To solve this problem I added code in the MailList constructor to look for a mm_cfg variable defining a global adapter module. But list creation/instantiation is a two step process, MailList object construction in __init__, followed by MailList.Create(). The adapter is not loaded in __init__ unless a name is provided, which is only done after Create() has been called.
Most importantly an adapter cannot correctly initialize itself until the work of Create() is complete. This is very important, but more on this later.
This sequence in TestBase.py leaves the adapter unloaded and uninitialized, it does not work.
mlist = MailList.MailList()
mlist.Create('_xtest', 'test@dom.ain', 'xxxxx')This idiom below which is found in most of the code does work because the list has already been created, but now we're back to the chicken and egg problem of when does extend.py get into the list directory.
mlist = MailList.MailList(listname, lock=0)This is a good moment to go back to the issue of per list adapter initialization (or rather the lack of it). After the mlist is instantiated as above it might very well call an adapter method, for example mlist.addNewMember(). But the adapter has never been called as part of list creation. If the adapter must create entities in a backend database to service the list data it has never been given that opportunity. Thus the adapter is dependent on asking the question "does this list exist in my database, if not create it" every time the adapter object is instantiated, this is not only a big performance problem but it completely fails with the calling sequence in TestBase.py.
If you've followed along so far pat yourself on the back :-) here's the payoff :-)
Summary:
Unless I'm out to lunch with a horrible misunderstanding I propose 3 changes:
Provide a site wide variable that specifies an adapter for every list which is overridden by the presence of extend.py in the list directory. If the site wide adapter is used then MailList.__init__() loads it and calls an initialization function in the module (extend()??, __init__()??)
At the end of MailList.Create() it should test for the existence of an adapter and if so it should attempt to call the adapters Create() method.
Just as the adapter is not informed of list creation is is not informed of list destruction either, calls need to be added to invoke Remove() as well.
I'm happy to provide patches for all this, I'm not interested in suggesting anyone else do more work, I'm mostly interested in knowing if I'm on the right track or if I have a misunderstanding.
PostScript (or a few more questions):
Now after having fully implemented the adapter interface I have to admit I don't really understand what its buying you over the existing OldMemberAdapter. My initial thought was to capitalize on existing user information at a site, but given the way mailman data is structured (a set of lists, each list may contain both local and unknown foreign users, and user properties are per list) then there seems to be little value in intermingling site user data and mailman list data.
Also, it was not clear how an adapter might implement just a subset of the methods via inheritance, I suppose it would copy the function pointers from the mlist._memberadapter into its own methods before resetting mlist._memberadapter to itself. yes/no?
-- John Dennis <jdennis@redhat.com>
On Wed, 2005-09-21 at 18:07, John Dennis wrote:
Good news, bad news: good news is I've implemented a LDAP based membership adapter that implements the complete MemberAdapter interface (the previous LDAP adapter was incomplete and limited in application). In stand alone testing my adapter seems to work quite nicely.
Skip to the last paragraph if you don't need or want the details.
Now the bad news, I'm having trouble integrating it because of either structural problems with adapters or a lack of understanding of how adapters are supposed to work. I'm looking for guidance or insight. Here are the issues I've run into:
John, I think you understand the current state of affairs exactly right. At the time I designed the extend.py stuff, it wasn't a requirement to support site-wide list specialization.
- Adapters are discovered by the presence of extend.py in the list's directory. How does extend.py get into the list directory? The only way I could figure was manual copying after list creation, this isn't a viable mechanism for a running site.
Right. I was thinking you might be able to hook into the create() and remove() methods in the MTA modules, but that doesn't work for several reasons. First, you've have to implement the same code for each module, and second, the call to those functions doesn't really happen at the right time.
Provide a site wide variable that specifies an adapter for every list which is overridden by the presence of extend.py in the list directory. If the site wide adapter is used then MailList.__init__() loads it and calls an initialization function in the module (extend()??, __init__()??)
At the end of MailList.Create() it should test for the existence of an adapter and if so it should attempt to call the adapters Create() method.
Just as the adapter is not informed of list creation is is not informed of list destruction either, calls need to be added to invoke Remove() as well.
You might consider instead creating these functions in the Mailman/Site.py module. The intention of that module is to implement site-wide customizations (or at least, provide the hooks for that). Right now, there's nothing in there that hooks into the list creation and instantiation process, but it definitely could.
You could add these functions:
* list_create(): called when a new mailing list is first created.
* list_open(): called when an existing mailing list is opened
(i.e. its __init__() is called).The above each take the MailList instance and can stuff new methods into the instance to override any behavior you want. I don't think you'll need to hook into MailList.Load() or MailList.Save() because those you'll do in your site-wide adapter.
Also, hooking into list deletion is trickier because as you can see in bin/rmlist and Mailman/Cgi/rmlist.py, they don't actually call anything on the MailList instance before the low-level removal procedure is performed (i.e. unlinking and such). That's a flaw that would take more work to fix, because what you'd really want to do is move the bulk of that code into the MailList class, and have those two other files call something like MailList.Delete(). The latter would then be hooked by a Site.py list_delete() function.
I'm happy to provide patches for all this, I'm not interested in suggesting anyone else do more work, I'm mostly interested in knowing if I'm on the right track or if I have a misunderstanding.
I think you're on the right track. Hopefully the above is helpful.
PostScript (or a few more questions):
Now after having fully implemented the adapter interface I have to admit I don't really understand what its buying you over the existing OldMemberAdapter. My initial thought was to capitalize on existing user information at a site, but given the way mailman data is structured (a set of lists, each list may contain both local and unknown foreign users, and user properties are per list) then there seems to be little value in intermingling site user data and mailman list data.
Also, it was not clear how an adapter might implement just a subset of the methods via inheritance, I suppose it would copy the function pointers from the mlist._memberadapter into its own methods before resetting mlist._memberadapter to itself. yes/no?
I'm not sure I understand exactly what you're trying to do here. The intention was that using a different adapter was an all-or-nothing proposition. I.e. if you were going to use an LDAP or MySQL adapter, then you wouldn't use any of the config.pck based OldMemberAdapter. I have my doubts that Mailman would be able to intermix the two.
-Barry
Thank you Barry for the thoughtful response. Sorry for the delayed response on my end.
On Sat, 2005-09-24 at 18:50 -0400, Barry Warsaw wrote:
[ implementation issues snipped for brevity ]
Now after having fully implemented the adapter interface I have to admit I don't really understand what its buying you over the existing OldMemberAdapter. My initial thought was to capitalize on existing user information at a site, but given the way mailman data is structured (a set of lists, each list may contain both local and unknown foreign users, and user properties are per list) then there seems to be little value in intermingling site user data and mailman list data.
Also, it was not clear how an adapter might implement just a subset of the methods via inheritance, I suppose it would copy the function pointers from the mlist._memberadapter into its own methods before resetting mlist._memberadapter to itself. yes/no?
I'm not sure I understand exactly what you're trying to do here. The intention was that using a different adapter was an all-or-nothing proposition. I.e. if you were going to use an LDAP or MySQL adapter, then you wouldn't use any of the config.pck based OldMemberAdapter. I have my doubts that Mailman would be able to intermix the two.
Here was my thinking:
Without adding new list creation/deletion hooks previously discussed above but elided for brevity alternate MemberAdapters are of marginal value.
After fully implementing the MemberAdapter interface with an alternate backend I don't see any particular advantage over the existing python pickle model.
The seductive enticement of a MemberAdapter would seem to be for it to integrate with an organizations existing user database. But because mailing list members can be both local (member of the organization) or foreign (external to the organization) a MemberAdapter whose view is only of local users is restrictive. This is why I thought perhaps the per list extend.py was invented, because certain lists might be "local" only.
MemberAdaptor wants to organize users as email addresses in a list. As far as I can tell the same user can belong to several lists, the user information is unique to the list, its not shared between lists (i.e. a list member does not point to a member record). This model also permits the same user having unique preferences per list and to use different email addresses per list. The data organization tends to be opposite of how organizations model their user data whereby one record exists per user sharing as much common data about that user as is possible. Trying to map on data model onto the other rapidly becomes awkward and does not address the issue of foreign list members.
What did seem to make sense to me was a hybrid model where some per user data (e.g. fullname, password, email address) and list membership was kept in the organizations database. But mailman specific list information (e.g. bounce info, digest, flags, etc.) are kept in mailman's database (e.g. pickle). The interface to MemberAdaptor would be a object inheritance model, the MemberAdaptor is a subclass of OldMembershipAdaptor. This would allow one to pick which methods to override and then delegate to the superclass those it did not want to handle. For example a subclassed MemberAdaptor might choose to implement getMemberName(), authenticateMember(), etc. but leave handling of bounce info, mailman flags, etc. to the parent adaptor. I did make a go at implementing this "inherited model" and it seems to work just fine (albeit very limited testing). What I did was to copy the function pointers from the existing member adaptor when my adaptor was instantiated. For any function I wanted to delegate to the parent I just called the saved function pointer, otherwise I called my own routine. This was a nice mix and match solution.
However, for what its worth after having done all this work (a full LDAP member adaptor, and then the subclassed version) we've decided to take an entirely different direction and we no longer have need for these pieces of code. I would like to contribute what I've done as a patch on the SF site. The only problem is its a 95% solution, the list creation/deletion hooks are the missing 5% and I doubt I'll finish that work. I wonder if 95% is useful to people.
-- John Dennis <jdennis@redhat.com>
participants (2)
-
Barry Warsaw -
John Dennis