[Barry A. Warsaw]
"SH" == Steven Hazel <cherub@azrael.dyn.cheapnet.net> writes:
Me> The problem here is that we don't actually keep that Me> information around in the database. This is bogus, and is Me> something that I hope will get fixed when we have a real User Me> database (don't ask when ;) SH> What are the current plans for this? I'd like to help. I SH> should have some time over the coming holidays.Well, there are a number of questions (if it was an easy hack it'd already be done :).
This is my cue, I guess ;)
First, do we want to use a real OODB underneath (such as ZODB), or do we cook our own?
I tried using ZODB at first, but found that my try at that scaled very poorly (The FileStorage file was ~160MB even before there were 2000 entries in my database, and the time needed to open the database seemed to scale with the size of it...). Dunno if this was due to me being boneheaded when implementing stuff, or to actual problems with ZODB, but I kinda lost my spirit for it anyway :-\
So, I surfed to Parnassos, searched the vault there, and came up with a copy of bplustree.py. Implemented purely in Python, as opposed to ZODB, which relies on some C Zope modules. The disk space used per record seems very reasonable. Open time appears to be pretty much unrelated to file size. Record access time seems fast enough. One problem, though -- this module does not handle multiple simultanous writes, so we have to be careful with locking -- which *could* make the user db into a bottleneck.
[ As we put Mailman into production with the ~3.500 lists we have here at uio.no, we discovered that the current locking code didn't work very well on a system that got so busy it started swapping -- so I rewrote the entire file locking module to be more robust, and added some code to have the "post" mail injection script exit with temporary error codes if it couldn't get a lock on the list within a reasonable amount of time -- leaving it to the MTA to try injecting the mail again later. Things have been running smooth ever since. ]
A fresh entry (created to be added as member of LIST-A) in the resulting user db have these attributes:
{'admin_for': {}, 'admin_pwd': None, 'aliases': {'ADDRESS@DOMAIN': 1}, 'member_of': {'LIST-A': 1}, 'member_pwd': 'PASSWORD', 'preferred_address': 'ADDRESS@DOMAIN'}
Currently 'aliases' and 'preferred_address' isn't used, but the intention is to use them for associating several mail addresses with the same user account (probably by some means of email confirmation).
'member_pwd' is stored in cleartext, while 'admin_pwd' (when set) is stored as a MD5 digest.
An "username" field has been added to all login pages -- currently we're running with this as a text entry field, but I'll probably install some changes to use a <SELECT> menu of the list's admins instead (mainly to reduce the amount of typing an admin has to do when logging in, but also partly due to the fact that it's common for a browser to SUBMIT on pressing Enter in a text field iff it's the only one in the form).
Logins are still done per list (but with your own admin password), as the cookie secret business has been a bit complicated by all this -- I'm planning to have a cron job generating random list- and admin-independent secrets ever so often, and use these when issuing new cookies -- thus removing the complications and making cross-list single login viable.
It is all rather non-generalized (and part of the code is... uhmmm... not quite as clean as could be wished for), but it has been working fine for over a month now (modulo the various bugfixes I've made along the way, of course). I agree with those wanting a generalized API to use whatever database backend appropriate -- but with all these local changes I'm feeling rather removed from the rest of Mailman development already, so I don't think I'm likely to start working on that soonish.
I'm not sure how it's best to proceed with all this from here -- I think the stuff I've made could prove usable for others beside me, and I'd sure love to get my CVS tree closer to being in sync with the rest of you. However, if I just go ahead and check stuff in, that might mean pushing 1.2 a fair bit away.
So, for now, I've made my current development tree (which, Right Now, isn't quite the code we have in production here; I've been holding off installing changes since I started implementing the login-with-menu stuff mentioned above. As some list admins might not want to be listed as list admins this openly, I've asked for opinions before installing that change) available at <URL: http://www.uio.no/~hmeland/tmp/mailman-userdb/>, or if one prefers to download it all as a single .tar.gz: <URL: http://www.uio.no/~hmeland/tmp/mailman-userdb.tar.gz>. This tree should be up-to-date with current CVS mailman. I'd appreciate it a lot if anyone have the time to have a look at it tell me what they think.
I'll try explaining all this in a more structured fashion later, but please don't hesitate to ask any specific questions you might have.
[ Apart from the user database stuff, the tree has a fair bit of other new stuff in it as well.
Perhaps most notable is "meta-members", i.e. special list members which are dynamically expanded to real mail addresses at list post time. Currently there is only one type of meta-member, MailmanList, which expands to the appropriate set of members (digest or regular members, based on whether what kind of post this is) on the given list. Other types are planned.
Other nice stuff is the ability to name lists with full email addresses internally (useful when housing lists in several mail domains), the use of a cache for building the list overview pages, and (useful when having lots of lists, to avoid loading _all_ of them to build the overview), the "this list is busy, return incoming mail to MTA" hack, and bin/move_to has been extended so it can create a copy of list X with another name Y -- to help renaming lists internally. ]
-- Harald