Load-balancing Mailman in LVS cluster
Hi there,
I'm a new user of Mailman, and a new subscriber to this list.
I want to set up Mailman in a load-balanced fashion on two servers running within an LVS (linux virtual server) cluster. I want the two servers to act as the same Mailman instance. The two servers will mount an NFS shared directory from a third server, and the NFS mount will, I guess, contain the shared Mailman stuff. I've found a few things on the web about load-balancing Mailman, but not much.
The following directories exist in $prefix/mailman: archives * bin cgi-bin cron data * icons lists * locks * logs mail Mailman * messages pythonlib qfiles scripts spam templates tests
Can anybody suggest which of these directories should be NFS shared between the two servers? To me, it looks like good candidates would be those that I have marked with an * ...
Further, is anyone running Mailman in a load-balanced fashion like this? Any hints/tricks/tips?
Thanks in advance, Guy.
At 2:10 PM +1000 2004-06-29, Guy Waugh wrote:
I want to set up Mailman in a load-balanced fashion on two servers running within an LVS (linux virtual server) cluster. I want the two servers to act as the same Mailman instance.
Okay, so far.The two servers will mountan NFS shared directory from a third server, and the NFS mount will, I guess, contain the shared Mailman stuff.
There's a problem. NFS sucks at operations that require lots of file locking. Of course, all queueing type operations involving mail servers involve lots of file locking.
This is why you do not ever put /var/spool/mqueue on NFS, whereas you can find ways to kinda-semi-sorta deal with the NFS locking problem for user mailboxes on /var/spool/mail (all solutions to this problem involve trade-offs that hurt performance in other areas).
Likewise, Mailman does file locking with the queueing process that it manages.
If you can find a way to put things on a proper clustered filesystem, that should work much better. Sistina GFS may work for you, or possibly Veritas VxFS plus their VCS tools.
I've found a few things on theweb about load-balancing Mailman, but not much.
The thing is that you can run the web server for Mailman separate from the mail servers. So, you could cluster the mail servers in a scalable way that will work for you, and then separately work on the issue of building a scalable web farm solution for the rest.
Further, is anyone running Mailman in a load-balanced fashion like this? Any hints/tricks/tips?
If you are interested in splitting of the mail cluster part of this equations, then I suggest you look at the parts of the FAQ that talk about performance tuning, especially the stuff in section 6. The most extensive entry is at <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.003.htp>.
-- Brad Knowles, <brad.knowles@skynet.be>
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin, Historical Review of Pennsylvania.
SAGE member since 1995. See <http://www.sage.org/> for more info.
Brad Knowles wrote:
At 2:10 PM +1000 2004-06-29, Guy Waugh wrote:
I want to set up Mailman in a load-balanced fashion on two servers running within an LVS (linux virtual server) cluster. I want the two servers to act as the same Mailman instance.
Okay, so far.The two servers willmount an NFS shared directory from a third server, and the NFS mount will, I guess, contain the shared Mailman stuff.
There's a problem. NFS sucks at operations that require lots offile locking. Of course, all queueing type operations involving mail servers involve lots of file locking.
This is why you do not ever put /var/spool/mqueue on NFS, whereasyou can find ways to kinda-semi-sorta deal with the NFS locking problem for user mailboxes on /var/spool/mail (all solutions to this problem involve trade-offs that hurt performance in other areas).
The system I'm building already has apache on each of the two application servers in the cluster, and the web docroot is NFS-shared between the two from the third server I mentioned above, so there shouldn't be any dramas with the web archives that Mailman generates (unless there are file locking issues with these...?). Similarly, sendmail is a standalone app on both servers, so actually sending mail shouldn't be a problem. Mailman will be sending mail to other servers outside the cluster (i.e. no user accounts exist within the cluster). So, my only problem (I think) with this is going to be with Mailman...
Likewise, Mailman does file locking with the queueing process thatit manages.
If you can find a way to put things on a proper clusteredfilesystem, that should work much better. Sistina GFS may work for you, or possibly Veritas VxFS plus their VCS tools.
I wasn't aware of those, so thanks for letting me know. We do run RHEL3, so GFS would be an option, but for US$2,200, I think I'd have an uphill battle justifying it. I see that NFS has an option of 'noac' (no attribute caching) which sounds potentially useful for me - I don't know whether that directly relates to file locking, though.
I've found a few things onthe web about load-balancing Mailman, but not much.
The thing is that you can run the web server for Mailman separatefrom the mail servers. So, you could cluster the mail servers in a scalable way that will work for you, and then separately work on the issue of building a scalable web farm solution for the rest.
Further, is anyone running Mailman in a load-balanced fashion like this? Any hints/tricks/tips?
If you are interested in splitting of the mail cluster part ofthis equations, then I suggest you look at the parts of the FAQ that talk about performance tuning, especially the stuff in section 6. The most extensive entry is at <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq06.003.htp>.
OK, thanks, I'll check that out...
Regards, Guy.
At 9:32 AM +1000 2004-06-30, Guy Waugh wrote:
The system I'm building already has apache on each of the two application servers in the cluster, and the web docroot is NFS-shared between the two from the third server I mentioned above, so there shouldn't be any dramas with the web archives that Mailman generates (unless there are file locking issues with these...?).
I wouldn't expect the web archives would have problems with locking, no.
But keep in mind that this is a very small part of what Mailman does.Similarly, sendmail is a standalone app onboth servers, so actually sending mail shouldn't be a problem. Mailman will be sending mail to other servers outside the cluster (i.e. no user accounts exist within the cluster). So, my only problem (I think) with this is going to be with Mailman...
That seems likely.
The other pieces of this puzzle are pretty well understood on the scalability side of the picture, and you can pull out a whole host of known workable solutions, depending on your particular needs.
I wasn't aware of those, so thanks for letting me know. We do run RHEL3, so GFS would be an option, but for US$2,200, I think I'd have an uphill battle justifying it. I see that NFS has an option of 'noac' (no attribute caching) which sounds potentially useful for me - I don't know whether that directly relates to file locking, though.
You need noac when sharing filesystems like this for other reasons, but it has nothing to do with file locking.
The problem is that locking is handled outside of the NFS protocol per se. You have lock manager daemons running on both the server and the client, and while NFS is supposedly stateless, they are not. And it is not uncommon for the server and client lock manager daemons to get out-of-sync in a busy environment. In addition to handling locking, these daemons also handle mount requests.
NFSv4 is being re-written to become more stateful and to bring the management of locks inside the base protocol, so that you don't have to worry about lock managers that lose their minds or simply roll over and die, and filesystems that can be read from and written to because the NFS side of the server is still working fine, but which cannot handle locking or be mounted or unmounted because the lock manager daemon has died.
Scaling NFS servers in a write-intensive environment is a very hard task. You end up doing all sorts of crazy things to avoid any kind of lock creation (much less contention).
Proper cluster-aware filesystems avoid these kinds of issues, and make it much easier to scale the systems involved. However, as you noted, they are expensive. The question becomes how much is your time worth, and how much do you lose when everything goes Tango-Uniform (T**ts-Up)? Here, you've got to look not only at your direct loss of revenue, but also the cost of lost opportunities.
There's a reason why cluster filesystems are so expensive -- this is hard to get right. Moreover, people will pay big money for those applications which *do* get it right. They've done the cost/benefit analysis and they figured out that if it takes one of their engineers an extra month to build the system, their MTBF is 1/100th what it would be, and their time/cost to repair is higher due to the custom nature of the solution, then the stuff pays for itself in the first outage -- or the first outage that they avoid.
I *still* haven't seen anything to compare with VaxCluster solutions in this field that were created something like fifteen or twenty years ago. Some of those things are still running, for that reason.
Don't get me wrong, NFS is great. But if you're trying to build a scalable network solution, it can be a very poor choice, depending on the application.
-- Brad Knowles, <brad.knowles@skynet.be>
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin, Historical Review of Pennsylvania.
SAGE member since 1995. See <http://www.sage.org/> for more info.
Brad Knowles wrote:
At 9:32 AM +1000 2004-06-30, Guy Waugh wrote:
The system I'm building already has apache on each of the two application servers in the cluster, and the web docroot is NFS-shared between the two from the third server I mentioned above, so there shouldn't be any dramas with the web archives that Mailman generates (unless there are file locking issues with these...?).
I wouldn't expect the web archives would have problems withlocking, no.
But keep in mind that this is a very small part of what Mailman does.Similarly, sendmail is a standalone app onboth servers, so actually sending mail shouldn't be a problem. Mailman will be sending mail to other servers outside the cluster (i.e. no user accounts exist within the cluster). So, my only problem (I think) with this is going to be with Mailman...
That seems likely. The other pieces of this puzzle are pretty well understood on thescalability side of the picture, and you can pull out a whole host of known workable solutions, depending on your particular needs.
I wasn't aware of those, so thanks for letting me know. We do run RHEL3, so GFS would be an option, but for US$2,200, I think I'd have an uphill battle justifying it. I see that NFS has an option of 'noac' (no attribute caching) which sounds potentially useful for me - I don't know whether that directly relates to file locking, though.
You need noac when sharing filesystems like this for otherreasons, but it has nothing to do with file locking.
The problem is that locking is handled outside of the NFS protocolper se. You have lock manager daemons running on both the server and the client, and while NFS is supposedly stateless, they are not. And it is not uncommon for the server and client lock manager daemons to get out-of-sync in a busy environment. In addition to handling locking, these daemons also handle mount requests.
NFSv4 is being re-written to become more stateful and to bring themanagement of locks inside the base protocol, so that you don't have to worry about lock managers that lose their minds or simply roll over and die, and filesystems that can be read from and written to because the NFS side of the server is still working fine, but which cannot handle locking or be mounted or unmounted because the lock manager daemon has died.
Scaling NFS servers in a write-intensive environment is a veryhard task. You end up doing all sorts of crazy things to avoid any kind of lock creation (much less contention).
Proper cluster-aware filesystems avoid these kinds of issues, andmake it much easier to scale the systems involved. However, as you noted, they are expensive. The question becomes how much is your time worth, and how much do you lose when everything goes Tango-Uniform (T**ts-Up)? Here, you've got to look not only at your direct loss of revenue, but also the cost of lost opportunities.
There's a reason why cluster filesystems are so expensive -- thisis hard to get right. Moreover, people will pay big money for those applications which *do* get it right. They've done the cost/benefit analysis and they figured out that if it takes one of their engineers an extra month to build the system, their MTBF is 1/100th what it would be, and their time/cost to repair is higher due to the custom nature of the solution, then the stuff pays for itself in the first outage -- or the first outage that they avoid.
I *still* haven't seen anything to compare with VaxClustersolutions in this field that were created something like fifteen or twenty years ago. Some of those things are still running, for that reason.
Don't get me wrong, NFS is great. But if you're trying to build ascalable network solution, it can be a very poor choice, depending on the application.
Thanks for the great info Brad - saved me a lot of running around... luckily, I have a few weeks up my sleeve before I need to get this sorted out, so I might play around with NFS and see what occurs. If NFS does fail, I'll investigate running Mailman out of one of the two app servers, and keeping an rsynced backup somewhere else for redundancy. GFS or Veritas sound like just what I need, but I can visualise my manager's lips screaming the word 'NO...' when I request the money 8-)
Cheers, Guy.
participants (2)
-
Brad Knowles -
Guy Waugh