The problem is when anyone who knows the list moderator password can
access and moderate the list.
That is the way it is designed to work. See, e.g., http://wiki.list.org/x/5YA9.
Further, the owner addresses are listed in the footer of the admin and moderator login pages, so they aren't in any way secret.
I have marked this as "wishlist" for possible inclusion in a future release.
** Changed in: mailman Importance: Undecided => Wishlist
** Changed in: mailman Status: New => Triaged