Brad Knowles <brad@stop.mail-abuse.org> writes:

At 2:23 PM +0100 2006-03-03, Hans Ulrich Niedermann wrote:

...

Our goal is to give this branch to the Mailman project if they want it. If they do not for any reason, we'll have to maintain our vhost branch until Mailman 3 with all the necessary features has been released. :)

If you upload the code to the SourceForge "patches" page for Mailman, the core developers will be much more likely to look at what you've got and to consider it for inclusion in a future version.

I'll do that as soon as I have something a) working b) clean (code) c) pretty (concept) d) as small a change as absolutely necessary

The key people you've got to convince are Tokio and Mark -- Barry is primarily working on his real job, and doing what he can for Mailman3 in his non-existent "spare time", although he does try to at least keep abreast of what is going on with Mailman 2.x.

I'll keep that in mind, thank you.

Gruß,

Hans Ulrich Niedermann

Mark Sapiro <msapiro@value.net> writes:

Anyway, my intent is not to critique other solutions, but to say that I would much prefer to see a directory inserted rather than a host_name appended. This has performance advantages and might fit better with a scheme of "hashed" intermediate directories to help sites with very large numbers of lists. Once you add the infrastructure for one intermediate directory, it may become simpler to add more intermediate directories.

I.e, for the initial cut, split the lists/<listname> structure into lists/<hostname>/<listname> and similarly for archives/private and archives/public.

Actually, implementing the lists/<hostname>/<localpart> and archives/(private|public)/<hostname>/<localpart> was one of the first things I have implemented, as it is one of the easiest parts and had been mentioned at http://www.zope.org/Members/bwarsaw/MailmanDesignNotes/VirtualHosting

It's not as clean as I'd like it to be yet, though: There is the very same code to create those directories in both Mailman/Site.py and Mailman/Archives/Archives.py (two nested try: blocks with the os.umask() where the inner try finally calls os.makedirs()).

OTOH, one can argue that Site.py and Archives.py handle fundamentally different things and just happen to need the same code. Still, I always become wary when I see the same code in different places :)

Anyway, it's your project, but I think we'd all welcome a complete solution to this problem.

Good. Let's see how far I can get.

Gruß,

Hans Ulrich Niedermann

On Mar 9, 2006, at 9:55 AM, Ian Eiloart wrote:

...

Actually, implementing the lists/<hostname>/<localpart> and archives/(private|public)/<hostname>/<localpart> was one of the first things I have implemented, as it is one of the easiest parts and had been mentioned at http://www.zope.org/Members/bwarsaw/MailmanDesignNotes/VirtualHosting

I'd propose a slight variation on that, to better accommodate campus/corporate setups, as well as hosting providers:

lists/tld/subdomain/.../listname

[...snip...]

With all these variations, it might be reasonable to consider dumping
the path macros altogether and switch to a functional model. The path mapping functions could be supplied by mm_cfg and modified
as appropriate for the installation.

For example, the existing 2.1 storage is functionally equivalent to: def list_dir ( domain, list_localpart ) { return os.path.join( VAR_PREFIX, 'lists', list_localpart )

Whereas what you're asking for is: def list_dir( domain, list_localpart ){ leaf = list_localpart for dom_part in domain.split('.'): leaf = os.path.join( dom_part, leaf ) return os.path.join( VAR_PREFIX, 'lists', leaf )

And what works for _my systems_ would be: def list_dir( domain, list_localpart ){ return os.path.join( '/var/vdomains/', domain, '/mailman/ lists', list_localpart)

IF one simply _MUST_ have different mapping regimes for different
domains....well, then encode it into your mm_cfg.list_dir().

Rich Johnson <rjohnson@dogstar-interactive.com> writes:

On Mar 9, 2006, at 1:02 PM, Rich Johnson wrote:

[ lists/<hostname>/<localpart> vs. lists/tld/subdomain/.../listname vs. ... ]

...

With all these variations, it might be reasonable to consider dumping the path macros altogether and switch to a functional model. The path mapping functions could be supplied by mm_cfg and modified as appropriate for the installation.

Oh wait!

Mailman/Site.py already provides this function--but the rest of the
system never propagates the domain--neither do the support utilities
(like check_perms)

There are multiple places in the code where pathnames and URLs are built outside of Mailman/Site.py. In the vhost branch, I need to change all the code parts generating a path anyway, so I'm moving the actual generation into a function in Mailman/Site.py while I'm at it.

Then if someone needs special file system layout later, he can rewrite Mailman/Site.py into "from MySite import *" and implement his own functions mapping objects to files and vice versa in a MySite.py module. However, I'm not overly concerned with the details of all possible layouts - that stuff is easy to change once it is concentrated in a single module.

Gruß,

Hans Ulrich Niedermann

I don't follow why one would need that, if you already separated based on email domain. In other words, if you had list@sussex.com, list@physics.sussex.com, and list@english.sussex.com, why not:

/lists/sussex.com/list /lists/physics.sussex.com/list /lists/english.sussex.com/list

?

Bob

---------- Original Message ----------- From: Msquared <sub1.dev.mailman@msquared.id.au> To: mailman-developers@python.org Sent: Fri, 10 Mar 2006 08:37:53 +0800 Subject: Re: [Mailman-Developers] Virtual Domains Redux (w proposal)

On Thu, Mar 09, 2006 at 02:55:14PM +0000, Ian Eiloart wrote:

...

I'd propose a slight variation on that, to better accommodate campus/corporate setups, as well as hosting providers:

lists/tld/subdomain/.../listname

for example, I'd have:

lists/uk/ac/sussex/ lists/uk/ac/sussex/physics lists/uk/ac/sussex/chemistry lists/uk/ac/sussex/english

I second that.

Regards, Msquared...

---

Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/bob%40nleaudio.com

Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp ------- End of Original Message -------

---------- Original Message ----------- From: Msquared <sub1.dev.mailman@msquared.id.au> To: mailman-developers@python.org Sent: Fri, 10 Mar 2006 10:00:07 +0800 Subject: Re: [Mailman-Developers] Virtual Domains Redux (w proposal)

On Thu, Mar 09, 2006 at 08:16:39PM -0500, Bob Puff wrote:

...

I don't follow why one would need that, if you already separated based on email domain. In other words, if you had list@sussex.com, list@physics.sussex.com, and list@english.sussex.com, why not:

/lists/sussex.com/list /lists/physics.sussex.com/list /lists/english.sussex.com/list

Scope of ownership and management.

These might have different owners:

/lists/mine.example.net/ /lists/yours.example.net/

but these might have the same owner:

/lists/dept1.foo.org/ /lists/dept2.foo.org/

If it were possible to control access/ownership/etc according to subdirectories, this would allow you to grant control of subdirectories from the appropriate level down:

/lists/org/foo/ is for *.foo.org /lists/net/example/mine/ is for me /lists/net/exmaple/yours/ is for you

Of course I don't know if mailman yet allows such delegation of control, but it would be nifty...

You could still control access like that through unix/linux directory ownerships, for example, though.

I think we're drifting here from a logical format. If there is to be shared ownership, I think that needs to be done with a different database. It could be a nightmare from the admin side if you group things by who wants to own what lists, rather than by domain. I believe the initial point behind the start of this is that list@foo and list@bar should be able to cohabitate on the same install.

Bob

At 9:04 PM -0500 2006-03-09, Bob Puff wrote:

I think we're drifting here from a logical format. If there is to be shared ownership, I think that needs to be done with a different database. It could be a nightmare from the admin side if you group things by who wants to own what lists, rather than by domain.

Actually, his proposed scheme does make sense for a larger-scale 

environment. Indeed, that's the second-most scalable way I know of to do it. The most scalable would have some sort of side index system and use good hashing algorithms to avoid having a billion subdirectories under /usr/local/mailman/lists/com/ and maybe just a few hundred under /usr/local/mailman/lists/info/.

For smaller environments, you might want a different option.


I think what we really need is some way to guarantee that any of 

several different suitable formats could be used, some of which might be more easily understood at a glance, others which might be more complex but also more scalable.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

LOPSA member since December 2005. See <http://www.lopsa.org/>.

[...snip...]

I think we're drifting here from a logical format. If there is to
be shared ownership, I think that needs to be done with a different
database. It could be a nightmare from the admin side if you group things by who wants
to own what lists, rather than by domain. I believe the initial point
behind the start of this is that list@foo and list@bar should be able to
cohabitate on the same install.

Not quite. It depends on your interpretation of what 'cohabitate'
means.

As the initiator, I'll chime in with the following:

_processed_ by the same install? yes. _cohabitate_? perhaps. But this is a site and/or domain
administrator configuration decision--and not imposed by the
developer. I've been pretty consistent about my aversion to
comingling data from separate domains in the same tree. I view the
domain split as _more_ important that the virtual service provided-- be it apache vhosts, exim vusers, or mailman. Ideally it could be a
mount point.

base-domain->site scoping [N.B. I'm still thinking about my

• Ephemeral data such as queues and logs might or might not cohabit.
• Persistent data such as list configuration, subscriber lists, and
  archives must be severable
• HTTP and mail message templates probably require list->sub-domain-

IMO: position on this issue]

A good part of this discussion seems to revolve around whether
mailman ''plays well with others''. Where the ''others'' are
amorphously defined.

On Mar 10, 2006, at 2:11 PM, Hans Ulrich Niedermann wrote:

Rich Johnson <rjohnson@dogstar-interactive.com> writes:

...

[...snip...] base-domain->site scoping [N.B. I'm still thinking about my

• HTTP and mail message templates probably require list->sub-domain-

position on this issue]

Let me cite the comment from MM's 2.1.7 Mailman/Utils.py findtext() function:

# When looking for a template in a specific language, there are  

4 places # that are searched, in this order: # # 1. the list-specific language directory # lists/<listname>/<language> # # 2. the domain-specific language directory # templates/<list.host_name>/<language> # # 3. the site-wide language directory # templates/site/<language> # # 4. the global default language directory # templates/<language>

Looks like your idea is already in place. :)

Hooray! The scoping exists!. (FWIW, I needed some good news today!)

Now, if I can use Site.py (or one of its siblings) to bring it all
into a domain-rooted fs by mapping: perhaps via: Site.list_template_dir(). Site.domain_template_dir(),

• lists/<listname>/<language> -> /var/vdomains/<list.host_name>/ var/lib/mailman/lists/<llist.local_name>/<language>
• templates/<list.host_name> -> /var/vdomains/<list.host_name>/ var/lib/mailman/templates/<language>

and so on....

......I'll declare victory and wait for the next release (...well,
maybe :-)

I hope that's not too much to ask.

On Sat, 2006-03-04 at 14:57 -0500, Bob Puff wrote:

...

I.e, for the initial cut, split the lists/<listname> structure into lists/<hostname>/<listname> and similarly for archives/private and archives/public.

Definite +1 on that!

Yep, that's definitely where my thinking was leading the last time I worked on this. FWIW, in Mailman 3 I've been thinking that the "list id" will essentially be the full posting address, so that it's pretty clear what "foo@dom1.org" and "foo@dom2.org" point to.

Let me make two other suggestion for those of you working on patches. First, generate your patches against the Mailman trunk, for what will eventually be Mailman 2.2. There's no way we can add this stuff to MM2.1, but Tokio did the heavy lifting a while back to whip the trunk into shape, and all patches are being applied to both the trunk and the 2.1-maint branch. At some point, they'll start to diverge but for now they should be pretty similar (modulo some i18n stuff).

Also, now that SF provides Subversion support, I very strongly want to migrate to it. Among many other advantages, this will make it easier to create experimental branches to work on things like this.

-Barry

Barry Warsaw <barry@python.org> writes:

Also, now that SF provides Subversion support, I very strongly want to migrate to it. Among many other advantages, this will make it easier to create experimental branches to work on things like this.

Is there a time schedule for migration SVN, and would I get to play with a branch there? :) I think my vhost branch is ready to be turned into an experimental SVN branch.

I would first start on MM 2.2 and introduce the necessary abstractions for file locations and the like, without changing the actual behaviour. Once those abstrations are in place, I would then build the vhost branch on that and thus make the vhost patch much smaller and easier to comprehend.

Even if you do not want my vhost changes in official Mailman, those abstractions could still be beneficial for both official Mailman and for independent maintenance of a vhost branch.

The following is a more detailed list which shows the current state of the vhost branch. It is not complete yet (there will still be a number of issues to solve, and I will do so), but the basics are there:

• Support for three list types:

  1. site wide lists without "postfix style virtual hosting":
     • internal_name()=="foo", like in 2.1.7)
     • uses the DEFAULT_EMAIL_HOST and DEFAULT_URL_HOST
  2. site wide lists with "postfix style virtual hosting":
     • internal_name()=="fox", like in 2.1.7
     • "fox" here conflicts with site wide "fox" even in case of different host_name
     • uses special email_host and (possibly) url_host
  3. vhost lists
     • internal_name()=="foo@bar" (new in vhost branch)
     • no conflicts with site-wide "foo" or vhost list "foo@bleh"
     • requires registration of the url_host<->email_host relation in mm_cfg (cf. VIRTUAL_HOSTS, add_virtualhost()) by the site admin Note: Type 2 lists may be migrated to type 3 lists, so I have not tested that type 2 lists work all that well. If we drop type 2 lists altogether, that work would have been in vain, so I'm not very much concerned about them. The architecture itself does not prevent their continued use.

• Web and E-Mail templates produce the expected results.

• The Web interface works as expected, as far as one can tell without executing an automated test with complete coverage:

  • (un)subscribing to lists works
  • administrating the list settings works
  • moderating postings works The URL space looks like this (for the example lists from above): http://lists.site-wide/listinfo/foo http://lists.site-wide/listinfo/fox http://lists.bar/listinfo/foo http://lists.bleh/listinfo/bleh Possible issues I don't care about on my installation:
  • There is no single web site which lists all lists.
  • All lists using the same url_host must use the same email_host.
  • You can/must use a single Apache <VirtualHost> for all lists.* URLs. (They do necessarily have to be called "lists.*").
  • You cannot have different basic URLs for different url_hosts, i.e. integration into existing websites is difficult/impossible.

• Command line list creation works.

• Sending mail to site wide lists and vhost lists works.

• Generate proper aliases and virtual aliases for Postfix.

A few things are still to do:

• Introduce the role of a vhost-domain admin between the site admin and the list admins. This requires per vhost-domain storage modifiable by the vhost-domain admin which we do not have. (The vhost-domain admin should be able to modify his password, right?)

• Enforce the requirement of one mailman@vhost-domain list per vhost-domain.

• Should we migrate the site wide lists with "postfix style virtual hosting" to vhost lists? I have not done this yet in order to avoid additional work in the proof-of-concept phase.

• Backing out unnecessary stuff introduced while getting to know the code and generally clean up stuff.

• CGI list creation/removal do not work yet.

• ...

If you want to examine the vhost branch yourself, here are a number of links:

Current patch against mailman-2.1.7 with the 20060114 patch: http://nix.lauft.net/mailman/mailman-2.1.7-20060114-to-vhost-current.patch The README.VHOST containing more text on issues and solutions: http://nix.lauft.net/mailman/README.VHOST The overview page: http://nix.lauft.net/mailman/ Cogito/Git repository: http://git.n-dimensional.de/mailman-virtualhost.git/

Gruß,

Hans Ulrich Niedermann

On Mar 6, 2006, at 12:44 AM, Msquared wrote:

On Fri, Mar 03, 2006 at 01:39:18PM -0800, Mark Sapiro wrote:

...

I.e, for the initial cut, split the lists/<listname> structure into lists/<hostname>/<listname> and similarly for archives/private and archives/public.

Just a thought on this: might it be better to mangle the directories like so:

<hostname>/lists/<listname> <hostname>/archives/<listmame> etc

so as to group things on a per-host basis? One reason for doing so is that it (to me) makes more sense especially if you are hosting many domains that are not related or are different clients, etc. It
simplifies the view of things from a per-host perspective.

Perhaps even this:

<hostname>/<listname>/list <hostname>/<listname>/archive etc

Even better, perhaps make it configurable. :o)

Regards, Msquared...

---

Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users% 40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman- developers/rjohnson%40dogstar-interactive.com

Security Policy: http://www.python.org/cgi-bin/faqw-mm.py? req=show&file=faq01.027.htp

I'll second this!

In general I object to co-mingling domain specific data. it was the
main motivation for my initial post. I'm glad it triggered a bit
more discussion that I expected. :-)

I'd encourage all to consider also: lists in a single domain.

• domains are likely to have different requirements and/or be
  separately administered.
• The storage format is not necessarily a file-system. (as other
  discussions have indicated)
• There's even an argument that storage format may vary between

In any case we have: archives <= f(domain,list,fmt) list <= f(domain,list, fmt)

How about a macro capability with $hosthame and $listname (ala Exim's
$local_part, $domain) for filesystem storage? One could specify: LIST_DATA_DIR = /var/lib/mailman/%hostname/lists or LIST_DATA_DIR = /var/vdomains/%hostname/mailman/lists

in the config file according to your system/site conventions.

Another, more configurable, mechanism would be implementing
''repository'' objects (initially implemented for filesystem
storage): These ''repositories'' would be indexed by ''hostname''.
Their specification could be modeled after Apache's <VirtualHost>.
To wit (assuming a way to interpret this snippit): <VirtualHost low-volume.tld> ArchivesRepository filesystem ListsRepository filesystem FilesystemRoot /var/vdomains/low_volume/mailman </VirtualHost>

<VirtualHost high-volume.tld> ArchivesRepository filesystem ListsRepository database DatabaseType Postgres FilesystemRoot /var/vdomains/high_volume/mailman DB_info mailman_db/mailman_usr/mailman_dbpwd </VirtualHost>

Or even (tying these two together):

FilesystemRoot /var/vdomains/%hostname/mailman DatabaseType postgres DB_info mailman-db/db-user/db-pwd <VirtualHost low-volume.tld> ArchivesRepository filesystem ListsRepository filesystem </VirtualHost> <VirtualHost high-volume.tld> ArchivesRepository filesystem ListsRepository database </VirtualHost>

--rich