Mailman 3 November 2000 - Mailman-Developers

Re: [Mailman-Developers] Re: Future of pipermail?
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Tue, 21 Nov 2000 22:43:40 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: >> Sad. I already don't allow posts from services that append >> advertising to their messages (eg Hotmail, Yahoo, etc). I guess >> I'll have to explicitly add AOL to that list. Annoying. > or strip to the pieces you find acceptable, using something like > demime. Which I hope to be able to get into 2.1 in some form.... > Rather than reject -- adapt. I've spent considerable time thinking about this and believe it is to the lists' advantage in this case to hold firm. My core promise is signal. In the final analysis it is the only promise I make and keep for the lists: there will be signal, always. To achieve that I need fairly repsectable norming forces. To achieve that rites of participation, minor barriers to entry, identifying characteristics, etc are all cultural assistants to establishing what is in essence a club (as in professional or gentlemans) atmosphere and culture. Avoiding HTML, or in this case AOL, would seem a minor and reasonable fillip on this. Aside: Less than 5% of my members are from AOL. Less than 1% of my semi-regular posters are from AOL. The impact here (for me) is quite small. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

3 2

Re: [Mailman-Developers] Re: Future of pipermail?
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Tue, 21 Nov 2000 22:42:02 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > At 10:31 PM -0800 11/21/00, J C Lawrence wrote: >> The archiver deconstructs messages into a stasndardised and well >> documented templatised form on to the file system. > XML. Think XML. then it can be repurposed by using different > definitions without having to convert from form to form based on > context. If you bug my offices or meeting rooms one more time I am going have to come on over there and shove those damned microphones right up... <<Of course, but I've been getting so many clueless people wittering at me about XML lately that I try and drop it in only after I've established the beachhead>> -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

1 0

Re: [Mailman-Developers] Re: Future of pipermail?
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Tue, 21 Nov 2000 22:13:12 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: >> At 10:04 PM -0800 11/21/00, J C Lawrence wrote: >> Ahh. That explains why I've been rejecting so many postings from >> AOL members on my lists lately (I don't allow HTML as an aid to >> promoting high signal lists). > yup. and you can't turn it off. There's no option. You HAVE to > deal with multipart-alternative and pull the right pieces out, or > their dead in the water. Any list that can't handle MIME e-mail is > incompatible with AOL 6 for windows now... Sad. I already don't allow posts from services that append advertising to their messages (eg Hotmail, Yahoo, etc). I guess I'll have to explicitly add AOL to that list. Annoying. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] Re: Future of pipermail?
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Wed, 22 Nov 2000 00:49:46 -0500 Bill Bumgarner <bbum(a)codefab.com> wrote: > I don't think archival should be treated as a database centric > operation. The concept of archival falls very naturally into a > static hierarchy of collections/directories containing > resoruces/files with a bit of additional meta information > associated with some resources. This is exactly the kind of > information archive that a web server is *designed* to optimally > serve. Adding extra layers here or abstracting to a DBI really > doesn't buy us much. While true in general, this ignores the benefits and ease of dynamically generated reports on archives. Now, perhaps I should talk a little about what I'm spending my copious spare time on lately (other than attempting to chase heisenbugs that crash machines for unknowable reasons). Among other things I'm attempting to meld the basic concepts of WikiWiki, mail archives, and advogato/slashdot-style commentary systems into a form where the basic tenets of Wiki noun definition are retained such that every individual comment and archive page has a dynamically generated/assigned WikiName and there is a moderately sound karma/trust net running as a meta layer atop this as an abstracted report layer. The fact that further abstractions can be made, and that rule-based reports/abstractions can be created as dynamically filled out WikiPages on the other side of this is just, well, really nice. I realise that's pretty turgid. Without spending considerable time and verbiage I don't want to spend right now I'm not sure I can do better. If you're interested in details, and I don't mean just casually, please ask away. But, getting back to the point, to do this properly (ie so that its scalable and reasonably system efficient), especially given the fact that any given data item (be it an archived message, a comment, a report, or whatever) has at least presentation modes each of which has very different surrounding capabilities, the apparancy is that I'm going to have to shove everything into a set of SQL tables. <shrug> The current half-implementation is a mix of SQL and static files (mostly wrapped around my current PHPLib template based PHP4+MHonArc based archiving setup). > WebDAV adds the ability to do advanced locking, easy meta > information storage, etc... but-- most importantly-- does not take > away the very effecient presentation of data naturally present > within a filesystem of stuff served by a web server. I'm actually considering WebDav slightly longer term as a potential Wiki-ish posting method. I'm not at all convinced its a good way to go, but I'm willing to look into it. > As well, a filesystem centric storage/presentation solution-- > webdav or raw filesystem-- solves *most* peoples archiving > problems *most* of the time. Very true. > I feel *very strongly* that the archival solution-- whether it be > raw messages or decoded messages-- should be centric to storing > files in directories and serving files from directories. In the general case, I agree. > This is *not* to say that the DBI approach isn't the right way to > go; if a generic DBI->filesystem, DBI->WebDAV, DBI->DB capable API > were put together and was relatively hidden from the user and > casual developer, it might be a huge win. I'd like to see something trivial: The archiver deconstructs messages into a stasndardised and well documented templatised form on to the file system. A standard CGI-like toolset is then provided to reconstruct those fragments back into a message (please resend this message to me, I didn't get it the first time), into an HTML page (web archives), etc. This is done by having standard small scripts which do >whatever< by assembling simple call sequences to the toolset. End users may redefine this by re-writing said scripts, acessing the deconstructions or reconstructions as they wise, to then do anything they want with the resulting archives. This could be SQL inserts, report generation, or black voodoo, it really doesn't matter, and more importantly, Mailman, and Mailman's archiver doesn't need to know about it, only the user does. The other nice thing is that we don't do it by defining APIs (which then reuires programmers to do useful things with), but by defining small useful atomic tools that may be assembled/ordered in interesting fashions wihtou having to subject the users to Python, or programming much beyond writing shell scripts and editing canned templates. > But an order of magnitude less effecient than downloading the BLOB > off of disk via a webserver! <koh> Greenspun <kof> > Generic access with simple access control is what *most* > users/administrators want *most* of the time. More > complex/abstracted/portable access is less of a requirement and *a > lot* of the people with such requirements also have other issues-- > real or imagined-- that dictate that they really just want Mailman > to hand the stuff to them as quickly/easily as possible and be > done with it. Precisely. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] Mailman with sendmail = performance bottleneck
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Wed, 22 Nov 2000 00:13:14 -0600 Bob Tanner <tanner(a)real-time.com> wrote: > Quoting Chuq Von Rospach (chuqui(a)plaidworks.com): >> You need to go into your system, and set >> >> SMTP_MAX_RCPTS = 10 > I had it down to 50, but I'll give 10 a try. Thanks. FWLIW I usually use either 5 or 10. Depending on your MTA (Historically I've been using Exim, recently moved to Postfix for non-MTA related reasons) and the distribution of your membership lists across domains this can significantly speed queue emptying speed. BTW: I haven't checked and I don't have anything in my spools to check. Does Mailman currently pre-sort the RCPT_TO list by domain? BTW I recently wrote the following in another forum on Postix and Exim: --<cut>-- I've spent the last week or so playing with both fairly extensively. They're sweet systems. More interestingly the performance curves for both are remarkably similar once your mail loads become large (ie your spool is always active (mail to deliver)). The different mostly lies in the curve up to continual activity, and the curve as they approach saturation (mail is arriving more quickly than they can get rid of it on a near continual basis). Loosely, Postfix' initial curve is very steep, but they both reach similar delivery rates for the continually active point. I've had some problems simulating sustained saturation (test networks et al), but Exim seems to be friendlier to the localhost in such circumstances. Exim appears to implicitly assume that such queue congestion is temporary and that given enough time things will slow down enough so it can empty the queue. As such its less interested in "get that damned mail out of here now" than it is in, "be nice to the local machine and get the mail out when you can". There are obvious problems with this if your loads stay at or above capacity for extended periods, but equally, its rather nice from an admin perspective for the local machine. Postfix conversely seems to operate on the "get that mail out of here now", and can be rather brutal on the local host during that process. Postfix under sustained saturation makes for a very heavily loaded system. Depending on what else you are doing there, this may be a problem, or not. It is this nature however which gives it the steep attack on delivery rates as versus Exim. The fact that their sustained delivery rates are so similar at the end of that curve is the really interesting bit however. --<cut>-- Unfortunatly I didn't have time to keep careful metrics, and my process wasn't really rigorous enough for that either a I was just trying to get some basic parameters on their behaviour. > I think this should be an FAQ. <nod> > FAQ #2 states to turn off synchronous DNS resolution. Anyone know > how to do this via sendmail's .mc files? Sorry, no idea. I gave up sendmail 5 years ago. -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

1 0

Mailman with sendmail = performance bottleneck
by Bob Tanner Nov. 22, 2000

Nov. 22, 2000

I have to say I am very disappointed at mailman's performance with sendmail. Especially qrunner and how it works. I am running sendmail with 10 queues, but it looks like mailman only understands and utilizes 1 queue. It also looks like qrunner is single threaded AND multiple instances cannot be running at the same time. This lends itself to a very slow delivery system. If a message is not deliverable for whatever reason, mailman puts files into a directory called qfiles. Then onces every minute a process called qrunner is invoked to walk the qfiles directory trying to re-deliver the messages. What sucks is qrunner just tries to deliver each message one at a time. If message 5 takes forever on the rcpt to: command (for example), no other messages deliveries are attempted because qrunner sits on message 5 until it times out. I have turned the rcpt to timeout down to 2mins, but I am still constantly running 1000+ message in the qfiles directory. Any ideas on speeding things up? Will mailman support multiple "qfiles" directories in the future? -- Bob Tanner <tanner(a)real-time.com> | Phone : (612)943-8700 http://www.mn-linux.org | Fax : (612)943-8500 Key fingerprint = 6C E9 51 4F D5 3E 4C 66 62 A9 10 E5 35 85 39 D9

3 3

Re: [Mailman-Developers] Re: Future of pipermail?
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Tue, 21 Nov 2000 20:52:26 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > At 11:23 PM -0500 11/21/00, Bill Bumgarner wrote: >> - archival of messages is a lot more than just writing the bodies >> to a web server and then generating some kind of automatic >> TOC/index. > agreed completely. I'd take it a step further and say it probably > shouldn't generate indexes at all, but that indexes should be > generated when a user wants to access the archives, dynamically. > That's probably the single major weakness of mhonarc. I have half a design and some basic code roughed out to have the PHP+MHonArc setup I used on Kanga.Nu instead inject the deconstructed data into an SQL DB with the message presentation code rather than reading from the PHP variable assignements sucking everything out of the DB etc etc etc. It wouldn't be very difficult to do and MHonArc has excellent MIME parsing and handling features to act in support of this. >> MIME based email content is not only a reality, but it is often >> the default. > and since AOL 6.0 for windows no longer has any provision for > text-only operation, you now have no choice but to deal with MIME > in some way. they no lnoger HAVE the option to turn MIME off. So > the server has to figure it out. Ahh. That explains why I've been rejectting so many postings from AOL members on my lists lately (I don't allow HTML as an aid to promoting high signal lists). -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1

Towards 2.1
by barry＠digicool.com Nov. 22, 2000

Nov. 22, 2000

There have been lots of great discussion on mailman-developers about the directions Mailman should be taking. This is all very cool stuff (and I want it to continue), but I view it as being more for Mailman 3.0, for which I have no time frame at the moment. Now that 2.0 final is imminent[*], I'd also like to open up some discussion about what we want to see in 2.1. My hard commitment, long overdue, is to integrate Juan Carlos's and Victoriano's I18N work. Juan Carlos has sent me patches, which I intend to start unpacking and looking at, probably after the US Thanksgiving holiday weekend. I'd also like to start using some of the nicer Python 2.0 features to clean up the code in places. Other than that, I'd like to get people's feedback on the really crucial stuff that needs to get fixed for 2.1. I want to keep to a bare minimum anything that requires re-architecting, pushing that into 3.0 after more discussions. I'd like to try to get a 2.1 beta out by the end of the year, with 2.1 final some time in January. I don't know if that's realistic or not. >From the TODO list, things I can imagine tackling for 2.1 include: - Plain text digests should conform to RFC 1153. - If a message has MIME parts and the header/footer is going to be added, the message should be transformed into a mulitpart/mixed with the header and footer added as text/plain parts. (note that this will require improving Python's standard MIME message handling. b.bum astutely <wink> observes that it basically sucks right now and fixing this dovetails with work i want to do for Python 2.1 as well). - Allow "urgent" postings to all members by the list admin which bypasses normal digest delivery. - A button that will bundled and deliver a digest Right Now. - Allow the list-admin to require approvals for unsubs - Allow the user to be excluded from postings if they're getting them in the to: or cc: headers. Be smarter about filtering out duplicate deliveries. - Allow users to subscribe without selecting a password and have Mailman create a password for them. - Allow the site admin to define list styles or themes, and list admins to choose one of the canned styles to apply to their list. - Full creation, deletion, renaming, etc. of lists through the web (and email?), including fixing aliases file updates. - add_members should have a switch to disable admin notifications - Allow individuals to turn off password reminders - Don't use the first public mailing list as the `originator' of password reminders. - Provide an email interface to all administrative commands - Add -join and -remove addresses for easy subscription, unsubscription - For email subscribes, keep an audit of where requests are coming from, and send the original request headers in the confirmation message. Helps track down subscribe bombs. - Support the `which' command. - archive link should do *something* reasonable before the first message has been posted to the list. We should also do /something/ about qrunner, but again, I want to keep radical changes to a minimum. Stability is crucial, but performance counts too. Maybe Chuq's idea of splitting the queue into 3 parts can be done without much disruption. I don't think /everything/ can be done for 2.1, and maybe other people have different priorities. I'm going to keep this list up-to-date on the Mailman2.1 Wiki: http://www.zope.org/Members/bwarsaw/MailmanDesignNotes/MailmanTwoDotOne Feel free to contribute. -Barry [*] The astute observer will notice the tarball on SourceForge already. I'm waiting for the mirror sites to update, but will announce tomorrow even if they don't.

2 1

Re: [Mailman-Developers] Mailman with sendmail = performance bottleneck
by Dan Mick Nov. 22, 2000

Nov. 22, 2000

> > What sucks is qrunner just tries to deliver each message one at a > > time. If message 5 takes forever on the rcpt to: command (for > > example), no other messages deliveries are attempted because > > qrunner sits on message 5 until it times out. > > Translation: You are doing name lookups on messages delivered via > SMTP from the localhost. There's not much reason for this. Turn it > off. Just out of curiosity: anyone know how to do this in sendmail? Is this the 'delay_checks' feature? I'm sure I haven't configured it, but my list isn't big enough that I care about such delays. If I were to want to change it, though, it's not at all clear to me how I would, and since it's become such a refrain....anyone know?

2 1

Re: [Mailman-Developers] Re: Future of pipermail?
by J C Lawrence Nov. 22, 2000

Nov. 22, 2000

On Tue, 21 Nov 2000 16:04:31 -0500 Bill Bumgarner <bbum(a)codefab.com> wrote: > The key with WebDAV is that use therein means that you can point > WebDAV enabled Mailman at Zope, IBM's websphere, Apache+mod_dav, > or any of the other WebDAV enabled HTTP servers out there and it > should "just work" (assuming they did the implementation right). > There is nothing proprietary about the solution and choosing it as > a foundation for archival/management of messages/attachments opens > doors (instead of closing doors). For me WebDAV raises concerns centering around authentication and access security. That said, I fail to see either the use or value in having what is essentially (if I understand it correctly) a posting tool (ability to post content to a site) *UNLESS* the archiver is running on a different machine than Mailman (in which case a networked filesystem would seem a safer/lighter approach). -- J C Lawrence claw(a)kanga.nu ---------(*) : http://www.kanga.nu/~claw/ --=| A man is as sane as he is dangerous to his environment |=--

2 1