slave@codegrunt.com wrote:

The quick answer would be to not bother and wait until version 3 of Mailman

That would be the easy answer, but since there is currently no ETA (as far as I know) for MM3, that certainly wouldn't be the quick answer.

-Dale Newfield Dale@Newfield.org

At 4:30 PM -0500 2005-11-16, Kevin McCann wrote:

...

...

Hello. How might one configure Mailman (version 2.1.6) list settings using PHP scripts?

Mailman is a Python project.  PHP is always going to be an alien 

in this environment.

...

The quick answer would be to not bother and wait until version 3 of Mailman which will use a MySQL backend and instead manipulate that directly (if I have been reading past discussions correctly).

I don't mean to be a pessimist, but I don't think this will ever happen.

The better integration with MySQL?  No, that's definitely going 

to happen. It may not happen as early as you (or Barry) would like, but it will happen.

• The Mailman project is not as open as it could be. There is too much control over who can contribute code, how they can do it, when they can do it. I understand wanting to maintain quality and stability, but effort and goodwill are going to waste.

All open source projects have to have a source code control 

system, otherwise they can't survive. You also have to have some controls over who is allowed to do what within the source code control system. Again, it's a matter of simple survival.

If you, or anyone else, is interested in contributing to the code 

base, it's not hard to add you to the system. Once Barry saw that Tokio was posting patches and being a useful contributor to the project, he got added to the system, and is now the primary person responsible for the 2.x code base. There are a number of other people who also have write access to the code base, but whose names you may not have seen very much of recently.

There is a certain lack of involved developers who are able to 

contribute a good deal of time to the project. If you're prepared to do that, then we will all benefit.

• Not everyone sees the need for a highly integratable MLM, despite the fact that people have been begging for it for half a decade. They beg for it on the Mailman list. They beg for it on the Sympa list.

They're begging for this kind of stuff in the wrong places. 

There a whole host of CRM and other related tools out there.

But for whatever reason, people seem to be unwilling or unable to 

go look for them. Instead, they glom onto tools like Mailman and instead expect us to make it do backflips, because they can't be bothered to go find the right tool for the job.

                                                             But MLM

developers apparently do not work with organizations or people who see the need to integrate MLM's with other collaborative tools.

Before you throw stones, you should make sure that you know what 

it is that you're going to be throwing stones at.

In the case of Mailman, you clearly do not.

                                                          MLM

developers do not buy into the concept of making the three main data stores available in SQL (those being list config, member and message archives)

Oh?  That's news to me.

• Backward compatability is an issue and puts the kebosh on dramatic departures from exisitng MLM versions.

For Mailman 2.x, that's true.  But Mailman 3.x is a complete 

ground-up re-design and re-write. That's part of why it's taking so long to get there.

They just couldn't cut ties to "the old way." With a Mailman 3, radical changes would be needed, in my opinion, but are developers willing to have MM3 be a new, different, separate beast than MM2? I have a feeling that there is not enough wilingness to let MM3 be a fresh, new start.

Obviously you haven't been tracking the Mailman3 development very closely.

Personally, I think a new MLM is needed, built from the ground up, and taking into account today's wants and needs. An MLM built for the 21st century. Completely open source and well-managed by people who have the time and the inclination to do it. New Project with New Ideas and Eager People.

If that's what you want, and Mailman3 isn't doing it for you, 

then you need to go off and set up your own project.

      Either MM3 needs to start happening or a new MLM project needs

to be created. If some people don't have time, fine, but then loosen the reins. Let it happen. And if it comes down to money, well, some people may be surprised at the amount of funding that is available for these kinds of projects. But the projects need to be able to move forward, unencumbered by control and competing commitments. Otherwise nobody will fund it.

Again, I'm not trying to throw out criticisms, just stating realities.

That sounds to me like criticism.  Badly placed and very poorly 

informed criticism.

Please do not continue to spread falsehoods and FUD about a 

project you obviously haven't been paying enough attention to.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

At 11:49 PM -0500 2005-11-16, Kevin McCann wrote:

                                                             I'm not

going to try to make you "get it" but I will say that the right tool for "the job" just doesn't exist yet. It would be nice if it did.

Mailman is not, and never will be, the be-all and end-all of 

community collaboration tools. By necessity, those tools must be all-encompassing, and mailing list management is just a small part of the much greater picture. You can't just take a bunch of independent tools and throw them into a blender and hope that something useful comes out the other end.

If you want the be-all and end-all of community collaboration 

tools, you're not just barking up the wrong tree, you're in the wrong bloody forest on the wrong damn planet in the wrong friggin' solar system in the wrong galaxy in the wrong Universe.

Mailman will move towards greater integration with database 

tools, yes. But it is not now, nor will it ever be, the be-all and end-all of community collaboration tools.

Moreover, if it was going to be such a thing, it would be highly 

commercially saleable, and you wouldn't be getting it for free. You wouldn't be getting it for anything remotely approaching free.

The ultimate community collaboration tool or free.  Pick one.

I'm not throwing stones, Brad. I've tried to make that clear.

I'm sorry, despite all your claims to the contrary, whether or 

not you say you are throwing stones does not necessarily have any bearing on whether or not you actually are throwing stones.

Took a while to convince some on list config and member data. Still work to do on message archives.

I don't think there's any problem at all convincing anyone that 

message archives would be very useful to have indexed in a database. The issue is how to change the code so as to make that happen.

You might as well just cross this one off your list now, because 

otherwise you're just going to make yourself look even more foolish.

No, Brad. That is not why. The reason is that people have other commitments and the project is not as open as it could be so that those who do have more time or MONEY could help move things along.

It's an open source project.  If you don't like the way it's 

being run, you can always take a copy of the code and go play in your own GPL sandbox.

But this project is as open as any other I've seen, and 

considerably more open than most. Do you have any idea what kind of work it takes to get a mod bit in FreeBSD that allows you to directly make any changes you want to the code and without having to have them approved by a mentor? And even then, you're only allowed to make changes within your small part of the kingdom?

Yes, there are a relatively large number of people who have 

achieved that goal, but they are a much, much, much bigger project and much better financed, and they've been around a lot longer.

Give us a thousand or a million times as many good programmers, 

millions of dollars, and a decade to put it all together, and we might be able to do something that could potentially begin to achieve some progress towards the ultimate collaboration tool.

I'm not at all surprised it sounds like criticism to you. You just don't get it and I doubt you ever will..

When you're being abused and the person doing the abuse tells you 

that you're not being abused, who and what are you going to believe?

The facts are the facts.  Your claims regarding the facts do not 

necessarily have any bearing whatsoever on the facts themselves.

I have been watching this project closer than I'd like to admit for 5 years. You're happy with the status quo, I'd prefer to try to get things happening.

Once again, you have mis-characterized the situation to fit your 

own agenda.

It's not that I'm "happy with the status quo", it's that I'm a 

realist and I understand that short of being able to take people like Barry and Tokio and pay them money to do little but work on Mailman full-time, we can't expect to see a Universe of change overnight.

This is going to take some time.  More time than I would like, 

much more time than Barry would like, and obviously a lot more time than you would like.

The question is what are you going to do to try to help that 

situation, or are you going to continue to just bluster and make noise?

Actions speak louder than words.  So, let's see some action.

If you can't come up with the money to pay for Barry to work on 

Mailman3 full-time, then come up with code to fix all the problems you've outlined. If you can't come up with code, then come up with something else to help break your perceived logjam.

         That's where we differ. No need to be snide and arrogant.

Then stop doing one thing and claiming something else.

If you want to criticize, then please at least attempt to be 

constructive in your criticism.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

At 4:10 AM -0500 2005-11-17, Kevin McCann wrote:

A design that supports independent functionality *and* integration is a Good Thing. It's how a lot of software works.

There are cases where this is possible, and cases where it is 

not. The Unix philosophy of tying a multitude of tiny little tools together with pipes doesn't work for everything.

Show me examples of stand-alone tools that are intended to be 

part of a complete community collaboration system, but which are also fully functional on their own.

I'm sorry.  This is a case where you have to own all the pieces 

to the puzzle yourself before you can have a reasonable hope of being able to make them all fit together.

Or did you think that you could just take arbitrary pieces from a 

dozen random puzzles and they'd all magically work together and create a beautiful picture?

                                                             But if it

can integrate easily with a collab tool that offers most of the other desired functionality, then that's a good step in the right direction. So what exactly is wrong with that premise?

That you'd get that utopia for free?  That you could simply 

continue to badger and browbeat us until we provided it for you, for free?

                                         If that puts me in a

different universe, well, I have a lot of company out here.

Yeah, on the badgering front, you certainly have a lot of company 

there. Not that this is a good thing, however.

...

 Mailman will move towards greater integration with database tools,

yes. But it is not now, nor will it ever be, the be-all and end-all of community collaboration tools.

Your first sentence above: that is all I'm looking for!

Not according to everything else you've said.  Among other 

things, greater integration with database tools says nothing for sharing the database schema designs with other projects. Nor does it say anything for sharing the supported database packages with other projects -- maybe we'd like MySQL and they'd like PostgreSQL. And that's just the tip of the ice shelf.

Or did you expect that just because you get everyone in the UN to 

speak English (or at least understand it through an interpreter), that they would automatically agree on everything and work together towards the same set of goals?

I think you're expecting a level of integration here that cannot 

possibly be provided by any one project, unless they own all the various pieces to their puzzle.

                                                      I don't want or

expect MM to be a be-all and end-all tool! I want it to be able to be PART of a very good solution. I hope to see that in MM3. Why can't you understand that?

Why can't you understand that you're speaking out of both sides 

of your mouth at the same time, and saying different things?

Once again, paying money for Barry (and others) to do work on MM3 is precisely what I tried to arrange. Availability was always the main issue.

Pay them enough money, and I'm sure that they'll solve the 

availability problem. But you'll have to guarantee them the payment -- they're not going to quit their day jobs just because you claim to have ten million dollars in a vault.

What the? I really don't understand your hostility. And you haven't read a word I've written have you? Understand this: I have put A LOT of time and effort into trying to move MM3 develoment forward through funding. These are ACTIONS, whether you recognize it or not.

I don't see any action there.  I see failure to act, sure.  Claim 

of action, sure. Possibility of attempted action, maybe. If so, then there was certainly a failure to complete the action -- regardless of who you try to blame that on.

Brad, try treating people a) like you would have them treat you,

I might do that, if you would have done the same.  Instead, you 

post your diatribe and expect all problems to be solved for you.

I'm sorry.  I'm not going to react well to that kind of 

treatment, and I doubt anyone else is, either.

and b) as if they were in the same room as you.

Again, Physician -- heal thyself.  Once you've done that, maybe 

others will follow suit.

You know, I'm reminded by all of this that no good deed goes unpunished.

Yup, you're certainly providing lots of punishment here.  We must 

have done lots of good deeds.

I did a lot of lobbying for MM3 development funding. It didn't work out because of Barry's schedule, but Lord knows I put in the effort.

I've put in lots of effort on lots of projects, some of which was 

successful and some of which wasn't.

But I don't recall ever posting a diatribe of the sort you did, 

and then expect everyone else to kowtow to my wishes, cater to my every desire, and magically solve all my problems.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

At 7:16 AM -0500 2005-11-17, Kevin McCann wrote:

For what it's worth, the kind of tool that I'm hoping to see--from a functional point of view--has already been created. At Bellanet (my former org.) we created something called Dgroups (see www.dgroups.org) several years ago. The problem is that it relies on commercial software (Lyris, ColdFusion, MS SQL).

And the folks at Kabissa.org have put together a somewhat 

comparable system based on open-source components (including an old version of Mailman), primarily for the people on the African continent. I'm sure it's much more kludgey than your stuff, and I know it's a hell of a lot more brittle than anyone would like, and I know it doesn't do anywhere near as much stuff as people would like.

However, there's a great deal of integration work that has been 

done there and part of the reason I agreed to volunteer my time with the group and help run the mailing list side of their site was to try to learn as much as I could about what they had done and how they did it, and then bring that back to the Mailman project.

If you look at the Mailman FAQ Wizard entries related to this 

subject (and the threads that they link to), most of the useful information regarding integration with CMSes, web board discussion systems, etc... has come from Tobias Eigen at Kabissa and various things that he's said on the mailman-users mailing list.

If you had information that was useful in these areas and you 

were willing to share it with us, I guarantee that this would go into the FAQ and would benefit the larger community.

                           We wanted other international development

organizations, especially in developing countries, to be able to have a dgroups for themselves. Essentially decentralize the service and build capacity in the south. But commercial software was not practical, and we really had moved toward open source policies by this point, anyway.

The folks at Kabissa have taught me that commercial stuff can be 

practical, if it is the best way to achieve the desired goals. They have commercial hosting. They have commercial support for their platform. They have some commercial software that they have used as part of their system.

It's not forbidden to spend money.  They try to avoid commercial 

stuff and use open source where possible, but sometimes the only viable solution is commercial -- and that's okay. What's important is to minimize the overall total cost of development and support of the system, and sometimes to keep TCO down it's better to spend a bit more money up-front.

Of course, Kabissa is just one organization that is trying to 

help under-privileged groups in more impoverished nations, and there are presumably other groups that don't have as much money to spend as Kabissa has had.

But I would hope that these other groups would take the same 

approach -- don't just look for open source solutions to the exclusion of everything else, but instead try to keep a lid on TCO as a whole, and make use of what services are available from what sources when and where possible.

So, yes, I'm disappointed in the lost opportunity but for exactly two reasons: 1) it means missed resources for MM3, 2) it means that international development groups will wait longer for the tool we had hoped to provide them with. In international development, time is an issue because quicker solutions means less suffering.

From the sound of it, you've already got a tool that does the 

job. Okay, it's based on commercial software, but one thing that using commercial software does usually buy you is speed and improved time-to-delivery.

So now you're going back and trying to re-work things so as to do 

everything with open source, and you're frustrated that it's not as easy to do with open source as it was with commercial software. That's easy to understand -- a lot of stuff can be frustrating with open source if you're trying to compare it to commercial software.

                                                           But I want

you to know is that my interest is solely in the greater good and nothing else.

Fair enough.

            So, go ahead, kick me in the head again if it'll make you

feel better, but you should know that you completely misunderstand my motivations and my agenda.

You meant well, although you chose a method of expressing 

yourself that I found objectionable. I think we can get past this.

I still feel that there's going to be a great deal more work that 

needs to be done to integrate any version of Mailman into a larger community collaboration system, and tighter database integration with future versions of Mailman is going to be just one small part of this overall picture.

The database integration is a necessary, but not sufficient, condition.


That integration is coming, albeit slower than anyone would like, 

and with less regularity than anyone would like. But it is coming. You can choose whether or not you believe in that, and you can choose your level of involvement in helping to make that happen. You can also ask for perspective or guidance with regards to the best way that you can help make that happen.

But none of this changes the fact that the improved database 

integration is going to be just the first step in a long road towards that combined community collaboration system.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

At 2:49 PM -0500 2005-11-17, Kevin McCann wrote:

For me, the most frustrating part is knowing that the time is not right for technical reasons but having to do it anyway because of political reasons (and, of course, the political reasons are borne from good intentions: helping the needy).

Indeed, that is extremely frustrating.  One thing I've always 

really hated is when bad technical decisions get made because of bad political decisions. Long-term, that's been one of my biggest hot buttons.

I think we can get past this, too. In the end we're on the same team, even if our goals are a bit different.

Admittedly, I responded as a frustrated person. See, I tried very hard to get MM3 funding to happen behind the scenes.

And I have to confess that I wasn't aware of the things you had 

tried to do behind the scenes -- From what I read, it seemed to me like you didn't know what was going on within the project and were making unreasonable complaints.

With this irreconcilable difference between our starting 

positions, I think the outcome we arrived at was at least somewhat unavoidable.

                           And when I saw someone on the list suggest

that the quick solution is to wait for MM3,

I'd have to go back and check the exact wording, but I don't know 

if it was intended as being the "quick solution". I think it was intended to be taken as the "least slow" solution, in the overall scheme of open source alternatives. But that's neither here nor there.

                                         well, I guess I let my

frustration show. :-( I do apologize for that.

And I was too quick to lump your response in with others that I 

had seen in the past, and my response was harsher than it should have been. I apologize for both of those mistakes.

Nevertheless, I still want to help. Eventhough I'm no longer with my previous org. (or in international development at the moment), I still want to see solutions for the Dgroups and the Kabissas of the world who want to do good things for people who need a helping hand. So, when MM3 gets moving, I'll be sure to listen in, try to learn what I can from the group, and offer anything that might be of value.

I think that would be very welcome.  We each contribute in our 

own ways, and it sounds like you've got some ideas and methods that are relatively rare. Ideas which might help open further doors for others, and allow them to be able contribute even more, better and faster.

The frustrating part here will be waiting until we can make our 

contributions.

Sound fair? Are we good?

Yeah, I think we're good.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

On Wed, 2005-11-16 at 23:49 -0500, Kevin McCann wrote:

This kind of response is precisely what makes me realize there is a huge disconnect between people who run mailing lists and ONLY want to run mailing lists and those who want to have a mailing list be a critical component of something more useful in the way of comunication and collaboration tools tha thelp build communities of practice. I'm not going to try to make you "get it" but I will say that the right tool for "the job" just doesn't exist yet. It would be nice if it did.

A quick bit of history might help explain why Mailman is the way it is. John Viega wrote Mailman to support mailing lists for some of his favorite bands. We (Ken Manheimer first, then joined by me) resurrected it because we were tired of using Majordomo to support the python.org mailing lists. So Mailman clearly comes out of the open discussion technical and fanboy mailing list tradition. I believe that's why it's very easy for people who want wide-ranging open discussion lists to adopt Mailman and just use it, and why it's more difficult for people to make it do things it wasn't originally intended to do. Not that I think those things are impossible, but they're more painful than they need to be.

I'm not throwing stones, Brad. I've tried to make that clear. And I know precisely what Mailman is and isn't. When it comes to looking at what it isn't, I'm hoping for solutions. It won't happen in MM2, I know that. MM3? I've seen no developments on the MM3 list. I have heard Barry mention he's too busy to really do anything (I think he's hoping for spare time soon but "soon" is elusive for many). If there are significant developments, why are they not mentioned on the MM3 list?

It's a valid point. I will make it a priority over the holidays to discuss MM3 project management on the mailman3-dev list.

I went to the MM3 sprint. I'm on the MM3 list. I tried to get tens of thousands of dollars in funding for it, and came damn close, but things fell through because there was no movement on the project and the potential funders were nervous about it. Barry can confirm that I was trying to get things to happen from my end. Unless there is some sort of MM3 Secret Society that I was not invited to, I think I have followed things as closely as possible.

There is no MSU @$(X...bEDF1abz... (carrier lost)

-Barry

Stephen J. Turnbull wrote:

On the other hand, the ad hoc DB integration ideas I've seen discussed to date all have a very strong "works for me" flavor to them. Ie, "if you want something generalizable, then MM people will have to do that, but here's a proof of concept."

Kevin> despite the fact that people have been begging for it for Kevin> half a decade. They beg for it on the Mailman list. They Kevin> beg for it on the Sympa list.

Beg, build, or buy. Plan A has failed. Try Plan B or Plan C.

I guess I have to watch the words I choose. Erase the word "beg", cease any mental imagery it might be conjuring up. I simply mean that there is a clear requirement for easier integration and people have expressed that need. That's all. Should it be articulated better than it has been so far on this list? Yes, probably.

My suggestion is Plan B, because Plan C probably requires a fork (as you mention, Barry is committed elsewhere). If I needed what you need, instead of posting the call for discussion _here_, I would post on *Mailman-Users*:

(1) summarize the current facilities (like MemberAdapter, however weak they may be),

(2) find out what John Dennis has been up to and what ad hoc patches have been submitted, and summarize their interfaces (not the implementations!),

(3) find out what the objections to the methods described in (1) and (2) are, both from would-be users and from the leading developers, and summarize them.

Rather than look at what has been done in the way of patches I'd be more inclined to contribute toward a design for MM3. I have actually worked on the specs for a SQL-enabled MLM system, based on user and admin requirements. Picture an open source version of Lyris but with better integration capabilities. I'd be happy to share them. I worked on it in 2002, so it might take a bit of digging.

And if in fact you're right, the MM developers don't care, and snub you ... you've lost nothing! You've got the spec you need to attract both developers and backing (including funding)---and the MM2.1 code base to start from.

Absolutely right. It would be nice to see developments happen in the MM project, but ultimately another project may be required to make certain things happen.

• Kevin

Stephen J. Turnbull wrote:

Kevin> Rather than look at what has been done in the way of Kevin> patches I'd be more inclined to contribute toward a design Kevin> for MM3. I have actually worked on the specs for a Kevin> SQL-enabled MLM system, based on user and admin Kevin> requirements.

That would be a fine step forward! Whether it works better than what I suggested depends on how fast you get past the singularity of "I".

Is this fair, Stephen? The word "I" was used to indicate who had worked on the specs, not who would be involved in implementing them. Why insinuate a personality trait here?

The point of the process I described (and its model, ie the Python Enhancement Proposal process) is to show that there's broad or deep support in the community. Whatever else you do, you need to get concrete discussion going to show that there is a group of "enough" users with consensus on what the "important" requirements are, and what those requirements are.

I'd love to engage in such discussion, provided I not get beaten up at every comment, question, idea.

Kevin> It would be nice to see developments happen in the MM Kevin> project, but ultimately another project may be required to Kevin> make certain things happen.

Oh, it probably will end up being another project, and it'll get to something merely useful in about the same effort and time that would have taken the Mailman project to excellence. We've seen that happen often enough. :-(

I'd prefer it not be another project. Which is why I'm hoping for MM3 success. So aside from my initial, poorly-crafted message, which has obvioulsy hit a raw nerve, are we really at odds when all is said and done?

• Kevin

I'd just like to pop in for a second about one thing:

So it's very difficult for them to specify the requirements. They've tried: MemberAdapter was supposed to enable integration of databases for membership management. But it evidently didn't work very well, and it doesn't seem to have gotten a very enthusiastic reception. John Dennis has been pushing that line forward, but I don't get the impression that he is getting tons of good input on the requirements. Instead, he posts here asking the same leadership that didn't really know what the requirements were the first time what they think the requirements might be!

On the other hand, the ad hoc DB integration ideas I've seen discussed to date all have a very strong "works for me" flavor to them. Ie, "if you want something generalizable, then MM people will have to do that, but here's a proof of concept."

I'll admit I've been on the scene for only a few weeks, and I'd not like to speak for fil or Kev Green, but the MySqlMemberAdaptor they developed works fairly easily with the intended design of MemberAdaptor. You are spot on that they have a "works for me" flavor, and I look forward to more MemberAdaptors coming out. I'd disagree with any word that "this will never reach the next Mailman release"; the whole idea of an adaptor is that it will.

Peter

The MySQLMemberAdaptor you guys produced is quite good. IMHO, a final version to be included in 2.2 branch should include:

1. Abstractions of SQL queries on a per list basis, i.e. I don't want to necessarily have to use your schema. Your schema should be the default, but I want to be able to override it to best integrate it into existing user databases.

2. Ability to switch MemberAdaptors through the web interface as well as configure SQL queries through the web interface.

But yeah, that's my pipe dream. I'd be happy to help hack some of that out if that would be helpful.

-jag

On Thu, 2005-11-17 at 19:38 +0100, Fil wrote:

...

I'll admit I've been on the scene for only a few weeks, and I'd not like to speak for fil or Kev Green, but the MySqlMemberAdaptor they developed works fairly easily with the intended design of MemberAdaptor. You are spot on that they have a "works for me" flavor, and I look forward to more MemberAdaptors coming out.

Well, it currently has a "works *not* for me" flavor and I'd appreciate som help on the two outstanding bugs I still have (in my version, but I guess Kev's version has them too).

1. if the DB connection dies or fades away, ping() is never called. I suspect this is due to the fact that the function self._prodServerConnection is not called with parenthesis. However if I add the parenthesis, I get bumped by an error because the function ping() itself does not exist at this point.

2. There's an issue with the bounce daemon not properly calling the __init__ sequence (or so it seems), and I get failures when I activate this daemon.

In both cases, I'd appreciate feedback, a hint to documentation, plain tests or even a full patch :)

                                 ***

As for the main topic of this thread, I think the issues of building a developers' community are very similar in many projects. In brief, you need to be familiar with the code to enter the community, and at the same time you almost need to belong to the developers' community to (have the incentive to) get familiar with the code.

This kind of loop has plagued my main pet project for years, and we're trying to solve it by opening parts of the development, in two directions:

1. modular code (MemberAdaptors are a good idea, in this sense)
2. a common and open development platform for modules (imho SourceForge sucks) where people test each other's patches and cooperate a lot.

-- Fil

---

Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/jag%40fsf.org

Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

-- Joshua Ginsberg <jag@fsf.org> Free Software Foundation - Senior Systems Administrator

On 11/17/05 11:24 AM, "Joshua Ginsberg" <jag@fsf.org> wrote:

2. Ability to switch MemberAdaptors through the web interface as well as configure SQL queries through the web interface.

This comment led me to think "ah, most people working on the development will be running Mailman on servers to which they have root access. The needs of the increasing body of CPanel installation users and other hosted situations need to be accounted for, and that will be hard for the developer group."

--John (running Mailman on servers to which I have root access and never having "sat in front of" CPanel)

Mark Sapiro wrote:

in _prodServerConnection(), you can't do

if ping(self.connection) == 0:

you need to do

if self.connection.ping() == 0:

If looked at this a bit more closely, and it turns out not to be quite that simple. The major issue is the way things currently exist in your MysqlMemberships.py, rev 19, 'connection' is a local variable in the MysqlMemberships class __init__ method. It is not visible to _prodServerConnection() at all.

I've made a few changes to MysqlMemberships.py, rev 19 which I _think_ will make it work. I did a test of a highly stripped down version which worked, but this didn't come close to the real thing.

My suggestions are attached as MysqlMemberships19.patch.

-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

I'm running the patch as is, and there's trouble. Here is the error log:

OK, good. A quick fix is to add on line 153 self.cursor = self.connection.cursor()

The correct fix would be to isolate the connection/reconnection procedure in a unique function, but that'll be for later.

-- Fil

Fil wrote:

Unfortunately this still doesn't succeed reconnecting to the server: I get this traceback:

File "/var/local/mailman/Mailman/MysqlMemberships.py", line 141, in _prodServerConnection if self.connection.ping() == 0: OperationalError: (2006, 'MySQL server has gone away')

Apparently the ping() method is not the way to test if the connection is still viable. I suppose you could wrap it in a try: and treat the exception the same as a non-zero result, but I don't know if that's appropriate.

I don't think I can be of further help on this one. Sorry.

You need someone who's familiar with the methods in the MySQLdb module. Maybe Kev Green can help.

-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 11/23/05 3:38 AM, "Fil" <fil@rezo.net> wrote:

...

Unfortunately this still doesn't succeed reconnecting to the server: I get this traceback:

File "/var/local/mailman/Mailman/MysqlMemberships.py", line 141, in _prodServerConnection if self.connection.ping() == 0: OperationalError: (2006, 'MySQL server has gone away')

A little bit of hacking makes this problem less painful, but I'm sure I got it wrong, or it is that my MySQL server "goes away" quite a lot. I don't know for sure, anyway, but if you're currently trying my version of MysqlMemberships.py you might want to update

the patch is @ http://trac.rezo.net/trac/rezo/changeset/59

-- Fil

What sort of setting does the MySQL server you are running have for timing out idle connections? Could it be that you aren't "hitting" it often enough? In which case, going away a lot is normal.

(I have that problem regularly with a desktop query client (CocoaMySQL for Mac OS X). It responds to a vanished connection by hanging unresponsively until killed...your code is clearly better than that. ;-))

--John

Hello,

I have updated the MysqlMembership extension http://trac.rezo.net/trac/rezo/browser/Mailman/MySQLMemberAdaptor/MysqlMembe...

I would appreciate it if someone could verify that this patch indeed potentially fixes the "TypeError" bug we used to get on some SetBounceInfo: http://trac.rezo.net/trac/rezo/changeset/72 (it was, I think, a question of priority between the + and % operators)

As usual see the readme file @ http://trac.rezo.net/trac/rezo/browser/Mailman/

-- Fil

On 17 Nov 2005, at 10:49, Ian Eiloart wrote:

On 16 Nov 2005, at 21:30, Kevin McCann wrote:

...

MLM developers do not buy into the concept of making the three main data stores available in SQL (those being list config, member and message archives)

Add to that Sender privacy rules, so that the Mail Transport Agent can do SMTP time rejections. Bouncing messages isn't really acceptable these days, since sender addresses are so frequently
forged.

Er, sorry, I didn't mean here to support the "MLM developers do not
buy into" part of the above. I think that what's planned for MM3 IS
to support SQL access for those things. SQL provision should be
enough that portal projects can then build list modules around
Mailman. Actually, they already could - at a push.

Seems to me the big misunderstanding in this debate lies around the
differences between Mailman 2 and Mailman 3. As far as I can see,
they are big differences - of the sort that Kevin is looking for.
Unfortunately, they're not highly visible - probably due to the fact
that they're at the early stages of development.

Perhaps someone should point Kevin at the Mailman 3 codebase, so he
can develop views from there.

-- Ian Eiloart Postmaster, IT Services University of Sussex

Warning: The Surgeon General Has Determined That Windows Can Be
Dangerous To Your Health <http://news.bbc.co.uk/2/hi/uk_news/4405042.stm>

At 10:49 AM +0000 2005-11-17, Ian Eiloart wrote:

...

MLM developers do not buy into the concept of making the three main data stores available in SQL (those being list config, member and message archives)

Add to that Sender privacy rules, so that the Mail Transport Agent can do SMTP time rejections. Bouncing messages isn't really acceptable these days, since sender addresses are so frequently forged.

I'm not sure I understand.  Are you talking about integrating the 

MTA into Mailman, so that it can directly access the list of allowed senders for a given mailing list and take appropriate action for other addresses before actually accepting a message?

It seems to me that this concept is moving even further towards 

the "kitchen sink" approach, where one group has to have complete control over all aspects of the system.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

On 17 Nov 2005, at 15:00, Brad Knowles wrote:

At 10:49 AM +0000 2005-11-17, Ian Eiloart wrote:

...

...

MLM developers do not buy into the concept of making the three main
data stores available in SQL (those being list config, member and
message archives)

Add to that Sender privacy rules, so that the Mail Transport Agent can do SMTP time rejections. Bouncing messages isn't really acceptable these days, since sender addresses are so frequently
forged.

I'm not sure I understand. Are you talking about integrating the
MTA into Mailman, so that it can directly access the list of
allowed senders for a given mailing list and take appropriate
action for other addresses before actually accepting a message?

Erm, I'm not sure what you mean by "integrating the MTA into
Mailman". I certainly don't mean that Mailman should replace my MTA.
I do mean that my MTA should work closer with Mailman, either by
sharing data, or by letting Mailman reject (not bounce) messages
somehow. "Integrating with" might be a better way of expressing it.

This is going to be an absolutely essential requirement. It's no
longer acceptable to generate email bounces for spam. Blackholing
spam isn't a very good idea either, since false positives go
undetected. The only acceptable options for spam are markup, and SMTP
time rejection. SMTP time rejection has the added advantage of
discouraging spam-bots.

The UK's Joint Academic Network (JANET) recently published an article
"Spam Bounces Considered Harmful". http://www.ja.net/newcert/email/ bounceUKN.html

Let's be clear. The *requirement* is that my MTA should never accept
an email that isn't going to prove undeliverable. Currently there are
two main reasons why it might: a full mailbox, or a Mailman rule. I'm
agnostic as to how the requirement is met, but there are issues of
performance, and configurability to be considered.

It seems to me that this concept is moving even further towards
the "kitchen sink" approach, where one group has to have complete
control over all aspects of the system.

No, I just need to be able to access the rules from some other
system. That means that the rules need to be stored in a manner
that's accessible to my MTA. SQL, LDAP, flat files, I don't really
care, but they do need to be accessible. They also need to be
represented in a manner that's easy to understand. For example, a
list of regular expressions. Exim can easily manage that.

Alternatively, I should be able to have my MTA ask Mailman whether it
will accept the message. It might need an SMTP or LMTP interface to
do that - I don't know LMTP well enough to know whether it can reject
messages.

Anyway, if the intention is to have open rosters (ie rosters that I
can access from any application), then we're pretty close to what's
required. It would help if a roster can include regular expressions
to say things like *@sussex.ac.uk or not *@sussex.ac.uk, and if ban
lists can be expressed as rosters. My MTA (Exim) has ACL rules that
can say things like: if /local/mailman/lists($local_part) is a file, then if $sender is in the list (SQL query|LDAP query) then accept the message else reject the message

Actually, I *thought* this was all in the plan for Mailman 3, but I'm
glad to get the opportunity to make clear a requirement that - for me

• is the single most important requirement that Mailman doesn't
  currently meet easily.

I say "easily", because I *could* have my MTA run a python script to
get the list of permitted senders, but I've not invested any time
into doing that, mainly because I've never done any python programming.

-- Ian Eiloart Postmaster, IT Services University of Sussex

On 19 Nov 2005, at 06:35, Brad Knowles wrote:

At 12:57 PM +0000 2005-11-18, Ian Eiloart wrote:

...

I do mean that my MTA should work closer with Mailman, either by sharing data, or by letting Mailman reject (not bounce) messages somehow. "Integrating with" might be a better way of expressing it.

How do you propose to do that? Each MTA has their own way of
working. Should we pick one MTA and tell everyone that they have
to use that one exclusively and we don't allow any other MTAs to be
used/supported?

Please give me details on precisely what level and kind of
integration would be required.

No, I'm not suggesting that Mailman prefer one mail client over
another. Simply that the ACLs for posting to the list are accessible
in some open format. That is, they should be stored in some
accessible database like an SQL or LDAP database, or even in flat files.

The ACLs themselves should be represented in some open format, like
regular expressions.

...

Let's be clear. The *requirement* is that my MTA should never accept an email that isn't going to prove undeliverable. I'm sorry. That sentence just isn't parsing right now. It seems
like a double or maybe triple negative or something. Can you try
re-wording that?

OK, the MTA should ONLY accept email that it can deliver.

...

No, I just need to be able to access the rules from some other system. That means that the rules need to be stored in a manner that's accessible to my MTA. SQL, LDAP, flat files, I don't really care, but they do need to be accessible. They also need to be represented in a manner that's easy to understand. For example, a list of regular expressions. Exim can easily manage that. Okay, do everything in your MTA. Problem solved. We don't need
any Mailman integration for that.

Well, I'm still hoping that the MLM can handle list creation, and
configuration, and subscription, and bounce handling, and content
filters (if the sender is permitted to post to the list, then I don't
mind bouncing mail to them), and topics, and delivery suspension, and
all the other things that Mailman does.

My MTA can't manage creation of ACLs. Now, I know that Mailman 3
development is focussed on operation, not list configuration, but my
point here is that I want my MTA to be able to share Mailman's sender
ACLs - and that's a design requirement.

Ian Eiloart Postmaster, University of Sussex () ascii ribbon campaign - against html mail /\ - against microsoft attachments

On 21 Nov 2005, at 20:27, Robby Griffin wrote:

On Nov 21, 2005, at 05:25, Ian Eiloart wrote:

...

OK, the MTA should ONLY accept email that it can deliver.

FWIW, Mailman currently avoids doing anything about this for a
documented reason (scripts/post):

# Immediately queue the message for the incoming qrunner to  

process. The # advantage to this approach is that messages should never get
lost -- # some MTAs have a hard limit to the time a filter prog can
run. Postfix # is a good example; if the limit is hit, the proc is SIGKILL'd
giving us # no chance to save the message.

Well, this is OK if the MTA can see the ACL. Then, it can reject or
accept the message before passing it to Mailman.

-- Ian Eiloart Postmaster, IT Services University of Sussex

Warning: The Surgeon General Has Determined That Windows Can Be
Dangerous To Your Health <http://news.bbc.co.uk/2/hi/uk_news/4405042.stm>

At 10:25 AM +0000 2005-11-21, Ian Eiloart wrote:

No, I'm not suggesting that Mailman prefer one mail client over another. Simply that the ACLs for posting to the list are accessible in some open format. That is, they should be stored in some accessible database like an SQL or LDAP database, or even in flat files.

Show me a single open data format that all MTAs understand. 

Hell, there aren't many file formats that they all understand.

The ACLs themselves should be represented in some open format, like regular expressions.

Again, show me a single open format that all MTAs understand.


Trust me, this issue is far more complex than you think it is.

OK, the MTA should ONLY accept email that it can deliver.

Considering that there are anti-spam rules inside of Mailman 

itself that can take on various actions, including rejection of the e-mail, that's simply not physically possible. At least, not without gutting a considerable amount of existing functionality in Mailman.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

At 2:53 PM +0900 2005-11-22, Stephen J. Turnbull wrote:

...

...

...

...

...

"Brad" == Brad Knowles <brad@stop.mail-abuse.org> writes:

 Brad> 	Show me a single open data format that all MTAs
 Brad> understand. Hell, there aren't many file formats that they
 Brad> all understand.

C'mon, Brad, don't let the perfect be the enemy of all improvement.

For access to the ACL database, we really need only to consider two MTAs (at most): Exim and Postfix.

You have to give the MTA direct access to the internal filters of 

Mailman in some sense. I don't think we can restrict ourselves to just these two MTAs.

Moreover, who owns this code?  It crosses the boundary between 

Mailman and the MTAs -- do we have to continue to track their development ad infinitum? Do we have to patch their code?

                                Sendmail has milters for this

purpose; you don't need to do surgery on sendmail itself, just configure the mailman-acl milter.

Mailman-acl milter?  This is the first I've heard of it.  Is this 

a new thing? Who maintains this code?

The point is that if Mailman does a reasonable job of specifying access to a database of list ACLs, people can and will write adapters for their drug of choice. Other MLMs will follow the Mailman spec if it's decent and we get there first. The whole Internet wins.

Okay, I can see Mailman providing a single, hopefully reasonably 

well-specified specification, and letting everyone else adapt. That's a far cry from what Ian was talking about.

The ACL format is a much tougher requirement, and will require a lot of thought. Do we want to specify archive ACLs in the same database? How do we condition access on the various authentication methods that users may use? "Like regular expressions" means exactly what? Etc, etc.

Again, if we specify and mostly implement such an interface well, the users will come and they'll make their other tools work with it.

I'm willing to go that route.  But you do seem to agree with me 

that this problem is going to be a lot tougher to solve than Ian implies, yes?

If you want to say that specifying these formats _well enough_ to attract users and to make related products (both substitutes like Majordomo and complements like Postfix) willing to support them is "too hard" or "is a project for MM3 or maybe MM3.1", I'll defer to you on that.

I am not convinced that this is a question that can be answered 

within the overall scheme of MM3. I'm willing to be proven wrong, but what I know of the problem is that it's a much bigger mountain than I think it's being given credit for.

       But saying "it's gotta be perfect on introduction or it's no

good" is not a way to communicate.

That is not an accurate characterization of what I was saying. 

Re-read my previous message to Ian -- I said to him:

	Trust me, this issue is far more complex than you think it is.

Now, if you can agree that this is a big issue that will need to 

have a lot of thought and work put into it, and it not something you're likely to knock out on a single late-night hacking session, well that's all I'm really asking for.

It's one thing to identify a desired long-term design goal.  It's 

another to imply that this will be trivially easy to implement.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

On Tue, 2005-11-22 at 17:18 +0900, Stephen J. Turnbull wrote:

...

...

...

...

...

"Brad" == Brad Knowles <brad@stop.mail-abuse.org> writes:

To get all the way to "MTA doesn't accept anything that MM is going to refuse to deliver", yes. But we certainly could define an interface to an auxiliary service such that if the MTA tells us (AuthType, AuthCredentials, EnvelopeSender, Sender, From, To) we'll say "nuke it" or "OK, we'll handle that". That would allow us to push a lot of simple cases (such as "authenticated user not a list member") back to the MTA, and never invoke the heavy equipment of Mailman itself.

I think that's an acceptable goal. My own feeling is that the way to accomplish this is through an out of band (though well-specified) communication protocol, probably based on some standard interoperable IPC, like XMLRPC or some such.

It's so new it's vapor.<wink> The point is that while I worry that Exim and Postfix may require source patches, sendmail can be done as a separate milter module. This is exactly the kind of thing that the milter API was designed for.

I'm fairly confident you can extend Postfix in this direction, though I haven't done it myself. And I'd be very surprised if you couldn't do it in Exim, that being traditionally a very extensible MTA.

Well, you may have a bigger problem in mind, but 90% of Mailman users would get a big bonus from getting only halfway there. Or maybe you know something I don't.

Actually, I'll bet 90% of Mailman users won't care. :) But the other 10% will care A LOT, so it's worth thinking about.

Speaking for myself, I see nothing wrong with asking the bleeding edge adopters to change MTAs, and I see nothing wrong with restricting Mailman-supplied MTA code to a couple of MTAs that we "like". Of course we help other MTAs to incorporate the feature, but we don't need to promise that they can use it in advance. Some MTAs may never get it.

I definitely want us to think about ways to improve the entire tool chain, from MTA to archiver. I'd prefer solutions that are geared more toward interoperable specifications first, followed by implementation and/or patches to the tools in those chains. For example, we can publish X-* headers, or XMLRPC specs, etc. By striving for MTA, web server, and even MLM agnosticism where possible, we can push the entire state of the art forward.

A good example of something I've looked for for a long time is an interface between MLM and archiver, where the MLM could anticipate the archiver URL without having to actually archive the message first. If we had that spec, we could include the archive URL in an X-header or a footer when we send the copy to the list members.

Definitely. That's been my main theme both in this thread and in the SQL database thread---simply specifying requirements is a big job. The users know the effects they want, but UI/API/implementation constraints haven't been presented at all. And users won't really be able to express them until they've got a prototype to work with.

One of the things I plan on doing is to get a new project wiki up where we can start collecting all this information. That's all part of my plan to improve project management, which I'm personally committed to addressing by year's end.

-Barry

At 5:18 PM +0900 2005-11-22, Stephen J. Turnbull wrote:

To get all the way to "MTA doesn't accept anything that MM is going to refuse to deliver", yes. But we certainly could define an interface to an auxiliary service such that if the MTA tells us (AuthType, AuthCredentials, EnvelopeSender, Sender, From, To) we'll say "nuke it" or "OK, we'll handle that". That would allow us to push a lot of simple cases (such as "authenticated user not a list member") back to the MTA, and never invoke the heavy equipment of Mailman itself.

The problem is that this process is basically what Mailman goes 

through anyway, and I don't see any reason why we would want to develop code to duplicate all that functionality -- you'd still have to touch all the same Python pickles, deal with file locking contention, etc.... Moreover, that's going to be a heavy process, and MTAs have some very tight constrictions upon them to get a response back to the sender in a reasonable amount of time.

I agree that the feature is desirable, but short of a complete 

ground-up rewrite, I don't see any way to do it. Morever, as stated in my previous response, I think that a lot of thought needs to go into this before we even get to the design stage.

Given that MM3 is already early in the implementation phase, we 

may have to wait quite a bit longer, especially since the designers and developers are few and far between, and what time they have is already very tightly spread.

Not "have to", but that may be the most effective way to jump start it, may even be necessary for jump starting. It's not our "job", but it's something we may need to budget resources for if we want to do it. Of course if those resources are large and won't come from fresh blood, that kills the idea. But we haven't shown that the needed resources are large, and there are a couple of people who I don't recognize as frequent code contributors who are pushing hard---there might be fresh blood out there to do this.

If there are people who want to push forward on this now, I think 

they'll have to wait until the specifications and design phase are done, before they can start hacking on their favourite MTAs. Either that, or everyone starts inventing their own APIs and hacking their MTAs to support those APIs, and then we'll have a half-dozen different competing solutions that we've got to try to support.

People should definitely get some real-world experience in how to 

do this sort of stuff, but no one should have any real hope of getting their personal pet version incorporated into Mailman. The Mailman developers might take advantage of their experience in designing their own API, but that's about as far as they could hope to go.

Was it? I don't know. My feeling is that Ian is lacking some necessary facts, and consequently expressed his requirements in terms that can't be satisfied. But if we give him the details and withhold judgment on the substance until he's reformulated, we might find that what he really wants is a lot closer to what would be do-able than our initial impressions are.

Okay, fair enough.  Let's give him time to re-formulate what he's 

asking for.

I think so. Solving them for Ian probably is not too hard.

That depends on what MTA Ian is using, and how much code in the 

MTA that Ian is willing (and able) to hack himself, or get someone else to hack for him. Likewise, there will probably be a fair amount of hacking on Mailman that would need to be done to support his particular requirements, and a lot would depend on who's going to be doing that hacking and how much time (and experience) they've got to put towards that effort.

                                                          Solving

them in a way that generalizes is going to be hard. But Python is a very good environment for such generalization.

The Python part is not the hard part of the equation.  The hard 

parts of the equation are going to be the API specification and design process, hacking on the MTA code, etc....

Well, you may have a bigger problem in mind, but 90% of Mailman users would get a big bonus from getting only halfway there. Or maybe you know something I don't.

I disagree.  I don't think a 50% solution is going to help many 

people, and it certainly won't help them for long.

It will just push out the event horizon a bit, and since we have 

exponential growth of spam and other forms of mail abuse, it's going to be a very short period of time before we're right back where we were -- and where there are no more easy 50% solutions possible, and where we have yet to spend any time stepping back from the problem a bit and working on a better 80-90% solution.

     Of course you meant something substantially weaker than "all

MTAs must understand as an absolute requirement", but surely it was something at least as strong as "we cannot ask users to change MTAs to get this feature." Agreed?

If they're using any of the mainstream MTAs, then they should be 

able to expect that they can use virtually all of the features of Mailman. They might be able to use other programs to fulfill other components of the system (e.g., the people who use mhonarc instead of pipermail), but that should be an optional additional extra enhancement that they can add to the base system.

As I see it, the kind of thing you're talking about would be a 

core component, one that could not be easily changed or substituted, and if it's going to be useful at all it will need to support all the major MTAs -- or be supported by them.

Speaking for myself, I see nothing wrong with asking the bleeding edge adopters to change MTAs, and I see nothing wrong with restricting Mailman-supplied MTA code to a couple of MTAs that we "like". Of course we help other MTAs to incorporate the feature, but we don't need to promise that they can use it in advance. Some MTAs may never get it.

The problem is that there are way too many whackos who then take 

those bleeding edge features and throw them into CPanel or a vendor-proprietary version, and then dump off the after-sales support onto us.

And way too many whackos will take whatever code is available and 

blindly install it, without really understanding that they need to have a certain minimum level of competence before they take bleeding edge code and try to start working with it.

Many vendors restrict who has access to development beta code, 

and for good reason. If you're a MM3 developer or want to be, then you should be free to play with that code about as much as you want -- after all, the only risk should be to you and your mailing lists.

But you should anticipate that many people may expect you to hack 

on the code yourself to fix whatever problems you may end up creating for yourself.

OTOH, I wonder if you're not overestimating the degree of difficulty and underestimating the benefits of the structural work required.

I don't think I'm underestimating the benefits of the structural 

work. We won't know whether I'm overestimating the degree of difficulty until the actual work is done and we can look back and see what it took to get where we did.

The "database backend" is _not_ structured that way yet,

Which is a large component of my caution on this topic.  Code 

hacking is easy. Database hacking is harder, because you've not only got to hack the code which accesses the database, but also the database structures that would be accessed. More importantly, since you're talking about exposing some aspect of this database structure to the outside world, you can't just take the easy way out and directly access everything but you instead have to work through a published API layer.

In programming, global variables make a lot of things easier in 

one aspect, but they greatly complicate life in others. It takes more work to write code in such a way to avoid global variables, but in the end it usually pays off.

Working through published API layers has a lot of benefits too, 

but it also takes a lot more work to make that happen -- compare the classic C version of "Hello, world" to what you have to do to get the same results with a standard X11 library. The source of the X11 version is likely to be many times larger than the compiled code of the C version.

In this case, you're asking us to create the equivalent of the 

X11 API and library, plus the associated database structures, etc....

Yes, we most likely have a smaller overall problem space, but we 

also have many fewer programmers, each of whom is probably spending less time on this work, and there are many classes of pre-existing programs that we will have to somehow find interoperability with.

-- Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

SAGE member since 1995. See <http://www.sage.org/> for more info.

...

...

...

...

"Ian" == Ian Eiloart <iane@sussex.ac.uk> writes:

Ian> No, you've evidently completely misunderstood me. All I do
Ian> want is for Mailman to use *exactly* the same mechanisms for
Ian> sender ACLs that it does for rosters. If that's LDAP, that's
Ian> fine. If it's SQL, that's fine. If it's flat files, that's
Ian> fine.

As Brad says, that's easier said than done. As it happens, Mailman does (by default) use exactly the same method---it's called a "Python pickle".

But a Python pickle is an _object_ database. LDAP is about tree-structured databases, SQL about relational databases. It's non-trivial to put most of the stuff in Mailman into those formats.

The problem isn't really that pickles are Python-specific, that's what Fred Brooks would call an "accidental" obstacle. You're right, we could, at a fairly predictable amount of effort, convert from a Python-specific data store to a standard one accessible by a well-defined protocol such as LDAP or SQL. The problem is that the data used by Mailman doesn't necessarily fit those models, and we won't know how hard the conversion is (ie, what effects there will be on various parts of Mailman) until we do it. It's a good bet that it will be harder than it looks.

Regards,

-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

...

...

...

...

"Ian" == Ian Eiloart <iane@sussex.ac.uk> writes:

Ian> Now, sender ACL's already are *just* lists of regular
Ian> expressions. So, what's so hard?

You're aware that "regular expression" is not exactly well-defined, and that python REs are a unique dialect among regular expression languages?

-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

...

...

...

...

"Ian" == Ian Eiloart <iane@sussex.ac.uk> writes:

Ian> I think Python regular expressions have enough in common with
Ian> the PCRE (perl compatible regular expressions) library to be
Ian> useful. For example, I think ^[^@]*@(.*\.)?sus(se)?\.ac\.uk$
Ian> has the same meaning in both dialects.

Ian> If I'm wrong, then I can make do with .*sussex\.ac\.uk$ and
Ian> .*susx\.ac \.uk$

The semantics of what you just posted are "I would like Mailman to impose and maintain a hack appropriate to my site on all users of this facility." I see the attraction of this proposal for a fair number of users like you, and it's precisely what open source is for: you can maintain your own variant, in cooperation with those other users.

But when I suggested a 50% solution would be good enough, the 50% I meant was precisely what you're leaving out: a specification of the syntax (you deliberately leave the syntax vague!) and semantics of the data that Mailman passes to the MTA.

-- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

On Fri, 2005-11-18 at 12:57 +0000, Ian Eiloart wrote:

Let's be clear. The *requirement* is that my MTA should never accept
an email that isn't going to prove undeliverable. Currently there are
two main reasons why it might: a full mailbox, or a Mailman rule. I'm
agnostic as to how the requirement is met, but there are issues of
performance, and configurability to be considered.

The problem as I see it is either that the MTA has to make this decision synchronously in the middle of its SMTP conversation. It's unlikely that Mailman will store its data in anything other than a natural format for itself (e.g. using Python regexps), so either your MTA has to be taught to understand that Mailman-native data, or we need some out-of-band way to get Mailman do that calculation.

One of the things I'd really like to move to in 3.0 is using the maildir format as the preferred way to get messages into Mailman's incoming queue. Actually, that all works in the current 3.0 code. What's nice is that it completely decouples Mailman from the MTA, but from your perspective above, that's also what sucks.

It's definitely a possibility that we could prevent Mailman from ever bouncing unacceptable messages.

Anyway, if the intention is to have open rosters (ie rosters that I
can access from any application), then we're pretty close to what's
required. It would help if a roster can include regular expressions
to say things like *@sussex.ac.uk or not *@sussex.ac.uk, and if ban
lists can be expressed as rosters. My MTA (Exim) has ACL rules that
can say things like: if /local/mailman/lists($local_part) is a file, then if $sender is in the list (SQL query|LDAP query) then accept the message else reject the message

Actually, I *thought* this was all in the plan for Mailman 3, but I'm
glad to get the opportunity to make clear a requirement that - for me

• is the single most important requirement that Mailman doesn't
  currently meet easily.

The current MM3 schema does express rosters in a table, and regexps are allowed in those rosters (IIRC), but be aware that they are going to be Python regexps, and that may not match the regexp language that Exim (or any other non-Python application) will understand.

I say "easily", because I *could* have my MTA run a python script to
get the list of permitted senders, but I've not invested any time
into doing that, mainly because I've never done any python programming.

That's the easy part. :)

-Barry

I'm finishing up a drupal module (in PHP) that wraps the CLI of mailman. Everything works EXCEPT getting at the moderation bit. And there is a bit of duplication with the userbase, since the pickle file is a bit inaccessible to anything outside Python.

There is an option in the script to use the MYSQL patch of fil and Kev Green, which is quite a nice patch and hopefully will be in the next mailman release. If you use this option, then full mailman functionality is available.

http://cvs.drupal.org/viewcvs/drupal/contributions/modules/mlist/#dirlist

Peter

On Wed, 16 Nov 2005, slave@codegrunt.com wrote:

...

Hello. How might one configure Mailman (version 2.1.6) list settings using PHP scripts?

The quick answer would be to not bother and wait until version 3 of Mailman which will use a MySQL backend and instead manipulate that directly (if I have been reading past discussions correctly). Calling the existing Python code via PHP exec() calls is plain out yuck.

=)

Cheers

http://industrial.org

---

Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/peterjh%40mennonot...

Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp