Mailman 3 May 2014 - Mailman-Developers

before next release: disable backscatter in default installation
by Jo Rhett 07 Nov '22

07 Nov '22

Hi. There's a fairly simple problem here that needs to be addressed. And it's mostly a documentation/install problem. I'm hoping we can get this resolved before the next release. PROBLEM: Mailman comes out of the box ready to backscatter spam people. Yes, it's easy enough to fix. But because it comes stock this way, and is documented to install this way, most people install it to do this. Those of us who work in abuse departments are tired of hearing "well that's how Mailman works". We also object to having to teach people how to fix their mailman installations because it's not documented in the current manual. This is *exactly* like Sendmail 14 years ago. We didn't accept it then, and Sendmail fixed the problem. RESOLUTION: Mailman default installation should not backscatter in a default configuration. 1. Don't create backscatter aliases for subscribe/unsubscribe/etc by default. Nearly everyone uses web based signup. 2. Discard or hold messages from non-subscribers by default. I would think that it would be perfectly reasonable to have documentation on how to enable the 1980s-style -request / -subscribe etc aliases. However this documentation should have a note that this is against the AUP of nearly every network provider, and enabling it will likely cause them to get listed in various blacklists as a backscatter source. FYI: I know that this goes against the instincts of many old-time mailing list advocates here. But after dealing with a 10k/hour backscatter DoS my tolerance for this problem is understandably limited. Yes, it was a sweet day back in the 1980s. I was running a mailing list server and several UUCP gateways at the time, so I remember them well. But those days are past, and we need to deal with the reality of today. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness

21 168

Thinking about list footers
by Stephen J. Turnbull 02 Jun '14

02 Jun '14

Murray S. Kucherawy writes: > What's the expertise on the idea of adding footers in a new MIME text/plain > part rather than just bolting it onto the text as-is? (Or is that already > done?) What do MUAs generally do with multipart text/plain bodies? As Mark and Barry point out, the MUAs-for-people-most-vulnerable-to- email-fraud often handle them poorly. Also, the last time partial signatures came up, it was pointed out that there are *no* MUAs that differentiate between signed parts and unsigned parts. You don't get a warning when your eyes move from a signed part to an unsigned part or vice-versa the way you do when following a link from an HTTP URL to an HTTPS URL in a browser. The DKIM advocates have not liked the idea of signatures that don't apply to the whole message at all. > And along those lines, do any MUAs do useful things with the > various List-* fields, other than permitting one to sort on them? I think many do. I had a proposal some years back that I discussed with the Mozilla people. The idea was to devise an algorithm for MUAs that would get rid of Reply-To munging in most cases, and an optional header field that would allow lists to express a preference. They thought it would be nice if someone would write up a document but weren't much interested in helping or implementing, they thought their products already did a good job. There were some refinements but the basic idea was 1. If there is a Reply-To:, use that address, otherwise 2. if there is a List-Post:, use that address, otherwise, 3. reply to the address in From:. The optional field (I even forget the name, it was something like List-Prefer-Reply) allowed giving From: priority over List-Post:. I'm pretty sure that Thunderbird, Mutt, and Emacs/Gnus implement either a "smart" reply algorithm or or a reply-to-list function. Other Emacs-based MUAs probably do, and it would be trivial to add in most cases. I don't know about KMail, Sylpheed, and Evolution. On the other hand Windows and web-based MUAs didn't do much useful at the time, and probably don't now, either.

2 1

Re: [Mailman-Developers] Thinking about list footers
by Andreas Schulze 31 May '14

31 May '14

Murray S. Kucherawy: > And along those lines, do any MUAs do useful things with the various List-* > fields, other than permitting one to sort on them? Hello, I think, that is the real problem: There are RFCs 2369 and 2919 and virtually no MUA implement them *usefull*. I do sort my inbound messages by list-id too. But no MUA 'show' me * unsubscribe * request list-help * open list-archive And because no MUA show such information to endusers MLM still has to modify subject and append a footer :-/ Let's talk to MUA developers! Andreas

1 0

Re: [Mailman-Developers] [GSoC 2014] Mailman CLI Project
by Abhilash Raj 30 May '14

30 May '14

On Fri, 30 May 2014 18:12:18 +0530 Rajeev S <rajeevs1992(a)gmail.com> wrote: > I agree that an override for the confirmation message is necessary so that > the CLI > commands can be used in scripts and I would like to follow Barry's > suggestion of using > a `--yes` or `--force` switch. (Isn't --force more common than --yes, as in > `rm -f` ?) No, as I think, `--yes` and `--force` or `-f` have different use cases. For conformations `--yes` is used most of the times, it says 'hey I know you are going to ask this question and my answer would be yes'. While `--force` is to actually *force* any operation that cannot be completed otherwise. > About the create domain suggestion, Is the follow up question on `*create > domain*`, upon missing domain, > such a bad idea? A user already can `add a domain` by using the web > interface. If that is not ambiguous, > neither is this, right? > > However I see this issue, if a newbie user types in an email address in > place of the list name, probably > on misreading the command format, he would end up creating a domain for his > mail provider. > > *$./mmclient create list --help* > *Creates the list list(a)domain.org <list(a)domain.org>* > *$./mmclient create list user(a)gmail.com <user(a)gmail.com>* > *The domain gmail.com <http://gmail.com> does not exist.Create one?[y/n] y* > *$* > > @Abhilash > > I have written unittests for methods like connect and create, but have to > refine them more. > Hence the tests won't come up in this revision. Good to hear that. Did you give TDD a try? > Also, I would like to ask how the tests should be triggered. Should > there be a `mmclient test` with a set of possible switches like `all` > ,`domain`, `list`, `user` etc? > Or should the customary `python setup.py test` be used? You can use `nose2` for that, like mailman does. It automatically discovers all the tests in directories specified and runs them. > > I also propose to build a a *`describe user`* command that gives a detailed > profile of the user. > Verbose listing using tables is not so effective users as most of the > attributes are lists. What detail profile are you talking about? Apart from addresses and name(optional) does mailman store any other data about a user? You can show the lists he is subscribed to, but `describe user` may not be an appropriate command for that. Maybe you can add it in users scope itself like `mmclient user a(a)b.org --list-subscriptions` Others may have different thoughts though. -- thanks, Abhilash Raj

2 1

Re: [Mailman-Developers] Thinking about list footers
by Mark Sapiro 30 May '14

30 May '14

On 05/30/2014 10:35 AM, Murray S. Kucherawy wrote: > > What's the expertise on the idea of adding footers in a new MIME text/plain > part rather than just bolting it onto the text as-is? (Or is that already > done?) What do MUAs generally do with multipart text/plain bodies? This is what Mailman currently does with msg_header and msg_footer in all cases *except* the common one where the message after content filtering is a single-part text/plain message with known character set. Mailman could easily do this in all cases, but there would be many complaints due to common MUAs that don't display this well. MUA's deal with this differently. Some are quite reasonable and just display all the parts inline in the sequence in which they appear, although they may also indicate that there are 'attachments'. Others display only the 'body' and one is required to open or download and open the attached footer. Really pathological ones, including IIRC MS Outlook will in cases where a msg_header part is added, display the msg_header part as the text of the message with the original body and footer as attachments requiring additional user action to see. Also, it is not clear what the legal situation would be in jurisdictions that require an unsubscribe link in list mail. Is a link in a separate MIME part that may not be displayed inline by some MUAs satisfaction of that requirement. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

1 0

Re: [Mailman-Developers] Thinking about list footers
by Barry Warsaw 30 May '14

30 May '14

I'm really sorry I haven't had time to catch up on the various DMARC threads, but I'm hoping to do so early next week. On May 30, 2014, at 10:35 AM, Murray S. Kucherawy wrote: >What's the expertise on the idea of adding footers in a new MIME text/plain >part rather than just bolting it onto the text as-is? (Or is that already >done?) What do MUAs generally do with multipart text/plain bodies? We do that already if we can't just bolt them on, e.g. if the charset or MIME type for the footer doesn't match the text body. Unfortunately, we also get complaints from users because MUAs don't handle them very well. http://wiki.list.org/pages/viewpage.action?pageId=4030707 >And along those lines, do any MUAs do useful things with the various List-* >fields, other than permitting one to sort on them? Some do. Mine at least handles List-Post pretty well. Cheers, -Barry

1 0

Thinking about list footers
by Murray S. Kucherawy 30 May '14

30 May '14

Right now list footers are a common feature of Mailman and other MLMs. The typical way of doing this is to just append some content, probably after "--", that indicates the location of archives and maybe some administrative information (unsubscribe instructions, for example). What's the expertise on the idea of adding footers in a new MIME text/plain part rather than just bolting it onto the text as-is? (Or is that already done?) What do MUAs generally do with multipart text/plain bodies? And along those lines, do any MUAs do useful things with the various List-* fields, other than permitting one to sort on them? I'm brainstorming on some possible solutions to DMARC and DKIM matters, and I don't want to reinvent any wheels. Thanks, -MSK

1 0

Re: [Mailman-Developers] [GSoC 2014] Mailman CLI Project
by Abhilash Raj 30 May '14

30 May '14

On Fri, 30 May 2014 18:12:18 +0530 Rajeev S <rajeevs1992(a)gmail.com> wrote: When you are not citing any context from previous posts, you should start a new thread instead of re-posting on same old one. It makes it difficult to find a specific post with this long thread. -- thanks, Abhilash Raj

1 0

Re: [Mailman-Developers] [GSoC 2014] Mailman CLI Project
by Rajeev S 30 May '14

30 May '14

Hi, I have pushed the revision 53, that has the following changes. 1. Refactoring as mentioned in the comments by Steve 2. Grouping of options and sub commands using subparsers 3. Changed the format of the command from *mmclient <scope> <action>* to *mmclient <action> <scope>* 4. Replaced ambiguous action *list* with *show* The change mentioned in point 2 solves many/most of the problems raised during the recent discussions. The options are now paired with their corresponding sub commands. The implications of this change is that the extra options/arguments raise command syntax errors, which were ignored earlier. Further, an great extent of leniency can be allowed in the command format. To be specific, the parser can support the commands *mmclient create list --list list1 --domain domain.org* and *mmclient set --key "foo" --value "bar"* at the same time. Note that the second command has not specified a scope.(PS: scope as in list, domain, user etc, and not site-wide, domain etc). The change also prevents argparse from generating a dictionary with all possible options (which may be huge). The agreed change to add a positional argument for the *name* (commands of the form *mmclient create list list(a)domain.org*) has been postponed to the next revision, as this revision mostly contains code refactoring. http://bazaar.launchpad.net/~rajeevs1992/mailman.client/mailmancli/revision…

5 13

Re: [Mailman-Developers] [GSoC 2014] Mailman CLI Project
by Rajeev S 28 May '14

28 May '14

On Friday 09 May 2014 11:12 PM, Stephen J. Turnbull wrote: > I would like to post this to Mailman-Developers, with your permission. > > Rajeev S writes: > > > Hi Steve,Can I go forward with the current design? > > Looking at the code you've produced so far[1] I can say it's clean and > (mostly) Pythonic, easy to read, making good use of packaged modules > (and good choices of the ones to use). > > But to be honest, you haven't really *presented* a design yet. A file > tree as in your blog[2] isn't a design, and the descriptions of > commands really amount to a set of example use cases, but hardly > complete. The design is implicit in patterns in the examples, and I > have a feeling that it isn't yet coherent in your mind. I only meant to expose my approach towards the project. Now, at least you know how I have planned to approach the task at hand. > So you can go forward, but if you continue to write code "as you go > along", I think every couple of weeks we're going to have to refactor. > That's OK if that's what you want (as you say in your blog > "discussions over a piece of code would be more effective than just > talk"). The alternative is to systematically explore the space of > command + argument + option combinations, and try to come up with a > language that compactly expresses the most common operations. I > recommend this approach if it works for you, because various options > are starting to proliferate, and I would rather we keep them to a > minimum so that users don't need to remember them all or be > continuously consulting the documentation. It may *not* work for you, > and that's OK, we can go with the refactor-until-happy strategy then. I too would prefer to go by the systematic approach. I felt the best way to express my thoughts was to implement some part of it. I am refactoring too much already. > A couple of examples of the kind of things I think it would be useful > to think about in advance: > > (1) You already have --domainname and --listname options, and of > course the pattern suggests --username for sure, and maybe there > will be additional --foonames. I wonder if it wouldn't be better > to have a single --name option so that instead of > > mmclient list create --domainname list.example.com --listname foo > > the user types > > mmclient list create --name foo --in-domain list.example.com > > or > > mmclient list create --namefoo(a)list.example.com I doubt the usability of a common option `name`. As per the above snippet, you have used a `in-domain` to specify the domain name. There are many such instances where you would have to use more than one `name`, for instance adding moderators for a list. In such cases, options like `domainname`, `listname` and `username` would be more intuitive than using options like `in-domain`. Further, the option `name` is ambiguous for a `user`. > or even > > mmclient list createfoo(a)list.example.com > > Note that the last is "Pythonic" in the sense that your > ArgumentParser will check for the right number of arguments. This is a good way to do it, a more pythonic way,as mentioned. Imposing a fair enough rule like the list must be specified only in `list@domain` format can reduce a number of arguments in a *lot* of commands. By using a third (optional) positional argument that specifies the `name` of the instance being managed, the commands look more cleaner. However, some commands still need more names to appear on the command which I prefer *not* to replace by a single `--name`. The commands are mentioned below. > (2) Should the scope (in [domain, list, user]) come first as in your > current design, "mmclient list create", or should the action come > first as in English, "mmclient create list"? I think the latter > will be easier for non-technical admins to get used to, but it may > not be -- we need a list of "things we want to say" to decide that. Commands that makes sense in English would suit most situations, in fact, all situations in current list of use cases. As the first step of the systematic design process, I present the list of things we would want to say and how the command must look like and what possible arguments it should accept. I have modified the CLI to use English like commands and hence will use them hereafter. *list* The command lists the entities and should be available for users,mailing lists and domains. mmclient list list [list(a)domain.org] [-v for verbose] mmclient list user [list(a)domain.org] [-v for verbose] [--role ROLE] mmclient list domain [domain.org] [-v for verbose] The ROLE would be any of member,owner or moderator. *create* Used to create an instance and should be available for domains, lists and users. mmclient create domain domain.org [--contact CONTACT_EMAIL] mmclient create list list(a)domain.org mmclient create user foo(a)bar.com --name DISPLAY_NAME --password PASSWORD *delete* Removes a domain, list or user. mmclient delete domain domain.org mmclient delete list list(a)domain.org mmclient delete user foo(a)bar.com *Role management* User roles can be managed by two actions, addrole and removerole, rather than 6 separate actions for subscribe, unsubscribe, addowner, addmoderator, removeowner, removemoderator mmclient addrole user foo(a)bar.com --list list(a)domain.org --role <member,owner,moderator> mmclient removerole user foo(a)bar.com --list list(a)domain.org --role <member,owner,moderator> The above commands are an argument longer than the commands like mmclient addmoderator list list(a)domain.org --user foo(a)bar.com but I feel former approach better, as it looks better and reduces the number of commands to learn/remember. *Preferences/settings* Settings would form a new `scope`, commands for which can be implemented in a fairly straightforward way. mmclient set setting <setting name> --value <value> *Edit (user)* Changing user credentials like email, password and display name mmclient edit user foo(a)bar.com --email foo1(a)bar.com mmclient edit user foo(a)bar.com --name foo mmclient edit user foo(a)bar.com --password bar A common method that will support multiple arguments together can be built for this action. > I feel that it's likely to be quite annoying to refactor a ton of code > if we decide to change those aspects of the design. That's why I feel > it's better to do a more detailed design now. > > A few unconnected comments on various aspects: > > - In your code you have some short variable names (d, l). This is > mildly un-Pythonic. Try reading diffs (eg, cmdparser.py in r52) and > I think you'll see. If you don't see right off, I can explain > precisely why I stumbled in a couple of places, but I'd rather let > you see it for yourself. Fixed. > - Short names for long options like --ll are a definite no-no. They > make the code harder to understand, and on the command line they are > arcane. We need to cater to users who may not be so flexible. I have replaced --ll with -v and --verbose. > - The change you make in r52 is interesting. I'm a little leery of > the use of getattr(), it's ugly, but that's probably easily fixed. > - Once you have the parsed command, you look up the command and then > apply it to arguments retrieved from your ArgumentParser instance. > *These lookups can fail*, and you need to catch them (for > interactive shell mode). I don't think ArgumentParser will check > that all syntactically required arguments are present, so you need > to do it yourself. Think carefully about this -- you may want to > revise or even revert the change in r52 because it implies doing > this argument checking in each action rather than in the command > parser. Why not leave the argument verification to the action method itself? I feel that it's better to leave it there as it would make future addition of actions easier. All the methods in the current version have a initial argument checking part. The current version already `prints` error messages if required arguments are absent. I will be replacing it with an `exception` so that it would be easier to integrate it with the interactive shell. Also argparse returns a dictionary with keys for *all* the possible options, whether or not they are specified by the user. If the value of option is not specified by the user, then the value of corresponding key would be `None`. However, extra arguments are ignored without warning. > - For detailed listings you add header in get_listing. But I think > you should have a no-header option -- if you ever want to pipe that > output to a script the header is annoying. My instinct was to move > it out of that function, but I guess you have to do it there to get > columnization right. > Adding --no-header requires just a trivial change.Will do. > Some nitpicking about names: > > > - "Operate" is pretty general, the conventional word for managing > lists of things is "Manage". > > - "Long list" is ambiguous ("one 'long list'" vs. a "paged list", for > example). "Detailed" (your word) or maybe "verbose" (conventional) > is better. > > - I prefer the word "scope" for [domain, list, user] rather than > "type" or "instance". Fixed. > - Having "list" be both the name of a scope and the name of a command > in the same ArgumentParser means that you need to check syntax > (order of the words), not just presence of the words, to be sure the > user has given all the required arguments. If the scope names and > command names are disjoint, then you could even let the user choose > whether to type "create list" or "list create". (That may not be a > great idea.) The so called positional arguments are parsed in the order that they are listed in the initialize_options method. Hence the order checking is performed by the argparse itself. `create domain` is not same as `domain create`. The presence of all required arguments is verified at the method corresponding to an action. > Footnotes: > [1]http://bazaar.launchpad.net/~rajeevs1992/mailman.client/mailmancli/change… > > [2]http://myfossblog.blogspot.in/2014/05/lets-talk-over-cup-of-code.html > > In addition, I will be partitioning the commands/arguments to groups based on the entity, by making use of the argparse sub-parsers. This would enhance the readability of the help strings. TODO list from this mail: - no-header option for list - positional argument for instance name - exceptions on argument lookup failure - printing error messages on lookup failure - partitioning arguments using sub-parsers All parts mentioned fixed are fixed and will form r53, after some more cleaning, like the docs, probably by tomorrow.The items mentioned in TODO lists will be built once the design process is complete.

3 12