On 2/8/2013 11:44 AM, Richard Damon wrote:

On 2/8/13 12:52 PM, Chris Nulk wrote:

...

Hello all,

I am trying to stop a particular group of messages from going to several of my lists. I am looking at the Subject line of the message to do the filtering and I am using the header_filter_rules in Spam Filtering. My regex filter is:

  ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*

and the Subject lines I am trying to catch and reject the message look like:

  Subject: [MYLIST] Invitation: Test event @ Thu Feb 28,
      2013 10:00 - 11:00 (mylistg@example.com)

or

  Subject: [MYLIST] Canceled Event: Test event @ Thu Feb 28,
      2013 10:00 - 11:00 (mylist@example.com)

I have set the action for the rule to reject, however, the messages still get posted to the list.

I have tested the regex above using the python interactive interpreter and the regex does match the various inputs used.

Can anyone tell me what the problem is with my regex?

Also, can any of the sender/recipient filters cause the message to bypass the spam filters? The message pipeline is unchanged.

Thanks, Chris Make sure you have set the "Action" for the filter to some other than Defer or Accept.

I did set the "Action" to "Reject".

A previous Accept that matches the message will override this filter.

Not a problem. Yet. I only have the one filter.

Also, note that I think the filter it tested BEFORE the [MYLIST] is added to the subject by the list, so is only there to be matched if it was in the original email.

This was the answer. Thank you. I should have realized that but I just wanted to get it done so I can move on to other critical work. Taking out the [MYLIST] check from the regex worked and I could add back in addition constraints to the regex to make it more targeted.

Thanks again Richard.

Chris

On 2/8/2013 4:28 PM, Mark Sapiro wrote:

Chris Nulk wrote:

...

My regex filter is:

   ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*

I know you've resolved this and that the issue was that header_filter_rules are processed way before the subject_prefix is added so you didn't want the '\[.*\]'part of the pattern, but I'm confused by the '[\s\+]*' part of the pattern which says match 0 or more occurrences of the character class consisting of any white space character (\s) and the plus sign (either \+ or just + would be equivalent here, + loses its special meaning inside []).

So, is [\s\+]* a typo or did you just mean \s+ or \s*?

The [\s\+]* was deliberate. Most of our lists but the list name in brackets ('[', ']'), however, we have a few lists that are members of another list. For those lists the list subject prefix is a plus-sign ('+'). For example, we have a student list that has the [Student] prefix when the messages go out. Faculty and staff are not members of the list and don't receive those messages. Yet, there are a few faculty and staff who have an interest in seeing the messages so those faculty/staff are put on a "listening" list which is a member of the student list. When a message goes to the student list, the prefix [Student] is added to the subject line. The message is then sent to the "listening" list and it adds its own subject prefix. Instead of having '[Student-Listen][Student]' show in the subject line, I changed it to a '+'. The long and short of it is that I was trying to match both types of lines and the [\s\+]* was the bit for matching the "listening" list's subject prefix.

...

Also, can any of the sender/recipient filters cause the message to bypass the spam filters? The message pipeline is unchanged.

No. header_filter_rules are processed by SpamDetect which is the first module in the pipeline. As you discovered, the CookHeaders module which adds the subject_prefix comes much later, and the sender/recipient filters are processed by Moderate which also comes later.

Great information to know, thanks.

The only issue I have now is the boss wants to be able to send a reject message if a message is rejected when it matches a rule. The only option I have is to hold the message and let the moderator send the rejection notice. And since, the header rule is for our global lists, I am the lucky moderator.

Thanks for all the great help, Chris

On 2/11/2013 6:40 PM, Mark Sapiro wrote:

Chris Nulk wrote: {SNIP}

...

The only issue I have now is the boss wants to be able to send a reject message if a message is rejected when it matches a rule. The only option I have is to hold the message and let the moderator send the rejection notice. And since, the header rule is for our global lists, I am the lucky moderator.

If you mean you want to reject the message with something other than the "Message rejected by filter rule match" message, then you are correct unless you want to modify the message in the SpamDetect.py module, i.e. change the message in

                 raise Errors.RejectMessage(
                     _('Message rejected by filter rule match'))

Making it something like:

                 raise Errors.RejectMessage(
                     _("""Some long winded,
                       Multiple line
                       Rejection message
                       """))

However, I hope these are just legitimate notifications that shouldn't have been sent to a list and not spam. It is a very bad idea to reject anything that might be spam because the rejection almost certainly goes to some innocent 3rd party whose address was spoofed in the message's headers. This is known as backscatter and is bad.

The messages I am trying to keep from getting to the list are Google Calendar invitations (and cancellations). For these messages or specifically targeted non-spam type messages where I have to use the spam filters to get rid of them, I would like to send a rejection notice giving a reason why the posting is not allowed. Any spam filters I set up for known spam and like messages, I usually just discard the message. I certainly don't want to do any backscatter.

Thanks, Mark.

Chris

On 2/8/2013 11:44 AM, Richard Damon wrote:

On 2/8/13 12:52 PM, Chris Nulk wrote:

...

Hello all,

I am trying to stop a particular group of messages from going to several of my lists. I am looking at the Subject line of the message to do the filtering and I am using the header_filter_rules in Spam Filtering. My regex filter is:

  ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*

and the Subject lines I am trying to catch and reject the message look like:

  Subject: [MYLIST] Invitation: Test event @ Thu Feb 28,
      2013 10:00 - 11:00 (mylistg@example.com)

or

  Subject: [MYLIST] Canceled Event: Test event @ Thu Feb 28,
      2013 10:00 - 11:00 (mylist@example.com)

I have set the action for the rule to reject, however, the messages still get posted to the list.

I have tested the regex above using the python interactive interpreter and the regex does match the various inputs used.

Can anyone tell me what the problem is with my regex?

Also, can any of the sender/recipient filters cause the message to bypass the spam filters? The message pipeline is unchanged.

Thanks, Chris Make sure you have set the "Action" for the filter to some other than Defer or Accept.

I did set the "Action" to "Reject".

A previous Accept that matches the message will override this filter.

Not a problem. Yet. I only have the one filter.

Also, note that I think the filter it tested BEFORE the [MYLIST] is added to the subject by the list, so is only there to be matched if it was in the original email.

This was the answer. Thank you. I should have realized that but I just wanted to get it done so I can move on to other critical work. Taking out the [MYLIST] check from the regex worked and I could add back in addition constraints to the regex to make it more targeted.

Thanks again Richard.

Chris

On 2/8/2013 4:28 PM, Mark Sapiro wrote:

Chris Nulk wrote:

...

My regex filter is:

   ^Subject:[\s\+]*\[.*\]\s+(Invitation|Canceled Event):.*

I know you've resolved this and that the issue was that header_filter_rules are processed way before the subject_prefix is added so you didn't want the '\[.*\]'part of the pattern, but I'm confused by the '[\s\+]*' part of the pattern which says match 0 or more occurrences of the character class consisting of any white space character (\s) and the plus sign (either \+ or just + would be equivalent here, + loses its special meaning inside []).

So, is [\s\+]* a typo or did you just mean \s+ or \s*?

The [\s\+]* was deliberate. Most of our lists but the list name in brackets ('[', ']'), however, we have a few lists that are members of another list. For those lists the list subject prefix is a plus-sign ('+'). For example, we have a student list that has the [Student] prefix when the messages go out. Faculty and staff are not members of the list and don't receive those messages. Yet, there are a few faculty and staff who have an interest in seeing the messages so those faculty/staff are put on a "listening" list which is a member of the student list. When a message goes to the student list, the prefix [Student] is added to the subject line. The message is then sent to the "listening" list and it adds its own subject prefix. Instead of having '[Student-Listen][Student]' show in the subject line, I changed it to a '+'. The long and short of it is that I was trying to match both types of lines and the [\s\+]* was the bit for matching the "listening" list's subject prefix.

...

Also, can any of the sender/recipient filters cause the message to bypass the spam filters? The message pipeline is unchanged.

No. header_filter_rules are processed by SpamDetect which is the first module in the pipeline. As you discovered, the CookHeaders module which adds the subject_prefix comes much later, and the sender/recipient filters are processed by Moderate which also comes later.

Great information to know, thanks.

The only issue I have now is the boss wants to be able to send a reject message if a message is rejected when it matches a rule. The only option I have is to hold the message and let the moderator send the rejection notice. And since, the header rule is for our global lists, I am the lucky moderator.

Thanks for all the great help, Chris

On 2/11/2013 6:40 PM, Mark Sapiro wrote:

Chris Nulk wrote: {SNIP}

...

The only issue I have now is the boss wants to be able to send a reject message if a message is rejected when it matches a rule. The only option I have is to hold the message and let the moderator send the rejection notice. And since, the header rule is for our global lists, I am the lucky moderator.

If you mean you want to reject the message with something other than the "Message rejected by filter rule match" message, then you are correct unless you want to modify the message in the SpamDetect.py module, i.e. change the message in

                 raise Errors.RejectMessage(
                     _('Message rejected by filter rule match'))

Making it something like:

                 raise Errors.RejectMessage(
                     _("""Some long winded,
                       Multiple line
                       Rejection message
                       """))

However, I hope these are just legitimate notifications that shouldn't have been sent to a list and not spam. It is a very bad idea to reject anything that might be spam because the rejection almost certainly goes to some innocent 3rd party whose address was spoofed in the message's headers. This is known as backscatter and is bad.

The messages I am trying to keep from getting to the list are Google Calendar invitations (and cancellations). For these messages or specifically targeted non-spam type messages where I have to use the spam filters to get rid of them, I would like to send a rejection notice giving a reason why the posting is not allowed. Any spam filters I set up for known spam and like messages, I usually just discard the message. I certainly don't want to do any backscatter.

Thanks, Mark.

Chris