bernholdtde@ornl.gov wrote:

If I use a simple regex, like '^x-spam-flag: *yes' it seems to show up properly in the configuration. But if I use a more precise regex that accounts for all forms of whitespace with the "\s" sequence, like this '^\s*x-spam-flag:\s*yes\s*$' then what appears in the config file is '^\\s*x-spam-flag:\\s*yes\\s*$' in other words the backslashes have been quoted and are now literals.

Are you saying you see this with bin/dumpdb of the config.pck. If so, that's just the way python is showing the representation of the string. It is not the actual value of the string. If you doubt that, try 'strings' instead of 'bin/dumpdb'.

Also note the initial \s* in '^\s*x-spam-flag:\s*yes\s*$' really does nothing since if 'x-spam-flag:' is preceded by whitespace, it isn't a header. I.e., none of the headers presented to header_filter_rules regexps have leading whitespace.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

bernholdtde@ornl.gov wrote:

On Tue, 19 Feb 2008 08:35:12 -0800 Mark Sapiro wrote:

...

Are you saying you see this with bin/dumpdb of the config.pck. If so, that's just the way python is showing the representation of the string. It is not the actual value of the string. If you doubt that, try 'strings' instead of 'bin/dumpdb'.

Ah, that seems to be it. dumpdb, config_list, and the web u/i all show the quoting of the backslashes, but strings on the pickle show it as I entered it. (Too bad such things don't "roundtrip" properly through tools that are supposed to do that, like config_list and the web u/i.)

What Mailman version is this. In my case, the web u/i shows the value I enter without doubling the '\'.

Also, the 'round trip' issue seems OK, at least for config_list -o followed by config_list -i. config_list reads the \\s and converts \\ to a literal \ so what gets put in the config is '\' followed by 's' which is exactly what you want. If you gave config_list something like

header_filter_rules = [('^x-header:\s+some value$', 0, False)]

it would interpret '\s' as a literal 's' and you would lose the '\'. config_list needs either

header_filter_rules = [('^x-header:\\s+some value$', 0, False)]

or

header_filter_rules = [(r'^x-header:\s+some value$', 0, False)]

Note that there is a big difference between '\s' in a string and say '\n'. '\s' is two characters. '\' and 's' whereas '\n' is a single newline character.

So the next question is why doesn't the header_filter_rules appear to be working? Message are getting held (which is what I'm doing for testing purposes), but the indicated reason is non-subscriber posting rather than the header filter. (Both conditions are true for the majority of junk that comes through, but there are some lists where I really do need to allow legitimate non-subscriber posts, with moderator approvals.)

Good question. I think we need to determine why the web u/i is showing the doubled '\'. It shouldn't, and whatever is making it do so, may also be the reason for the rules not matching.

As I understand it, SpamDetect runs before Hold, and I thought that the first exception kicked it out of the handler processing.

That is correct.

Is there some logging I can turn on to see more details as to what's going on in here?

Unfortunately, no. You have to actually code additional logging in the handler.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

bernholdtde@ornl.gov wrote:

Here's the result: In applying header_filter_rules, it looks like the entire set of headers is being treated as a single multiline string.

That is correct.

For reasons I don't entirely understand (remember I'm not a python expert), "^" and "$" are not matching the beginning and end of individual lines of a multiline string, even though I interpreted http://www.python.org/doc/current/lib/matching-searching.html to say that they should, and a colleague who's very familiar with Python also thought they should.

It works for me. I just set a header_filter_rules regexp on a test list to "^subject:.*hello.*$" with a reject action, and my test post with

Subject: test Hello in subject

was rejected with "Message rejected by filter rule match". In case you're wondering, the Subject: was the 15th of 21 headers in the message delivered to Mailman.

I then tested the regexp "^subject:\s+yes\s*$" with a post with

Subject: yes

and it too was caught.

If I don't have the line beginning/ending constraints in the regex, there is a risk (albeit small) that a subject header could match. So I ended up with "\nx-spam-flag:\s+yes\s*\n".

That should be equivalent to "^x-spam-flag:\s+yes\s*$" in header_filter_rules. Does the actual search in your SpamDetect.py say

        if re.search(pattern, headers, re.IGNORECASE|re.MULTILINE):

By the way, MM FAQ entry 3.32 and 3.51 are inconsistent about "^" and "$", and based on my experience 3.51 is wrong.

I have revised 3.32 which I think was the incorrect one, but we still have to resolve why it works for me and not for you.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On Fri, 29 Feb 2008 19:53:15 -0800 Mark Sapiro wrote:

I suspect you have an old Mailman version. Prior to Mailman 2.1.7, the regular expression search was not multiline

That's the answer, I'm using 2.1.5 from RHEL 4.

Thanks very much for your help with this!

David E. Bernholdt | Email: bernholdtde@ornl.gov Oak Ridge National Laboratory | Phone: +1 (865) 574 3147
http://www.csm.ornl.gov/~bernhold/ | Fax: +1 (865) 576 5491