Some subscriber's e-mail adresses contains line break unicode character ​
In some occasions, a few subscribers of different lists contains the character in the mail-address, like me@example.com<mailto:me@example.com> preventing them from receiving any mails from Mailman. The Unicode is always the same on affected subscribers.
As what I've read it is a zero space line break unicode character. So far the only think I found was about HTML pages containing the character and a single post explains something about the character being added to a page through the Office 365 Public Website editor. But since subscribing to a Mailman list does not involve an editor, I assume that the character could have been added by a webbased mail client.
From what I've learned, the issue is associated with a random number of subscribers on different lists, having different admins in most cases - I currently have the issue on two lists having the same admin though, but two different subscribers.
I don't know whether the subscribers have subscribed through mail or through the Mailman web interface, but since the problem is random, I assume that the Mailman web interface is not associated. I guess that it must be some kind of common web based MUA they have used, like Gmail or Hotmail or similar.
What could course this and how can I prevent it?
Yours faithfully
Henrik Rasmussen
Hi,
Is there a way to automate the adding and removing subscribers from another system? such that the list is kept concurrent?
regards
Steven
Hello,
It depends on how much work you want to do and how timely the changes need to be made. Here is what we have and do...
We have two lists for each group of users. Both lists are used. One list is dynamically built using LDAP. That list is considered the master list. Members are not allowed to remove themselves nor change any options. The second list is a regular list built using data from the master list. Users can removed themselves and change their options.
To automate the adding and removing of subscribers on the second list, we: the address to be added and a '-' before the address to be removed.
- if <listname>.today exists, rename <listname>.today to <listname>.yesterday
- dump the list membership of the masters list to <listname>.today
- diff -i --context=0 <listname>.today <listname>.yesterday -> <listname>.diff so that each line of <listname>.diff has a '+' before
- Split <listname>.diff into <listname>.add and <listname>.remove where <listname>.add has all the '+' addresses (without the '+') and <listname>.remove has the equivalent for address to remove.
- Use the regular Mailman tools, add_members and remove_members, to add and remove subscribers to the second list.
We run this process once a day. When a new user shows up in the master
list, it takes a day to make it to the second list. Acceptable for us.
Run more often if you need faster updates to the second list.
The master list doesn't have to be a list. It can be a list of address from a database, LDAP, or anything else. All the processing to the master list information can be done on another system or the same system. Only send the <listname>.add and <listname>.remove files to the Mailman system for final processing.
The process works for us. Diff occasionally glitches on closely matched names but a follow on process run weekly fixes the problem.
Good Luck, Chris
On 10/20/2015 2:42 PM, Steven Jones wrote:
Hi,
Is there a way to automate the adding and removing subscribers from another system? such that the list is kept concurrent?
regards
Steven
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/cnulk%40scu.edu
Hi,
No, this sounds good enough and what I thought of doing as the external file/list will be updated once a week which I'll "suck" onto the server.
I just wondered if there was something already written in/for mailman to do this.
thanks
regards
Steven
From: Mailman-Users <mailman-users-bounces+steven.jones=vuw.ac.nz@python.org> on behalf of Chris Nulk <cnulk@scu.edu> Sent: Wednesday, 21 October 2015 11:40 a.m. To: mailman-users@python.org Subject: Re: [Mailman-Users] automating adding and removing subscribers
Hello,
It depends on how much work you want to do and how timely the changes need to be made. Here is what we have and do...
We have two lists for each group of users. Both lists are used. One list is dynamically built using LDAP. That list is considered the master list. Members are not allowed to remove themselves nor change any options. The second list is a regular list built using data from the master list. Users can removed themselves and change their options.
To automate the adding and removing of subscribers on the second list, we: the address to be added and a '-' before the address to be removed.
- if <listname>.today exists, rename <listname>.today to <listname>.yesterday
- dump the list membership of the masters list to <listname>.today
- diff -i --context=0 <listname>.today <listname>.yesterday -> <listname>.diff so that each line of <listname>.diff has a '+' before
- Split <listname>.diff into <listname>.add and <listname>.remove where <listname>.add has all the '+' addresses (without the '+') and <listname>.remove has the equivalent for address to remove.
- Use the regular Mailman tools, add_members and remove_members, to add and remove subscribers to the second list.
We run this process once a day. When a new user shows up in the master list, it takes a day to make it to the second list. Acceptable for us. Run more often if you need faster updates to the second list.
The master list doesn't have to be a list. It can be a list of address from a database, LDAP, or anything else. All the processing to the master list information can be done on another system or the same system. Only send the <listname>.add and <listname>.remove files to the Mailman system for final processing.
The process works for us. Diff occasionally glitches on closely matched names but a follow on process run weekly fixes the problem.
Good Luck, Chris
On 10/20/2015 2:42 PM, Steven Jones wrote:
Hi,
Is there a way to automate the adding and removing subscribers from another system? such that the list is kept concurrent?
regards
Steven
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/cnulk%40scu.edu
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/steven.jones%40vuw.ac....
On 10/20/2015 02:42 PM, Steven Jones wrote:
Is there a way to automate the adding and removing subscribers from another system? such that the list is kept concurrent?
Please don't hijack threads by replying to unrelated posts. Your post is still threaded with the one you replied to even if you change the Subject:
If you have two lists on the same server/mailman installation
bin/list_members -f <source_listname> | bin/sync_members -f - [other options] <dest_listname>
all on one line, run by cron as often as you like.
On two different hosts, a cron on the destination that runs bin/list_members on the source with output to a file (via ssh) followed by rsync to copy the file to the destination followed by bin/sync_members on the destination.
Give the list_members and sync_members commands with --help for documentation.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Henrik Rasmussen writes:
In some occasions, a few subscribers of different lists contains the character in the mail-address, like me@example.com<mailto:me@example.com>
This example doesn't make sense to me. me@example.com can't be a real address so you didn't cut and paste, and I don't understand why a plain-text message would have HTML escapes in it. (I guess maybe your mail was pure HTML and autoconverted by mailman@mail.python.org?) Do these addresses really have an embedded "mailto:" URL?
Regarding the address itself. I don't understand why an address would have ZWSP stuck on the end. I would expect something like "First Last{ZWSP}<me@example.com>".
preventing them from receiving any mails from Mailman. The Unicode is always the same on affected subscribers.
It appears that pasting from (some versions of? some plug-ins of?) Google Chrome to other applications can have this effect. Eg,
http://stackoverflow.com/questions/7055600/u200b-zero-width-space-characters...
As what I've read it is a zero space line break unicode character.
Yes. Technically, a zero-width space, which indicates a word boundary and *may* be treated as a line-break point or a point for adding space when text is fully justified.
What could course this and how can I prevent it?
Broken software (not Mailman) is injecting the character, and there's nothing you can do to prevent it at present. I expect that it's occurring in the web interface, and my guess would be a list admin copy-pasting into "mass add members" or a user copy-pasting a full address with display name into the address field on the listinfo page.
To figure out what to do about it, it would help to have confirmation that your me@example.com example accurately represents the configuration that is occuring.
Mark may have a patch to filter these at subscription time and/or a withlist script to remove them shortly.
On 10/20/2015 08:29 PM, Stephen J. Turnbull wrote:
Mark may have a patch to filter these at subscription time and/or a withlist script to remove them shortly.
As I said in another reply, MM >= 2.1.15 will not allow these addresses to be subscribed. With MM >= 2.1.15, the pipe
bin/list_members -i <listname> | bin/remove_members -f - <listname>
may remove them from <listname>, but it may not because stripping of leading and trailing whitespace in remove_members may remove the offending character. There is also a script at <https://www.msapiro.net/scripts/remove_bad_address.py> that should work in any case.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 10/20/2015 01:52 AM, Henrik Rasmussen wrote:
In some occasions, a few subscribers of different lists contains the character in the mail-address, like me@example.com<mailto:me@example.com> preventing them from receiving any mails from Mailman. The Unicode is always the same on affected subscribers.
I am confused. Are you saying the addresses end with an actual Unicode zero width space character or that they end with the ascii string ''. If the latter, I don't see how this can happen. In no version of Mailman 2.1 back to pre 2.1.1 has a ';' been allowed in an email address. If the former, an address with '\u200B' (the python escape for that character) in the domain would not have been allowed since Mailman 2.1.15, but may have been allowed before that.
As what I've read it is a zero space line break unicode character. So far the only think I found was about HTML pages containing the character and a single post explains something about the character being added to a page through the Office 365 Public Website editor. But since subscribing to a Mailman list does not involve an editor, I assume that the character could have been added by a webbased mail client.
That is probably correct, but gmail's web mail in particular (which I once thought was good about stuff like this) and probably others when composing in 'rich text' and creating a multipart/alternative message, sometimes creates a text/plain alternative littered with these characters.
Thus, if one composes an email in Gmail's web client to the list-request address with a body line
subscribe me@example.com
It is possible the text/plain alternative will have a zero width space appended to the address.
I don't know whether the subscribers have subscribed through mail or through the Mailman web interface, but since the problem is random, I assume that the Mailman web interface is not associated. I guess that it must be some kind of common web based MUA they have used, like Gmail or Hotmail or similar.
Have you looked at Mailman's subscribe log?
What could course this and how can I prevent it?
Conjecture on cause above. Prevention - upgrade Mailman to >= 2.1.15
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Sorry, my bad. Output from list_members LISTNAME shows email addresses containing like me@example.com, but without semicolon (;) used in HTML.
All the affected members seems to have been added by mass subscribtion from an admin so maybe the Mailman web interface form have been altered somewhere. I wonder why it's random though.
Thank you for the command. I will look further in to the problem.
Yours faithfully Henrik Rasmussen Systemadministrator, Core Services
-----Oprindelig meddelelse----- Fra: Mailman-Users [mailto:mailman-users- bounces+her=adm.ku.dk@python.org] På vegne af Mark Sapiro Sendt: 21. oktober 2015 05:59 Til: mailman-users@python.org Emne: Re: [Mailman-Users] Some subscriber's e-mail adresses contains linebreak unicode character
In some occasions, a few subscribers of different lists contains the character in the mail-address, like me@example.com<mailto:me@example.com> preventing
On 10/20/2015 01:52 AM, Henrik Rasmussen wrote: them from receiving any mails from Mailman. The Unicode is always the same on affected subscribers.
I am confused. Are you saying the addresses end with an actual Unicode zero width space character or that they end with the ascii string ''. If the latter, I don't see how this can happen. In no version of Mailman 2.1 back to pre 2.1.1 has a ';' been allowed in an email address. If the former, an address with '\u200B' (the python escape for that character) in the domain would not have been allowed since Mailman 2.1.15, but may have been allowed before that.
As what I've read it is a zero space line break unicode character. So far the only think I found was about HTML pages containing the character and a single post explains something about the character being added to a page through the Office 365 Public Website editor. But since subscribing to a Mailman list does not involve an editor, I assume that the character could have been added by a webbased mail client.
That is probably correct, but gmail's web mail in particular (which I once thought was good about stuff like this) and probably others when composing in 'rich text' and creating a multipart/alternative message, sometimes creates a text/plain alternative littered with these characters.
Thus, if one composes an email in Gmail's web client to the list-request address with a body line
subscribe me@example.com
It is possible the text/plain alternative will have a zero width space appended to the address.
I don't know whether the subscribers have subscribed through mail or through the Mailman web interface, but since the problem is random, I assume that the Mailman web interface is not associated. I guess that it must be some kind of common web based MUA they have used, like Gmail or Hotmail or similar.
Have you looked at Mailman's subscribe log?
What could course this and how can I prevent it?
Conjecture on cause above. Prevention - upgrade Mailman to >= 2.1.15
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail- archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman- users/her%40adm.ku.dk
On 10/21/2015 12:59 AM, Henrik Rasmussen wrote:
Sorry, my bad. Output from list_members LISTNAME shows email addresses containing like me@example.com, but without semicolon (;) used in HTML.
That address would be invalid in Mailman >= 2.1.15, but not before.
All the affected members seems to have been added by mass subscribtion from an admin so maybe the Mailman web interface form have been altered somewhere. I wonder why it's random though.
Perhaps the admin has copied and pasted the addresses from a webmail page to the Mailman admin page.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (5)
-
Chris Nulk -
Henrik Rasmussen -
Mark Sapiro -
Stephen J. Turnbull -
Steven Jones