This mail got blocked at first because I was not subscribed with my current email address.
Some of you may remember that I announced the release of the harvester script. We discussed the issue and as the result was, not to change mailman, I now released the script to raise public awareness of the problem.
Bernhard
-------- Original Message -------- Subject: mailman email harvester Date: Mon, 07 Feb 2005 23:48:44 +0100 From: Bernhard Kuemel <bernhard@bksys.at> To: full-disclosure@lists.netsys.com, bugtraq@securityfocus.com, mailman-developers@python.org
Hi!
Tons of email addresses from mailman mailing lists are vulnerable to be collected by spammers.
They are "protected" by obfuscation (user@example.com -> user at example.com) and access to the subscriber list can be restricted to subscribers. The obfuscation is trivially reversed and harvester scripts can subscribe to gain access to restricted lists.
I suggested a graphical turing test that would bar scripts but the mailman developers argued spammers might hire a couple of temps that would solve the test as it already happened for the creation of email accounts. The only solution would be not to have the desired information available. This is already an option by restricting access to the member list to the list administrator.
However, still many lists either have the member list openly published, or available to the list members. To raise awareness to this issue I wrote a script that collects addresses from openly accessible lists. It stops after processing 1000 (the maximum allowed) search results from google and collects 76772 email addresses (61124 unique). It is attached as mmxp1.
An improved version that collects addresses that are restricted to subscribers, processes more lists and works more parallelized is planned.
Bye, Bernhard
#!/usr/bin/perl -w
#http://www.google.com/search?q=%22list+is+only+available+to+the+list+members%22+mailman/listinfo&start=600&num=100 #2.1.4 "current archive" "private list which" mailman/listinfo site:org
$n=0;
$u=0;
for ($i=0;1;$i+=10) {
$#urls=-1;
$google=wget -qO - -U 'any browser' 'http://www.google.com/search?q=%22Click+here+for+the+list%22+mailman%2Flistinfo&start=$i';
# print $google;
@urls=($google=~m*<p class=g><a href=(http://\S+?)>*g);
# print join("\n",@urls);
if ($#urls==-1) {last;}
# print "\naoeu $#urls\n";
foreach $url (@urls) {
$u++;
$url=~s*/listinfo/*/roster/*;
print STDERR "$url...\n";
$roster=`lynx -connect_timeout=10 -dump $url`;
# print $roster;
@mails=$roster=~/^ +\* \(?\[\d+\](.* at .*?)\)?$/mgo;
foreach $mail (@mails) {
$mail=~s/ at /@/;
print "$mail\n";
$n++;
}
print STDERR "mails=".($#mails+1).", total=$n, url=$u, google=$i\n";# exit; } #foreach url
} #while google
Hi all,
Is there a way to change the setting to restrict access to the roster
for all lists, globally? If there isn't one, would one of you be
willing to write one quickly? The only other option I see is to remove
the ~mailman/cgi-bin/roster script which would be a pity.
Given the risk, now made worse by Bernhard's very helpfully
distributing this script for spammers, this is a really urgent issue.
Cheers,
Tobias
On Feb 9, 2005, at 6:08 AM, Bernhard Kuemel wrote:
This mail got blocked at first because I was not subscribed with my
current email address.Some of you may remember that I announced the release of the harvester
script. We discussed the issue and as the result was, not to change
mailman, I now released the script to raise public awareness of the
problem.Bernhard
-------- Original Message -------- Subject: mailman email harvester Date: Mon, 07 Feb 2005 23:48:44 +0100 From: Bernhard Kuemel <bernhard@bksys.at> To: full-disclosure@lists.netsys.com, bugtraq@securityfocus.com, mailman-developers@python.org
Hi!
Tons of email addresses from mailman mailing lists are vulnerable to be collected by spammers.
They are "protected" by obfuscation (user@example.com -> user at example.com) and access to the subscriber list can be restricted to subscribers. The obfuscation is trivially reversed and harvester scripts can subscribe to gain access to restricted lists.
I suggested a graphical turing test that would bar scripts but the mailman developers argued spammers might hire a couple of temps that would solve the test as it already happened for the creation of email accounts. The only solution would be not to have the desired information available. This is already an option by restricting access to the member list to the list administrator.
However, still many lists either have the member list openly published, or available to the list members. To raise awareness to this issue I wrote a script that collects addresses from openly accessible lists. It stops after processing 1000 (the maximum allowed) search results from google and collects 76772 email addresses (61124 unique). It is attached as mmxp1.
An improved version that collects addresses that are restricted to subscribers, processes more lists and works more parallelized is planned.
Bye, Bernhard #!/usr/bin/perl -w
#http://www.google.com/search? q=%22list+is+only+available+to+the+list+members%22+mailman/ listinfo&start=600&num=100 #2.1.4 "current archive" "private list which" mailman/listinfo site:org
$n=0; $u=0; for ($i=0;1;$i+=10) { $#urls=-1; $google=
wget -qO - -U 'any browser' 'http://www.google.com/search? q=%22Click+here+for+the+list%22+mailman%2Flistinfo&start=$i'; # print $google; @urls=($google=~m*<p class=g><a href=(http://\S+?)>*g); # print join("\n",@urls); if ($#urls==-1) {last;} # print "\naoeu $#urls\n";foreach $url (@urls) { $u++; $url=~s*/listinfo/*/roster/*; print STDERR "$url...\n"; $roster=
lynx -connect_timeout=10 -dump $url; # print $roster; @mails=$roster=~/^ +\* \(?\[\d+\](.* at .*?)\)?$/mgo; foreach $mail (@mails) { $mail=~s/ at /@/; print "$mail\n"; $n++; } print STDERR "mails=".($#mails+1).", total=$n, url=$u, google=$i\n"; # exit; } #foreach url} #while google
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives:
http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe:
http://mail.python.org/mailman/options/mailman-developers/ tobias%40kabissa.orgTobias Eigen Executive Director
Kabissa - Space for Change in Africa http://www.kabissa.org
- Kabissa's vision is for a socially, economically, politically, and
environmentally vibrant Africa, supported by a strong network of
effective civil society organizations. *
At 11:40 AM -0500 2005-02-10, Tobias Eigen wrote:
Is there a way to change the setting to restrict access to the roster for all lists, globally?
That should be possible to do using "withlist". However, I don't know the specifics of how to do that.
-- Brad Knowles, <brad@stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
On Thu, Feb 10, 2005 at 11:40:29AM -0500, Tobias Eigen wrote:
Hi all,
Is there a way to change the setting to restrict access to the roster
for all lists, globally? If there isn't one, would one of you be
willing to write one quickly? The only other option I see is to remove
the ~mailman/cgi-bin/roster script which would be a pity.Given the risk, now made worse by Bernhard's very helpfully
distributing this script for spammers, this is a really urgent issue.
Not that hard to write such a script. I expect the spammers already have several alternatives to choose from. So, it's quite likely no harm has been done, and some good, arising from Bernhard's raising the issue in public.
I'd go further and mention that while Berhhard's script harvests membership rosters, it isn't that much more difficult to write a script that gets around the obfuscation of email addresses in the list archives. A list I used to manage until a few weeks ago (Hey, anybody got a lead on a Seattle-area opportunity for a rabid Python developer? Who also does C, SQL, HTML, CSS and various assemblers?) apparently had its archives harvested recently by some bank phishing folk. Emails were obscured in the archives using the "user at wherever.domain" option, and the archives had been regenerated quite some time ago back to their beginning, with that option in force. The roster has never been open to anybody but the list admin, so I don't believe it was the roster. Hence, likely it was the archives that were harvested.
There are a pretty fair number of good reasons for keeping list archives open. My opinion is a person posting to a list assumes the risk of having his or her email address harvested, and that one unwilling to assume this risk should refrain from posting. However I understand if others do not subscribe to that belief, and that there may be circumstances where there are reasonable grounds for wanting to manage a list by some other policy.
My suggestion is that an option be considered to redact all email addresses whatsoever from a list archive. Including anything mentioned in-line in the text of the post that even vaguely looks like an email address.
No doubt somebody on this list manages a list where users are quite sensitive to public exposure, who might care to advocate for such an option, and even code it, should the idea meet with sufficient approval.
-- Dan Wilder <wilder@eskimo.com>
Hi Dan,
Given the risk, now made worse by Bernhard's very helpfully distributing this script for spammers, this is a really urgent issue.
Not that hard to write such a script. I expect the spammers already have several alternatives to choose from. So, it's quite likely no harm has been done, and some good, arising from Bernhard's raising the issue in public.
Ok - granted. Sorry for sounding a little passive aggressive there. :-)
I'd go further and mention that while Berhhard's script harvests membership rosters, it isn't that much more difficult to write a script that gets around the obfuscation of email addresses in the list archives.
Mhonarc on my system actually removes the email address completely in the archives.
The mbox archives are still intact though, so if someone knows how mailman works they could probably hack their way into there. MBOX files generally are a motherlode for spammers.
My idea eventually is to replace mhonarc archives with a forum for discussion groups and newsletter archives, all integrated with my Mambo CMS. It's great to have the mbox files ready to be imported. This way you don't end up with disabled archives and you don't open up subscribers to having their email addresses harvested.
(Hey, anybody got a lead on a Seattle-area opportunity for a rabid Python developer? Who also does C, SQL, HTML, CSS and various assemblers?)
Try introducing yourself to the nonprofit open source initiative folks. They're very into drupal etc, and of course also use mailman. http://www.nosi-net
There are a pretty fair number of good reasons for keeping list archives open. My opinion is a person posting to a list assumes the risk of having his or her email address harvested, and that one unwilling to assume this risk should refrain from posting. However I understand if others do not subscribe to that belief, and that there may be circumstances where there are reasonable grounds for wanting to manage a list by some other policy.
I think the answer to this is that not everybody is as familiar with these tools as we are, and we can't assume that the people we are trying to serve are willing or able to put up with confusion about this sort of thing. Hence, again, my plans to develop a yahoo-like service that is more transparent and easy to navigate and use. These types of questions simply shouldn't even come up.
My suggestion is that an option be considered to redact all email addresses whatsoever from a list archive. Including anything mentioned in-line in the text of the post that even vaguely looks like an email address.
Yes. This is what mhonarc does.
No doubt somebody on this list manages a list where users are quite sensitive to public exposure, who might care to advocate for such an option, and even code it, should the idea meet with sufficient approval.
Yes. We do. We also often find ourselves in the awkward position of having to manually remove postings from our archives because people are being defamed etc or have posted something that they didn't realize was going to be archived and are now regretting it. Many people using our lists are African activists and this can (to be dramatic) be a matter of life and death.
Cheers,
Tobias
-- Tobias Eigen Executive Director
Kabissa - Space for Change in Africa http://www.kabissa.org
- Kabissa's vision is for a socially, economically, politically, and environmentally vibrant Africa, supported by a strong network of effective civil society organizations. *
Tobias Eigen schrieb:
Given the risk, now made worse by Bernhard's very helpfully
distributing this script for spammers, this is a really urgent issue.
Since it is known for many *years* that spammers are harvesting addresses from ML-archives, and since anybody can see that replacing "at" with "@" is ... not a very hard task, I fail to see any urgency here (or any problem in the very simple script Berhard distributed).
-thh
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Thomas Hochstein wrote: |> Given the risk, now made worse by Bernhard's very helpfully |> distributing this script for spammers, this is a really urgent |> issue. | | Since it is known for many *years* that spammers are harvesting | addresses from ML-archives, and since anybody can see that | replacing "at" with "@" is ... not a very hard task, I fail to | see any urgency here (or any problem in the very simple script | Berhard distributed).
There may be no urgency but something should be done. Obviously there is a problem (as can also be seen by the emotions). Since the only solution we found for now is not to publish the email addresses, we should do that.
I pointed this out over a year ago and the number of vulnerable lists only grew. Probably because being able to see who else is on the list is a nice feature which we don't want to give up. We repress the problem: We think, spammers don't exploit it because they find enough addresses elsewhere. But spammers are smart: They play a lot of tricks to pass spam filters, they defeat graphical turing tests to semiautomatically sign up email accounts which the use for spamming, they make worms which act as mail relays.
They probably already harvest mailing list subscriber addresses and if they don't do so by now, they sure will, sooner or later. But they would be fools to tell us about it. We would lock our email addresses away from them.
I am writing the exploit code not for the spammers. They may already have one. I'm writing it to wake us up and treat this problem properly.
Brad Knowles wrote: |> However, still many lists either have the member list openly |> published, or available to the list members. | | True enough. However, even if we changed the default in Mailman | to be accessible only to the list administrator, it would take a | very, very long time before 50% of all Mailman installations were | secured in this manner.
I hope my exploit code will speed this up. I plan to release the improved version, which harvests addresses restricted to subscribers of about 100.000 mailing lists in several (3-6) months.
| That said, changing the default is probably the right thing to | do.
Please include a note of the upcoming exploit. The current exploit harvests about 600 lists where the addresses are published unrestricted.
| Moreover, it would be trivially easy for spammers to subscribe to | the list and silently collect all address information that comes | across. | | There's enough schemes out there for finding addresses that no | one simple scheme is going to work, and the methods that we know | will work are going to take a long time to become the default | standard.
If hashcash (http://www.hashcash.org/) gets integrated in our mail systems we no longer need to hide or obfuscate our email addresses.
Bernhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Debian - http://enigmail.mozdev.org
iD8DBQFCDWCH9zL78+QhnUgRAhSfAJ9WpPLARJ4bTG6ZPGH7anxc4FA5YwCdGn0C nwSeZoHoitZKRA+6rE1hlFU= =lM5z -----END PGP SIGNATURE-----
participants (5)
-
Bernhard Kuemel -
Brad Knowles -
Dan Wilder -
Thomas Hochstein -
Tobias Eigen