Quoting Skip Montanaro <skip@pobox.com>:
>> We definitely should NOT crawl the site, just in case it really is an >> innocent url. The load can crush a site, particularly if it's >> hosted.
Richard> Nah. You need to throw thousands of requests at a half-decent Richard> web server before it gives up the ghost. And if they're sending Richard> out 10 million mail pieces, they should expect their http Richard> server to take some load. These are definitely NOT innocent Richard> emails. They come from bogus senders, have minimal headers Richard> (deliberately), and contain *nothing* but a url. Which points,
You can't make that judgement beforehand. If the site you are poking is a valid site and the email received was not spam, none of what you said holds. If I remember correctly, you said this was only to be performed in circumstances where certain criteria were met, none of which included a conclusion the mail was spam.
Anyone who includes a URL in a mail message will probably be prepared for some load based on the number of people receiving the message. If I send a message to a client asking him to look at a web site on a staging server, I expect a dozen or so hits, followed by a phone call. If I send a message to my family mailing list, I expect a couple hundred hits (followed by a complaint from my brother that his picture looks ugly (What can I do? 8-) ). If an evil spammer sends a URL to 50 million addresses, it might expect (hope for) a decent slashdot spike. Interpreting the results of the http request opens a new can of worms. All of the tricks we use to mangle addresses (javascript, formmail honeypots, user-agent based web-pages, funky encodings, etc.) can now be used by the spammer against us. hmmm. I think it will take a while for that to become a major problem. In a server-side deployment where the same spam is likely to reach many hosted mailboxes, a specialized proxy server might be able to reduce the perceived response rate and the wasted bandwidth. -- Terrel