Anti-Spam Techniques
TECHNIQUE PROS CONS

Keyword Filtering –
Specific strings are searched for in an E-mail message. Simple to implement. Already available in many E-mail packages. Not very accurate. Frequent false positives (good E-mail messages flagged as spam)

Rule-Based Filtering –
E-mail messages are filtered and scored by keyword and context using a set of rules. Can be very accurate. Requires ongoing rule updating. May miss well-crafted spam messages.

Bayesian Filtering –
A type of statistical analysis derived from Bayesian logic. Learns to recognize spam. Returns percentage probabilities instead of scores. Can be very accurate. Depends somewhat on the quality of spam used to teach it. May need to be customized for each user.

Blacklists –
Lists of IP addresses that have been identified to originate spam. Good at blocking known spam sources. Requires constant updating. Misses well-crafted spam messages and spam that isn't coming from known mail relays.

Fingerprinting –
Identification of spam based on similarity to previously received spam messages. Can quickly recognize spam with a high rate of accuracy. Doesn't recognize new spam messages. Requires constant updating from a central source.

Challenge-Response –
E-mail senders are challenged to prove their identity before E-mail messages are delivered. Very high degree of spam prevention. Creates more E-mail traffic. May block valid E-mail sent by machines. May require foreign-language support.

Secure Messaging –
E-mail is sent with encrypted credentials or on an encrypted transport, so that the sender or sending machine can be authenticated. Very high degree of spam prevention. Reliable identification of E-mail source. Requires widespread acceptance and support for digital-identity standards. May require expensive infrastructure enhancements.

Good summary, thanks!

A couple of questions though:

Where would you place the FireTrust CFS system? Fingerprint or blacklist server ... or something else?

Could you give some examples of a challenge-response package? How do such systems avoid requiring that a sender needs anything special to respond to the challenge?

I'll work on it this week....Ihope.....I threw this thread out there before we started having server issues today.......

OK, first you need to ask rusticdog about the firetrust issues.(Where would you place the FireTrust CFS system? Fingerprint or blacklist server ... or something else? )

Second on challenge response.....
If you are not visually jmpaired.......here is a whole article with loads of additional links.....

Spam-bot tests flunk the blind
By Paul Festa
Staff Writer, CNET News.com
http://news.com.com/2100-1032-1022814.html
Learn more about fighting spam

An increasingly popular technique for preventing e-mail abuse is frustrating some visually impaired Net users, setting the stage for a conflict between spam busters and advocates for the disabled.
Many companies have recently begun requiring users to pass a verification test in order to access their services--typically by typing into a Web form a few characters that appear on the form in a guise that prevents a computer or software robot from recognizing and copying them.

The technique, now used by Web giants Yahoo, Microsoft, VeriSign and others, seeks to block software bots from signing up for Web-based e-mail accounts that can be used to launch spam and from scraping e-mail addresses from online databases.

The scheme is winning high marks in the battle against unwanted junk e-mail. But it is also increasingly hindering the progress of Web surfers with visual disabilities--raising the ire of advocates for the blind, spurring plans for alternatives from a key Web standards group, and eliciting warnings from legal experts who say that the practice could expose companies to lawsuits brought under the Americans with Disabilities Act.

"It seems that they have jumped on a technological idea without thinking through the consequences for the whole population," said Janina Sajka, director of technology research and development for the American Foundation for the Blind in Washington, D.C. "These systems claim to test whether there's a human on the other end. But it's only technology that can challenge certain human abilities. So someone who doesn't have that particular ability is excluded from participation. That's really inappropriate."

Efforts to create tests aimed at distinguishing humans from machines go back decades, with the most famous formulation of the problem posed in 1950 by the English mathematician and World War II "Enigma" code breaker Alan Turing. Turing's controversial hypothesis was that a machine could be defined as "intelligent" if a questioner could be fooled into believing it was a person.

Visual tests in a sense turn that theory on its head, assuming that a machine is defined by its inability to perform a task that is easy for most humans to accomplish.

The increase in use of visual tests--Yahoo in recent weeks has started springing them on users of its mail service--comes as Internet service providers and other companies are acknowledging and attacking the spam problem with unprecedented energy. Assaults on spam have come fast and furious this year on the litigation, legislation and technology fronts.

Companies that have implemented the technique call it a winner. Microsoft last month said it had achieved a 20 percent reduction in e-mail account registrations after implementing the test.

VeriSign, in another example, uses the technique to prevent automated queries to its WhoIS database of Web addresses and their registrants, in part to keep bots from harvesting the database for potential spam recipients.

Some Web sites using visual tests provide work-arounds for the visually impaired; some don't. But existing work-arounds are less than perfect and less than universally implemented.

Of the three above-mentioned companies with visual verification tests on their Web sites, only VeriSign's provided no alternative for the visually impaired. The company did not respond to requests for comment.

Microsoft's Hotmail service provides an audio alternative to its visual test, in which letters are read aloud instead of being displayed in a graphical file. But one such audio file--deliberately garbled to prevent its being read by a computer--was unintelligible to four out of four CNET News.com reporters, all with good hearing, who tried to decipher it.

Microsoft said it would review the audio work-around and defended its accessibility efforts generally.

"Microsoft has been exploring and evolving accessibility solutions that are integrated with products for more than a decade and takes its responsibility here very seriously," wrote a Microsoft representative in an e-mail exchange. "They're committed to raising the standard for the whole industry in the making of accessible technology."

Microsoft maintains a Web page listing its resources and products for the visually impaired. "It seems they have jumped on a technological idea without thinking through the consequences for the whole population."
-- Janina Sajka, director of technology research and development, American Foundation for the Blind

Yahoo lets people who can't see its visual verification test fill out a Web form that it promises it will process within 24 hours. But even that slower work-around is not available to all Yahoo sign-up services--for example, for people signing up for a new ID through Yahoo's instant messenger application.

Yahoo said engineers were working on a customer support option for YIM, and that it would be added to the next version. The company added that the option is currently available for those who register through the IM Web site.

Looking for a better way
The increasingly popular visual test, and the difficulty of using current work-arounds, has raised enough hackles among advocates for the disabled that working groups within the World Wide Web Consortium's (W3C) Web Accessibility Initiative have begun discussions on how to standardize an alternative.

Two WAI working groups are hashing out proposals to guide Web sites in designing blind-friendly bot repellants, and the WAI hopes to address the issue in the next working draft of its Web Accessibility Guidelines, Version 2.0, which is due by year's end. So far, published working drafts of the guidelines are silent on the issue.

"What visual verification is testing is whether someone is a sighted human, even if that's not the intent of the organizations using it," said Judy Brewer, director of the WAI. "This has been a known problem for several years, and I know that we've received different complaints about it. But it's not necessarily an easy problem to solve."

Brewer did not specify what alternatives the WAI working groups were debating.

The American Foundation for the Blind's Sajka, who is himself blind, stressed that the technique posed problems also for those with less than total visual impairment. The camouflaged characters common to the tests are often impossible for the color blind to make out. They also thwart people who have trouble with contrast, he said.

Sajka raised the specter of bringing discrimination lawsuits against companies that implement similar tests.

In light of an October ruling that said the Americans with Disabilities Act did not apply to Web sites, and a May 15 ruling by a federal court that distinguished Web sites from the "public accommodations" that fall under federal civil rights statutes, Sajka acknowledged that suing implementers of visual verification might require asking Congress to pass additional legislation mandating accessibility for the disabled.

"The industry would like to avoid regulation, and if that's the case, thinking through this kind of thing would be a good idea," Sajka said. "I think we would rather they realized they have a responsibility, than our having to go up to the Hill or go to court. The technology is so entrenched in day-to-day living, and just because something is a cool idea doesn't mean it's the right thing to do."

But one lawyer with extensive experience in discrimination law said Web companies shouldn't consider themselves out of the ADA woods just because of the October ruling in "Access Now and Robert Gumson v. Southwest Airlines," decided by the U.S. District Court for the Southern District of Florida.

"That is something for which sites would almost certainly be required to make an accommodation for people with visual impairments who want to access them," said Kerry Scanlon, a partner with Kaye Scholer, and formerly Deputy Assistant Attorney General in the Civil Rights Division of the U.S. Department of Justice.

Noting that the Southwest Airlines decision had been appealed, Scanlon also cited comments by Judge Richard Posner indicating his opinion that the ADA covered the Internet, and predicted that the courts would ultimately overturn the Southwest Airlines ruling.

"I think it's unlikely that the courts will ultimately hold that the ADA does not apply to the Internet," Scanlon said. "I don't think, given the role the Internet plays in commerce today, that the courts are going to say that the provisions passed to protect 50 million people with disabilities in this country aren't going to apply to the Internet."

One spam opponent came to the defense of companies with visual tests, calling the tests crucial weapons against the growing legions of spam-sending machines.

"Anything that restricts people's ability to use e-mail lessens its usefulness as a communications medium," said Laura Atkins, president of the SpamCon Foundation. "On the surface, it's not a good thing. But there's so much abuse out there that the (Internet service providers) have to do it. Any site that does it should provide an alternate way for a blind person to sign up. But you can't condemn the ISPs for doing what they're doing to minimize the abuse."



Related News
Court denies Mailblocks injunction request June 12, 2003
http://news.com.com/2100-1032-1016250.html

Yahoo sets up spam roadblock June 10, 2003
http://news.com.com/2100-1032-1015247.html

Spam blockers may wreak e-mail havoc May 27, 2003
http://news.com.com/2010-1071-1009745.html

In-boxes that fight back May 19, 2003
http://news.com.com/2010-1032-1003921.html

Microsoft unveils antispam tools May 7, 2003
http://news.com.com/2100-1025-1000417.html

Get this story's "Big Picture"
http://news.com.com/2104-1032-1022814.html


Copyright ©1995-2003 CNET Networks, Inc. All rights reserved.





The Earthlink Challenge-Response System and How it Affects your Campaigns.
http://www.emaillabs.com/resources_articles.html

by Kirill Popov

July 2003 Issue - EmailLabs Newsletter


As the spam problem increases, ISPs are testing new ways of giving their users control over the email they receive. Earthlink is the first large ISP to offer a system of spam control called "Challenge-Response" to their users.

The challenge-response system works by inversing the way email processes. Currently, most ISPs accept all email from all senders and allow the user to select which email to delete or block. Challenge-response blocks all email, letting the user decide which to accept.

Earthlink's version of the Challenge-Response system does deliver all email to the user, potentially to a "Suspect" folder rather than the inbox. When an email arrives in the Suspect folder, Earthlink will send a challenge email back to the sender, instructing the sender to click on a link and add themselves to the user's whitelist. This allows future email from that sender to go directly to the user's inbox. This will theoretically prevent spammers using a forged "from" address from reaching the Inbox.

Unfortunately, the challenge email only applies to single individuals. In the case of email marketing messages, Earthlink will not send the challenge email. They have ways of detecting mass email transmissions and have opted not to send floods of challenge emails that may or may not be read.

Email marketers have to rely on the user to actively whitelist email messages that end up in the Suspect folder. The users receive an automatic email every day summarizing the contents of the Suspect folder. This summary email also contains an easy way to manually whitelist senders, by clicking an embedded link.

As Earthlink and more ISPs begin to proactively find ways to stop spam from being delivered, it becomes more important than ever to clearly brand your From and Subject lines so your customers will readily identify your email messages.


Spam blockers may wreak e-mail havoc
By Declan McCullagh
http://news.com.com/2010-1071-1009745.html

Here's an unhappy prediction: The explosion of spam-blocking technology could herald the death of much legitimate e-mail.
I wrote about patents relating to this technology, known as challenge-response technology, last week. Basically, when your mailbox is protected by a challenge-response system, people who try to contact you will be greeted with a response saying something like "click on this link to deliver this message" or "type in the word you see in the box above." The idea is to block increasingly obnoxious spam bots but still let actual humans get in touch with you.

In theory, well-designed challenge-response utilities won't challenge mail from known correspondents or mail that you've actually asked to receive. Unfortunately, many current challenge-response systems are poorly designed, which could wreak havoc on mailing lists and other legitimate communications. This could make e-mail far less useful than it is today.

It's already starting to happen. SpamArrest.com began challenging mailing list messages last year. Recently Mail-block.com and iPermitMail.com followed suit.

When that happens, the operator of the mailing list receives a message--from each subscriber using the poorly designed challenge-response utility--that asks the list operator to respond to the challenge. Replying to a handful of challenges is no big deal, but if many subscribers start using poor challenge-response software, it will pose a serious problem for mailing list operators. Big corporations may be able to afford to hire someone to sit in front of a computer and spend all day proving they're not a spam bot, but nonprofit groups, individuals and smaller companies probably can't.

Challenge-response systems, ironically, share some characteristics with spam: In small quantities, both are only mildly annoying to the recipient. But as quantities increase, they make it more difficult to use e-mail at all. MailFrontier.net is a good example: It prevents its users from signing up to mailing lists unless the list operator manually intervenes to answer the challenge, a process that is exactly backward.

The enormous growth in spam means that challenge-response technology will become more popular. EarthLink recently announced it would make a challenge-response system available to its customers by the end of May, and the field is wide open, with no market leader so far.

EarthLink's announcement has alarmed veteran list operators, who view it as a model that other Internet service providers may follow. Dave Farber, the University of Pennsylvania computer scientist who runs the "interesting people" list, warned his subscribers: "If I start getting a flood of challenges from EarthLink IPers that require my response I will most likely declare them spam and you will stop receiving IP mail. I fully expect this to be the case for almost all the legitimate mailing lists you are on and count on."

This could make e-mail far less useful than it is today.
Editors at TidBits, the popular Macintosh newsletter that boasts about 50,000 subscribers, wrote a message on May 13 to readers: "Be warned that we will not answer any challenges generated in response to our mailing list postings. Thus, if you're using a challenge-response system and not receiving TidBits, you'll need to figure that out on your own."

It's worth remembering that, while they may not be as glamorous as the Web, peer-to-peer applications, or instant messaging software, mailing lists are the Internet's oldest form of mass communication. They date back to the original "MsgGroup" list in 1975, which the same Dave Farber--then at the University of California at Irvine--helped to create. Then the famous "sf-lovers" list came along, and the rest is, well, history.

Nowadays just about every organization uses mailing lists of some type, from Hotwire.com's cheap airfare announcements to the left-leaning activists at MoveOn.org who organized a massive e-mail campaign against the Iraq war. Professional organizations use them to contact members; companies offer deals to existing customers; and advocacy groups rely on lists to rally support for political causes. And that's not counting services like Yahoo Groups and Topica.

Another downside to challenge-response systems is that they can be exploited by spammers, yielding false negatives in addition to false positives. Some challenge-response systems require only that the sender reply to the challenge; others require only that a hyperlink in the challenge be followed.

A more pernicious problem is that challenge-response systems trust the "From:" line of a message. If challenge-response systems become sufficiently widespread, spam bots may start trying to guess at who your correspondents are--and then forge the "From:" header appropriately--by subscribing to discussion lists or following links from your personal or company home page. Digital signatures are probably the only way to prevent that kind of attack.

John Levine, an author, moderator of the comp.compilers Usenet newsgroup and veteran Internet hand, offers a gloomy worst-case prediction. "So what will the effect of this be?" Levine asks. "You won't be able to trust that mail from your friends is actually from your friends, since an increasing fraction will be spam leaking through your challenge system. What will people do? Given the basic principle of challenge systems, which is that it's someone else's job to solve your spam problem, people will dump their white lists and start challenging every message."

At least right now, because challenge-response systems are so easy for programmers to create, there are plenty of them, and the potential for market dominance has attracted some companies of dubious virtue. SpamArrest spammed advertisements to people who e-mailed its customers (imagine if AOL or MSN claimed the right to spam anyone who's ever sent you mail). Mail-block.com has been blocked by Outblaze.com, a large mail provider, for spamming. And MailWiper.com has been caught spamming.

For a challenge-response system to work properly, it will need to be tightly integrated with the mail client--so it knows who you contacted--and it should understand popular mailing list software such as Majordomo, Mailman and Listserv. It's easier for challenge-response companies that sell Web-based e-mail. For people using software like Eudora and Outlook, that probably means plug-ins or an e-mail proxy server that let the challenge-response system keep track of your outgoing messages.

Brad Templeton, chairman of the Electronic Frontier Foundation and author of one of the first challenge-response systems, compiled a useful list of design principles for challenge-response systems earlier this month. Templeton's list has some recommendations: Never challenge any mail that's a reply to a private message you sent; use multiple e-mail addresses; and never challenge mailing-list messages.

All these should be obvious, but many challenge-response systems just don't follow them. Fortunately, the Internet Engineering Task Force's Anti-Spam Research Group is spending some time trying to devise a reasonable standard.

Challenge-response systems may turn out to be the only way to inoculate ourselves against the spam epidemic. Or they may not. But their designers and users should think twice before trusting the future of Internet e-mail to buggy and problematic technology.
Copyright ©1995-2003 CNET Networks, Inc. All rights reserved.


May 19, 2003
A Challenging Response to Challenge-Response:
http://www.freedom-to-tinker.com/archives/000389.html

One of the trendy ideas these days is challenge-response (CR) anti-spam technologies. The idea is simple: incoming email is intercepted before you see it, and a “challenge” email is returned to the sender. If the sender replies to the challenge message, then the original message is forwarded on to you; otherwise it is discarded. The idea is to require some kind of human involvement in the sending of each message. Sometimes the sender has to answer some kind of puzzle that is supposed to be easy for people but hard for computers.

Whenever we analyze a security technology – and that is what CR is – we need to look not only at the immediate effect of the technology, but also at how people will adapt to it. We need to look especially at how the bad guys will adapt. Will they adjust their attack strategy to defeat the new defense? Will the new defense create new opportunities for malicious attacks? Will the technology lead to an arms race between defenders and attackers? If so, can we predict the outcome of the arms race?

CR stands up poorly to this kind of analysis. To see why, suppose that Alice sends an email to Bob, and Bob is using CR. Bob’s computer sends a challenge message back to Alice and awaits her response. This challenge message had better get through to Alice; if it doesn’t, the whole scheme breaks down. If Alice is using anti-spam technology that blocks the challenge message, then she’ll never see the challenge -- her original message won’t get through to Bob, and she won’t know what went wrong.

We can fix this problem by making sure that Alice’s anti-spam technology has a loophole for challenge messages, to make sure they are never blocked. (Note that although Bob is the one using CR, it is Alice who has to create the loophole.) If CR is going to succeed, most of the Alices out there will have to open the loophole. Messages with certain “challenge-ish” attributes will be mostly immune from spam controls.

At this point, the bad guys’ response is obvious: create spam that can exploit the loophole, spam that looks like a challenge message. If they can do this, then CR will have made things worse – spam will pour in through the loophole.

We might try to solve this problem by narrowing the loophole, requiring the challenge messages to be so narrowly stylized that they cannot carry a spam. This too creates an opportunity for the spammers. If the challenges are so predictable, then the spammers will be able to develop computer programs that spot the challenges and auto-send the required responses. If they can do this, then the spammers can just add automated CR responses to their automated email-sending software, and continue to pollute our inboxes.

Given all of this, I’m skeptical of CR as a response to email. If you’re the first on your block to adopt CR, and if nobody else uses anti-spam technology, then CR might provide you some modest benefit. But it’s hard to see how CR can be widely successful in a world where most people use some kind of spam defense.


Topic(s): Spam

About Mailblocks
Mailblocks, Inc., founded by WebTV co-founder and former Microsoft executive Phil Goldman, was started with a simple idea: improve the consumer email experience; make it faster, more manageable and — above all else — free from the aggravation of spam (unsolicited, computer-generated email).

http://about.mailblocks.com/about.html
How We Block Spam
With our new spam-blocking solution Challenge/Response 2.0, you only have to answer one Challenge message the first time you send email to Mailblocks. As long as you remain a sender in good standing, you can email Mailblocks customers and never have to complete more than one Challenge message (Mailblocks customers are already senders in good standing and don’t have to answer a Challenge). To see how Challenge/Response works, send an email to goodstanding@mailblocks.com and answer the Challenge message. The next time you mail to a Mailblocks customer, your email will go straight to their Inbox.

How does Challenge Response Work?


To see what the Challenge/Response process is like for new senders not in your address list, click here. ( http://support.mailblocks.com/tab_howto/Validation/detail_privacy_howitworks.asp ) if this image should disappear.

How Challenge Response Works
Here is an example of the email that will be sent from your account to senders who are not in your Addresses list:


What is a sender in good standing?
A sender’s standing is determined by their affiliation with Mailblocks and/or the frequency and content of their email activity.
©2003 Mailblocks Inc. All rights reserved. Mailblocks is a service mark of Mailblocks, Inc.


**Note I nor ccsp endorse or promote the use of this product........it is merely used here as one example of challenge/response.




Challenge-response email authentication

I've been thinking about a challenge-response email authentication scheme for quite a while. The idea is simple: the first time that someone who isn't on your whitelist sends you an email, they receive an automated response that says "click on this link to actually send me the email". If you don't blacklist that sender, all subsequent emails get through.

Well, apparently a company called MailBlocks has already done this. There are, however, two problems with their approach: 1) they don't integrate in with an existing enterprise-class email system and 2) they claim to hold the patent on the concept for which there is apparently ample evidence of prior art.

On a related note, my friend Aaron Skonnard has a similar setup on his voice line. If your number isn't on a recognized list of allowed callers or you are calling with caller id blocked, you will hear a recorded message that says "hit 1 if you are not a telemarketer". This has apparently blocked all unsolicited phone calls from his home, since it is illegal for a telemarketer to misrepresent themselves this way.

Posted by John at May 21, 2003 08:30 AM
http://www.iunknown.com/000252.html

CFS would certainly come under the Fingerprinting –
Identification of spam based on similarity to previously received spam messages. Can quickly recognize spam with a high rate of accuracy. Doesn't recognize new spam messages. Requires constant updating from a central source.

absolutely amazing grace.........i whip up a report the width of the English Channel and the Ikester doesn't even grace me with a reply.........I thought I answered his TWO questions.......Chris??

Oh Canada...................

Very good info. I am going to work on this week.