Monday, October 25, 2010


Challenge/response still lives (barely)

Wow; I haven’t gotten one of these in a long time:


A message you recently sent to a user with the subject "[redacted]" was not delivered because they are using the anti-spam service. Please click the link below to confirm that this is not spam. When you confirm, this message and all future messages you send will automatically be accepted.

I wrote about challenge/response anti-spam systems about three years ago, but probably haven’t seen a challenge message in at least two years. I thought people had given up on them.

Alas, no. But if the last two years is something to judge by, they’ve at least fallen further into disfavour.

Anyway, it’s worth a re-post, then, of my three-year-old item about them. All the problems, all the reasons one shouldn’t use them, are still valid now. So, here’s the link again: head over and read (or re-read) it.


Nathaniel Borenstein said...

I'm a little surprised at the blanket dismissal of C/R systems from such a sophisticated source. I agree that they often do more harm than good, and that they simply shouldn't be used in many cases, such as the customer support example in your 2007 post.

But in most cicrumstances, once you've done everything else you can in filtering spam, and the dust settles, and you've let 99% of the good mail through and blocked 99% of the bad mail, you still typically have a few messages left to deal with. At that time -- when you've done everything else that you mention in your post, and that you and I mentioned in our paper on the subject -- there may be no better alternative than challenge/response. Is an occasional C/R really worse than occasionally guessing wrong about whether or not a particularly ambiguous message is spam?

Barry Leiba said...

« Is an occasional C/R really worse than occasionally guessing wrong about whether or not a particularly ambiguous message is spam? »

Yes, I think it is.

Some end user (Barry's mother, say) installs someone's "Ultimate Anti-Spam Solution" on her computer, and it happens to be a C/R system (because those are the ones that market themselves as perfect solutions). Mail starts to get challenged.

Now, each challenged message is either spam or not. If it's spam, the challenge is almost certain to amount to "blowback", just like a bounce message saying "I couldn't deliver the spam you supposedly sent."

If it's not spam, it might be mom's long-lost best friend trying to get back in touch with her, in which case the challenge will work as advertised. The friend will happily reply to the challenge, not really annoyed at all because she's happy to be reconnecting with mom. That's the good case.

On the other hand, it could be an opt-in confirmation from a mailing list that mom's trying to subscribe to, and she'll never see it (because the list will never respond to the challenge). Or it could be a "We need more information about your problem," request from the help desk she tried to ask a question to, and she'll never get the help she needs (because the address the challenge goes to is unmonitored).

Or we can come up with a bunch of other possibilities, here. The point is that there's no value in sending a challenge to a non-spam message (which is why I call this a "false positive", though the C/R software makers steadfastly don't), and several use cases where it's really a problem. And, while sending challenges to spam does generally keep the spam out of mom's inbox, it does so at the expense of spamming someone else with challenges that they didn't ask for and don't know why the #*$% they're getting.

And, in fact, it might not even keep the spam out of mom's inbox: I know people who, just to be ornery (or perhaps out of cluelessness, as well) respond to spam challenges, causing the spam to be delivered anyway.

The bottom line is that you shouldn't be keeping your inbox clean at other people's expense, and you could be shooting yourself in the foot, as well. No spam filtering is perfect, and sometimes we just have to accept a few spam messages here and there.

Brent said...

For what it is worth, Im with Barry on this one. I really hate C/R systems and consider them socially obnoxious.

Nathaniel Borenstein said...

So would those of you (Barry, Brent) who hate C/R systems feel the same about a payment based system, where the email you got back said in essence "I'll pass your mail through for a dollar" or something like that?

I'm guessing that you would, but I'm also getting that such a system would be tremendously appealing to anyone in the public eye. It would be useful in ways utterly unrelated to spam control, actually. Part of the problem is how much users differ in what they want from their email service.

I rarely defend C/R, because it's so badly used in general, but I'm not convinced it couldn't be useful if it were ever done right.

Barry Leiba said...

No. I don't like any system that autoresponds. If someone could make it work (and we've discussed this many times, you and I), I'd happily have a system where senders could offer payment in advance. But anything that says "I'm a robot, I received your message, and you have to do something before I deliver it to the intended recipient," suffers from the problems I've talked about. It doesn't matter what that challenge wants you to do.

Nathaniel Borenstein said...

Alas, since you can't people from setting up robots, you can't win. Anyway robots are our future. Farm animals don't like humans telling them what to do any more than you want to be pushed around by a robot. I fear you'll be standing by the side of the road once human drivers are banned from cars.

Brent said...

Again...I'm with Barry. I would rather pay 1/1000 or 1/100 cent per email, with no auto-response. The robotic "velvet rope" around a friend's email just annoys me - come on folks, we all get spam, just deal with it.

Barry Leiba said...

Oh, Nathaniel, that last one is a straw man. I'm happy with robots when the collateral damage is tolerable (or absent). I would similarly reject car-driving robots if they caused the accident rate and/or road fatalities to go up. A challenge/response robot is more like a car-driving robot that happily gets you where you want to go, but does it by running other drivers off the road.

And you're right that I can't stop people from installing robots. On the other hand, I'm pleased to see that their use does appear to be dwindling. It probably means that most ISPs are doing pretty well at blocking spam in other ways, so fewer users are willing to pay for separate spam filtering.

Nathaniel Borenstein said...

A strawman? Ten years ago a car that drove itself was almost unthinkable. Today it looks like a virtual inevitability. This, like speech recognition, is being driven by techniques utterly unlike those investigated by early "artificial intelligence" researchers, favoring instead the brute force and statistical decision making that modern computing makes possible. They learn from massive data sets to make decisions that appear intelligent, but without any underlying reasoning of the type we understand.

As often as I talk about how complex it is, I don't really believe spam filtering is as complex as speech recognition or driving a car. Therefore I suspect it is only a matter of time before email robots can act highly intelligently. At that point, not only might they be able to do C/R well, they might be able to challenge us in such a way that we didn't know whose robot was challenging us, or even that we were being challenged in the first place.

The problem isn't robots. It's dumb robots. Once they're smarter than us, this debate becomes even more pointless than it already is. :-)

Barry Leiba said...

No, the straw man was that I wouldn't accept car-driving robots. I liked them when I wrote about them here.

And, yes, I agree with you that the problem is, specifically, dumb robots. As I've said a few times, the issue I have with C/R systems is that the challenges cause more damage than they fix. Turn that greater-than sign around, and then I'll be happy with them.

I'm not seeing attempts to improve C/R systems, though. If anyone's working on them, they're not submitting their papers to any place where I'm looking (CEAS, MIT spam conference, CHI...).

Nathaniel Borenstein said...

OK, sorry to have misunderstood. It sounds like we agree on everything except the likelihood of smarter email robots. And even I don't think it's very likely, just possible.