This all started when my husband got fed up and refused to be held hostage. "Screw this!" he told me over lunch. "I am not going to take those forms down and I am not going to treat our users like criminals."
To be honest his language was a little more colorful than “screw this” but let me back up a bit and explain.
We started our Web development firm in 1999 and spent the next few years building dataWe started our Web development firm in 1999 and spent the next few years building
database-driven Web sites for clients and managing a few sites of our own. Like
many companies, we had moved to using Web contact forms instead of just an email
address link, since spammers harvested email links so quickly. It wasn’t long
though before our contact forms (and other Web forms) started getting spam, too.
Spam or no spam...just read 'em all
Form spam created a unique dilemma. Since form results were being emailed and posted by us from our own Web sites, traditional email filters couldn't determine what was spam and what wasn’t. And since every contact form theoretically meant communication from a client or potential client, we had to open and read them all, spam or not.
We tried the usual form tricks like validating email addresses, adding hidden fields, making sure forms were submitted from the correct URL, and checking IP addresses against known blacklists.
It sort of worked.
Well, not really. Everything we tried slowed down (but didn’t eliminate) the spam for a short time, but within a few days or weeks we’d be bombarded with as much spam as ever.
So for a while we just put up with it.
CAPTCHA enters the scene (and hubby gets ticked)
By 2002, we were building more complex Web sites and databases, and forms were becoming more critical. We needed usable Web forms for subscriptions, lead generation, memberships, registration forms, and a whole bunch of user interactions. Often we’d need to post form data directly to a database, and that meant we were getting more and more spam clutter mixed in with valid data.
To make matters worse, the volume of spam had ramped up considerably. It wasn’t unusual for 90-95% of form submissions to be spam.
We certainly weren’t the only ones struggling with form spam, either. Large Web sites like Yahoo! had already started tackling the problem with CAPTCHA images of distorted words and letters that users had to identify (read: guess) before doing anything else. I personally found this annoying, but hubby took it as an insult.
“I’m their customer, not the spammer! Why are they assuming I’m the criminal here? And why am I being treated as guilty until proven innocent? Last time I checked we lived in America where it works the other way around. If I walked into a brick and mortar store, the sales staff wouldn’t demand that I ‘prove’ I’m not a shoplifter before browsing around. This is screwed up.”
Turkey sandwiches inspire a new goal
By Thanksgiving of that same year, form spam on one of our Web sites had become overwhelming, with several hundred spam entries per day. Everything we tried hadn’t worked for very long and, although CAPTCHA had become more and more common, we’d ruled it out as an option just on principle.
In desperation, we took down the forms on that site completely.
About two days later we were eating leftover turkey sandwiches for lunch, when my husband made the “Screw this!” pronouncement.
“I am not going to cave in and let those damn spammers force me to abandon our forms,” he said. “Who are they to tell me what I can or can’t do with my Web site? It’s like negotiating with terrorists. You just don’t do it. And forget about treating our users like criminals and using those crazy CAPTCHA images. There’s got to be a better way.”
The research road
So the forms went back up and hubby set about compiling and analyzing data being submitted through a whole mess of different forms and Web sites.
As we collected spam entries from more and more Web sites and different types of forms, and as we analyzed more and more data, we started finding new ways to counter the spam attacks.
Much like before, spammers defeated our early tactics pretty quickly. But over time the gap between “this stops spam” and “this doesn’t stop spam anymore” grew larger and larger. Clearly we were on the right track.
Fast forward four years to 2006, when we had accumulated a pretty ridiculous amount of data and a pretty ridiculous amount of obscure knowledge about form abuse. By this time we had started to develop a fairly sophisticated detection matrix and set of algorithms to identify spam and abusive submissions vs. the real thing.
We’d learned that form spam isn’t at all similar to email spam, so it’s no wonder those early filter-based tactics had failed. Form spam defied simple heuristics, and most information submitted through Web forms (especially lead generation forms) was just too critical to rely on an educated guess anyway. We needed a way to sort out the good entries from the bad ones with 100% reliability.
After more research, analysis and testing we finally struck upon a method that took an entirely new approach to protecting forms. And for site after site, and form after form, it started working.
The status quo
CAPTCHA variations had by this time become entrenched as the status quo for dealing with form spam and form abuse. To add insult to injury, we knew that CAPTCHA didn’t work. No one would disclose “official” numbers but we heard rumors that CAPTCHA failed to stop spam as much as 80% of the time. (Hubby’s comment: “All of this user abuse and it doesn’t even work anyway!?!”)
A few new CAPTCHAs had even started making the rounds – asking users to identify pictures of kittens and animals; solve math problems; answer silly questions like ‘Is the sky blue or green?’ and otherwise defend themselves as being actual humans as opposed to, I don’t know, androids?
CAPTCHA reaches the end of the road...
Meanwhile Form Armor, as we had started calling it, worked great and just kept getting better. Success rates varied, but in some cases Form Armor stopped 100% of spam submissions — and maintained that success rate for several months in a row. Even better, we’d determined how to identify "good" submissions from bad ones, so virtually none of the valid submissions were being flagged incorrectly as spam.
In early 2008, mainstream news outlets started reporting that CAPTCHA images were being defeated for major services like Gmail, Yahoo! and Hotmail as often as 90% of the time. Users had started becoming more and more vocal about their hatred of CAPTCHA, and as a spam-fighting tool it had obviously reached the end of the road. We already had a solid base of subscribers using Form Armor as the ideal alternative, but clearly it was time to ramp up and make Form Armor available to a larger market.
...and Form Armor blazes a new trail
So here we are, seven years after the original “Screw this!” pronouncement. And Form Armor is thriving as a stable, fast-growing Web service (or Software-as-a-Service, if you like that buzzword) that stops form abuse in its tracks without abusing users in the process.
As Form Armor continues to grow, we’re looking forward to harassing more spammers, and helping to protect more Web site forms, databases and Web-enabled applications.
If you’d like to give Form Armor a try for your own Web site or client sites, please sign
up for an account and see the results for yourself.
If you’d like to discuss partnership opportunities or find out how to integrate Form Armor with your own Web and software applications, give me a ring at 1-866-433-2638 x110, or send a note via our contact form (that’s protected by Form Armor, of course).
I'd be glad to chat and look forward to hearing from you.
Cheers,
Larissa Reynolds
Idea Catchers Group, LLC