Bots are better at beating ‘are you a robot?’ tests than humans are

Bots are better and faster than humans at online CAPTCHA tests designed to keep them out of websites. The finding calls into question whether we should continue using these kind of security measures, given how much frustration they can cause for people.

A cat-and-mouse game has played out for almost two decades between website developers who want to keep bots out of their sites and the hackers who want to bypass those protections using these troublesome chunks of computer code to scrape content, create fake accounts and post fraudulent comments or reviews.

Tests designed to be easy for humans to pass, but to trip up software, have long been a feature of websites. Over time, types of CAPTCHA – which stands for Completely Automated Public Turing test to tell Computers and Humans Apart – have become more advanced, and gradually trickier to solve, at least for us.

Now, Gene Tsudik at the University of California, Irvine, and his colleagues have shown that bots have little problem solving the current crop of tests faster than humans, and suggest the tests are more trouble than they are worth.

“We do know for sure that they [the tests] are very much unloved. We didn’t have to do a study to come to that conclusion,” he says. “But people don’t know whether that effort, that colossal global effort that is invested into solving CAPTCHAs every day, every year, every month, whether that effort is actually worthwhile.”

The researchers scoured the world’s 200 most popular websites and found that 120 of them used CAPTCHA tests. They then recruited 1000 people online of varied age, sex, location and educational level to each take 10 CAPTCHA tests on these sites. Their findings showed that the various bots coded by researchers and published in journals tended to beat humans in accuracy and speed.

When humans solved distorted text CAPTCHA tests, for instance, they took between 9 and 15 seconds and achieved accuracy of only 50 to 84 per cent. Bots taking the same test completed it in less than a second with 99.8 per cent accuracy.

Team member Andrew Searles, also at UC Irvine, says CAPTCHA tests have become less useful. “There’s no easy way using these little image challenges or whatever to distinguish between a human and a bot any more,” he says.

Shujun Li at the University of Kent, UK, isn’t surprised by the results, because of the recent progress of automated CAPTCHA solvers due to more powerful machine learning techniques. “In general, as a concept CAPTCHA has not met the security goal, and currently is more an inconvenience for less determined attackers,” he says. “New approaches are needed, like more dynamic approaches using behavioural analysis.”

Searles says firms should use intelligent algorithms to identify and weed out bot interactions on websites, rather than relying on tests.

GeeTest, which creates CAPTCHAs and whose tests were among those studied, didn’t respond to a request for comment.

Google reCAPTCHA tests were also included in the study, and Jess Leroy, senior director of product management at Google Cloud, says: “We are increasingly focused on recognising and interrupting malicious activity, whether perpetrated by bots or humans. As such, we are able to help our customers prevent loss even as AI bots become better at masquerading as humans. Further, we have a very large focus on helping our customers protect their users without showing visual challenges, which is why we launched reCAPTCHA v3 in 2018.”