Google sues Chinese cybercrime network that used Gemini to automate scams

Google loves telling us all the ways people are using its generative AI products to build new things, grow businesses, and save the world. Supposedly. Of course, people are also using AI for crime. Google has announced a new legal salvo aimed at a Chinese group called Outsider Enterprise, which is allegedly responsible for a massive AI-powered scam campaign. Google says it’s working with law enforcement and mobile carriers to fight back.

According to Google’s legal filing, Outsider Enterprise operates through Telegram. The group offers phishing-as-a-service to individuals who may not be technically savvy enough to set up fraudulent websites and text campaigns on their own. In its Telegram channels, Outsider Enterprise reportedly provided instructions on how to use Google’s Gemini AI to create websites that imitate those of Google, YouTube, and government agencies such as New York’s E-ZPass. The group offered nearly 300 scam templates.

Google says that scams enabled by Outsider Enterprise resulted in more than 2.5 million text messages being sent to Android users. About 55,000 of those messages happened in a two-week period last month. In all, Google has tracked 9,000 fake websites and 1 million URLs connected to the scam network.

The text messages often made claims about account problems or issues with a package delivery. When users clicked on the links, they ended up on one of those fraudulent websites, designed by Gemini to look legitimate. The cybercriminals used these sites to steal personal data and banking details. Google’s filing does not estimate the amount of money stolen through Outsider Enterprise scams, but the blog post notes that hundreds of people have lost some amount of money.

Google worked with AT&T, Verizon, and T-Mobile to block many of these malicious text messages, and Google notes that its on-device scam detection in Google Messages probably helped reduce the number of successful phishing attempts, too. This AI-powered feature apparently stops 10 billion scam texts every month, so it’s fair to expect it caught at least some Outsider Enterprise activity.