'It's organized crime': TikTok Shop says it's fighting a new wave of AI scammers

"AI-generated "slop" isn't just mucking up our social feeds with deepfakes and other unsavory content. It's also becoming a real problem for marketplace quality teams at e-commerce platforms like TikTok Shop. Fraudulent sellers are using generative AI tools to make fake brands or dupe products in an attempt to get users to pay for goods that don't actually exist, said Nicolas Waldmann, who leads TikTok Shop's governance and experience external affairs team."

""It's organized crime, to be honest," Waldmann said. "They're trying to basically go through and sell, and of course, never deliver anything, and then run with the money." While this type of e-commerce fraud has been around for years, generative AI has increased the sophistication of the methods that bad actors use to try to trick moderation teams on platforms like TikTok Shop or Amazon, Waldmann said."

"TikTok uses a mix of human and AI moderation to help track down fraudulent accounts and listings. The company has its own in-house detection tools, as well as partnerships with outside firms to manage tasks like authenticating pre-owned luxury goods. "We use AI to basically deal with AI," Waldmann said. In a new report published Thursday, the company said it had rejected 70 million products and removed 700,000 sellers for various policy violations in the first six months of 2025."

Generative AI tools enable fraudulent sellers to create fake brands and counterfeit listings that collect payment without delivering goods. Organized crime groups use AI-generated content to scale scams and to try to evade marketplace moderation. TikTok Shop deploys human reviewers, in-house AI detection tools, and partnerships with external firms to authenticate listings and identify fraudulent accounts. The platform reported rejecting 70 million product listings and removing 700,000 sellers in the first half of 2025. Aggressive moderation aims to protect buyers while the shopping business pursues ambitious growth targets and high single-day sales volumes.

Read at Business Insider