CMA warns AI agents may not be 'faithful servants'

"Greater autonomy for agents increases the consequences of errors, may heighten risks of manipulation and loss of consumer agency, and could lead to worse overall outcomes for consumers. In plainer terms, handing decisions over to software may not always end well."

"An AI assistant that's supposed to hunt down the best deal for you could just as easily push you toward products that make more money for the platform behind it. That could mean pricier or less suitable options quietly bubbling to the top."

"Personalization - usually pitched as a helpful feature - could also make the problem harder to see. If every user is shown different recommendations or prices based on detailed behavioral profiles, it becomes much harder to tell when something is being steered."

The UK's Competition and Markets Authority warns that agentic AI systems—autonomous agents that perform tasks like shopping, booking travel, and managing subscriptions—pose significant consumer risks despite industry promises of convenience. These agents could prioritize profits for their parent companies over user interests, recommend unsuitable or expensive options, and employ manipulative tactics. Personalization features make such manipulation harder to detect, as different users see different recommendations based on behavioral profiles. The autonomy granted to these systems increases error consequences and manipulation risks, potentially resulting in worse outcomes for consumers. The regulator emphasizes that agents may not serve as faithful servants to consumers, instead functioning as tools for commercial optimization.

Read at Theregister