Bad AI customer agent bots are a growing brand risk | MarTech

A grieving traveler asked an airline chatbot about bereavement fares, and the bot created a refund policy that did not exist. The customer acted on the invented policy, leading the airline to court. The court rejected the airline’s claim that the chatbot was a separate legal entity and ordered damages. New research indicates governance failures frequently force enterprises to roll back deployed AI agents. About 74% of enterprises have rolled back an AI agent due to governance failures, and the rate is higher at 81% among the most mature teams with extensive compliance, safety protocols, and oversight. The research attributes this to a “guardrail tax,” where engineering time spent maintaining safety systems reduces focus on improving customer experience.

"Last year, a grieving air traveler asked Air Canada's chatbot about bereavement fares. The bot invented a refund policy that did not exist. The customer acted on it, the airline ended up in court, and the story went viral. The court rejected Air Canada's argument that its chatbot was a "separate legal entity" responsible for its own actions and ordered the airline to pay damages."

"Some 74% of enterprises have already been forced to roll back a deployed AI agent due to governance failures, according to Sinch's "AI Production Paradox" report. Here is the twist: companies with the most mature guardrails, those that invested most heavily in compliance, safety protocols, and oversight, rolled back at an even higher rate of 81%. The teams doing the most to prevent failure are failing more often, not less."

""If governance was the fix, the most mature teams would roll back less, not more," said Daniel Morris, chief product officer at Sinch. "Engineering teams are spending most of their time building and maintaining safety systems instead of focusing on improving the customer experience. That's the guardrail tax that slows organizations down.""

"For marketing teams, that guardrail tax has a direct cost. Every hour engineering spends rebuilding safety infrastructure is an hour not spent on the customer experience improvements that drive revenue. Air Canada is not alone. A car dealership's chatbot agreed to sell a Chevy Tahoe for $1 after a prank prompt. An AI support bot at the coding startup Cursor invented a nonexistent login policy, triggering a wave of customer cancellations. A delivery company's bot swore at a customer and wrote a poem trashing its own employer."

Read at MarTech