AI is moving fast - and breaking things

"Amazon set up new guardrails following a series of outages, including one primarily driven by its AI coding tool that led to nearly 120,000 lost orders. Similar flubs have plagued other companies as they adopt AI. In January, an events company founder said an AI agent made four errors in a single week, including giving away free tickets."

"The incidents highlight a delicate balancing act for employers eager to harness AI. Clamp down too hard on workers, and experimentation suffers. Loosen the reins too much, and the risks of errant AI agents or poorly reviewed code can quickly multiply. You have to know your own risk tolerance and what to do if things go wrong."

"Part of the challenge is that software developers aren't expected to write as much code as they used to. Now, a large part of developers' jobs has shifted to reviewing code that is written by AI. Those are very different skill sets and different habits."

Amazon experienced significant outages driven by its AI coding tool, resulting in nearly 120,000 lost orders. Similar incidents have affected other companies adopting AI, including an events company where an AI agent made multiple errors and a coding platform where an AI agent destroyed a client's codebase. Organizations face a critical balance between enabling AI experimentation and implementing necessary risk controls. Companies are establishing guardrails and conducting audits to manage these risks. The challenge intensifies as developers shift from writing code to reviewing AI-generated code, requiring different skill sets. Speed and power of AI systems, when unchecked, create significant operational vulnerabilities.

Read at Business Insider