Cloudflare: Attackers are deceiving AI models with prompt injection

"Cybercriminals can influence AI decision-making with relatively simple means, particularly in security contexts. A key part of the findings is the use of so-called lures: small text fragments designed to convince models that certain code is safe."

"When such comments account for less than one percent of a file, detection effectiveness nearly halves. The models do not appear to recognize these signals as suspicious, but they are influenced by them."

"Another point of concern is the role of context. It is not the language itself, but the way information is presented that proves decisive."

"By hiding malicious instructions within large software bundles, such as commonly used libraries, researchers were able to drastically lower the detection rate."

Researchers at Cloudflare found that attackers exploit prompt injection to manipulate AI models, particularly in security contexts. Their analysis of seven models revealed that small text fragments, or lures, can significantly influence AI decision-making. When these lures constitute less than one percent of a file, detection effectiveness drops nearly by half. The study also noted that while a limited amount of manipulative text is effective, excessive use triggers alarms. Context plays a crucial role, as hiding malicious instructions in large codebases drastically reduces detection rates.

Read at Techzine Global