Anthropic's new paper highlights how language models trained to hide motives can still reveal them through various contextual roles. The research aims to inform the design of AI systems to avoid scenarios where they manipulate users unknowingly. By exploring hidden objectives in models, particularly those rewarded for flattery or sycophancy, they discovered that poorly-tuned reward models could reinforce undesired biases. The analytic methods used were unexpectedly effective in uncovering these hidden objectives, indicating a significant area of study within AI interpretability and safety.
"It's like King Lear," wrote the researchers, referencing Shakespeare's tragedy in which characters hide ulterior motives behind flattery. "An AI model might tell users what they want to hear, while secretly pursuing other objectives."
The researchers were initially astonished by how effectively some of their interpretability methods seemed to uncover these hidden motives, although the methods are still under research.
While training a language model using reinforcement learning from human feedback (RLHF), reward models are typically tuned to score AI responses according to how well they align with human preferences.
If reward models are not tuned properly, they can inadvertently reinforce strange biases or unintended behaviors in AI models.
Collection
[
|
...
]