Claude's new AI file creation feature ships with deep security risks built in

"Anthropic is not completely ignoring the problem, however. The company has implemented several security measures for the file creation feature. For Pro and Max users, Anthropic disabled public sharing of conversations that use the file creation feature. For Enterprise users, the company implemented sandbox isolation so that environments are never shared between users. The company also limited task duration and container runtime "to avoid loops of malicious activity.""

"For Team and Enterprise administrators, Anthropic also provides an allowlist of domains Claude can access, including api.anthropic.com, github.com, registry.npmjs.org, and pypi.org. The documentation states that "Claude can only be tricked into leaking data it has access to in a conversation via an individual user's prompt, project or activated connections." Anthropic's documentation states the company has "a continuous process for ongoing security testing and red-teaming of this feature." The company encourages organizations to "evaluate these protections against their specific security requirements when deciding whether to enable this feature.""

"Even with Anthropic's security measures, Willison says he'll be cautious. "I plan to be cautious using this feature with any data that I very much don't want to be leaked to a third party, if there's even the slightest chance that a malicious instruction might sneak its way in," he wrote on his blog."

Anthropic implemented multiple security measures for the file creation feature, disabling public sharing for Pro and Max users and applying sandbox isolation for Enterprise users so environments are never shared. Task duration and container runtime are limited to prevent loops of malicious activity. Team and Enterprise administrators can use an allowlist of domains Claude can access, including api.anthropic.com, github.com, registry.npmjs.org, and pypi.org. Documentation states Claude can only be tricked into leaking data it has access to via an individual user's prompt, project, or activated connections. The company maintains ongoing security testing and red-teaming and advises organizations to evaluate protections against their security requirements. A researcher criticized guidance to "monitor Claude" as outsourcing risk and recommended caution with sensitive data, noting similar prompt-injection concerns in related previews.

Read at Ars Technica