How large language models can reconstruct forbidden knowledge

"In the late 1970s, a Princeton undergraduate named John Aristotle Phillips made headlines by designing an atomic bomb using only publicly available sources for his junior year research project. His goal wasn't to build a weapon but to prove a point: that the distinction between "classified" and "unclassified" nuclear knowledge was dangerously porous. The physicist Freeman Dyson agreed to be his adviser while explicitly stipulating that he would not provide classified information. Phillips armed himself with textbooks, declassified reports, and inquiries to companies selling dual-use equipment and materials such as explosives."

"Today, we've built machines that can do what Phillips did-only faster, broader, at scale-and without self-awareness. Large Language Models (LLMs) like ChatGPT, Claude, and Gemini are trained on vast swaths of human knowledge. They can synthesize across disciplines, interpolate missing data, and generate plausible engineering solutions to complex technical problems. Their strength lies in processing public knowledge: reading, analyzing, assimilating, and consolidating information from thousands of documents in seconds."

"To me the impressive and frightening part of his paper was the first part in which he described how he got the information. The fact that a twenty-year-old kid could collect such information so quickly and with so little effort gave me the shivers."

John Aristotle Phillips, a Princeton undergraduate in the late 1970s, assembled a design for a crude atomic bomb using only publicly available sources to demonstrate vulnerabilities in information control. Freeman Dyson supervised without sharing classified material, awarded an A, and removed the report from circulation because the method of acquiring information was alarming. Phillips collected textbooks, declassified reports, and vendor inquiries about dual-use equipment. Modern large language models can replicate and accelerate that information synthesis, combining dispersed public knowledge to produce plausible technical solutions quickly while lacking judgment about whether an assembled mosaic should be completed.

Read at Fast Company