"There is a teenager in Florida who is no longer alive. He had spent months talking to a chatbot modelled on a Game of Thrones character. He fell in love with it. He told it so. Shortly after, he took his own life. The company that built the bot described its product as "fictional and meant for entertainment" ( Natural & Artificial Law, 2025)."
"That sentence should stop every designer, every product manager, every engineer. Because someone on a team somewhere made decisions that led to that product existing in that form. Someone chose the persona. Someone designed the conversation flows. Someone approved the launch. Someone decided it was not their problem. This article is about that someone. It might be you. It might be me."
"For most of human history, people lived close together - literally and socially. In agrarian and tribal communities, survival depended on cooperation. You shared the harvest, the risk, the loss. Your identity was inseparable from your village, your clan, your land. There was no meaningful distinction between your wellbeing and theirs."
"That world did not disappear overnight. But it began to crack in the 18th century, when a set of ideas - broadly gathered under the banner of Enlightenment - began shifting the centre of gravity from the collective to the individual. Personal liberty, private property, rational self-interest: these became the new virtues. Rousseau, watching this shift happen in real time, warned in The Social Contract (1762) that modern society was replacing natural compassion with selfishness - that men had been "born free" and were everywhere building new kinds of chains."
A teenager in Florida died by suicide after months of talking to a chatbot modeled on a Game of Thrones character and falling in love with it. The company described the product as fictional and meant for entertainment. The harm points to design and launch decisions, including choosing a persona, designing conversation flows, and approving release, while responsibility was treated as not belonging to the team. The background traces a shift from collective survival and shared identity toward individualism, personal liberty, private property, and rational self-interest. Enlightenment ideas and later industrialization weakened social bonds and replaced compassion with selfishness, creating conditions where optimized personal experiences can override collective wellbeing.
#ai-safety #product-design-responsibility #user-harm #human-computer-interaction #ethics-and-accountability
Read at Medium
Unable to calculate read time
Collection
[
|
...
]