The rise and fall of robots.txt
Briefly

For three decades, a tiny text file has kept the internet from chaos. This text file has no particular legal or technical authority, and it's not even particularly complicated.
Now AI has changed the equation: companies around the web are using your site and its data to build massive sets of training data, in order to build models and products that may not acknowledge your existence at all.
Read at The Verge
[
add
]
[
|
|
]