The Text File that Runs the InternetRobots.txt is a handshake deal among internet pioneers to regulate web access.Robots.txt allows website owners to control web indexing and access.