Ken, hired at Amazon as a Linux sysadmin despite being unqualified, undertook the job of upgrading a tape backup application. He meticulously planned and tested, believing everything to be fine after initial success. Hours later, however, both Ken and senior executives learned that Amazon.com had gone down due to his backup application mishap. Ken discovered that while the backup process was operational, it inadvertently led to a total failure of the database system.
I spent months planning and testing because with this upgrade configuration files changed and we were required to make new ones and push them out with the update.
We sat and watched for several hours after the update, everything worked great, so we patted ourselves on the back, called it a job well done, and went home.
This, many considered, was bad.
Ken knew the backup app he built would copy the database's logs to tape, then delete the logs on the servers that hosted the database.
Collection
[
|
...
]