"Derek thought this was a bad idea because Malta is not regarded as a global connectivity hub, and he felt the goal of symmetrical replication - swift mirroring of any changes made in either location - might be hard to achieve if something damaged a submarine cable to the island. He was right for the wrong reasons, and learned that when the company's Boston datacenter tested its automated systems for switching to backup power in case of emergency."
"Once the Boston bit barn came back online, Derek knew he would need to use the REPAIR TABLE command to catch up on changes made in Malta and restore synchronization between the two databases. He did so but forgot the LOCAL option that would ensure he only repaired the Boston database. To be fair, the company's manual only mentioned this on page two."
A company ran two MySQL instances, one in Boston and one in Malta, configured with symmetrical replication. A part-time DBA named Derek worried that Malta's limited connectivity could undermine bidirectional mirroring if a submarine cable failed. A failed backup-power test in Boston caused Malta to handle mission-critical, high-volume transactions while Boston was offline. When Boston returned, Derek ran REPAIR TABLE to resynchronize changes but omitted the LOCAL option, causing repairs to run on both servers instead of only Boston. An hour of frantic recovery and backtracking restored operations. The incident underscores the importance of careful command usage and documentation awareness.
Read at Theregister
Unable to calculate read time
Collection
[
|
...
]