The reliability cost of default timeouts

"In user-facing distributed systems, latency is often a stronger signal of failure than errors. When responses exceed user expectations, the distinction between "slow" and "down" becomes largely irrelevant, even if every service is technically healthy."

"What stood out was not the slowness itself, but how "infinite by default" waiting quietly drained capacity long before anything crossed a traditional failure threshold."

"The system crossed the user's pain threshold long before it crossed any paging threshold. Our alerts were optimized for traditional failures, not for the cascading effects of unbounded waiting that turned slowness into an outage."

In user-facing distributed systems, latency serves as a stronger failure signal than error rates. When response times exceed user expectations, the practical distinction between slow and down becomes irrelevant despite technical health. An incident demonstrated how default infinite waiting quietly consumed capacity before crossing traditional failure thresholds. Support tickets preceded alarms as CPU, memory, and thread pools filled while error rates remained low. Product pages hung intermittently, causing user abandonment and refresh loops. A recent deployment rollback had no effect, revealing the issue stemmed from system behavior under sustained slowness rather than specific code changes. By day's end, the incident caused measurable business impact including double-digit conversion drops and significant user trust erosion.

Read at InfoWorld