"Spotify has introduced the Experiments with Learning (EwL) metric on top of its Confidence experimentation platform to measure how many tests deliver decision-ready insights, not just how many "win." EwL captures both the quantity and quality of learning across product teams, helping them make faster, smarter product decisions at scale. A successful experiment under this framework is both valid (correctly implemented, with healthy traffic splits and no sample mismatches) and decision‑ready."
"The outcome must definitively support one of three actions: ship, abort, or iterate. This metric redefines experimentation success as learning that informs decisions-even when the result isn't positive. Experiments classified as "no learning" fail one or more of these standards. They are separated into three types: invalid (failed health checks or setup errors), unpowered (neutral results with insufficient data on any key metric), and aborted early (tests stopped mid-run, with experimenter feedback collected for analysis)."
Spotify added the Experiments with Learning (EwL) metric to its Confidence platform to measure how many experiments produce decision-ready insights rather than just 'wins.' An experiment qualifies as EwL when it is valid—properly implemented with healthy traffic splits and no sample mismatches—and decision-ready, clearly indicating to ship, abort, or iterate. Experiments that are invalid, unpowered, or aborted early are classified as 'no learning.' Focus shifted from test velocity to test quality and business impact. Across R&D, the learning rate averages 64% compared with a win rate of about 12%, showing learning better reflects experimentation health.
Read at InfoQ
Unable to calculate read time
Collection
[
|
...
]