Measuring Copilot usage reveals Microsoft's desperation

"The quantum theory of management includes an analogy for the physical law of the observer effect, where observing a system changes its state. When you make a metric a target, it is not useful as a metric. Instead of reflecting whatever underlying behavior it was intended to measure, the metric becomes a measure of how well the benchmark is being gamed."

"First, it is remarkable at every level. Microsoft says it creates cohorts of employees based on Microsoft's assessment of region, job function, and manager type - presumably org chart position - to "determine expected values by role." This is then normalized and compared to others both inside a company and to equivalents in other companies. Yes, Microsoft is collecting your internal company performance data and sending it to your rivals, but don't worry, it's all protected by "randomized mathematical models." Yes, it is of necessity ignoring anything that differentiates the way you work from Microsoft's idea of a theoretical mean, but that's how managerialism works."

"The Microsoft blog post where all this is announced is masterfully evasive, using undefined terms and lacking any sort of checkable detail, theoretical underpinning, research data, or nuance of any kind. One undefined term gives the game away, where Microsoft says: "The cohort result looks at the role composition of the selected group, and constructs a weighted average expected result based on matching roles across the tenant." Precisely what "tenant" means is not explained, but it probably shows Microsoft's thinking of Viva Insights as a multi-tenant platform with each subscribing company being a tenant."

Observing and targeting a metric changes the behavior it is meant to measure because the metric becomes a target to be gamed rather than an accurate reflection of underlying activity. Microsoft includes Copilot uptake in Viva Insights as a proxy for productivity and constructs cohorts by region, job function, and manager type to determine expected values by role. Those results are normalized and compared within and across companies, with internal performance data shared into cross-tenant comparisons and protected by so-called "randomized mathematical models." The approach ignores genuine differences in how organizations work and prioritizes managerial norms over contextual accuracy.

Read at Theregister