"Muse helps creative and launch teams at Netflix understand which artwork and video assets resonate with audiences, and its growth required supporting advanced filtering and audience-affinity analysis at massive scale. To meet these demands, Netflix reports that it redesigned the data serving layer, cutting query latencies by about 50% while maintaining accuracy and responsiveness. Muse started as a Spark-powered dashboard backed by a modest Apache Druid cluster."
"Another challenge came with two of Muse's key metrics, impressions: the number of times an asset is shown and qualified plays, which link playback events back to impressions. Both require counting distinct users, an expensive operation at Netflix scale. To address this, the team adopted HyperLogLog sketches from Apache DataSketches, which provide estimates within about one percent error. Sketches are built in two places: during Druid ingest and in Spark ETL jobs that merge daily sketches into all-time aggregates."
Netflix's Muse application analyzes which artwork and video assets resonate with audiences and needed to scale to trillions of rows. Creative teams requested features like outlier detection, notifications, and media comparisons, increasing data volumes and latency requirements. Audience-affinity labels created many-to-many relationships that increased complexity. Two key metrics—impressions and qualified plays—require counting distinct users, an expensive operation at scale. The team adopted HyperLogLog sketches from Apache DataSketches with about one percent error and built sketches during Druid ingest and in Spark ETL to merge aggregates. These changes cut query latencies by roughly 50% while preserving accuracy and responsiveness.
Read at InfoQ
Unable to calculate read time
Collection
[
|
...
]