Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Abstract and Introduction | HackerNoon
Briefly

from Hackernoon4 months ago
Apparate is a system that automatically applies and manages early exits (EEs) in ML models, allowing certain inputs to exit with results at intermediate layers.
Hackernoonhttps://hackernoon.com/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-abstract-and-introduction
By providing continual feedback through repurposed exits, Apparate enables several novel runtime monitoring and adaptation strategies that are crucial for optimizing ML inference.
Hackernoonhttps://hackernoon.com/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-abstract-and-introduction
Our evaluation shows that Apparate can lower median response latencies by 40.5-91.5% for computer vision workloads and 10.0-24.2% for natural language processing tasks.
Hackernoonhttps://hackernoon.com/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-abstract-and-introduction
Despite the advances in latency reduction, Apparate maintains throughputs and adheres to strict accuracy constraints, addressing the fundamental challenges of ML inference platforms.
Hackernoonhttps://hackernoon.com/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-abstract-and-introduction
Read at Hackernoon
#machine-learning #inference-optimization #early-exit-models #latency-reduction #throughput-management
[
|
]