Existing self-distillation schemes for multi-exit networks primarily enhance performance on clean samples by distilling knowledge from the most effective exit. Our NEO-KD algorithm innovates by utilizing multiple exits for more robust distillation, leading to better overall performance in adversarial scenarios.
In our experiments, we demonstrated that using an ensemble of exits for distillation not only improves resilience against adversarial attacks but also consistently surpasses traditional single-exit distillation methods like SKD and ARD in multi-exit networks.
Collection
[
|
...
]