A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

We use game-theoretic interactions to analyze how knowledge distillation (KD) changes the inference behavior of large language models. Decomposing output scores into interactions among input variables reveals a common pattern across the studied KD methods: interaction sparsification. Students retain fewer salient interactions, preferentially inherit simple teacher interactions, and suppress many complex interactions.

Differences in how KD methods handle complex interactions help explain their performance differences. Motivated by this analysis, we introduce Complex Interaction Penalty (CIP), a plug-and-play loss that promotes sparsity in complex interactions during distillation. Experiments across multiple model families show improvements for diverse KD methods on both in-domain and out-of-distribution benchmarks.

Recommended citation: Qingzhuo Wang*, Ruiyang Qin*, Zhenxin Qin, Wen Shen, Zhihua Wei. (2026). "A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions." ICML 2026.
Download Paper