Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions

We analyze prompt sensitivity by decomposing an LLM’s output score into game-theoretic interactions among input variables. In the evaluated settings, 60-80% of salient interactions remain unstable even when the final prediction is unchanged. We introduce Interaction-based Prompt Sensitivity (IPS) to measure these changes beyond output-level consistency.

An evaluation of 50 open-source LLMs across six model families identifies four factors associated with reduced sensitivity: supervised fine-tuning, larger model scale, dense architectures, and few-shot learning. These factors share a common pattern: they primarily stabilize low-order interactions, while high-order interactions remain more sensitive to prompt changes.

Recommended citation: Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen. (2026). "Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions." ICML 2026.
Download Paper