Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift

Using explicitly harmful multimodal inputs, we find that VLM representations distinguish harmful from benign requests, while jailbreak and refusal samples form separable internal states. These observations suggest that harmfulness recognition alone does not reliably trigger refusal. We define the jailbreak-related representation shift as the component of the image-induced shift along an identified jailbreak direction.

Jailbreak-Related Shift Removal (JRS-Rem) removes this component at inference time without additional model training. Experiments across three VLMs and seven safety datasets cover explicitly harmful inputs, implicitly harmful inputs, and adversarial attacks. The results show improved safety while preserving performance on benign tasks.

Recommended citation: Zhihua Wei, Qiang Li, Jian Ruan, Zhenxin Qin, Leilei Wen, Ruiyang Qin, Qingzhuo Wang, Dongrui Liu, Wen Shen. (2026). "Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift." NeurIPS 2026.
Download Paper