Depth Selective Adapter Allocation for Polarity Robust Medical Vision Language Inference

Abstract

Medical visual inference models often receive questions that contain polarity cues such as absence, presence, exclusion, or uncertainty. These cues can be short, syntactically fragile, and clinically important, because a mistaken polarity interpretation can reverse the intended answer. This study evaluates whether adapter parameters should be distributed uniformly across a multimodal language backbone or assigned according to measurable layer-level sensitivity to polarity transformations. We introduce a depth-selective adapter allocation strategy for binary medical vision-language inference and test it on 18,720 image-question pairs derived from radiography, ultrasound, and dermoscopy tasks. The evaluation uses paired affirmative, negative, exclusion, and uncertainty-framed prompts with identical image labels. We estimate layerwise polarity transport through Jacobian energy, centered kernel alignment, and signed margin propagation, then allocate low-rank adapter capacity using a constrained optimization rule. Compared with uniform adapter placement, final-block placement, and random layer allocation, the proposed allocation reduces polarity reversal errors from 10.8% to 4.9% while preserving mean task accuracy within 0.4 percentage points. It also improves negative predictive calibration in exclusion-framed prompts and reduces the variance of signed margin changes across prompt forms. Ablation experiments indicate that adapter rank distribution matters more than total trainable parameter count once a minimum capacity threshold is reached. These findings suggest that layer-sensitive parameter allocation can improve robustness to polarity-bearing clinical wording without increasing model size.