Selective tie learning suppresses spurious correlations in preference optimization with minimal annotation

Moya et al. show that augmenting preference datasets with targeted ties mitigates shortcut learning, deriving influence bounds and using implicit reward margins to identify optimal pairs.

Primary source: arXiv:2610.35787: https://arxiv.org/abs/2610.35787

What’s new: Direct preference optimization algorithms such as DPO, IPO, and KTO align language models with human preferences, but they frequently exploit spurious correlations present in feedback datasets (such as response length, verbosity, formatting artifacts, or stylistic tokens) rather than learning true task competence. While augmenting pairwise preference datasets with “tied” responses (pairs where two candidate completions are judged equally acceptable) can suppress reliance on spurious cues, uniformly or randomly adding ties requires excessive annotation and dilutes optimization efficiency. In Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization (arXiv:2610.35787, 26 Oct 2026), Christian Moya, Purav Matlia, Elliott Thornley, and Guang Lin formulate a theoretical and empirical framework for selectively identifying ties that maximize debiasing impact.

For log-linear policies, the authors analytically derive each tie’s exact influence on spurious reliance and prove that selecting candidates with the largest predicted reductions is first-order optimal among all subsets of equal size. Because computing exact influences for deep neural networks requires knowing hidden spurious features and computing the Hessian curvature of the training objective, the authors propose a practical proxy: using the magnitude of a candidate tie’s implicit reward margin alongside a generalized Gauss-Newton curvature approximation. Evaluated across synthetic tasks, standard neural models, and large language models, Selective Tie Learning matches the debiasing performance of full uniform tie augmentation while utilizing a fraction of the tied examples, effectively breaking shortcut reliance without degrading general conversational capabilities.

Why it matters: Spurious correlations in alignment data represent a persistent safety and reliability hazard, leading models to reward superficial verbosity or sycophancy while failing to penalize factual errors. By identifying exactly which tied pairs exert the strongest mathematical leverage to cancel out spurious gradients, Selective Tie Learning offers alignment researchers a principled, cost-effective method to audit preference data and harden reward models against shortcut learning.

Caveats: Computing generalized Gauss-Newton curvature approximations on billion-parameter models introduces non-trivial computational overhead during the dataset curation phase. Furthermore, the efficacy of the implicit reward margin proxy depends on the model having already learned partial representations of the spurious features during earlier training stages.