Speaker:  Yichi Zhang, Assistant Professor, Department of Statistics, Indiana University

Title:  Random-walk Debiased  Inference for Contextual Ranking Model with Application in  Large Language Model Evaluation

Abstract:

We present a debiased inference framework for item comparison in the  contextual Bradley-Terry-Luce   model, where each item  is associated with a context-dependent preference   function. We first  use a maximum likelihood approach for the  preference  function learning with ReLU neural networks. By aggregating   pairwise scores via random walks on the comparison graph, we introduce a novel   random-walk estimating score invoking our proposed random-walk debiased estimator. When preference functions are well-approximated, our debiased estimator ensures   statistical inference and asymptotically attains the  semiparametric efficiency bound. We further extend our method with multiplier bootstrap for the uniform inference across a large number of  comparisons, and adapt it to handle the distribution shift of contextual variables. Simulations confirm the theoretical advantages of the proposed method, and we demonstrate its application for the contextual  large language model evaluation with human preferences.